SpiderMonkey to V8, mongometer and the Aggregation Framework

I previously posted a comparison that covered running some simple queries against versions 2.2.2 and 2.3.2 of MongoDB. They were pretty basic examples, I just wanted to demonstrate one of the uses of mongometer; comparing the relative performance of MongoDB releases and the MongoDB scripts you write to run on them.

Now I'm going to knock the complexity up a notch and perform a comparison between the different releases of MongoDB, their underlying JavaScript engines using another relatively new feature (added in version 2.1), the Aggregation Framework. I'm going to use the example they have in the documentation, mainly so I don't have to make something up.

{
title : "this is my title" ,
author : "bob" ,
posted : new Date () ,
pageViews : 5 ,
tags : [ "fun" , "good" , "fun" ] ,
comments : [
{ author :"joe" , text : "this is cool" } ,
{ author :"sam" , text : "this is bad" }
],
other : { foo : 5 }
}

db.articles.aggregate(
{ $project : {
author : 1,
tags : 1,
} },
{ $unwind : "$tags" },
{ $group : {
_id : { tags : "$tags" },
authors : { $addToSet : "$author" }
} }
);

I'm populating the collection with data in the form as described above. The only thing I'm adding is that I'm using the JMeter CSV Data Set to populate the author attribute.

Version 2.2.2
So let's make sure we're starting from a clean slate.

$ /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log

$ ps -ef | grep mongo
4974 /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log

$ ./mongo --port 27000

> show dbs
local 0.078125GB
test (empty)



> show dbs
aggregation 0.203125GB
local 0.078125GB
test (empty)

> use aggregation
switched to db aggregation
> db.dropDatabase()
{ "dropped" : "aggregation", "ok" : 1 }

$ sudo kill -15 4974

Version 2.3.2
Let's ensure we have that same clean slate as with Version 2.2.2

$ /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log

$ ps -ef | grep mongo
1463 /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log

$ ./mongo --port 27001

> show dbs
local 0.078125GB
test (empty)



> show dbs
aggregation 0.203125GB
local 0.078125GB
test (empty)

> use aggregation
switched to db aggregation
> db.dropDatabase()
{ "dropped" : "aggregation", "ok" : 1 }

$ sudo kill -15 1463

Conclusions
I ran this a few times and the results were consistent. I'll knock it up another notch over time, and hopefully draw out some useful conclusions, until then, you can draw your own.

Suggestions and comments welcome.

SpiderMonkey to V8 and mongometer

With 10gen switching the default JavaScript engine for MongoDB 2.3/2.4 from SpiderMonkey to V8 I thought I'd take the opportunity to compare the relative performances of the releases using mongometer. Being a Security bod, I really should have looked at the Additional Authentication Features first... Hey ho.

I'll document the steps taken during the comparison, including the set up, so this can be repeated and validated - just in case anyone is interested - but mainly so I can remind myself of what I did; memory, sieve.

The set up
I'm going to install 2.2.2 and 2.3.2 side-by-side on a dedicated machine. I'll then use the latest version of the Java driver with mongometer.

$ wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.3.2.tgz
$ wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.3.2.tgz.md5

I got a 403 response for this request...

$ wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.2.2.tgz
$ wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.2.2.tgz.md5

$ md5sum -c mongodb-linux-x86_64-2.2.2.tgz.md5
md5sum: mongodb-linux-x86_64-2.2.2.tgz.md5: no properly formatted MD5 checksum lines found

Grrr. An md5 file is supposed to be the checksum (then x2 spaces) and then the filename of the file being checksummed. I'll have to eyeball them instead, well, eyeball the one that I could actually download...

$ md5sum mongodb-linux-x86_64-2.2.2.tgz
be0f5969b0ca23a0a383e4ca2ce50a39 mongodb-linux-x86_64-2.2.2.tgz
$ cat mongodb-linux-x86_64-2.2.2.tgz.md5
be0f5969b0ca23a0a383e4ca2ce50a39

Configure
$ tar -zxvf ~/mongodb-linux-x86_64-2.2.2.tgz
$ sudo mkdir -p /usr/lib/mongodb/2.2.2
$ sudo mv mongodb-linux-x86_64-2.2.2/* /usr/lib/mongodb/2.2.2/
$ rm -r mongodb-linux-x86_64-2.2.2
$ sudo mkdir -p /data/db/2.2.2
$ sudo chown `id -un` /data/db/2.2.2
$ /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log

$ tar -zxvf ~/mongodb-linux-x86_64-2.3.2.tgz
$ sudo mkdir -p /usr/lib/mongodb/2.3.2
$ sudo mv mongodb-linux-x86_64-2.3.2/* /usr/lib/mongodb/2.3.2/
$ rm -r mongodb-linux-x86_64-2.3.2
$ sudo mkdir -p /data/db/2.3.2
$ sudo chown `id -un` /data/db/2.3.2
$ /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log

Let's check they are running.

$ ps -ef | grep mongod
1795 /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log
2059 /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log

Now, let's kill one (gracefully) and move on to the interesting stuff.

$ sudo kill -15 2059
$ ps -ef | grep mongod
1795 /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log

Now I'm jumping on to another box.

$ wget https://github.com/downloads/mongodb/mongo-java-driver/mongo-2.10.1.jar
$ cp mongo-2.10.1.jar /usr/lib/jmeter/2.8/lib/ext
$ cp ~/IdeaProjects/mongometer/out/artifacts/mongometer_jar/mongometer.jar /usr/lib/jmeter/2.8/lib/ext
$ /usr/lib/jmeter/2.8/bin/jmeter.sh

The tests
The tests are really rather basic; I'll perform an insert into two different databases, and perform finds against those databases.

Version 2.2.2

> show dbs
local 0.078125GB













> show dbs
jmeter 0.203125GB
jmeter2 0.203125GB
local 0.078125GB

> use jmeter
> db.jmeter.find().count()
1000
> db.dropDatabase()

> use jmeter2
> db.jmeter.find().count()
1000
> db.dropDatabase()

$ ps -ef | grep mongo
2690 /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log

$ sudo kill -15 2690
$ ps -ef | grep mongo

Nothing. Let's get the 2.3.2 instance up and running.

$ /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log

$ ps -ef | grep mongo 2947 /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log

Version 2.3.2

> show dbs
local 0.078125GB













> show dbs
jmeter 0.203125GB
jmeter2 0.203125GB
local 0.078125GB

> use jmeter
> db.jmeter.find().count()
1000
> db.dropDatabase()

> use jmeter2
> db.jmeter.find().count()
1000
> db.dropDatabase()

Conclusions
I guess you should draw your own. I ran this a couple of times and am considering scripting it so the environments are cleaned down prior to each run, I could probably add more complex queries too. Perhaps if I find some time next weekend then I will.

If you have any suggestions, please leave a comment.

mongometer v2.0

A while back I knocked up mongometer to compare the relative performance of MongoDB scripts. I then made some minor changes, and since then - and only recently - made additional changes based on feedback.

I've now made slightly more significant changes. I'll now cover them briefly.

The minimum JMeter version is now 2.8, this is because it is dependent on a change introduce in JMeter 2.8.

I've created a MongoSourceElement which is accessible from the context menu; Add -> Config Element -> MongoDB Source Config

This pulls all the MongoDB connection details up a level and allows you to share it between multiple MongoScriptSamplers. This means you only need to define the connection once and associate a source name with the instance. When you create a MongoScriptSampler you can then reference the source name as defined in MongoSourceElement.

MongoScriptSampler now only includes the fields that you need. It allows you to specify the mongo source, the database, the database credentials and of course the script to run.

The way I'm currently using it is to create a MongoSourceElement.
Under that I create a ThreadGroup.
And under that I add the MongoDB Script v2.0, View Results Tree, and Graph Results.

If I have multiple script to run, I use the same grouping of, MongoDB Script v2.0, View Results Tree, and Graph Results. See the image below for an example.



There is the added MongoUtils class which has a static method which allows you to specify multiple hosts (if you want) and breaks it out into each host and port, returning an ServerAddress, which to be honest, I'm surprised isn't available under the Java MongoDB driver.

The other semi-useful addition is the QuickEnvironment. It basically allows you to start up mongod, mongo, jmeter and a tail of the jmeter log. You could do this in a script, but hey ho, I did it in a helper class.

The latest version is available on github.

Please have a play, try and break it, make suggestions, give me feedback.