SpiderMonkey to V8, mongometer and the Aggregation Framework

I previously posted a comparison that covered running some simple queries against versions 2.2.2 and 2.3.2 of MongoDB. They were pretty basic examples, I just wanted to demonstrate one of the uses of mongometer; comparing the relative performance of MongoDB releases and the MongoDB scripts you write to run on them.

Now I'm going to knock the complexity up a notch and perform a comparison between the different releases of MongoDB, their underlying JavaScript engines using another relatively new feature (added in version 2.1), the Aggregation Framework. I'm going to use the example they have in the documentation, mainly so I don't have to make something up.

{
title : "this is my title" ,
author : "bob" ,
posted : new Date () ,
pageViews : 5 ,
tags : [ "fun" , "good" , "fun" ] ,
comments : [
{ author :"joe" , text : "this is cool" } ,
{ author :"sam" , text : "this is bad" }
],
other : { foo : 5 }
}

db.articles.aggregate(
{ $project : {
author : 1,
tags : 1,
} },
{ $unwind : "$tags" },
{ $group : {
_id : { tags : "$tags" },
authors : { $addToSet : "$author" }
} }
);

I'm populating the collection with data in the form as described above. The only thing I'm adding is that I'm using the JMeter CSV Data Set to populate the author attribute.

Version 2.2.2
So let's make sure we're starting from a clean slate.

$ /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log

$ ps -ef | grep mongo
4974 /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log

$ ./mongo --port 27000

> show dbs
local 0.078125GB
test (empty)



> show dbs
aggregation 0.203125GB
local 0.078125GB
test (empty)

> use aggregation
switched to db aggregation
> db.dropDatabase()
{ "dropped" : "aggregation", "ok" : 1 }

$ sudo kill -15 4974

Version 2.3.2
Let's ensure we have that same clean slate as with Version 2.2.2

$ /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log

$ ps -ef | grep mongo
1463 /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log

$ ./mongo --port 27001

> show dbs
local 0.078125GB
test (empty)



> show dbs
aggregation 0.203125GB
local 0.078125GB
test (empty)

> use aggregation
switched to db aggregation
> db.dropDatabase()
{ "dropped" : "aggregation", "ok" : 1 }

$ sudo kill -15 1463

Conclusions
I ran this a few times and the results were consistent. I'll knock it up another notch over time, and hopefully draw out some useful conclusions, until then, you can draw your own.

Suggestions and comments welcome.

SpiderMonkey to V8 and mongometer

With 10gen switching the default JavaScript engine for MongoDB 2.3/2.4 from SpiderMonkey to V8 I thought I'd take the opportunity to compare the relative performances of the releases using mongometer. Being a Security bod, I really should have looked at the Additional Authentication Features first... Hey ho.

I'll document the steps taken during the comparison, including the set up, so this can be repeated and validated - just in case anyone is interested - but mainly so I can remind myself of what I did; memory, sieve.

The set up
I'm going to install 2.2.2 and 2.3.2 side-by-side on a dedicated machine. I'll then use the latest version of the Java driver with mongometer.

$ wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.3.2.tgz
$ wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.3.2.tgz.md5

I got a 403 response for this request...

$ wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.2.2.tgz
$ wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.2.2.tgz.md5

$ md5sum -c mongodb-linux-x86_64-2.2.2.tgz.md5
md5sum: mongodb-linux-x86_64-2.2.2.tgz.md5: no properly formatted MD5 checksum lines found

Grrr. An md5 file is supposed to be the checksum (then x2 spaces) and then the filename of the file being checksummed. I'll have to eyeball them instead, well, eyeball the one that I could actually download...

$ md5sum mongodb-linux-x86_64-2.2.2.tgz
be0f5969b0ca23a0a383e4ca2ce50a39 mongodb-linux-x86_64-2.2.2.tgz
$ cat mongodb-linux-x86_64-2.2.2.tgz.md5
be0f5969b0ca23a0a383e4ca2ce50a39

Configure
$ tar -zxvf ~/mongodb-linux-x86_64-2.2.2.tgz
$ sudo mkdir -p /usr/lib/mongodb/2.2.2
$ sudo mv mongodb-linux-x86_64-2.2.2/* /usr/lib/mongodb/2.2.2/
$ rm -r mongodb-linux-x86_64-2.2.2
$ sudo mkdir -p /data/db/2.2.2
$ sudo chown `id -un` /data/db/2.2.2
$ /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log

$ tar -zxvf ~/mongodb-linux-x86_64-2.3.2.tgz
$ sudo mkdir -p /usr/lib/mongodb/2.3.2
$ sudo mv mongodb-linux-x86_64-2.3.2/* /usr/lib/mongodb/2.3.2/
$ rm -r mongodb-linux-x86_64-2.3.2
$ sudo mkdir -p /data/db/2.3.2
$ sudo chown `id -un` /data/db/2.3.2
$ /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log

Let's check they are running.

$ ps -ef | grep mongod
1795 /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log
2059 /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log

Now, let's kill one (gracefully) and move on to the interesting stuff.

$ sudo kill -15 2059
$ ps -ef | grep mongod
1795 /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log

Now I'm jumping on to another box.

$ wget https://github.com/downloads/mongodb/mongo-java-driver/mongo-2.10.1.jar
$ cp mongo-2.10.1.jar /usr/lib/jmeter/2.8/lib/ext
$ cp ~/IdeaProjects/mongometer/out/artifacts/mongometer_jar/mongometer.jar /usr/lib/jmeter/2.8/lib/ext
$ /usr/lib/jmeter/2.8/bin/jmeter.sh

The tests
The tests are really rather basic; I'll perform an insert into two different databases, and perform finds against those databases.

Version 2.2.2

> show dbs
local 0.078125GB













> show dbs
jmeter 0.203125GB
jmeter2 0.203125GB
local 0.078125GB

> use jmeter
> db.jmeter.find().count()
1000
> db.dropDatabase()

> use jmeter2
> db.jmeter.find().count()
1000
> db.dropDatabase()

$ ps -ef | grep mongo
2690 /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log

$ sudo kill -15 2690
$ ps -ef | grep mongo

Nothing. Let's get the 2.3.2 instance up and running.

$ /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log

$ ps -ef | grep mongo 2947 /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log

Version 2.3.2

> show dbs
local 0.078125GB













> show dbs
jmeter 0.203125GB
jmeter2 0.203125GB
local 0.078125GB

> use jmeter
> db.jmeter.find().count()
1000
> db.dropDatabase()

> use jmeter2
> db.jmeter.find().count()
1000
> db.dropDatabase()

Conclusions
I guess you should draw your own. I ran this a couple of times and am considering scripting it so the environments are cleaned down prior to each run, I could probably add more complex queries too. Perhaps if I find some time next weekend then I will.

If you have any suggestions, please leave a comment.

mongometer v2.0

A while back I knocked up mongometer to compare the relative performance of MongoDB scripts. I then made some minor changes, and since then - and only recently - made additional changes based on feedback.

I've now made slightly more significant changes. I'll now cover them briefly.

The minimum JMeter version is now 2.8, this is because it is dependent on a change introduce in JMeter 2.8.

I've created a MongoSourceElement which is accessible from the context menu; Add -> Config Element -> MongoDB Source Config

This pulls all the MongoDB connection details up a level and allows you to share it between multiple MongoScriptSamplers. This means you only need to define the connection once and associate a source name with the instance. When you create a MongoScriptSampler you can then reference the source name as defined in MongoSourceElement.

MongoScriptSampler now only includes the fields that you need. It allows you to specify the mongo source, the database, the database credentials and of course the script to run.

The way I'm currently using it is to create a MongoSourceElement.
Under that I create a ThreadGroup.
And under that I add the MongoDB Script v2.0, View Results Tree, and Graph Results.

If I have multiple script to run, I use the same grouping of, MongoDB Script v2.0, View Results Tree, and Graph Results. See the image below for an example.



There is the added MongoUtils class which has a static method which allows you to specify multiple hosts (if you want) and breaks it out into each host and port, returning an ServerAddress, which to be honest, I'm surprised isn't available under the Java MongoDB driver.

The other semi-useful addition is the QuickEnvironment. It basically allows you to start up mongod, mongo, jmeter and a tail of the jmeter log. You could do this in a script, but hey ho, I did it in a helper class.

The latest version is available on github.

Please have a play, try and break it, make suggestions, give me feedback.

JMeter Bug Bean

I came across a potential bug in JMeter when adding support for MongoOptions within mongometer. It's not a big deal in itself as I worked around it, but it was slightly frustrating at the time.

Implementing the MongoOptions seemed rather innocuous at the time; after all it's just another bunch of properties the user can alter. I ended up just adding them all at once. Imagine my disappointment when the extension failed to load in Jmeter as intended. To keep a long story short, the offender was wTimeout. Now, wtimeout (is one of a group a rather poorly named variables, others include j and w ), but why was there an issue with wtimeout?

The error of my ways

This is the error that was generated when starting up JMeter:


This is how it was defined, firstly in ScriptSamplerBeanInfo.java


Next, in ScriptSamplerResource.properties


And finally, in ScriptSampler.java


I checked and double checked the spelling. It didn't really make sense, it should've been fine. Out of all the properties that were defined, the wTimeout property was the odd one out, in that it was the only one that had the pattern of lowercase{1}uppercase{N}. All the others were either lowercase{1}, lowercase{N} or lowercase{N}uppercase{N}. So long story, short; switching from wTimeout to doubleuTimeout resolved the issue.


Next, in ScriptSamplerResource.properties


And finally, in ScriptSampler.java


It seems that when performing the creation of the property, if there is only a single lowercase character followed by an uppercase character, we get an error.

The moral of the story

Hmmmm. I don't really do morals, but if this is a bug, then it needs to be fixed. Either way j, w and wtimeout are really rather poor names for variables.

Creating your project with IntelliJ

Now we have a development environment configured, lets create the project and start taking it to the next level.

Creating a project in IntelliJ

New Project
New Project from Scratch
Enter the name: mongometer
Select Maven Module
Next
Select Create from archetype (I chose maven j2ee simple)

Arrrrg! Don't do this! Man, IntelliJ sat spinning the tyres for an age.  I ended up killing it and starting again.

To delete a project, close intellij and remove the project folder
projects are in IdeaProjects
$rm -r ~/IdeaProjects/mongometer

Second time around
New Project
New Project from Scratch
Enter the name: mongometer
Select Maven Module
Next
Deselect the create from archetype

Create a package
Create the classes and the properties file

Fix imports
Add dependencies to the pom.xml

This is what I need for mongometer

• junit4.10
  
• mongodb 2.7.2
  
• ApacheJMeter_core 2.6
  
• jorphan 2.6
  

Push to GitHub

$cd ~/IdeaProjects/mongometer
$git init
$git add *.java
$git add *.properties
$git add *.xml
$git commit -m 'Initial upload of the project'
$git remote add origin ssh://github.com/yourname/mongometer.git
$git push -u origin +master

Voilà! You can go to your repo, and the project files are there.  Happy days.

Points of interest

• The + performs a force push to github.  I'm guessing this happened because when I created a new repository on GitHub I opted to create the README file.  I won't do this again.
  
• IntelliJ complained about setting up the repo outside of IntelliJ.  No problem, you just add the repo inside IntelliJ.  I've been an Eclipse person for years (though we've decided that we'd like spend some time apart) so I'm not 100% sure of how to do this inside IntelliJ just yet.
  
• This is a great site for listing dependencies http://mvnrepository.com/ for setting up your pom.xml