Performance testing MongoDB

So, this morning I was hacking around in the mongo shell. I had come up with three different ways to aggregate the data I wanted, but wasn't sure about which one I should subsequently port to code to use within my application.

So how would I decide on which method to implement? Well, lets just choose the one that performs the best. Ok, how do I do that? Hmmm. I could download and install some of the tools out there, or I could just wrap the shell code in a function and add some timings. OR, I could use the same tool that I use to performance test everything else; JMeter. To me it was a no brainer.

So how do we do it?

There is a full tutorial here.

Simply put, you need to do the following:

1. Create a Sampler class. 
2. Create a BeanInfo class. 
3. Create a properties file. 
4. Bundle up into a jar and drop into the apache-jmeter-X.X\lib\ext folder 
5. Update search_paths=../lib/ext/mongodb.jar in jmeter.properties if you place the jar anywhere else. 

How I did it

I tend to have a scratch pad project set up in my IDE, so I decided just to go with that. Just to be on the safe side, I imported all the dependencies from:

• apache-jmeter-X.X\lib 
• apache-jmeter-X.X\lib\ext 
• apache-jmeter-X.X\lib\junit

I then created the two classes and the properties file.



I then exported the jar to apache-jmeter-X.X\lib\ext, and fired up jmeter.

Go through the normal steps to set the test plan up:

1. Right click Test Plan and add a Thread Group. 
2. Right click the Thread Group and add a Sampler, in this case a MongoDB Script Sampler. 
3. Add your script to the textarea; db.YOUR_COLLECTION_NAME.insert({"jan" : "thinks he is great"}) 
4. Run the test

Happy days.  You can then use JMeter as you would for any other sampler.


Future enhancements

This is just a hack that took me 37 minutes to get running, plus 24 minutes if you include this post. This can certainly be extended to allow you to enter the replicaset config details for instance and to pull the creation of the connection out so we're not initiating this each time run a test.

Mongodb MapReduce scope variables

I recently had a requirement for conditional emission from a map function. Essentially I only wanted to emit where the date was within a given range.

In SQL, grouping a count for a given time granularity within a date range would look something like this:

I wasn't 100% sure about the 'right way' to achieve this in NOSql/MongoDB. So this is a solution.
It requires you to know about scope variables. Problem is, I found that scope variables are not very well documented. You can find more about scope variables in the MongoDB documentation MapReduce-Overview. The relevant parts are:

      [, scope : <object where fields go into javascript global scope >]
and
      scope - can pass in variables that can be access from map/reduce/finalize.

Back to this example. First, let's define some data:

Before we implement this, lets get it working at the command-line, in our mongo shell:

What is happening here?

Well, we're selecting the sub-document of this collection where the value of "meh" is "meh". Then we've defined two dates; from and to to represent the boundaries of the date range, we're including these within the MapReduce function call. Basically what this means is that we can use what ever is defined here in the Map function (btw, we can also use them in the Reduce and Finalize functions).

Once we have this working from the shell, it is straight forward to implement it. This is the very same implemented in Java.

Caveat: This is an example of how to use scope variables, I'm sure if you go to any of the Events or gmane.comp.db.mongodb.user group you'll get some advice straight from the 10Gen guys.

Updating a document using the default ObjectId

Here are a few facts about keys and the way MongoDB deals with them.

• Documents in MongoDB require a key, _id, which uniquely identifies them.  
• This is a 12-byte binary value, read more from the source of truth, ObjectId.
• When inserting a new document, MongoDB will generate an ObjectId where one is not specified.
• There is a 'reasonable' chance that this will be unique at the time of creation.  
• All of the officially-supported MongoDB drivers use this type by default for _id values.

An annoyance I had with MongoDB, when I first started to use it, was the explicit update required on the default _id key/value pair.  To illusrate this here is a concrete example of what I mean.

I wanted to do this:
But I had to do this:
In this case, json is a JSON document passed from an HTML form via HTTP PUT, eventually ending up here at the DAO.  I was using the default _id implementation and thus would have expected save to do the figure that out and do the necessary work for me.

I just can't live those extraneous lines of code, it's too messy and too much to type every time.  I'll pull it out into a fromJson call, as I can see the pattern emerging and the odds of this reoccurring high.

I must have missed something in the API Docs as this cannot be an unusual requirement.  I would of at least expect to see it in a Util class...

Factory snippet:
Usage snippet:

I get to do what I wanted to now, but it still feels wrong.  It feels a little bit dirty.  It feels like a hack-o-la.  How can I do it better?

A simple guide to finding distinct array values in a MongoDB collection

So you want to find unique values within an array within a document in a collection? A reasonable request.

In ANSI SQL you'll be using DISTINCT, JOINS and GROUP BYs, stuff you're used to, but in the NoSQL realm your best bet is mapreduce.

It might seem a little bit like hard work, and probably a little intimidating at first, but it is certainly worth it; mapreduce is an extremely powerful tool.

Set up the collection, the map and reduce functions, and execute the mapreduce command:

But now you want to find unique values within an array within each document in an entire collection.

The map function in this example iterates over each of the items and emits the key/value of each array element.

The reduce function aggregates the key/value from each of the emits from the map function. In this example we're looking at unique keys and maintaining a count of the unique keys.

If you're looking to find the distinct array elements for a single document, simply specify the document index. For the entire collection, just leave the query out. *simples*

Simple mongodb equivalents

After making a noob error today, I realised that some of the documentation could use a boost.

By default mongo returns the entire document.

All fields for the matched document will be returned

And in java



To return only certain fields, you need to let mongo know which ones you want


The will only return the fields meh and feh from the matched document

And in Java


You can also say 'all fields except for...'



This will return all the fields in the matched document expect for meh

And in Java