Thursday, 26 April 2012

Performance testing MongoDB

So, this morning I was hacking around in the mongo shell. I had come up with three different ways to aggregate the data I wanted, but wasn't sure about which one I should subsequently port to code to use within my application.

So how would I decide on which method to implement? Well, lets just choose the one that performs the best. Ok, how do I do that? Hmmm. I could download and install some of the tools out there, or I could just wrap the shell code in a function and add some timings. OR, I could use the same tool that I use to performance test everything else; JMeter. To me it was a no brainer.

So how do we do it?

There is a full tutorial here.

Simply put, you need to do the following:
  1. Create a Sampler class. 
  2. Create a BeanInfo class. 
  3. Create a properties file. 
  4. Bundle up into a jar and drop into the apache-jmeter-X.X\lib\ext folder 
  5. Update search_paths=../lib/ext/mongodb.jar in jmeter.properties if you place the jar anywhere else. 

How I did it

I tend to have a scratch pad project set up in my IDE, so I decided just to go with that. Just to be on the safe side, I imported all the dependencies from:
  • apache-jmeter-X.X\lib 
  • apache-jmeter-X.X\lib\ext 
  • apache-jmeter-X.X\lib\junit
I then created the two classes and the properties file.



I then exported the jar to apache-jmeter-X.X\lib\ext, and fired up jmeter.

Go through the normal steps to set the test plan up:
  1. Right click Test Plan and add a Thread Group. 
  2. Right click the Thread Group and add a Sampler, in this case a MongoDB Script Sampler. 
  3. Add your script to the textarea; db.YOUR_COLLECTION_NAME.insert({"jan" : "thinks he is great"}) 
  4. Run the test

Happy days.  You can then use JMeter as you would for any other sampler.


Future enhancements

This is just a hack that took me 37 minutes to get running, plus 24 minutes if you include this post. This can certainly be extended to allow you to enter the replicaset config details for instance and to pull the creation of the connection out so we're not initiating this each time run a test.

49 comments:

  1. Thank you very much for your article, this is the only value article I could find on using JMeter for MongoDB calls.
    I am not Java guy, so may I ask you to share your mongodb.jar please?
    Thank you.

    ReplyDelete
  2. Thank you, Archie. I may, over the weekend, put it it up on github. Stay tuned.

    ReplyDelete
  3. Thanks for your reply, Jan.
    I stay tuned!

    ReplyDelete
  4. Check out the initial push of #MongoMeter to #GitHub https://github.com/JanPaulEttles/mongometer

    ReplyDelete
  5. Hi Jan. I've seen mongo benchrun, but there is little or no docs, and I don't see how to pipe results so they can be viewed grapihcally, so I have found this really useful. I built the jar from github, but are you likely to be sharing a jar for future releases?

    ReplyDelete
  6. Hi Anonymous.

    Hold tight, I've been working on getting this all up on GitHub. Next I'll be tidying up and creating jars.

    Thanks for posting.

    I agree with you on the documentation for benchrun. I didn't know it existed until a couple of months ago. I've seen very few posts on the subject.

    ReplyDelete
  7. @archie I've pushed a jar to github. Let me know how it goes for you. It's a java7 64bit linux build...

    ReplyDelete
    Replies
    1. Thank you very much Jan!
      Will try tomorrow and let you know.

      Delete
    2. No problem, archie. Have a play and let me know if there are further enhancements that could be made.

      Delete
    3. Hello Jan,
      I cannot make it work, sorry.
      This is the error:

      2012/06/22 18:07:24 INFO - jmeter.engine.StandardJMeterEngine: Running the test!
      2012/06/22 18:07:24 ERROR - jmeter.JMeter: Uncaught exception: java.lang.NoClassDefFoundError: com/mongodb/ServerAddress
      at jan.mongometer.ScriptSampler.testStarted(ScriptSampler.java:280)
      at jan.mongometer.ScriptSampler.testStarted(ScriptSampler.java:268)
      at org.apache.jmeter.engine.StandardJMeterEngine.notifyTestListenersOfStart(StandardJMeterEngine.java:224)
      at org.apache.jmeter.engine.StandardJMeterEngine.run(StandardJMeterEngine.java:347)
      at java.lang.Thread.run(Unknown Source)

      I have installed jre7 (win7-64) and jmeter2.7 as you say in readme file, nothing helps.

      Maybe I missed something?

      Delete
    4. Thank you Jan, I really appreciate your help.
      I put driver to lib\ext\, and now there are no errors in the log.
      I can insert documents into collections.
      But now there is another issue: I never receive any response, for example for db.XXX.find();
      In listener it always shows response size 16, but only thread name displayed.

      Any suggestions this time?

      Delete
  8. @archie, have you placed the mongo java driver somewhere that jmeter can see? I put the jar in the same place as the mongometer jar.

    ReplyDelete
  9. Yes, @archie I do have some suggestions.

    I wrote this to work via eval, as such as long as it runs in the shell, I'm happy, I was never interested in returning any results for display in jmeter.

    This is on github, so feel free to extend it in anyway you want. It's easy to create a fork.

    If you have any specific user cases, then feel free to send them to me, though I can't give any guarantees around when I can implement them.

    I'll probably commit some quick changes tonight, so stay tuned.

    Please note (and as I alluded to earlier) that you need to ensure that anything added to the script field needs to run via eval. You cannot return a cursor from db.eval() So you cannot do this db.jmeter.find(), but you can do this db.jmeter.find().toArray()

    ReplyDelete
    Replies
    1. Thank you for your time answering my stupid questions Jan, I really appreciate that.
      I am not a developer, I just use jmeter for QA; so stuff that is simple for you could be beyond my understanding.
      I was looking for a jmeter plugin to send requests to mongodb and receive responses, same as I do using JDBC driver.
      Sorry.

      Delete
    2. No need to apologise.

      When you say requests, what do you mean? What type of requests? Are you running insert/save/update/find? Or are you doing something more exotic?

      Delete
    3. Have you tried the latest version of the jar?

      You can use db.jmeter.find().toArray() to get results of a find.

      You can use return db.jmeter.find().count() to get the count of a find.

      Delete
    4. Nothing exotic Jan.
      I would like to request data from mongodb with find(), get response, extract values into variables using xpath or regex, and re-use variables in further requests.

      Delete
    5. I tried:
      return db.tbl1.find(); no response.

      But I did not try your latest jar.
      I will try it.
      Thank you very much.

      Delete
    6. OK.

      Remember to add toArray() as you cannot return a cursor from db.eval(). So db.tbl1.find() needs to be db.tbl1.find().toArray()
      Remember to wrap it in a function too, so the whole thing will look like this function() {return db.tbl1.find().toArray();}

      Delete
    7. Thank you Jan, it works now!
      At least I am getting response from my local mongodb.
      It still returns [] from network mongodbs, but I will try to debug it, there could be network issues.

      Thank you!

      Delete
    8. Could you elaborate on what you mean by network mongo instances? Do you mean anything that is not 127.0.0.1? Are you trying to connect to a list of mongo servers?

      Delete
    9. This comment has been removed by the author.

      Delete
    10. @archie, I've just set up mongo running on a different box (ie not localhost/127.0.0.1) and mongometer worked as expected.

      Do you have anything in the jmeter.log file?
      Can you ping the IP from the box running jmeter?

      A response of [] kinda seems to indicate that you're getting an empty array returned from mongo.

      If I do this:
      function() {
      return db.jmeter.find({"foo":"bar"}).toArray();
      }

      I get this in return:
      []

      The find is not matching anything the collection, so the array is empty. This is expected behaviour. Try a query for which you know there will definitely be a result.

      Delete
    11. Hello Jan,
      Finally with your help I have everything working for remote mongodb.

      I found how to reproduce the fail:
      If there are two or more mongodb samples in one .jms file, only the last sample returns results, the previous mongodb samples return empty array [].

      This is log file record for multiple mongodb samples:
      2012/06/26 10:22:34 WARN - jmeter.engine.StandardJMeterEngine: Error encountered during shutdown of jan.mongometer.ScriptSampler@69b3af04 java.lang.NullPointerException
      at jan.mongometer.ScriptSampler.testEnded(ScriptSampler.java:330)
      at jan.mongometer.ScriptSampler.testEnded(ScriptSampler.java:322)
      at org.apache.jmeter.engine.StandardJMeterEngine.notifyTestListenersOfEnd(StandardJMeterEngine.java:236)
      at org.apache.jmeter.engine.StandardJMeterEngine.run(StandardJMeterEngine.java:454)
      at java.lang.Thread.run(Unknown Source)

      But anyway, I am absolutely happy now. Thank you!

      Delete
    12. Great stuff @archie.

      Let me know if you come across any issues or if you find any limitations or think of a way you'd find value in further development of mongometer.

      Delete
    13. Hi Jan,
      I've beed using your extension for a week already; everything works as expected.
      Thank you again.

      It would be great if you ever have time to find out why whith multiple mongodb samples only the last one returns response data.

      Delete
    14. Hi @archie.

      If you can post some of your samples (obfuscated for security reasons of course) and let me know what you're getting and what you're expecting, then I'm happy to take a look.

      Delete
    15. Thank you Jan.
      This is the jmx file:
      https://www.dropbox.com/sh/jdwgub72wkrvsyd/5xzG4UvWIx

      There are two samples, sample-1 sends request to local db-1; sample-2 sends request to local db-2.
      If both samples enabled, only the second one gets response from the db; the first sample returns [].
      Each sample works ok if another is disabled.

      Delete
  10. Great post, everything I needed to know written extremely succinctly. Good job!

    ReplyDelete
    Replies
    1. Thanks Anonymous. Glad it could be of some help to you.

      Delete
  11. Hi Jan, I am getting the same symptom that archie mentions on 7 July 2012 00:58 and 26 June 2012 18:30. This prevents me from having multiple mongodb script instances in the same thread. Have you had a chance to fix this?

    ReplyDelete
  12. Hi Ralph,

    Would be great if you could give an example of what you're trying to do.

    Caveat, this is by no means a mature code base - in fact I'd say this is a first cut, just to see if it was possible and designed to work on relatively simple scripts.

    The source code is in gitub, and I encourage you to play with it and add functionality to cover any shortcomings (of which there will be many) and fix any bugs.

    That being said, if I find any time over the wekend, I'll take a look.

    ReplyDelete
  13. Hi Jan,

    Thank you for responding so quickly! I need to get this working soon. It will take me a while to get familar enough with the JMeter Sampler requirements to see if the mongoDB script code meets al the requirements so I will attempt it this weekend.

    For today, I am looking into other less pretty (more complicated) ways to implement getting mongoDB values into JMeter variables (e.g. BSF Sampler using Groovy script to call mongo directly and pass values back to JMeter)

    Below is more information in case you have time this weekend to look into it.

    This is the request for the first MongoDB script:
    db.useraccount.find( { "email" : "ralph@navarrocomputing.com" }, { _id:1 } ).toArray()

    This is the actual and expected response data for the first MongoDB script when the second MongoDB script is disabled:
    [ { "_id" : "b4a634aa-e71b-410e-9ead-cada44b26ea0"}]

    I pull the _id value into an ID variable using ATLANTBH's JSON Path Extractor ( https://github.com/ATLANTBH/jmeter-components ) with the following:
    $[0]._id

    Debug PostProcessor shows the variable has been initialized properly so the mongoDB script and JSON path extractor worked:
    ID=b4a634aa-e71b-410e-9ead-cada44b26ea0

    Here is the problem:
    As soon as I enable a second mongoDB script that does a similar query to get another variable initialized with data from another db/collection, both mongoDB scripts return null:
    []

    Debug PostProcessor shows the variables also get set to NULL:
    ID=NULL
    ID2=NULL

    There is a NullPointerException that shows up in the JMeter Log Viewer only during the second run when both mongoDB scripts are enabled:

    2013/01/11 12:09:56 INFO - jmeter.engine.StandardJMeterEngine: Running the test!
    2013/01/11 12:09:57 INFO - jmeter.gui.util.JMeterMenuBar: setRunning(true,*local*)
    2013/01/11 12:09:57 INFO - jmeter.engine.StandardJMeterEngine: Starting ThreadGroup: 1 : Content Thread Group
    2013/01/11 12:09:57 INFO - jmeter.engine.StandardJMeterEngine: Starting 1 threads for group Content Thread Group.
    2013/01/11 12:09:57 INFO - jmeter.engine.StandardJMeterEngine: Thread will continue on error
    2013/01/11 12:09:57 INFO - jmeter.threads.ThreadGroup: Starting thread group number 1 threads 1 ramp-up 1 perThread 1000.0 delayedStart=false
    2013/01/11 12:09:57 INFO - jmeter.threads.ThreadGroup: Started thread group number 1
    2013/01/11 12:09:57 INFO - jmeter.engine.StandardJMeterEngine: All thread groups have been started
    2013/01/11 12:09:57 INFO - jmeter.threads.JMeterThread: Thread started: Content Thread Group 1-1
    2013/01/11 12:09:57 INFO - jmeter.threads.JMeterThread: Thread finished: Content Thread Group 1-1
    2013/01/11 12:09:57 INFO - jmeter.engine.StandardJMeterEngine: Notifying test listeners of end of test
    2013/01/11 12:09:58 WARN - jmeter.engine.StandardJMeterEngine: Error encountered during shutdown of jan.mongometer.ScriptSampler@14607108 java.lang.NullPointerException
    at jan.mongometer.ScriptSampler.testEnded(ScriptSampler.java:330)
    at jan.mongometer.ScriptSampler.testEnded(ScriptSampler.java:322)
    at org.apache.jmeter.engine.StandardJMeterEngine.notifyTestListenersOfEnd(StandardJMeterEngine.java:224)
    at org.apache.jmeter.engine.StandardJMeterEngine.run(StandardJMeterEngine.java:436)
    at java.lang.Thread.run(Thread.java:722)

    2013/01/11 12:09:58 INFO - jmeter.gui.util.JMeterMenuBar: setRunning(false,*local*)

    ReplyDelete
    Replies
    1. When you say you enable a second script are you saying you add a second script to the ThreadGroup?
      Can you share the second mongo script.

      Delete
    2. Yes, I added the second script to the same Thread Group by duplicating the first mongoDB script JMeter component and then modifying its fields. The Thread Group is set to 'Continue' after Sampler error.

      In the second mongoDB script, the mongoDB server is the same as used in the first script, but the db and collection are different.

      Here is the second script:
      db.cpath.find( { "profileID" : "${ID}", "name" : "Default" }, { "_id" : 1 } ).toArray()

      As you can see, the second mongoDB script will utilize the ID variable which was initialized from the previous mongoDB script to build the query.

      I use another JSON Path Extractor (child to the second MongoDB script) to put the returned mongodb value into the variable ID2:
      $[0]._id

      The weird thing is that the first mongodb script works by returning a proper value when the second is disabled. Enable both, and the both return NULL.

      It seems like there is some incompatibility that happens when multiple mongoDB scripts are enabled in the same Thread Group.

      Delete
    3. Ralph / archie

      Sorry archie, kinda missed your post, then moved jobs and had a second kid.

      Anyhoo, you should now be able to run against multiple dbs.

      As I said before, it was never designed for this, bu tit should be good to go now.

      (spent more time recreating ssh keys and syncing git than it took to write the 'fix')

      Let me know how it goes, and let me know if you need more functionality.

      Jp

      Delete
    4. ♫♫♫ Happy days are here again ♫♫♫

      Thank you Jan! Your fix worked great. I was able to get my work completed today and on time.

      Your mongoDB script component for JMeter is the easiest solution to use for manipulating mongo databases from JMeter.

      I'll be sure to let you know what other features would be useful. I'll also try to setup my environment to develop and help any way I can.

      btw, Congratulations on your new job and second kid!

      Delete
    5. Awesome. Glad it worked for you, Ralph.
      Hope archie gets word that you can now add a second sampler to the ThreadGroup.

      Delete
    6. Jan,
      My congratulations on your family changes!
      I just tried the latest version of you mongometer.jar - it works excellent.
      Really useful tool, and it is the only one I could find.
      Thank you very much.


      Delete
    7. Thanks Archie. Glad you find it useful.

      I'm hoping you and Ralph can give me some ideas for additional functionality, as I may now have some time to spend on dev again.

      Delete
  14. archie, ralph.

    I've pushed an update to mongometer https://github.com/JanPaulEttles/mongometer

    I've add some narrative here:
    http://exceptionallyexceptionalexceptions.blogspot.co.uk/2013/01/mongometer-v20.html

    Let me know your thoughts.

    ReplyDelete
  15. Hi Jan,

    Thank you very much for the tool.
    I started using this tool, but i am getting this error.
    " jan.mongometer.config.MongoSourceElement: MongoDB Source Config testStarted
    2013/07/11 11:34:32 ERROR - jmeter.JMeter: Uncaught exception: java.lang.NoClassDefFoundError: com/mongodb/MongoOptions
    at jan.mongometer.config.MongoSourceElement.testStarted(MongoSourceElement.java:204)
    at org.apache.jmeter.engine.StandardJMeterEngine.notifyTestListenersOfStart(StandardJMeterEngine.java:209)
    at org.apache.jmeter.engine.StandardJMeterEngine.run(StandardJMeterEngine.java:325)
    at java.lang.Thread.run(Unknown Source)
    "

    Can you pls help me out of this.
    Thanks

    ReplyDelete
    Replies
    1. java.lang.NoClassDefFoundError: com/mongodb/MongoOptions

      The NoClassDefFoundError, means that the class MongoOptions cannot be found.

      This means you haven't added the mongodb to the classpath.
      Have you downloaded it and added it to the classpath?

      You can get it from here...
      http://central.maven.org/maven2/org/mongodb/mongo-java-driver/

      Delete
    2. Hi Jan,

      Thank you very much for the help.

      By miss i had downloaded some java doc file.

      Now again i downloaded the proper file. And its working fine.
      Thank you. Its a great tool

      Delete
    3. Glad you like it.

      It is only meant to compare relative performances of different scripts trying to achieve similar results; so it's probably not ideal for testing for absolute performances.

      Let me know you get on and if there are any features you think can be added.

      Delete
  16. Hi thanks for your tool.
    but when i try to insert some data i am getting the following error
    jmeter.JMeter: Uncaught exception: java.lang.NoSuchMethodError: com.mongodb.MongoClientOptions.builder()Lcom/mongodb/MongoClientOptions$Builder;
    at org.apache.jmeter.protocol.mongodb.config.MongoSourceElement.testStarted(MongoSourceElement.java:139)
    at org.apache.jmeter.engine.StandardJMeterEngine.notifyTestListenersOfStart(StandardJMeterEngine.java:214)
    at org.apache.jmeter.engine.StandardJMeterEngine.run(StandardJMeterEngine.java:336)
    at java.lang.Thread.run(Unknown Source)

    I am using 2.10 mongo in win7 64 bit

    ReplyDelete
  17. Have you added the MongoDB java client to your classpath? See previous comments.

    ReplyDelete
  18. Have you added the MongoDB java client to your classpath? See previous comments.

    ReplyDelete
  19. Hi
    Thank you sooooooo much.. It works like charm. Now only learning on mongo and jmeter and if any prob comes in future i will ask.

    Thanks once again for immediate reply

    ReplyDelete