Monday 12 March 2012

Mongodb MapReduce scope variables

I recently had a requirement for conditional emission from a map function. Essentially I only wanted to emit where the date was within a given range.

In SQL, grouping a count for a given time granularity within a date range would look something like this:

I wasn't 100% sure about the 'right way' to achieve this in NOSql/MongoDB. So this is a solution.
It requires you to know about scope variables. Problem is, I found that scope variables are not very well documented. You can find more about scope variables in the MongoDB documentation MapReduce-Overview. The relevant parts are:

      [, scope : <object where fields go into javascript global scope >]
and
      scope - can pass in variables that can be access from map/reduce/finalize.

Back to this example. First, let's define some data:

Before we implement this, lets get it working at the command-line, in our mongo shell:

What is happening here?

Well, we're selecting the sub-document of this collection where the value of "meh" is "meh". Then we've defined two dates; from and to to represent the boundaries of the date range, we're including these within the MapReduce function call. Basically what this means is that we can use what ever is defined here in the Map function (btw, we can also use them in the Reduce and Finalize functions).

Once we have this working from the shell, it is straight forward to implement it. This is the very same implemented in Java.

Caveat: This is an example of how to use scope variables, I'm sure if you go to any of the Events or gmane.comp.db.mongodb.user group you'll get some advice straight from the 10Gen guys.

4 comments:

  1. Comment from http://www.reddit.com/user/Madd0g

    is it just me or is there no example on how the to/from variables are created in java?

    anyway, great example, on the very few occassions when I needed to pass params, I just injected them to the js function... which always felt wrong

    jsCompareCode = @"if (" + jsWordArray + ".indexOf(this.Words[item]) != -1) continue;";

    ReplyDelete
  2. I'm a bit confused by your example. Specifically, this part...

    "this.arrayOfStuff.forEach(function(hit) {...}"

    You reference "hit" in your function but then inside the block you refer to "stuff"...am I missing something?

    ReplyDelete
    Replies
    1. Thanks for reading and taking the time to comment.

      Your're 100% correct - I would have be thoroughly confused myself.

      The 'hit' referenced in both the shell and the java implementations were both typos; this typo (heh) thing happens when you take actual code, reduce it and then obfuscate it for presentational purposes. Both errors have been fixed.

      Does it make sense now? Is there anything that could be explained in more detail?

      Delete
  3. Excellent study! I really like this. I really like this a lot, how the money grubbing visual-data gnome within me personally desires much more! Many thanks with regard to discussing!

    LOL Boost



    Cheap FUT 14 Coins

    ReplyDelete