Key Stretching; an example

I've had quite a few questions about my previous post (from June 2010) on Passwords since I recently reposted it.

More specifically the questions and comments were around key stretching.

Q.  But doesn't looping that many times slow the password verification step down?
A.  Well yes.  That's kind of the point.  The user may experience negligible latency during the authentication process, this can be tuned to have no or little effect on the user experience.  This same delay is multiplied by the number of brute force attempts made by the attacker.

Q.  The attacker doesn't need the salt or the algorithm if they are going through the front door?
A.  Agreed.  Which is why you'd have some controls and triggers at the front door to alert when abnormal or atypical behaviour is detected.

Q.  If an attack gains access to a system, surely it's game over for user PII?
A.  The thing I like about key stretching is that a hacker needs to have access to the salt, the hashed password and stretching algorithm.  So, if you store the salt in a different table or even DB than the hashed password, the attacker would need to get their mitts on both tables or both DBs. Gaining visibility of the algorithm means that the attacker would also need to have access to, or knowledge about the specific implementation details; ie the source code. Typical environment configurations ensure that there is no way to get to a development environment from a production environment (and vice versa), making it difficult to gain access to the source.

This is a quick and dirty example of what key stretching would look implemented in Java.


This runs in around 0.75 seconds on my local box. Which means every iteration of a dictionary attack or a brute force attack would take the same time.

Caveat: this is purely an example of how you could implement stretching.

Performance testing MongoDB

So, this morning I was hacking around in the mongo shell. I had come up with three different ways to aggregate the data I wanted, but wasn't sure about which one I should subsequently port to code to use within my application.

So how would I decide on which method to implement? Well, lets just choose the one that performs the best. Ok, how do I do that? Hmmm. I could download and install some of the tools out there, or I could just wrap the shell code in a function and add some timings. OR, I could use the same tool that I use to performance test everything else; JMeter. To me it was a no brainer.

So how do we do it?

There is a full tutorial here.

Simply put, you need to do the following:

1. Create a Sampler class. 
2. Create a BeanInfo class. 
3. Create a properties file. 
4. Bundle up into a jar and drop into the apache-jmeter-X.X\lib\ext folder 
5. Update search_paths=../lib/ext/mongodb.jar in jmeter.properties if you place the jar anywhere else. 

How I did it

I tend to have a scratch pad project set up in my IDE, so I decided just to go with that. Just to be on the safe side, I imported all the dependencies from:

• apache-jmeter-X.X\lib 
• apache-jmeter-X.X\lib\ext 
• apache-jmeter-X.X\lib\junit

I then created the two classes and the properties file.



I then exported the jar to apache-jmeter-X.X\lib\ext, and fired up jmeter.

Go through the normal steps to set the test plan up:

1. Right click Test Plan and add a Thread Group. 
2. Right click the Thread Group and add a Sampler, in this case a MongoDB Script Sampler. 
3. Add your script to the textarea; db.YOUR_COLLECTION_NAME.insert({"jan" : "thinks he is great"}) 
4. Run the test

Happy days.  You can then use JMeter as you would for any other sampler.


Future enhancements

This is just a hack that took me 37 minutes to get running, plus 24 minutes if you include this post. This can certainly be extended to allow you to enter the replicaset config details for instance and to pull the creation of the connection out so we're not initiating this each time run a test.

Mongodb MapReduce scope variables

I recently had a requirement for conditional emission from a map function. Essentially I only wanted to emit where the date was within a given range.

In SQL, grouping a count for a given time granularity within a date range would look something like this:

I wasn't 100% sure about the 'right way' to achieve this in NOSql/MongoDB. So this is a solution.
It requires you to know about scope variables. Problem is, I found that scope variables are not very well documented. You can find more about scope variables in the MongoDB documentation MapReduce-Overview. The relevant parts are:

      [, scope : <object where fields go into javascript global scope >]
and
      scope - can pass in variables that can be access from map/reduce/finalize.

Back to this example. First, let's define some data:

Before we implement this, lets get it working at the command-line, in our mongo shell:

What is happening here?

Well, we're selecting the sub-document of this collection where the value of "meh" is "meh". Then we've defined two dates; from and to to represent the boundaries of the date range, we're including these within the MapReduce function call. Basically what this means is that we can use what ever is defined here in the Map function (btw, we can also use them in the Reduce and Finalize functions).

Once we have this working from the shell, it is straight forward to implement it. This is the very same implemented in Java.

Caveat: This is an example of how to use scope variables, I'm sure if you go to any of the Events or gmane.comp.db.mongodb.user group you'll get some advice straight from the 10Gen guys.

Refamiliarising oneself with Selenium

My main focus over the last few years has been Information Security and more specifically Application Security.  As such I've not cut much production code of late and have probably let much of my development practices atrophy.  However, recently I've had the opportunity to get back on the development bike and ride around a little.

I'm used to, as of course we all are, writing unit tests at the service layer, the persistence layer and of course the presentation layer, so nothing new there.

As an aside, I remember many years ago having a conversation with a front-end 'Architect' that stated you cannot unit test a front-end; cue HTMLUnit/HTTPUnit/JWebUnit and some red faces.  (I wonder how those frameworks have progressed in the last few years - perhaps the topic for a future post).

Anyway, I've been kicking around some front end code whilst picking up jQuery and what not, so I thought I'd wheel out some of the old favourite *Unit frameworks and I thought I'd also reacquaint myself with Fitnesse and Selenium skills.  But one step at a time.  I thought I'd get back into Selenium slowly and rather than jumping straight into the client/server setup, I went with the Selenium IDE - the Firefox plugin.  This is a post recording said reacquaintance process over the last day or so.

Getting Started
Download http://seleniumhq.org/download/
  Selenium IDE 1.5.0
  I'm running Firefox 9.0.1

Once you restart Firefox you'll want to bring up the Selenium UI.

1. Firefox -> Web Developer -> Selenium IDE.

I have created a small 'admin app' for the purposes of this post - slightly contrived - that allows me to CRUD an entity.  Its mainly basic HTML with some jQuery, and so that I can include more of the Selenium commands I've included other elements such as some frames; I have a navigation frame at the top of the page and a main frame below that which essentially acts as a content container.

So to the first test suite
The navigation frame is a good place to start.  To keep this as simple as possible I'm going to create a test case for each of the items in the navigation frame and verify the result by testing for the existence of some text in the content frame.

                                               "Home", "Search", "Create", "List", "Report"

Let the testing commence

1. Bring up the context in the left panel and select New Test Case.  
2. Rename the test by selecting properties. 
3. Hit record - the red circle up on the right hand side Selenium. 
4. On the browser, select Home. 
5. Wait for the frame to display, select some unique text to that page. 
6. Bring up the context menu and select 'verifyTextPresent your_unique_text' 
7. Run the test case.  The bar is red...

Oh dear.  What happened there?   The error is:
[error] There was an unexpected Alert! [Error! Status = 0 Message = error Message = ]

So from that, you can deduce the cause; its simple.  Well, it is and its not depending on your Selenium experience.   You'll have to change the logging level to figure out more information on the problem.  I set this at 'Debug', it's a fair bit more verbose, but its worth doing it to begin with.  When you select this level it becomes much clearer.  Selenium is failing when its trying to select the content frame.  Okay, so lets clickAndWait; tell Selenium to wait for the page that we just requested to finish loading.  Let try running that test again.  The bar is red....

Grrrr.  What happened there?  The error is:
[error] There was an unexpected Alert! [Error! Status = 0 Message = error Message = ]

Same error as before.  I (thought I'd) found two distinct solutions for this.  The first is just to slow the speed at which the testsuite executes down.  You can do this by going to the slider and moving it slightly to the right.  This however does not address the root cause.

The second is, and I should have used this before trying clickAndWait tbh, to waitForFrameToLoad, or so I thought.  I added this after the clickAndWait (I could just revert this clickAndWait to just click), adding the name of the frame and a timeout, but I started to get the same error as above when I ran the tests at full pelt.  In the end I lost my patience, and as a rule of thumb I run the test at ~20% slower than full speed.  ISSUE_1 If someone know the solution to the root cause, please ping me; I'd rather not waste any more time on that particular issue.

So back to the tests

1. Repeat the creation of test cases for each of the navigation elements.  Now you have five simple tests to run.  If the bar is green... and it is.  Sweet.

Now to save your work so far

1. File -> Save Test Suite As... 
2. Select a name and save. 

Now, let's simulate a cold start

1. Firefox -> Web Developer -> Selenium IDE 
2. File -> Open Test Suite...

Oh oh!

Ouch!  So what is going wrong here?  A quick look at the file that was written to disk only raises further questions. The file size for a start is only 1k.   On opening the file I'm presented with some html with only the test case names defined, no content.  Pants.  But, I was saving the testsuite after every change.  It turns out I wasn't saving the individual testcases.  I'll get my coat...  (This is so embarrassing I almost omitted it for this post)  Ah well, lesson learned, lets start again.  It is a bit annoying that you have to save each case individually, surely by saving the testsuite you're implicitly saying '...and save the testcases'.

I'm going to repeat the above, but this time save each individual testcase as I create it.  I've ended up with the following files:

Running the testsuite produces the following results; Runs: 5, Failures 0.  The bar is green...  Happy days.

Knock it up a notch

So these tests were really rather trivial, (barring the issue with waiting for a page to load...) so next we should tackle a form.

1. Bring up the context menu and create a New Test Case.  
2. Rename the test by selecting properties. 
3. Hit record. 
4. Hit Create on the navigation frame. 
5. Fill in the form fields one by one.
6. And stop recording.. 

Hey wait a minute.  Some of the form-filling didn't get recorded.  FFS!  ISSUE_2  After spending a bit of time searching the interweb, I found nothing to help me.  Grrr!  This didn't seem to make any sense, the form  tag was pretty standard, as were the input tags.

Well, it seems I was the architect of my own downfall; whilst trying to be a bit clever and have the input field label in the form field itself I'd inadvertently hit a Selenium edge case.  The issue was with the "placeholder" attribute of the input tag, after removing it from the input tag the actions were recorded as expected.  That was annoying.

    <input id="description" type="text" name="description" placeholder="some description" />

I could be lazy and remove all the placeholder attributes and record myself entering values into the form, but I'm going to use this as an opportunity to exercise my xpath skills.

Manual form filling

Doing this manually isn't that bad, plus I get some practice with xpath and with the Selenium IDE.  So for the first field of the form I want to emulate the user typing a unique identifier.

                   Basically this is saying, for the input field with the id attribute value of id, type in 667.

All of the fields follow the same pattern now all that needs to be done is to click the form submit button and verify that certain text is present.  Continuing in that vein I built up a sizeable set of tests always running the suite after each additional testcase.

After a while, working out the xpaths for elements became tedious, even for the more complex paths, so I ended up using FireBug (https://addons.mozilla.org/en-US/firefox/addon/firebug/)

Some tips

1. Remember to select the correct frame
2. Don't be clever and use the placeholder attribute
3. Use a plugin that will help you with the xpaths
4. Don't run the testsuite at maximum speed

Testing your document structure for inconsistencies within MongoDB - Part II

In a previous post we looked at how to validate your collection for structural consistency. We did this by creating a DBObject with the 'template' document structure and comparing this against the relevant collection. Testing your document structure for inconsistencies within MongoDB
Now, it's not exactly a leap of faith to extend that and rather than code the template we can define this in a json document. Now all we need to do it to pass the json document and the collection name as parameters saving a recompile for each and every test. Happy days.

Updating a document using the default ObjectId

Here are a few facts about keys and the way MongoDB deals with them.

• Documents in MongoDB require a key, _id, which uniquely identifies them.  
• This is a 12-byte binary value, read more from the source of truth, ObjectId.
• When inserting a new document, MongoDB will generate an ObjectId where one is not specified.
• There is a 'reasonable' chance that this will be unique at the time of creation.  
• All of the officially-supported MongoDB drivers use this type by default for _id values.

An annoyance I had with MongoDB, when I first started to use it, was the explicit update required on the default _id key/value pair.  To illusrate this here is a concrete example of what I mean.

I wanted to do this:
But I had to do this:
In this case, json is a JSON document passed from an HTML form via HTTP PUT, eventually ending up here at the DAO.  I was using the default _id implementation and thus would have expected save to do the figure that out and do the necessary work for me.

I just can't live those extraneous lines of code, it's too messy and too much to type every time.  I'll pull it out into a fromJson call, as I can see the pattern emerging and the odds of this reoccurring high.

I must have missed something in the API Docs as this cannot be an unusual requirement.  I would of at least expect to see it in a Util class...

Factory snippet:
Usage snippet:

I get to do what I wanted to now, but it still feels wrong.  It feels a little bit dirty.  It feels like a hack-o-la.  How can I do it better?

Convert HTML form to JSON and POST using jQuery

Reinventing the wheel is a pet hate of mine; I could list quite a few actually, but that's not the point of this post.  Sometimes it takes me so long to look for a wheel that I think has or should have already been invented that it would have been easier just to invent the wheel myself.

Recently I was playing around with jquery, pulling fields from a form and posting them as a JSON object to a set of services.  Nothing fancy, nothing difficult there.  However, when the data model started to mature beyond the most basic structure I soon realised I need an easy, repeatable way to pull structured/nested data from a a form and convert it into a JSON object prior to posting.

To create the following structure:


You will need to create the following references:


This is a complete working example that will parse the form, create the JSON obbject, POST it to a service and display the returned object in a table.



The form above will produce the following JSON.

Testing your document structure for inconsistencies within MongoDB

One of the advantages with schema-less design is that it works well for prototyping; you can have a collection of documents with each of the documents of variable structure. You can modify the document structure for one, some or all documents within the collection all without requiring a schema for the collection or each and every document.

However, this is also a disadvantage during prototyping; there are no constraints to stop documents within the same collection having variable structure. Deliberate updates to a document succeed silently as do accidental updates; ie when you update a document with a subdocument hanging off the wrong node. So when you assume you have consistency across all the documents, within a collection, but don't, you will run into some issues. You could also argue here that you're not coding defensively enough if you're not checking consistency at the time of execution; I'm not going to go into that right now though.

That exact structure inconsistency happened to me, and I ended up going down a rabbit hole. The smart thing to do was to blat the DB and recreate it during each test, but there were reasons that I didn't do that, which again I'm not going to go into here. Additionally the error that was coming back from performing an operation on the inconsistent structure wasn't obvious and didn't indicate to me that there was document structure inconsistencies, but that's another story.

Anyhoo, I didn't want this to happen again, so to verify the structural consistency of a collection I now pull in a json template; an example of the structure of the document I'm expecting to find within the collection I'm working with, and compare it to the collection in the DB. You can define your template in a json/txt file or you can manually create the DBObject. Simply put, I perform a symmetrical diff on the documents that are contained in the collection(s) I'm working with, and report on any additional field not defined in the template, I also report on any field that is defined in the template but not the document.

This example creates a DBObject with a few NVPs at the root, a couple as subdocument NVPs and finally an array.

We use this 'template' to compare against the documents within the tests collection.

This is all very lightweight but as a method to verify crude consistency it is very handy, for me anyway.

Security and MongoDB

The MongoDB Security Model has some scope for improvement.

1. As default, Authentication is off. MongoDB is designed to run in a 'trusted' environment, depending on the network configuration to ensure the security of the environment.
2. Pre v2 authentication is not supported within a sharded deployment.
3. Once authenticated a user has full read-write to the entire DB. There is no concept of roles or groups or such like.

Now, I wanted to deploy with Security turned up to the max, so here I will present a practical example.

This is a typical set-up I'd use for my development environment; separate machines each running mongod.

Firstly, let's set up the replicaset.
On the primary we can define and run the replicaset config:

Nothing fancy here, all standard stuff. After a few minutes the dust settles and the primary and secondary identify themselves. You can check the status of the replicaset with:

Now, on each box in the replicaset you'll need to create a key (must be valid Base64 and 1K or less):

You can now bring down both instances. Either hit ctrl+c or do it the right way:

Now we can start each of the mongod instances with the keyFile:

Word of warning here. I spent a significant amount of time trying to set this up on v2.0.0 - yes I know it is a bit dumb to go with a x.0.0 version, but you'd think that something as basic/fundamental as this would be thoroughly tested. Well, suffice to say I ended up moving to the latest binary to get this basic functionality to work. It was annoying at the time, but it certainly made me read the available documentation multiple times.
So now on the primary we can create the admin user aka root, super...

As we have a replicaset these users will existing in admin and mydb on both instances.

This whole process should take about 5 minutes to configure provided you're using a stable, well tested version.

I got unlucky and wasted hours on a buggy version grappling with errors that I couldn't decipher, feeling a bit denvercoder9 (http://xkcd.com/979/).

I went through the pain so you don't have to...

A simple guide to finding distinct array values in a MongoDB collection

So you want to find unique values within an array within a document in a collection? A reasonable request.

In ANSI SQL you'll be using DISTINCT, JOINS and GROUP BYs, stuff you're used to, but in the NoSQL realm your best bet is mapreduce.

It might seem a little bit like hard work, and probably a little intimidating at first, but it is certainly worth it; mapreduce is an extremely powerful tool.

Set up the collection, the map and reduce functions, and execute the mapreduce command:

But now you want to find unique values within an array within each document in an entire collection.

The map function in this example iterates over each of the items and emits the key/value of each array element.

The reduce function aggregates the key/value from each of the emits from the map function. In this example we're looking at unique keys and maintaining a count of the unique keys.

If you're looking to find the distinct array elements for a single document, simply specify the document index. For the entire collection, just leave the query out. *simples*

Simple mongodb equivalents

After making a noob error today, I realised that some of the documentation could use a boost.

By default mongo returns the entire document.

All fields for the matched document will be returned

And in java



To return only certain fields, you need to let mongo know which ones you want


The will only return the fields meh and feh from the matched document

And in Java


You can also say 'all fields except for...'



This will return all the fields in the matched document expect for meh

And in Java