Exceptionally exceptional exceptions: February 2013

Tuesday 12 February 2013

JMeter multiline tooltip hack

I found something a bit annoying with mongometer; it doesn't handle tooltips very well.

Let me explain.

I wanted to show the full text for each of the settings for MongoOptions.

Take the autoConnectRetry.shortDescription property for example.


autoConnectRetry.shortDescription=If true, the driver will keep trying to connect to the same server in case that the socket cannot be established.  There is maximum amount of time to keep retrying, which is 15s by default.  This can be useful to avoid some exceptions being thrown when a server is down temporarily by blocking the operations.  It also can be useful to smooth the transition to a new master (so that a new master is elected within the retry time).  Note that when using this flag: - for a replica set, the driver will trying to connect to the old master for that time, instead of failing over to the new one right away - this does not prevent exception from being thrown in read/write operations on the socket, which must be handled by application Even if this flag is false, the driver already has mechanisms to automatically recreate broken connections and retry the read operations.  Default is false.

Now this may be seen as being overly verbose. Fair enough, but I wanted to show all of this text so that users don't keep having to refer to the API docs.

The problem is, because there is so much text on a single line, it extends off both ends of the screen and causes unwanted side effects when multiple virtual desktops are in use.

No problem I thought, as I cast my mind back to my Swing development days. I'll just wrap the text in html tags and create a mutliline tooltip.


autoConnectRetry.shortDescription=<html><b>autoConnectRetry</b><br><br>If true, the driver will keep trying to connect to the same server in case that the socket cannot be established.<br>There is maximum amount of time to keep retrying, which is 15s by default.<br>This can be useful to avoid some exceptions being thrown when a server is down temporarily by blocking the operations.<br>It also can be useful to smooth the transition to a new master (so that a new master is elected within the retry <br>Note that when using this flag: - for a replica set, the driver will trying to connect to the old master for that time, instead of failing over to the new one right away - this does not prevent exception from being thrown in read/write operations on the socket, which must be handled by application Even if this flag is false, the driver already has mechanisms to automatically recreate broken connections and retry the read <operations. <br><br>Default is false.</html>

Oh dear. It seems it isn't going to be that easy. The tooltip now simple displays the html tags.

Time to don my deerstalker and get out the magnifying glass. Let's work backwards.

I'm extending BeanInfoSupport and as such I'm depending on GenericTestBeanCustomizer to render the GUI. Let's look to see what is happening during the rendering process.


core\org\apache\jmeter\testbeans\gui\GenericTestBeanCustomizer.java(597)




text = propertyToolTipMessage.format(new Object[] { desc.getName(), desc.getShortDescription() });

OK. So we're passing the name and short description to the MessageFormat instance.

If we scroll up a bit we'll see the pattern that is being applied to this MessageFormat.


core\org\apache\jmeter\testbeans\gui\GenericTestBeanCustomizer.java(283)




propertyToolTipMessage = new MessageFormat(JMeterUtils.getResString("property_tool_tip")); //$NON-NLS-1$

Not quite. We have a little more work to do here. JMeterUtils is being used to fetch resources. Let's have a quick peek in there.


core\org\apache\jmeter\util\JMeterUtils.java(371)




ResourceBundle resBund = ResourceBundle.getBundle("org.apache.jmeter.resources.messages", loc); // $NON-NLS-1$

OK. We're finally there.


core\org\apache\jmeter\resources\messages.properties(696)




property_tool_tip={0}\: {1}

Let's update this to handle html. I don't want it to break anything else, so I'm going to apply the minimum I can get away with.


core\org\apache\jmeter\resources\messages.properties(696)




property_tool_tip=<html><b>{0}</b><br><br>{1}</html>

OK. I told a teenie weenie lie; I didn't need to add the bold tag and the two new lines, but I couldn't help it. Now the properties file looks like this:


autoConnectRetry.shortDescription=If true, the driver will keep trying to connect to the same server in case that the socket cannot be established.<br>There is maximum amount of time to keep retrying, which is 15s by default.<br>This can be useful to avoid some exceptions being thrown when a server is down temporarily by blocking the operations.<br>It also can be useful to smooth the transition to a new master (so that a new master is elected within the retry <br>Note that when using this flag: - for a replica set, the driver will trying to connect to the old master for that time, instead of failing over to the new one right away - this does not prevent exception from being thrown in read/write operations on the socket, which must be handled by application Even if this flag is false, the driver already has mechanisms to automatically recreate broken connections and retry the read <operations. <br><br>Default is false.

I've removed the opening and closing html tags, along with the bolding of the property name and the two new lines, as they are all now present in the format pattern which will be applied to all tooltips across the application. All we need to do when we want any markup within tooltips is simply add it to the property value and it will automagically be rendered. If you don't want markup, then don't add any.

This looks so much better and has the added benefit of addressing the unwanted side effects when using multiple virtual desktops.

I'd imagine this isn't the best way to implement this, but maybe it is; I didn't spend much time on this and I wouldn't be surprised if there is a well documented/clever-er way of achieving the same result without having to hack a global message format pattern.

If you don't like the bold and new lines, then the change to the pattern should be as follows:


core\org\apache\jmeter\resources\messages.properties(696)




property_tool_tip=<html>{0}\: {1}</html>

Saturday 2 February 2013

MongoDB Authentication

I recently updated mongometer to make it a bit more flexible. Shortly after releasing the new version, one of the users fed back an issue via a comment on the post. I booted up my machine, opened up my IDE, found the issue and had pushed the fix out to github within half-an-hour.

This isn't a quick turn-around, success story post. It quickly dawned on me that if I was going to do anything in the future with mongometer, I should really know a little more about how a user authenticates against a database within MongoDB. (I don't want to spend more than an hour or so on this as I've just cracked open a bottle of Nyetimber Classic Cuvee - I'm also cooking a chicken pie (ping me if you want the recipe) and I'd rather be finished this post before I finish the bottle.) Before diving into any documentation that may exist around MongoDB Security, I'll start with a few observations. So in typical man style, let's kick the tyres and then if required, RTFM.

Start up a mongod instance.


$ /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --fork --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log


$ ./mongo --port 27001

Create an admin user


> use admin


> db.addUser("mongouser","mongopass")


1

Restart mongod


$ sudo kill -15 $(ps -ef | grep mongo | grep -v grep | cut -f8 -d" ")


$ /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --fork --auth --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log


$ ./mongo --port 27001

Authenticate to admin


> use admin


switched to db admin


> db.aut("mongouser","mongopass")


Thu Jan 31 13:53:31.271 javascript execution failed (shell):1 TypeError: Property 'aut' of object admin is not a function


db.aut("mongouser","mongopass")


   ^




> db.aut("mongouser","mongopass")

Ooops. Fat-fingered it. Hang on, I think I've found Issue #1

Issue #1
If an admin user mistypes the auth command and not the credentials, then the actual credentials stay in the shell history, which persists across sessions. Any other user could potentially come along and view the shell history and pick the credentials up.

On the other hand, if the command is correct and either the username or password or both are incorrect, or indeed if the authentication attempts succeeds, then the command is not kept in the history. (The command history for the mongo shell is available in the same way as on a linux box - using the up arrow)


> db.auth("mongouser","mongopass0")


{ ok: 0.0, errmsg: "auth fails" }


0


> db.auth("mongouser0","mongopass0")


{ ok: 0.0, errmsg: "auth fails" }


0


> db.auth("mongouser0","mongopass")


{ ok: 0.0, errmsg: "auth fails" }


0

Ok. Let's authenticate against admin and continue.


> use admin


switched to db admin


> db.auth("mongouser","mongopass")


1

Oooops. I almost missed one there.

Issue #2
Until the mongod instance is restarted, any user can...


> use admin


switched to db admin


> db.system.users.find()


{ "_id" : ObjectId("510a58c6de50e136190f9ed7"), "user" : "mongouser", "readOnly" : false, "pwd" : "c49caa1cb6b287ff6b1deaeeb8f4d149" }

...grab the usernames and hashes.

So, now that I've restarted the mongod instance, any user is going to have to authenticate against admin to be able to view the contents of system.users.

Now, continuing on from entering incorrect credentials, I'm going to launch a dictionary attack and see what happens. Oh dear. Found another issue.

Issue #3
There is no lock-out. I wrote a quick hack to connect to the mongod instance, to switch over to admin and attempt to log in. Using a rather large dictionary (with "mongopass" tacked on at the end) I attempted to log in over a million times. This was only a crude single-threaded attempt that took around 17 seconds to complete, but it shows that there is no account lock out. I'm confident I could put together a multi-threaded brute-forcer if required. I'll need to look into this further to see if there is any brute forcing/dictionary attack alerting that can be configured or whether there is a lock-out policy that can be applied. I'm not ready to RTFM just yet.

Let's take a closer look at the format of the password in system.users.


c49caa1cb6b287ff6b1deaeeb8f4d149

That looks like an MD5 to me. Let's take a look in the code, which is available to cruise on github.

Wow! I got luck straight off-the-bat. db.js has the following method:


function _hashPassword(username, password) {


    return hex_md5(username + ":mongo:" + password);


}

With hex_md5 then referencing native_hex_md5 within utils.cpp:


void installGlobalUtils( Scope& scope ) {


    scope.injectNative( "hex_md5" , native_hex_md5 );


    scope.injectNative( "version" , native_version );


    scope.injectNative( "sleep" , native_sleep );


    installBenchmarkSystem( scope );


 }




    static BSONObj native_hex_md5( const BSONObj& args, void* data ) {


    uassert( 10261, "hex_md5 takes a single string argument -- hex_md5(string)",


    args.nFields() == 1 && args.firstElement().type() == String );


    const char * s = args.firstElement().valuestrsafe();




    md5digest d;


    md5_state_t st;


     md5_init(&st);


    md5_append( &st , (const md5_byte_t*)s , strlen( s ) );


    md5_finish(&st, d);




    return BSON( "" << digestToString( d ) );


}

Time for a quick recap. Just in case you missed anything:

the hashing algorithm is MD5; my least favourite hashing algorithm.
the string to be hashed is in the form username + ":mongo:" + password; using the same "salt" is non-optimal...
the string :mongo: is global; I'm not really sure why it's there at all tbh.

I think this is probably enough to go with for now, else this will turn into a tl;dr and I may exceed my self imposed time constraints.

Thinking back to any discussions I had with regards to MongoDB, the same statements always arose within the context of Security.

Authentication is off by default.
MongoDB was always meant to be deployed in a trusted environment

I have to say that even with authentication on, we still have some gnarly issues. Further, I don't think a trusted environment exists.

Right then, time to RTFM with regards to Security. I'm hoping to find a roadmap defined that will deal with the issues stated above or there are already some mitigating steps that can be taken.

So, there are some Authentication features coming out in the near future. It looks like the new authentication features are only available under the MongoDB Subscriber Edition, I'm not sure what that means tbh... I also came across this know issue, which forms the basis for...

Issue #4
"if a user has the same password in multiple databases, the hash will be the same on all database. A malicious user could exploit this to gain access on a second database use a different users’ credentials." [sic]

Let's break that down.

"if a user has the same password in multiple databases, the hash will be the same on all database."

Yes. Correct. Same username, same password and same "salt" (ie the ":mongo:" string") equals same hash. OK, cool, let's move on.

"A malicious user could exploit this to gain access on a second database use a different users’ credentials." [sic]

A malicious user could exploit this if, and only if they have a non-readonly user on both databases involved.

If they only have readonly access, then they cannot list the system.users collection. In which case they will never see that the hashes are the same across different databases in the first place.

If they are not readonly, then they could list the system.users collection and take the hashed passwords offline to crack.

You're going to have to move into cracking territory if the hashes don't match across databases, in summary:

the user attribute would have be the same. The odds of different users on different databases having the user could be high.
the pwd attribute would have be the same. The odds of different users creating the same pwd is probably quite high.
the "salt" is the same, so it has no real relevance here.

So the problem here is that a user (that is not readonly) can pull all the password hashes for a given database and take them offline to crack. The malicious user already has the user name and the "salt", all they have to find is the password.

Conclusions

Issue #1
This one is a bit of a pain tbh. When the command is entered correctly (ignoring whether the credentials are correct or not) the command is not shown in the history. When the command is not entered correctly, then it is difficult to know what to exclude from the command history. I guess you could retrospectively remove commands that resulted in errors (ie invalid commands) that preceded the authentication. That is not a solution...

Issue #2
There may be an argument that once the admin user is created in system.users in the admin database that a restart should be forced.

Issue #3
A no-brainer. I've written password policies on multiple occasions (what a fun life I live, eh?), account lock-out is password 101.

Issue #4
It seems that creating a "salt" (":mongo:") per database would resolve the issue. Looking at the code, it looks like the implementation is a doddle, a quick and easy win. Adding the option to manually set it would be grand. Implementing a unique "salt" under the covers such that users didn't have to think about it would be equally grand.

So, Nyetimber finished, post finished.

I'm not saying that there is anything in this post that is new or clever, it's a cursory glance. I'm not having a go; everything I've mentioned is merely observation. I install mongo on almost a daily basis because it's a great product, I do however like having a balanced view and identifying any elephants in the room. I'd be interested in any feedback.