tag:blogger.com,1999:blog-31327854149675393572023-11-16T06:33:14.121+00:00Exceptionally exceptional exceptionsAnonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.comBlogger45125tag:blogger.com,1999:blog-3132785414967539357.post-73161789163830027032013-04-14T23:15:00.000+01:002013-04-14T23:15:13.524+01:00A quick peek at nodeunit
I've been hacking around with <a href="http://nodejs.org/">node</a> for a while now (as many people have), but because much of it has been just that, hacking, and very little of the code base grows beyond a couple of hundred lines of code, I've not been writing unit tests. <i>tisk</i> <i>tisk</i>.
<br/>
<br/>
I'm not really planning anything big in the near future, but if I embark on building something that is more than a prototype or a hack I'm going have to unit test. So tonight I'm going to have a quick poke around to see what folks are using for unit testing node application.
<br/>
<br/>
I'm not sure what the de facto framework is, but I'm going to follow the pattern set over the years of _something_Unit. So, I'm going to punt at there being a NodeUnit. Google "nodeunit" -> Bang!
<br/>
<br/>
There is already a <a href="http://caolanmcmahon.com/posts/unit_testing_in_node_js/">good article</a> on getting started, so I'm not going to add any value here tbh. But I'll carry on and record my concise notes, just in case I need to refer to them myself in the future.
<br/>
<br/>
I'm on Ubuntu, though it shouldn't matter which platform you're on. (if truth be known, I've had some issues with code being cross-platform compatible with node)
<br/>
<br/>
<code>
$ sudo npm install nodeunit -g
</code>
<br/>
<br/>
I thought I'd knock up something that is at least vaguely security related. I'm not getting into arguments around random passwords vs memorable passwords, nor am I starting on hashing, salting or key stretching; this is simply something that allows me to test drive nodeunit quickly and simply.
<br/>
<br/>
The example is a simple, old-skool password policy. Stuff like:
<ul>
<li>Minimum length
<li>Upper case
<li>Lower case
<li>Numeric
<li>Special characters
</ul>
I'm keeping it simple. Ensure the above criteria are validated. So, here's the code. (there's nothing clever here)
<br/>
<br/>
<script src="https://gist.github.com/JanPaulEttles/5382227.js"></script>
The tests are simple too. I want to ensure the policy is enforced and that nodeunit can handle the tests I want to exercise.
<ul>
<li>If minimum length is not met, throw an error
<li>If it is not a string, throw an error
<li>If the criteria is not met, fail the test
</ul>
<script src="https://gist.github.com/JanPaulEttles/5382218.js"></script>
I added the random generation of test passwords - I thought about using <a href="http://en.wikipedia.org/wiki/Markov_chain">Markov Chains</a> to generate the passwords, but then thought, that's a good subject for a different post.
<br/>
<br/>
As an aside, I kinda like the fact that the validator is validating the test utility that generates passwords to be used in the tests, code validating the validation tests...
<br/>
<br/>
Anyhoo; running the tests...
<br/>
<br/>
<code>
$ nodeunit test_validator.js
</code>
<br/>
<br/>
The code ran first time, surprising since I wasn't using an IDE, it was the old fave TextPad. More surprising though was how quickly it ran; I upped the random generating of passwords to 5,000,000 and it took only 15 seconds to execute. Cool.
<br/>
<br/>
So, to summarise my fleeting introduction to nodeunit.
<ul>
<li>It's fast
<li>Easy to install
<li>Quick to install
<li>Easy to understand
<li>Fast
</ul>
I'll kick the tyres of nodeunit a little more in the future should I get more serious with node, and if there are any notable observations I'll add them as a comment to this post.
<br/>
<br/>
There are mode frameworks out there too, but from experience, you tend to stick with what works until it doesn't work. So, for me, I think nodeunit will be my de facto node...unit.Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com0tag:blogger.com,1999:blog-3132785414967539357.post-22148747844947708612013-03-19T20:22:00.000+00:002013-03-20T10:00:57.712+00:00CTF/DTF ideaI had a brief chat earlier today about your standard CTF experience at xyzSecCon. It's always been a bit biased towards breaking-in rather than defending, and biased towards networks/services/apps/etc rather than on just the app layer. <br />
<br />
Some initial thoughts for a potentially interesting and new(er) format are thus:<br />
<br />
There are teams of attackers<br />
There are teams of defenders<br />
<br />
Each defending team is given the same application to defend<br />
* The application is riddled with issues, you name it, it's there<br />
<br />
Each defending team has the same codebase with which to start from<br />
<br />
Each defending team is given a branch within the SCM repo<br />
* the build pipeline is in place<br />
* dependencies, project files and imports to different IDEs (eclipse, intellij) are all there<br />
* VMs are auto provisioned and releases are deployed to those VMs<br />
* essentially all the defenders have to think about is the code, everything else is in place<br />
<br />
Each branch and build track has the same tool set,<br />
* functional and non-functional unit tests including some security centric unit tests (xUnit, etc),<br />
* functional and non-functional user tests including some security centric user tests (Selenium or casper js, etc),<br />
* static code analysis (Sonar, Findbugs, Coverity, Fortify, etc),<br />
* automated web scanning (acunetix, burp, skipfish, etc)<br />
<br />
Each attacking team is given prior knowledge of all the flaws within said application<br />
* well in advance if required/to make it more fun<br />
<br />
Each attacking team can use any tool they see fit - this is about attack the app, not the network/service/OS<br />
* In all cases the attacking teams can add their own tools to the tool set<br />
<br />
Each defending team must deploy at least once an hour<br />
<br />
Each deployment goes to a new environment for that team - so if the CTF/DTF lasts over 2 (10 hour) days, then each team needs 20 different virtual envs.<br />
* This is so that any long running attacks have time to complete against any given 'release'<br />
<br />
The application contains your standard web functionality<br />
* Anonymous/Authenticated browsing<br />
* Registration<br />
* Login/Logout<br />
* Forgotten password<br />
* Common issues such as Captcha/Username already registered/etc<br />
* Register CC<br />
* Calls out to payment providers/etc<br />
* Shopping cart<br />
* Checkout<br />
* RSS feeds<br />
* Social network integration<br />
* etc etc, this is just a quick list<br />
<br />
One (of many) responses to consider<br />
* If there is a leakage of passwords - for instance - the defending team need to think about how they flick a switch so that all new logins are forced to change a password<br />
<br />
Obviously the attacking teams will find most vulns early on, so the idea is that the defending teams reduce the number of exploitable vulns over time. Attacking teams can attack all the deployments for all the defending teams, so the scoring is recorded in a matrix<br />
<br />
There can even be a set of dumb users; they can be socially engineered on an automagic basis - ie links in emails coming into email inboxes are automagically clicked<br />
<br />
Anyway, I've thought about this for a total of an hour or so, but I think there is something different in this, something different to your normal run-of-the-mill CTF<br />
<br />
It's usually easier to break stuff than it is to fix it. The initial challenge is in fixing rather than breaking.
<br />
<br />
Once the defending teams have gone through several release iterations, it will be harder to break stuff than it is to fix it. The longer challenge is in breaking rather than fixing.
<br />
<br />
Points will be awarded for the number of vulnerabilities detected; there will be a sliding scale for the level to which the vulnerability is exploited. For instance, displaying alert(1) on an attackers browser is 5 points, but harvesting the application user base session ids would be worth 50 points.
<br />
<br />
Different attacking teams will adopt different strategies. Do they go after the low hanging fruit that the defending teams will likely implement defences for in the first few releases; thus picking up some early points to add to their total but potentially wasting time developing exploits for the trickier issues that the defending team will not implement fixes for until release 10?Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com0tag:blogger.com,1999:blog-3132785414967539357.post-83872836024120494792013-02-12T15:59:00.000+00:002013-02-12T16:48:35.209+00:00JMeter multiline tooltip hackI found something a bit annoying with <a href="https://github.com/JanPaulEttles/mongometer">mongometer</a>; it doesn't handle tooltips very well.
<br />
<br />
Let me explain.
<br />
<br />
I wanted to show the full text for each of the settings for <a href="http://api.mongodb.org/java/2.10.1/com/mongodb/MongoOptions.html">MongoOptions</a>.
<br />
<br />
Take the <code>autoConnectRetry.shortDescription</code> property for example.
<br />
<br />
<code>
autoConnectRetry.shortDescription=If true, the driver will keep trying to connect to the same server in case that the socket cannot be established. There is maximum amount of time to keep retrying, which is 15s by default. This can be useful to avoid some exceptions being thrown when a server is down temporarily by blocking the operations. It also can be useful to smooth the transition to a new master (so that a new master is elected within the retry time). Note that when using this flag: - for a replica set, the driver will trying to connect to the old master for that time, instead of failing over to the new one right away - this does not prevent exception from being thrown in read/write operations on the socket, which must be handled by application Even if this flag is false, the driver already has mechanisms to automatically recreate broken connections and retry the read operations. Default is false.
</code>
<br />
<br />
Now this may be seen as being overly verbose. Fair enough, but I wanted to show all of this text so that users don't keep having to refer to the API docs.
<br />
<br />
The problem is, because there is so much text on a single line, it extends off both ends of the screen and causes unwanted side effects when multiple virtual desktops are in use.
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBXyfC1tgb5J6WB4-Cl9V9HJmtMzINtqSp1bPLnYOFGWH1QpJ-yVxsZj4vb6riAgq8_hZ5isvvFXU8X11woMQsrz7MovGFHc72_3iSGKWZqmmWptzLWgPOOvrfm6F3dVluYVcWYWAhBqQ/s1600/1.png" imageanchor="1" style=""><img border="0" height="238" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBXyfC1tgb5J6WB4-Cl9V9HJmtMzINtqSp1bPLnYOFGWH1QpJ-yVxsZj4vb6riAgq8_hZ5isvvFXU8X11woMQsrz7MovGFHc72_3iSGKWZqmmWptzLWgPOOvrfm6F3dVluYVcWYWAhBqQ/s400/1.png" /></a>
<br />
<br />
No problem I thought, as I cast my mind back to my Swing development days. I'll just wrap the text in html tags and create a mutliline tooltip.
<br />
<br />
<code>
autoConnectRetry.shortDescription=<html><b>autoConnectRetry</b><br><br>If true, the driver will keep trying to connect to the same server in case that the socket cannot be established.<br>There is maximum amount of time to keep retrying, which is 15s by default.<br>This can be useful to avoid some exceptions being thrown when a server is down temporarily by blocking the operations.<br>It also can be useful to smooth the transition to a new master (so that a new master is elected within the retry <br>Note that when using this flag: - for a replica set, the driver will trying to connect to the old master for that time, instead of failing over to the new one right away - this does not prevent exception from being thrown in read/write operations on the socket, which must be handled by application Even if this flag is false, the driver already has mechanisms to automatically recreate broken connections and retry the read <operations. <br><br>Default is false.</html>
</code>
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6nXqs_CO3s1owqjPftHsnYJdJt8XZ2_ewAxtTkHRUhP0r4rueTDLoKECWe_JD3M2IUd7yOqjQgnu9tuGV5FsV_HKMOfI3z07vZED7QQm-9hYZnW6zgtl83wxQKv5vVlcLh09kucQ_5TQ/s1600/2.png" imageanchor="1" style=""><img border="0" height="238" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6nXqs_CO3s1owqjPftHsnYJdJt8XZ2_ewAxtTkHRUhP0r4rueTDLoKECWe_JD3M2IUd7yOqjQgnu9tuGV5FsV_HKMOfI3z07vZED7QQm-9hYZnW6zgtl83wxQKv5vVlcLh09kucQ_5TQ/s400/2.png" /></a>
<br />
<br />
Oh dear. It seems it isn't going to be that easy. The tooltip now simple displays the html tags.
<br />
<br />
Time to don my deerstalker and get out the magnifying glass. Let's work backwards.
<br />
<br />
I'm extending <a href="http://jmeter.apache.org/api/org/apache/jmeter/testbeans/BeanInfoSupport.html">BeanInfoSupport</a> and as such I'm depending on <a href="http://jmeter.apache.org/api/org/apache/jmeter/testbeans/gui/GenericTestBeanCustomizer.html">GenericTestBeanCustomizer</a> to render the GUI. Let's look to see what is happening during the rendering process.
<br />
<br />
<code>
core\org\apache\jmeter\testbeans\gui\GenericTestBeanCustomizer.java(597)
<br />
<br />
text = propertyToolTipMessage.format(new Object[] { desc.getName(), desc.getShortDescription() });
</code>
<br />
<br />
OK. So we're passing the name and short description to the <a href="http://docs.oracle.com/javase/6/docs/api/java/text/MessageFormat.html">MessageFormat</a> instance.
<br />
<br />
If we scroll up a bit we'll see the pattern that is being applied to this <a href="http://docs.oracle.com/javase/6/docs/api/java/text/MessageFormat.html">MessageFormat</a>.
<br />
<br />
<code>
core\org\apache\jmeter\testbeans\gui\GenericTestBeanCustomizer.java(283)
<br />
<br />
propertyToolTipMessage = new MessageFormat(JMeterUtils.getResString("property_tool_tip")); //$NON-NLS-1$
</code>
<br />
<br />
Not quite. We have a little more work to do here. <a href="http://jmeter.apache.org/api/org/apache/jmeter/util/JMeterUtils.html">JMeterUtils</a> is being used to fetch resources. Let's have a quick peek in there.
<br />
<br />
<code>
core\org\apache\jmeter\util\JMeterUtils.java(371)
<br />
<br />
ResourceBundle resBund = ResourceBundle.getBundle("org.apache.jmeter.resources.messages", loc); // $NON-NLS-1$
</code>
<br />
<br />
OK. We're finally there.
<br />
<br />
<code>
core\org\apache\jmeter\resources\messages.properties(696)
<br />
<br />
property_tool_tip={0}\: {1}
</code>
<br />
<br />
Let's update this to handle html. I don't want it to break anything else, so I'm going to apply the minimum I can get away with.
<br />
<br />
<code>
core\org\apache\jmeter\resources\messages.properties(696)
<br />
<br />
property_tool_tip=<html><b>{0}</b><br><br>{1}</html>
</code>
<br />
<br />
OK. I told a teenie weenie lie; I didn't need to add the <b>bold</b> tag and the two new lines, but I couldn't help it. Now the properties file looks like this:
<br />
<br />
<code>
autoConnectRetry.shortDescription=If true, the driver will keep trying to connect to the same server in case that the socket cannot be established.<br>There is maximum amount of time to keep retrying, which is 15s by default.<br>This can be useful to avoid some exceptions being thrown when a server is down temporarily by blocking the operations.<br>It also can be useful to smooth the transition to a new master (so that a new master is elected within the retry <br>Note that when using this flag: - for a replica set, the driver will trying to connect to the old master for that time, instead of failing over to the new one right away - this does not prevent exception from being thrown in read/write operations on the socket, which must be handled by application Even if this flag is false, the driver already has mechanisms to automatically recreate broken connections and retry the read <operations. <br><br>Default is false.
</code>
<br />
<br />
I've removed the opening and closing html tags, along with the bolding of the property name and the two new lines, as they are all now present in the format pattern which will be applied to all tooltips across the application. All we need to do when we want any markup within tooltips is simply add it to the property value and it will automagically be rendered. If you don't want markup, then don't add any.
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1_qgEQ7HhaI25G3mbE4oVvNvBMQ3fy8ZGTHkhJEmFPxfrgXEhmYXID8dR03C6dZncQFT4KDsE8nxdRaZvGnle_qdNhL-z-29te9M3ZuavK9i_QCET-tWd2cXp3UddRjxHa72UdgiCP58/s1600/3.png" imageanchor="1" style=""><img border="0" height="238" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1_qgEQ7HhaI25G3mbE4oVvNvBMQ3fy8ZGTHkhJEmFPxfrgXEhmYXID8dR03C6dZncQFT4KDsE8nxdRaZvGnle_qdNhL-z-29te9M3ZuavK9i_QCET-tWd2cXp3UddRjxHa72UdgiCP58/s400/3.png" /></a>
<br />
<br />
This looks so much better and has the added benefit of addressing the unwanted side effects when using multiple virtual desktops.
<br />
<br />
I'd imagine this isn't the best way to implement this, but maybe it is; I didn't spend much time on this and I wouldn't be surprised if there is a well documented/clever-er way of achieving the same result without having to hack a global message format pattern.
<br />
<br />
If you don't like the bold and new lines, then the change to the pattern should be as follows:
<br />
<br />
<code>
core\org\apache\jmeter\resources\messages.properties(696)
<br />
<br />
property_tool_tip=<html>{0}\: {1}</html>
</code>
<br />
<br />
Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com0tag:blogger.com,1999:blog-3132785414967539357.post-80534614776119093672013-02-02T22:39:00.001+00:002013-02-02T23:38:51.001+00:00MongoDB AuthenticationI <a href="http://exceptionallyexceptionalexceptions.blogspot.co.uk/2013/01/mongometer-v20.html">recently</a> updated <a href="https://github.com/JanPaulEttles/mongometer">mongometer</a> to make it a bit more flexible. Shortly after releasing the new version, one of the users fed back an issue via a comment on the post. I booted up my machine, opened up my IDE, found the issue and had pushed the fix out to <a href="https://github.com/JanPaulEttles/mongometer/blob/master/src/main/java/jan/mongometer/mongo/MongoDB.java">github</a> within half-an-hour.
<br />
<br />
This isn't a quick turn-around, success story post. It quickly dawned on me that if I was going to do anything in the future with mongometer, I should really know a little more about how a user authenticates against a database within MongoDB. (I don't want to spend more than an hour or so on this as I've just cracked open a bottle of Nyetimber Classic Cuvee - I'm also cooking a chicken pie (ping me if you want the recipe) and I'd rather be finished this post before I finish the bottle.) Before diving into any documentation that may exist around MongoDB Security, I'll start with a few observations. So in typical man style, let's kick the tyres and then if required, RTFM.
<br />
<br />
Start up a mongod instance.
<br />
<br />
<code>
$ /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --fork --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log
<br />
$ ./mongo --port 27001
</code>
<br />
<br />
Create an admin user
<br />
<br />
<code>
> use admin
<br />
> db.addUser("mongouser","mongopass")
<br />
1
</code>
<br />
<br />
Restart mongod
<br />
<br />
<code>
$ sudo kill -15 $(ps -ef | grep mongo | grep -v grep | cut -f8 -d" ")
<br />
$ /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --fork --auth --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log
<br />
$ ./mongo --port 27001
<br />
</code>
<br />
<br />
Authenticate to admin
<br />
<br />
<code>
> use admin
<br />
switched to db admin
<br />
> db.aut("mongouser","mongopass")
<br />
Thu Jan 31 13:53:31.271 javascript execution failed (shell):1 TypeError: Property 'aut' of object admin is not a function
<br />
db.aut("mongouser","mongopass")
<br />
^
<br />
<br />
> db.aut("mongouser","mongopass")
<br />
</code>
<br />
<br />
Ooops. Fat-fingered it. Hang on, I think I've found Issue #1
<br />
<br />
<b>Issue #1</b>
<br />
If an admin user mistypes the auth command and not the credentials, then the actual credentials stay in the shell history, which persists across sessions. Any other user could potentially come along and view the shell history and pick the credentials up.
<br />
<br />On the other hand, if the command is correct and either the username or password or both are incorrect, or indeed if the authentication attempts succeeds, then the command is not kept in the history. (The command history for the mongo shell is available in the same way as on a linux box - using the up arrow)
<br />
<br />
<code>
> db.auth("mongouser","mongopass0")
<br />
{ ok: 0.0, errmsg: "auth fails" }
<br />
0
<br />
> db.auth("mongouser0","mongopass0")
<br />
{ ok: 0.0, errmsg: "auth fails" }
<br />
0
<br />
> db.auth("mongouser0","mongopass")
<br />
{ ok: 0.0, errmsg: "auth fails" }
<br />
0
</code>
<br />
<br />
Ok. Let's authenticate against <i>admin</i> and continue.
<br />
<br />
<code>
> use admin
<br />
switched to db admin
<br />
> db.auth("mongouser","mongopass")
<br />
1
</code>
<br />
<br />
Oooops. I almost missed one there.
<br />
<br />
<b>Issue #2</b>
<br />
Until the mongod instance is restarted, any user can...
<br />
<br />
<code>
> use admin
<br />
switched to db admin
<br />
> db.system.users.find()
<br />
{ "_id" : ObjectId("510a58c6de50e136190f9ed7"), "user" : "mongouser", "readOnly" : false, "pwd" : "c49caa1cb6b287ff6b1deaeeb8f4d149" }
</code>
<br />
<br />
...grab the usernames and hashes.
<br />
<br />
So, now that I've restarted the mongod instance, any user is going to have to authenticate against <i>admin</i> to be able to view the contents of <i>system.users</i>.
<br />
<br />
Now, continuing on from entering incorrect credentials, I'm going to launch a dictionary attack and see what happens. Oh dear. Found another issue.
<br />
<br />
<b>Issue #3</b>
<br />
There is no lock-out. I wrote a quick hack to connect to the mongod instance, to switch over to <i>admin</i> and attempt to log in. Using a rather large dictionary (with "mongopass" tacked on at the end) I attempted to log in over a million times. This was only a crude single-threaded attempt that took around 17 seconds to complete, but it shows that there is no account lock out. I'm confident I could put together a multi-threaded brute-forcer if required. I'll need to look into this further to see if there is any brute forcing/dictionary attack alerting that can be configured or whether there is a lock-out policy that can be applied. I'm not ready to RTFM just yet.
<br />
<br />
Let's take a closer look at the format of the password in <i>system.users</i>.
<br />
<br />
<code>
c49caa1cb6b287ff6b1deaeeb8f4d149
</code>
<br />
<br />
That looks like an <a href="http://en.wikipedia.org/wiki/MD5">MD5</a> to me. Let's take a look in the code, which is available to cruise on <a href="https://github.com/mongodb/mongo">github</a>.
<br />
<br />
Wow! I got luck straight off-the-bat. <a href="https://github.com/mongodb/mongo/blob/master/src/mongo/shell/db.js">db.js</a> has the following method:
<br />
<br />
<code>
function _hashPassword(username, password) {
<br />
return hex_md5(username + ":mongo:" + password);
<br />
}
</code>
<br />
<br />
With hex_md5 then referencing native_hex_md5 within <a href="https://github.com/mongodb/mongo/blob/master/src/mongo/scripting/utils.cpp">utils.cpp</a>:
<br />
<br />
<code>
void installGlobalUtils( Scope& scope ) {
<br />
scope.injectNative( "hex_md5" , native_hex_md5 );
<br />
scope.injectNative( "version" , native_version );
<br />
scope.injectNative( "sleep" , native_sleep );
<br />
installBenchmarkSystem( scope );
<br />
}
<br />
<br />
static BSONObj native_hex_md5( const BSONObj& args, void* data ) {
<br />
uassert( 10261, "hex_md5 takes a single string argument -- hex_md5(string)",
<br />
args.nFields() == 1 && args.firstElement().type() == String );
<br />
const char * s = args.firstElement().valuestrsafe();
<br />
<br />
md5digest d;
<br />
md5_state_t st;
<br />
md5_init(&st);
<br />
md5_append( &st , (const md5_byte_t*)s , strlen( s ) );
<br />
md5_finish(&st, d);
<br />
<br />
return BSON( "" << digestToString( d ) );
<br />
}
</code>
<br />
<br />
Time for a quick recap. Just in case you missed anything:
<ol>
<li>the hashing algorithm is <a href="http://en.wikipedia.org/wiki/MD5">MD5</a>; my least favourite hashing algorithm.
<li>the string to be hashed is in the form <code>username + ":mongo:" + password</code>; using the same "salt" is non-optimal...
<li>the string <code>:mongo:</code> is global; I'm not really sure why it's there at all tbh.
</ol>
I think this is probably enough to go with for now, else this will turn into a tl;dr and I may exceed my self imposed time constraints.
<br />
<br />
Thinking back to any discussions I had with regards to MongoDB, the same statements always arose within the context of Security.
<ol>
<li>Authentication is off by default.
<li>MongoDB was always meant to be deployed in a trusted environment
</ol>
I have to say that even with authentication on, we still have some gnarly issues. Further, I don't think a <i>trusted environment</i> exists.
<br />
<br />
Right then, time to RTFM with regards to Security. I'm hoping to find a roadmap defined that will deal with the issues stated above or there are already some mitigating steps that can be taken.
<br />
<br />
So, there are <i>some</i> <a href="http://docs.mongodb.org/manual/release-notes/2.4/#new-modular-authentication-system-with-support-for-kerberos">Authentication</a> features coming out in the <a href="http://docs.mongodb.org/manual/release-notes/2.4/">near future</a>. It looks like the new authentication features are only available under the <b>MongoDB Subscriber Edition</b>, I'm not sure what that means tbh...
I also came across this <a href="http://docs.mongodb.org/manual/tutorial/control-access-to-mongodb-with-authentication/">know issue</a>, which forms the basis for...
<br />
<br />
<b>Issue #4</b>
<br />
"if a user has the same password in multiple databases, the hash will be the same on all database. A malicious user could exploit this to gain access on a second database use a different users’ credentials." [sic]
<br />
<br />
Let's break that down.
<br />
<br />
"if a user has the same password in multiple databases, the hash will be the same on all database."
<br />
<br />
Yes. Correct. Same username, same password and same "salt" (ie the ":mongo:" string") equals same hash. OK, cool, let's move on.
<br />
<br />
"A malicious user could exploit this to gain access on a second database use a different users’ credentials." [sic]
<br />
<br />
A malicious user could exploit this if, and only if they have a non-readonly user on both databases involved.
<br />
<br />
If they only have readonly access, then they cannot list the <i>system.users</i> collection. In which case they will never see that the hashes are the same across different databases in the first place.
<br />
<br />
If they are not readonly, then they could list the <i>system.users</i> collection and take the hashed passwords offline to crack.
<br />
<br />
You're going to have to move into cracking territory if the hashes don't match across databases, in summary:
<ol>
<li>the <i>user</i> attribute would have be the same. The odds of different users on different databases having the <i>user</i> could be high.
<li>the <i>pwd</i> attribute would have be the same. The odds of different users creating the same <i>pwd</i> is probably quite high.
<li>the "salt" is the same, so it has no real relevance here.
</ol>
So the problem here is that a user (that is not readonly) can pull all the password hashes for a given database and take them offline to crack. The malicious user already has the <i>user</i> name and the "salt", all they have to find is the password.
<br />
<br />
<b>Conclusions</b>
<br />
<br />
<b>Issue #1</b>
<br />
This one is a bit of a pain tbh. When the command is entered correctly (ignoring whether the credentials are correct or not) the command is not shown in the history. When the command is not entered correctly, then it is difficult to know what to exclude from the command history. I guess you could retrospectively remove commands that resulted in errors (ie invalid commands) that preceded the authentication. That is not a solution...
<br />
<br />
<b>Issue #2</b>
<br />
There may be an argument that once the admin user is created in <i>system.users</i> in the <i>admin</i> database that a restart should be forced.
<br />
<br />
<b>Issue #3</b>
<br />
A no-brainer. I've written password policies on multiple occasions (what a fun life I live, eh?), account lock-out is password 101.
<br />
<br />
<b>Issue #4</b>
<br />
It seems that creating a "salt" (":mongo:") per database would resolve the issue. Looking at the code, it looks like the implementation is a doddle, a quick and easy win. Adding the option to manually set it would be grand. Implementing a unique "salt" under the covers such that users didn't have to think about it would be equally grand.
<br />
<br />
So, Nyetimber finished, post finished.
<br />
<br />
I'm not saying that there is anything in this post that is new or clever, it's a cursory glance. I'm not having a go; everything I've mentioned is merely observation. I install mongo on almost a daily basis because it's a great product, I do however like having a balanced view and identifying any elephants in the room. I'd be interested in any feedback.Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com2tag:blogger.com,1999:blog-3132785414967539357.post-91668376594117723802013-01-21T15:54:00.001+00:002013-01-21T16:08:56.680+00:00SpiderMonkey to V8, mongometer and the Aggregation FrameworkI previously <a href="http://exceptionallyexceptionalexceptions.blogspot.co.uk/2013/01/with-10gen-switching-default-javascript.html">posted</a> a comparison that covered running some simple queries against versions 2.2.2 and 2.3.2 of MongoDB. They were pretty basic examples, I just wanted to demonstrate one of the uses of <a href="https://github.com/JanPaulEttles/mongometer">mongometer</a>; comparing the relative performance of MongoDB releases and the MongoDB scripts you write to run on them.
<br />
<br />
Now I'm going to knock the complexity up a notch and perform a comparison between the different releases of MongoDB, their underlying JavaScript engines using another relatively new feature (added in version 2.1), the
<a href="http://docs.mongodb.org/manual/applications/aggregation/">Aggregation Framework</a>. I'm going to use the example they have in the documentation, mainly so I don't have to make something up.
<br />
<br />
<code>
{
<br />
title : "this is my title" ,
<br />
author : "bob" ,
<br />
posted : new Date () ,
<br />
pageViews : 5 ,
<br />
tags : [ "fun" , "good" , "fun" ] ,
<br />
comments : [
<br />
{ author :"joe" , text : "this is cool" } ,
<br />
{ author :"sam" , text : "this is bad" }
<br />
],
<br />
other : { foo : 5 }
<br />
}
</code>
<br />
<br />
<code>
db.articles.aggregate(
<br />
{ $project : {
<br />
author : 1,
<br />
tags : 1,
<br />
} },
<br />
{ $unwind : "$tags" },
<br />
{ $group : {
<br />
_id : { tags : "$tags" },
<br />
authors : { $addToSet : "$author" }
<br />
} }
<br />
);
</code>
<br />
<br />
I'm populating the collection with data in the form as described above. The only thing I'm adding is that I'm using the <a href="http://jmeter.apache.org/usermanual/component_reference.html#CSV_Data_Set_Config">JMeter CSV Data Set</a> to populate the author attribute.
<br />
<br />
<b>Version 2.2.2</b>
<br />
So let's make sure we're starting from a clean slate.
<br />
<br />
<code>
$ /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log
<br />
<br />
$ ps -ef | grep mongo
<br />
4974 /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log
<br />
<br />
$ ./mongo --port 27000
<br />
<br />
> show dbs
<br />
local 0.078125GB
<br />
test (empty)
</code>
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgdfbsgwb9r8HGAefwZzdYaQn0OIHP-yzNFYCF5Q2dTduKsEIR2fuo9YjwYG98awKj_sjQS9Ibi69tFB5CC6crL3QuiSUevEzsDlCgy-vrYbhD7yShi6pq-WGRYQ3MPBBuBvsAvWMGA-Y/s1600/27000+Source.png" imageanchor="1" style=""><img border="0" height="255" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgdfbsgwb9r8HGAefwZzdYaQn0OIHP-yzNFYCF5Q2dTduKsEIR2fuo9YjwYG98awKj_sjQS9Ibi69tFB5CC6crL3QuiSUevEzsDlCgy-vrYbhD7yShi6pq-WGRYQ3MPBBuBvsAvWMGA-Y/s400/27000+Source.png" /></a>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXAnf7qj_xP54hR1kPY1GbYh3M7OZDVAzU2I0PvXb_piCWcNO31Sv5gG6jCmoTDBwSvAHBwjLIrFQR7UDqjllSrfj5K6ceKJFrMEKYXKYFjVxnXOomN3V7u-ozSTba3wGOKzg-zVAtL8I/s1600/27000+Data.png" imageanchor="1" style=""><img border="0" height="255" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXAnf7qj_xP54hR1kPY1GbYh3M7OZDVAzU2I0PvXb_piCWcNO31Sv5gG6jCmoTDBwSvAHBwjLIrFQR7UDqjllSrfj5K6ceKJFrMEKYXKYFjVxnXOomN3V7u-ozSTba3wGOKzg-zVAtL8I/s400/27000+Data.png" /></a>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2WP21NdHQe5MszMDJOXyEdBSh7BGSNYyCQ1w-7Zx18cM2mLIZk2yk4ZZwgoeU82A5aLgNSekvkQFpra4KNSnH7T_AopOf8k1mnV9vZAi30LPj_5H_AWMCmuV5n13jmt9nLgr2Fhb7ccY/s1600/27000+Save.png" imageanchor="1" style=""><img border="0" height="255" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2WP21NdHQe5MszMDJOXyEdBSh7BGSNYyCQ1w-7Zx18cM2mLIZk2yk4ZZwgoeU82A5aLgNSekvkQFpra4KNSnH7T_AopOf8k1mnV9vZAi30LPj_5H_AWMCmuV5n13jmt9nLgr2Fhb7ccY/s400/27000+Save.png" /></a>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRdaGXrBYwNqh-BDtvRUZto2HTSEFQ3jPa0JMUxNd3omEGnwMSaBMFmflPwNiZEtFRGV4pCovmZpY1iGMfmA4lXouzYHay5JgbJSbOQ6H_p2GzxN5VG0DVLqVvzOeyO5azSJRr82ixXgQ/s1600/27000+Save+Graph.png" imageanchor="1" style=""><img border="0" height="255" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRdaGXrBYwNqh-BDtvRUZto2HTSEFQ3jPa0JMUxNd3omEGnwMSaBMFmflPwNiZEtFRGV4pCovmZpY1iGMfmA4lXouzYHay5JgbJSbOQ6H_p2GzxN5VG0DVLqVvzOeyO5azSJRr82ixXgQ/s400/27000+Save+Graph.png" /></a>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfmJDvV0SOsk7umFmlMi0-UpPr23NIjok6_09W-3TyxDkP85x6cIT3TaJEJDkwKqqNUHd5BeDmLsJ59jztpyDQ0U6ZRffxBjfXTY7jgZck7ToHZAIRIqDjRx7fEA9LOy42a-3ML3C9a_Q/s1600/27000+Aggregate.png" imageanchor="1" style=""><img border="0" height="255" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfmJDvV0SOsk7umFmlMi0-UpPr23NIjok6_09W-3TyxDkP85x6cIT3TaJEJDkwKqqNUHd5BeDmLsJ59jztpyDQ0U6ZRffxBjfXTY7jgZck7ToHZAIRIqDjRx7fEA9LOy42a-3ML3C9a_Q/s400/27000+Aggregate.png" /></a>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpViwqIDiKy3E_eYpYPSjuyCmsnbhqD8TU7VJ5N040nU2AUMCJ3mdsCd-mUT6nevKpsEN57OF9RHprSlg-jtfqjE9T4JD8SL90-FOS7R87ew4_YrunoM6Gej2l1dOFtxcYySh08bLypps/s1600/27000+Aggregate+Script.png" imageanchor="1" style=""><img border="0" height="255" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpViwqIDiKy3E_eYpYPSjuyCmsnbhqD8TU7VJ5N040nU2AUMCJ3mdsCd-mUT6nevKpsEN57OF9RHprSlg-jtfqjE9T4JD8SL90-FOS7R87ew4_YrunoM6Gej2l1dOFtxcYySh08bLypps/s400/27000+Aggregate+Script.png" /></a>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEit7bWTi8rOs7g1wC9iRQy1DsKerbyjr2fu0dz-DxrwPgZdGkBoxx1NkhSZ6HDjUjqYF_sEdD-Fv39tvIan3NuyB2wtD6btiDj_I69RQk6Ui9Dd9_ohUp9VbnepePRpHQ0kiucGocZgVWk/s1600/27000+Aggregate+Tree.png" imageanchor="1" style=""><img border="0" height="255" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEit7bWTi8rOs7g1wC9iRQy1DsKerbyjr2fu0dz-DxrwPgZdGkBoxx1NkhSZ6HDjUjqYF_sEdD-Fv39tvIan3NuyB2wtD6btiDj_I69RQk6Ui9Dd9_ohUp9VbnepePRpHQ0kiucGocZgVWk/s400/27000+Aggregate+Tree.png" /></a>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMkIaHj-WX-oQhDpXIVLD_2RLcTXXxLb03-uhJIDBnYDwm063YpMo32IKdOTRSecEa6zeudDT9n9zlY9OL9e66-yzIWezFOsA0k_-OhJemM2rgO7c5gW4MY4mZ-OqxyFEIkG6PoOVyQg4/s1600/27000+Aggregate+Graph.png" imageanchor="1" style=""><img border="0" height="255" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMkIaHj-WX-oQhDpXIVLD_2RLcTXXxLb03-uhJIDBnYDwm063YpMo32IKdOTRSecEa6zeudDT9n9zlY9OL9e66-yzIWezFOsA0k_-OhJemM2rgO7c5gW4MY4mZ-OqxyFEIkG6PoOVyQg4/s400/27000+Aggregate+Graph.png" /></a>
<br />
<code>
<br />
> show dbs
<br />
aggregation 0.203125GB
<br />
local 0.078125GB
<br />
test (empty)
<br />
<br />
> use aggregation
<br />
switched to db aggregation
<br />
> db.dropDatabase()
<br />
{ "dropped" : "aggregation", "ok" : 1 }
<br />
<br />
$ sudo kill -15 4974
</code>
<br />
<br />
<b>Version 2.3.2</b>
<br />
Let's ensure we have that same clean slate as with Version 2.2.2
<br />
<br />
<code>
$ /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log
<br />
<br />
$ ps -ef | grep mongo
<br />
1463 /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log
<br />
<br />
$ ./mongo --port 27001
<br />
<br />
> show dbs
<br />
local 0.078125GB
<br />
test (empty)
</code>
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSMCrHKO5r8Q8kjaiOJufo2VPDQtp26djILTpVpC0knZWxoU9DEMFcrKHhw3x57NpSy7UHBsbHyxv50nSlkXyIQvKTp3mx8YqMR8MhwdZSUYy7sdTPpYTdSWzYZcbUMLyNhDzZQpOu6FQ/s1600/27001+Source.png" imageanchor="1" style=""><img border="0" height="255" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSMCrHKO5r8Q8kjaiOJufo2VPDQtp26djILTpVpC0knZWxoU9DEMFcrKHhw3x57NpSy7UHBsbHyxv50nSlkXyIQvKTp3mx8YqMR8MhwdZSUYy7sdTPpYTdSWzYZcbUMLyNhDzZQpOu6FQ/s400/27001+Source.png" /></a>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigB1shYB3xFqQtLWSTFWmQvQSApad0O6fLN5MJZwiYqyBbGbHNm7NlBPonCEfYqWJY1sZtedg1T2siRzVX2tdmFNEd3DnRx4cxd-k1dP90l4XnacosHV9CSxhLff6Zvk_BkeLB6vCcXQA/s1600/27001+Data.png" imageanchor="1" style=""><img border="0" height="255" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigB1shYB3xFqQtLWSTFWmQvQSApad0O6fLN5MJZwiYqyBbGbHNm7NlBPonCEfYqWJY1sZtedg1T2siRzVX2tdmFNEd3DnRx4cxd-k1dP90l4XnacosHV9CSxhLff6Zvk_BkeLB6vCcXQA/s400/27001+Data.png" /></a>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjioPqRKTPiCmzdHFKCJz8ZWti2Fwyxn1m3Wewq88RGcJb-WVvLJm0VfxQR0YPX3Oae_I2dbgR40BIQ23EbZJnLx19rYTBm1gpbKXLgpNHm-3PIoKX94MS3LWW1Jej22W28ZkdP824gNFg/s1600/27001+Save.png" imageanchor="1" style=""><img border="0" height="255" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjioPqRKTPiCmzdHFKCJz8ZWti2Fwyxn1m3Wewq88RGcJb-WVvLJm0VfxQR0YPX3Oae_I2dbgR40BIQ23EbZJnLx19rYTBm1gpbKXLgpNHm-3PIoKX94MS3LWW1Jej22W28ZkdP824gNFg/s400/27001+Save.png" /></a>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSKqiiD-TFMlbK4QItZVBvkQ8RoMgTEiqiwkXuZOzpOIYsUf8YEDoN4-zz1RoLxpj1LEMM78gc6Wdtaq0U1AXNH17NEF8vPYd2WEWQIKiLh0PNPrJNxQvijs3nKG8Oxt5JQpIYKKibSUk/s1600/27001+Save+Graph.png" imageanchor="1" style=""><img border="0" height="255" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSKqiiD-TFMlbK4QItZVBvkQ8RoMgTEiqiwkXuZOzpOIYsUf8YEDoN4-zz1RoLxpj1LEMM78gc6Wdtaq0U1AXNH17NEF8vPYd2WEWQIKiLh0PNPrJNxQvijs3nKG8Oxt5JQpIYKKibSUk/s400/27001+Save+Graph.png" /></a>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhp0wbt1opCXw5bAs9ZjKAoOZn3s_Siu5K0Cj0TrH3P_WtXx3AUJpd4AAM-q1luSyr6CzKiRCVm1n9Gi4HyBKna4JAf08mb1kyxMkjSxjCoc4TMsgcbv2VzT6Kjf2sgRx1830zxeXbxqP8/s1600/27001+Aggregate.png" imageanchor="1" style=""><img border="0" height="255" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhp0wbt1opCXw5bAs9ZjKAoOZn3s_Siu5K0Cj0TrH3P_WtXx3AUJpd4AAM-q1luSyr6CzKiRCVm1n9Gi4HyBKna4JAf08mb1kyxMkjSxjCoc4TMsgcbv2VzT6Kjf2sgRx1830zxeXbxqP8/s400/27001+Aggregate.png" /></a>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1eeGabKka5JT15gbAgfNljqQ-bNWrpzyUqghjJy5CXPUFPRN8LPqjupso4aEexIMADR7Xvs0wg-1-3MJpfIePqDHaj_VnOQNGk73JGpLKLlskgIrD9bhADPylIaikBzTjmNIzUHtDE_o/s1600/27001+Aggregate+Script.png" imageanchor="1" style=""><img border="0" height="255" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1eeGabKka5JT15gbAgfNljqQ-bNWrpzyUqghjJy5CXPUFPRN8LPqjupso4aEexIMADR7Xvs0wg-1-3MJpfIePqDHaj_VnOQNGk73JGpLKLlskgIrD9bhADPylIaikBzTjmNIzUHtDE_o/s400/27001+Aggregate+Script.png" /></a>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEit6qRXl45x14g626lzyc10amXDi7gBLhrELEM6_w07UueKZAdQ-fq0tz_SPP7ICn686EWONoXz84vmV5kODwuPrWWz-cfbYkb7ssVXTjonMkzOx9-_Ik0Ummkqxui13lBrKrZmFwskhU0/s1600/27001+Aggregate+Tree.png" imageanchor="1" style=""><img border="0" height="255" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEit6qRXl45x14g626lzyc10amXDi7gBLhrELEM6_w07UueKZAdQ-fq0tz_SPP7ICn686EWONoXz84vmV5kODwuPrWWz-cfbYkb7ssVXTjonMkzOx9-_Ik0Ummkqxui13lBrKrZmFwskhU0/s400/27001+Aggregate+Tree.png" /></a>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQFEh_3e4Ger3mF-7WsT3KoLe3DYPH1twLwmBAXsRnnjXp0yptqdaRwIAlrLYiU2wWj6WxhzkhE2wA5h_6MpKnvzYQ0NO6nBAInx3UN7i5H3GuLuV3rLjVmlbOqhviVFiGf0JQ1_ZBi78/s1600/27001+Aggregate+Graph.png" imageanchor="1" style=""><img border="0" height="255" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQFEh_3e4Ger3mF-7WsT3KoLe3DYPH1twLwmBAXsRnnjXp0yptqdaRwIAlrLYiU2wWj6WxhzkhE2wA5h_6MpKnvzYQ0NO6nBAInx3UN7i5H3GuLuV3rLjVmlbOqhviVFiGf0JQ1_ZBi78/s400/27001+Aggregate+Graph.png" /></a>
<br />
<br />
<code>
> show dbs
<br />
aggregation 0.203125GB
<br />
local 0.078125GB
<br />
test (empty)
<br />
<br />
> use aggregation
<br />
switched to db aggregation
<br />
> db.dropDatabase()
<br />
{ "dropped" : "aggregation", "ok" : 1 }
<br />
<br />
$ sudo kill -15 1463
</code>
<br />
<br />
<b>Conclusions</b>
<br />
I ran this a few times and the results were consistent. I'll knock it up another notch over time, and hopefully draw out some useful conclusions, until then, you can draw your own.
<br />
<br />
Suggestions and comments welcome.Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com0tag:blogger.com,1999:blog-3132785414967539357.post-34988527968618124992013-01-20T20:38:00.000+00:002013-01-20T21:20:30.850+00:00SpiderMonkey to V8 and mongometerWith 10gen switching the default JavaScript engine for <a href="http://docs.mongodb.org/manual/release-notes/2.4/#default-javascript-engine-switched-to-v8-from-spidermonkey">MongoDB 2.3/2.4</a> from SpiderMonkey to V8 I thought I'd take the opportunity to compare the relative performances of the releases using <a href="https://github.com/JanPaulEttles/mongometer">mongometer</a>. Being a Security bod, I really should have looked at the <a href="http://docs.mongodb.org/manual/release-notes/2.4/#additional-authentication-features">Additional Authentication Features</a> first... Hey ho.
<br />
<br />
I'll document the steps taken during the comparison, including the set up, so this can be repeated and validated - just in case anyone is interested - but mainly so I can remind myself of what I did; memory, sieve.
<br />
<br />
<b>The set up</b>
<br />
I'm going to install 2.2.2 and 2.3.2 side-by-side on a dedicated machine. I'll then use the latest version of the Java driver with mongometer.
<br />
<br />
<code>
$ wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.3.2.tgz
</code>
<br />
<code>
$ wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.3.2.tgz.md5
</code>
<br />
<br />
I got a 403 response for this request...
<br />
<br />
<code>
$ wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.2.2.tgz
</code>
<br />
<code>
$ wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.2.2.tgz.md5
</code>
<br />
<br />
<code>
$ md5sum -c mongodb-linux-x86_64-2.2.2.tgz.md5
<br />
md5sum: mongodb-linux-x86_64-2.2.2.tgz.md5: no properly formatted MD5 checksum lines found
</code>
<br />
<br />
Grrr. An md5 file is supposed to be the checksum (then x2 spaces) and then the filename of the file being checksummed. I'll have to eyeball them instead, well, eyeball the one that I could actually download...
<br />
<br />
<code>
$ md5sum mongodb-linux-x86_64-2.2.2.tgz
<br />
be0f5969b0ca23a0a383e4ca2ce50a39 mongodb-linux-x86_64-2.2.2.tgz
</code>
<br />
<code>
$ cat mongodb-linux-x86_64-2.2.2.tgz.md5
<br />
be0f5969b0ca23a0a383e4ca2ce50a39
</code>
<br />
<br />
<b>Configure</b>
<br />
<code>
$ tar -zxvf ~/mongodb-linux-x86_64-2.2.2.tgz
</code>
<br />
<code>
$ sudo mkdir -p /usr/lib/mongodb/2.2.2
</code>
<br />
<code>
$ sudo mv mongodb-linux-x86_64-2.2.2/* /usr/lib/mongodb/2.2.2/
</code>
<br />
<code>
$ rm -r mongodb-linux-x86_64-2.2.2
</code>
<br />
<code>
$ sudo mkdir -p /data/db/2.2.2
</code>
<br />
<code>
$ sudo chown `id -un` /data/db/2.2.2
</code>
<br />
<code>
$ /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log
</code>
<br />
<br />
<code>
$ tar -zxvf ~/mongodb-linux-x86_64-2.3.2.tgz
</code>
<br />
<code>
$ sudo mkdir -p /usr/lib/mongodb/2.3.2
</code>
<br />
<code>
$ sudo mv mongodb-linux-x86_64-2.3.2/* /usr/lib/mongodb/2.3.2/
</code>
<br />
<code>
$ rm -r mongodb-linux-x86_64-2.3.2
</code>
<br />
<code>
$ sudo mkdir -p /data/db/2.3.2
</code>
<br />
<code>
$ sudo chown `id -un` /data/db/2.3.2
</code>
<br />
<code>
$ /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log
</code>
<br />
<br />
Let's check they are running.
<br />
<br />
<code>
$ ps -ef | grep mongod
<br />
1795 /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log
<br />
2059 /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log
</code>
<br />
<br />
Now, let's kill one (gracefully) and move on to the interesting stuff.
<br />
<br />
<code>
$ sudo kill -15 2059
<br />
$ ps -ef | grep mongod
<br />
1795 /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log
</code>
<br />
<br />
Now I'm jumping on to another box.
<br />
<br />
<code>
$ wget https://github.com/downloads/mongodb/mongo-java-driver/mongo-2.10.1.jar
</code>
<br />
<code>
$ cp mongo-2.10.1.jar /usr/lib/jmeter/2.8/lib/ext
<br />
$ cp ~/IdeaProjects/mongometer/out/artifacts/mongometer_jar/mongometer.jar /usr/lib/jmeter/2.8/lib/ext
<br />
$ /usr/lib/jmeter/2.8/bin/jmeter.sh
</code>
<br />
<br />
<b>The tests</b>
<br />
The tests are really rather basic; I'll perform an insert into two different databases, and perform finds against those databases.
<br />
<br />
Version 2.2.2
<br />
<br />
<code>
> show dbs
<br />
local 0.078125GB
<br />
</code>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVJlkze9T_9bDpu_wEzWCc94BpA7hyt_olwbKeDjtGsDcS1BRX4idZsi6Cv3NhWGIBnl8plOjaJyXCs5ABGTJCDAlgiaYqIirp2oJdO0ZIOrktGVjWLy529DIBSzftDVtx4zqHGx8RGxE/s1600/27000+Mongo+Source+Element.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVJlkze9T_9bDpu_wEzWCc94BpA7hyt_olwbKeDjtGsDcS1BRX4idZsi6Cv3NhWGIBnl8plOjaJyXCs5ABGTJCDAlgiaYqIirp2oJdO0ZIOrktGVjWLy529DIBSzftDVtx4zqHGx8RGxE/s400/27000+Mongo+Source+Element.png" /></a>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGMae3uuwhCnUVujXhgNdC9AaTtDQXQRGtfFf1F2uUzgw0p2ULHNewJeUr3GtWDrGzQhYpsWecw4Ib2dEB2hyphenhyphenDr8_yK3xu5Ww56qiD51ZGxJ8SIoa98pHk1tv8NibSMcDYqohST_91yn0/s1600/27000+find0.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGMae3uuwhCnUVujXhgNdC9AaTtDQXQRGtfFf1F2uUzgw0p2ULHNewJeUr3GtWDrGzQhYpsWecw4Ib2dEB2hyphenhyphenDr8_yK3xu5Ww56qiD51ZGxJ8SIoa98pHk1tv8NibSMcDYqohST_91yn0/s400/27000+find0.png" /></a>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgV63GFZ7gyn5b3UrkR_SFfQS9aUt-rw1G622Hjwy6BRfJmg1YYAaFtnt1VmS_0uJRz_4Gq3I51GYVrnT7OFDY5etBL5YOvuOqWjlq6IcRM3r15C47LS0M-qP7sUTXSVXfSi1k_Rpo5oKE/s1600/27000+find0+graph.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgV63GFZ7gyn5b3UrkR_SFfQS9aUt-rw1G622Hjwy6BRfJmg1YYAaFtnt1VmS_0uJRz_4Gq3I51GYVrnT7OFDY5etBL5YOvuOqWjlq6IcRM3r15C47LS0M-qP7sUTXSVXfSi1k_Rpo5oKE/s400/27000+find0+graph.png" /></a>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5nBEzJdBjOAVMu1gAxYYVAMPVTUDDCNlE7uNqOpQ-jJYtfnbdDfWInyZLtmHAEe_eHJlMT6PZsXPREa8HQL4GBkoLUiLMUDRiZoDDOMZofAXWRn0SmajAyEwXPTFobZ5HslFoYGnCW1g/s1600/27000+find1.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5nBEzJdBjOAVMu1gAxYYVAMPVTUDDCNlE7uNqOpQ-jJYtfnbdDfWInyZLtmHAEe_eHJlMT6PZsXPREa8HQL4GBkoLUiLMUDRiZoDDOMZofAXWRn0SmajAyEwXPTFobZ5HslFoYGnCW1g/s400/27000+find1.png" /></a>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfhvKzVLgv9vlhEgTSyTh6kxi9mOKKUwG2acDiCTzz-vq4SewX1yngpgUNAGIITpXhdbUlLc9xkIiuR75i8N_lr5eAb1ACqjq59skqlLuQPBSOUjRUwpAI6ssR9YGM9NzdV47n32OZn5U/s1600/27000+find1+graph.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfhvKzVLgv9vlhEgTSyTh6kxi9mOKKUwG2acDiCTzz-vq4SewX1yngpgUNAGIITpXhdbUlLc9xkIiuR75i8N_lr5eAb1ACqjq59skqlLuQPBSOUjRUwpAI6ssR9YGM9NzdV47n32OZn5U/s400/27000+find1+graph.png" /></a>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5TCz2g7_GrTLOnbjz4EROENy3MgJ_bI_VkQEGM5PLeNoIHLvAlMb_hWdmPGUs6iS-QebByhyphenhyphenSK1TC4VNQpqCy55Zdx0uC2tONasMMPKPG751Uo16bvy_HBZD6jlWQ_eHs_wUw_ee-IJ8/s1600/27000+insert0.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5TCz2g7_GrTLOnbjz4EROENy3MgJ_bI_VkQEGM5PLeNoIHLvAlMb_hWdmPGUs6iS-QebByhyphenhyphenSK1TC4VNQpqCy55Zdx0uC2tONasMMPKPG751Uo16bvy_HBZD6jlWQ_eHs_wUw_ee-IJ8/s400/27000+insert0.png" /></a>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidMbp8LLyOpGnHc60rwY0qgdET5wAzsCE_e0wIitYFzZqV9-vfJ6WCLIdNh6hEUvcQLWEYhrnww_KJ2bUZDPQDFvK0U5erVdNzRA4udeWeVzC7J9L1yy0Al_ASb8LaIP3oN6Nw8COmBuw/s1600/27000+insert0+graph.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidMbp8LLyOpGnHc60rwY0qgdET5wAzsCE_e0wIitYFzZqV9-vfJ6WCLIdNh6hEUvcQLWEYhrnww_KJ2bUZDPQDFvK0U5erVdNzRA4udeWeVzC7J9L1yy0Al_ASb8LaIP3oN6Nw8COmBuw/s400/27000+insert0+graph.png" /></a>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifxcX97n2RGZT5OGdKdt65viPPb5gqpA4nBLPn1DjqGKEGEMlU7hijHoBS0n0Ld-eWafHu37eLJlmga-xdSw5Sd3vCmvmMk-bqhfnfbUqPDZkIDH_oyaZwlYGiDEvUjvzo4wmk3kXUbJQ/s1600/27000+insert1.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifxcX97n2RGZT5OGdKdt65viPPb5gqpA4nBLPn1DjqGKEGEMlU7hijHoBS0n0Ld-eWafHu37eLJlmga-xdSw5Sd3vCmvmMk-bqhfnfbUqPDZkIDH_oyaZwlYGiDEvUjvzo4wmk3kXUbJQ/s400/27000+insert1.png" /></a>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoE3JIxMTfykiXJz3F3upbrhqSS2LYSmZhxuIkzI9DYhtUf6HKIDkbbAiTt1rKaGoD86oaJNarSlW28DsNxJu8_boDzUIBs_BxrB7Qzaz5KI2lqRRa0jUFMHpAFiRcUkwuJzWyACD8BEc/s1600/27000+insert1+graph.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoE3JIxMTfykiXJz3F3upbrhqSS2LYSmZhxuIkzI9DYhtUf6HKIDkbbAiTt1rKaGoD86oaJNarSlW28DsNxJu8_boDzUIBs_BxrB7Qzaz5KI2lqRRa0jUFMHpAFiRcUkwuJzWyACD8BEc/s400/27000+insert1+graph.png" /></a>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnK6p8BxloQAYRv6VaY-mvERH_bUHSc0V5jo-EyyRrX2JdhLBQbOXguw7c-pqOPn0bnFid5JcuSSoxQUPgBxRilQoRtyov6DFP2lFc7tMxW8v8rQ5yy4uMjXXOE0NcJYrQ8SRii4f0xeg/s1600/27000+all+graph.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnK6p8BxloQAYRv6VaY-mvERH_bUHSc0V5jo-EyyRrX2JdhLBQbOXguw7c-pqOPn0bnFid5JcuSSoxQUPgBxRilQoRtyov6DFP2lFc7tMxW8v8rQ5yy4uMjXXOE0NcJYrQ8SRii4f0xeg/s400/27000+all+graph.png" /></a>
<br />
<br />
<code>
<br />
> show dbs
<br />
jmeter 0.203125GB
<br />
jmeter2 0.203125GB
<br />
local 0.078125GB
<br />
<br />
> use jmeter
<br />
> db.jmeter.find().count()
<br />
1000
<br />
> db.dropDatabase()
<br />
<br />
> use jmeter2
<br />
> db.jmeter.find().count()
<br />
1000
<br />
> db.dropDatabase()
<br />
<br />
$ ps -ef | grep mongo
<br />
2690 /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log
<br />
<br />
$ sudo kill -15 2690
<br />
$ ps -ef | grep mongo
<br />
</code>
<br />
Nothing. Let's get the 2.3.2 instance up and running.
<br />
<br />
<code>
$ /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log
<br />
<br />
$ ps -ef | grep mongo
2947 /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log
</code>
<br />
<br />
Version 2.3.2
<br />
<br />
<code>
> show dbs
<br />
local 0.078125GB
<br />
</code>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMcLlR2o5LDiqORE-usUq56ZbOGKbPQe_hbiOsisbXOlkHpcGwkGy3zsrnqGdD_ZMkIA5NiP4l0EvuXlmERqdeJo4AvoXQgy8Hvgfb9ltTZIw11kcVXBGrAI2ZyV00R8sPBs0B05GYsd0/s1600/27001+Mongo+Source+Element.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMcLlR2o5LDiqORE-usUq56ZbOGKbPQe_hbiOsisbXOlkHpcGwkGy3zsrnqGdD_ZMkIA5NiP4l0EvuXlmERqdeJo4AvoXQgy8Hvgfb9ltTZIw11kcVXBGrAI2ZyV00R8sPBs0B05GYsd0/s400/27001+Mongo+Source+Element.png" /></a>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFoO3BcZtLJqgE7TRxp1y70TOma6p4Qk24HEDFt5-jC4BtRmaESm4r_4glbgNsSTXJxHVlfAoHmTakXXlvQvB13S5kcYJVprs9120DXieno1cgCuIQX-IA4_7S3yeYQrun-kzlkgej0Vc/s1600/27001+find0.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFoO3BcZtLJqgE7TRxp1y70TOma6p4Qk24HEDFt5-jC4BtRmaESm4r_4glbgNsSTXJxHVlfAoHmTakXXlvQvB13S5kcYJVprs9120DXieno1cgCuIQX-IA4_7S3yeYQrun-kzlkgej0Vc/s400/27001+find0.png" /></a>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7gfQbpG3L7DJm0pNUmoounQUuVe-QY0jRtmh7VBRyvS7FKHRFXI8qWrdO2zAxKO8fccYamJHHfg-DP7snWiMTmihZO_2Q99uOnVDy4xOj3eNqgKue929Q3E9Wj6v0wrHVMipmUYg7tO8/s1600/27001+find0+graph.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7gfQbpG3L7DJm0pNUmoounQUuVe-QY0jRtmh7VBRyvS7FKHRFXI8qWrdO2zAxKO8fccYamJHHfg-DP7snWiMTmihZO_2Q99uOnVDy4xOj3eNqgKue929Q3E9Wj6v0wrHVMipmUYg7tO8/s400/27001+find0+graph.png" /></a>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGBbmSeHxpafRh1tmRypZA8EOJriaAOUtgxICW2QUTYEBkKCYSTOCu-esPaa7aqdJE90Uc_D70it0NbLQadz5vicAMQiv2A7OoYM2g-RzhXhS-5wofsh28yRxSH2mZw_oeSHfto-eR1w8/s1600/27001+find1.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGBbmSeHxpafRh1tmRypZA8EOJriaAOUtgxICW2QUTYEBkKCYSTOCu-esPaa7aqdJE90Uc_D70it0NbLQadz5vicAMQiv2A7OoYM2g-RzhXhS-5wofsh28yRxSH2mZw_oeSHfto-eR1w8/s400/27001+find1.png" /></a>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLzx6qZT63PChkasi7Av5Xo2EJ5mn0URl4G-tOpywz5F5vu5B1qmzdRn95BZ1wBhtlhY_JX2ZiAXsY9RKmC6cYELeg-oN8A6sXdkghNOIqQzpEZbZQl0xIvquF1uUC4VgkXMuoUxtSlMU/s1600/27001+find1+graph.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLzx6qZT63PChkasi7Av5Xo2EJ5mn0URl4G-tOpywz5F5vu5B1qmzdRn95BZ1wBhtlhY_JX2ZiAXsY9RKmC6cYELeg-oN8A6sXdkghNOIqQzpEZbZQl0xIvquF1uUC4VgkXMuoUxtSlMU/s400/27001+find1+graph.png" /></a>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlYhkS2VIuPBEcMTMSA52M_hcHjbfoRkkFnEYu8ROC2sC-WM2R6K4LVc1ydwMxDHZ4gnVE8BCqJrf4OJyfpfZbboiNzviFUgzeWCUzTTagyUaNAWtJoGZiy31PiymUVC5ly1hG9XytFT4/s1600/27001+insert0.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlYhkS2VIuPBEcMTMSA52M_hcHjbfoRkkFnEYu8ROC2sC-WM2R6K4LVc1ydwMxDHZ4gnVE8BCqJrf4OJyfpfZbboiNzviFUgzeWCUzTTagyUaNAWtJoGZiy31PiymUVC5ly1hG9XytFT4/s400/27001+insert0.png" /></a>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFSmhmRKE9xoX7RAGYbUIK8iz2MuNEjZ38WjvzH-EPH6MaTHrrbaGKtBEOPRdhcq_pp2Qe6PCvdLq2_Rp7Vp1KPnzdLvTr4691Mviv72abQTTN591w29v02ENoqWhK293RqzbQc4Fn_1o/s1600/27001+insert0+graph.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFSmhmRKE9xoX7RAGYbUIK8iz2MuNEjZ38WjvzH-EPH6MaTHrrbaGKtBEOPRdhcq_pp2Qe6PCvdLq2_Rp7Vp1KPnzdLvTr4691Mviv72abQTTN591w29v02ENoqWhK293RqzbQc4Fn_1o/s400/27001+insert0+graph.png" /></a>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLpVTUdoTvV3cWvWrPNmDfJzLvU3pFlJNHXWGSyNdO2_PKh_QAbGExjhSXReI4KXhRKynHWkl4YNVnOXCkU-_AMWUBTTB9HG9b3uCqycONaCLAbv2w51p9rNJm8LZrAMkBKOLZVT0teIk/s1600/27001+insert1.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLpVTUdoTvV3cWvWrPNmDfJzLvU3pFlJNHXWGSyNdO2_PKh_QAbGExjhSXReI4KXhRKynHWkl4YNVnOXCkU-_AMWUBTTB9HG9b3uCqycONaCLAbv2w51p9rNJm8LZrAMkBKOLZVT0teIk/s400/27001+insert1.png" /></a>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEid-onq8umAOExpCWLTa1Oh1IFPVo3TRs_GQBpYbSHvYiH1bqxbindGqp8e05aIbgEO2AA8u7F9rSXfkWLTVg8Drn7MMjQ8TMag69A4Sf42UUtIAyamSxKQdq-pOdDm624_orRjPhnhojY/s1600/27001+insert1+graph.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEid-onq8umAOExpCWLTa1Oh1IFPVo3TRs_GQBpYbSHvYiH1bqxbindGqp8e05aIbgEO2AA8u7F9rSXfkWLTVg8Drn7MMjQ8TMag69A4Sf42UUtIAyamSxKQdq-pOdDm624_orRjPhnhojY/s400/27001+insert1+graph.png" /></a>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtQyBfldDsBOAp27H6HZFOEj0Q2pihCK3KzCX-wHM9rWg6p9JqzAq6n-DuDbZWB94o25Jyd1_lKnbw-NPYIpacb1f6UKn6vN110VZ-MhluI0Bt2hbMfetbejYqXTSsJWEeUMLc542UJWE/s1600/27001+graph+all.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtQyBfldDsBOAp27H6HZFOEj0Q2pihCK3KzCX-wHM9rWg6p9JqzAq6n-DuDbZWB94o25Jyd1_lKnbw-NPYIpacb1f6UKn6vN110VZ-MhluI0Bt2hbMfetbejYqXTSsJWEeUMLc542UJWE/s400/27001+graph+all.png" /></a>
<br />
<br />
<code>
<br />
> show dbs
<br />
jmeter 0.203125GB
<br />
jmeter2 0.203125GB
<br />
local 0.078125GB
<br />
<br />
> use jmeter
<br />
> db.jmeter.find().count()
<br />
1000
<br />
> db.dropDatabase()
<br />
<br />
> use jmeter2
<br />
> db.jmeter.find().count()
<br />
1000
<br />
> db.dropDatabase()
</code>
<br />
<br />
<b>Conclusions</b>
<br />
I guess you should draw your own. I ran this a couple of times and am considering scripting it so the environments are cleaned down prior to each run, I could probably add more complex queries too. Perhaps if I find some time next weekend then I will.
<br />
<br />
If you have any suggestions, please leave a comment.Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com1tag:blogger.com,1999:blog-3132785414967539357.post-66563023668310694732013-01-19T20:30:00.001+00:002013-01-20T10:16:07.644+00:00mongometer v2.0A while back I knocked up <a href="http://exceptionallyexceptionalexceptions.blogspot.co.uk/2012_04_01_archive.html">mongometer</a> to compare the relative performance of MongoDB scripts. I then made some minor <a href="http://exceptionallyexceptionalexceptions.blogspot.co.uk/2012/06/mongometer-unhacked.html">changes</a>, and since then - and only recently - made additional changes based on feedback.
<br />
<br />
I've now made slightly more significant changes. I'll now cover them briefly.
<br />
<br />
The minimum JMeter version is now <a href="http://jmeter.apache.org/download_jmeter.cgi">2.8</a>, this is because it is dependent on a change introduce in JMeter 2.8.
<br />
<br />
I've created a MongoSourceElement which is accessible from the context menu; Add -> Config Element -> MongoDB Source Config
<br />
<br />
This pulls all the MongoDB connection details up a level and allows you to share it between multiple MongoScriptSamplers. This means you only need to define the connection once and associate a source name with the instance. When you create a MongoScriptSampler you can then reference the source name as defined in MongoSourceElement.
<br />
<br />
MongoScriptSampler now only includes the fields that you need. It allows you to specify the mongo source, the database, the database credentials and of course the script to run.
<br />
<br />
The way I'm currently using it is to create a MongoSourceElement.
<br />
Under that I create a ThreadGroup.
<br />
And under that I add the MongoDB Script v2.0, View Results Tree, and Graph Results.
<br />
<br />
If I have multiple script to run, I use the same grouping of, MongoDB Script v2.0, View Results Tree, and Graph Results. See the image below for an example.
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhIDI-OrxVtTmDfu0P1ke5oeZ89kFZMVVfOYeak5NuEU5QwcDoaaT0Wc-yTDgelzpWvTe_utyFwsGuFWcp8Ez3tSxRAfxA3Ysi04QBkRt1LTkmaaGsyZ3ClJ86IG7LSKxFU4CBuztQkPzc/s1600/Screenshot+MongoDB+Source+Config.jmx.png" imageanchor="1" style=""><img border="0" height="241" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhIDI-OrxVtTmDfu0P1ke5oeZ89kFZMVVfOYeak5NuEU5QwcDoaaT0Wc-yTDgelzpWvTe_utyFwsGuFWcp8Ez3tSxRAfxA3Ysi04QBkRt1LTkmaaGsyZ3ClJ86IG7LSKxFU4CBuztQkPzc/s400/Screenshot+MongoDB+Source+Config.jmx.png" /></a>
<br />
<br />
There is the added MongoUtils class which has a static method which allows you to specify multiple hosts (if you want) and breaks it out into each host and port, returning an <ArrayList>ServerAddress, which to be honest, I'm surprised isn't available under the Java MongoDB driver.
<br />
<br />
The other semi-useful addition is the QuickEnvironment. It basically allows you to start up mongod, mongo, jmeter and a tail of the jmeter log. You could do this in a script, but hey ho, I did it in a helper class.
<br />
<br />
The latest version is available on <a href="https://github.com/JanPaulEttles/mongometer">github</a>.
<br />
<br />
Please have a play, try and break it, make suggestions, give me feedback.
Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com25tag:blogger.com,1999:blog-3132785414967539357.post-6149418659849472482012-12-23T09:20:00.000+00:002013-02-16T20:33:17.121+00:00S3 MODThe first thing I ask myself is why? Why flash a new phone, running the latest OS, voiding the warranty and risk bricking it... Two, well, three reasons really.
<br/ >
<br/ >
The first reason; I don't like the feel of the Samsung gumpf on top of JellyBean; I'd had the phone for a month, but I still wasn't getting on with the Samsung layer. I guess I was so used to the LAF of my Nexus 7, which after 6 months, I'm still loving btw.
<br/ >
<br/ >
The second reason is, the battery life on the S3 is far worse than I expected. I'd go a day, without doing much on the phone, and I'd burn through the entire battery - the top two offenders being the screen and the OS. I'd had my HTC Desire for ages, as soon as it first came out in fact. I was getting used to having it on charge pretty much any time I sat down at my desk and was thinking, hoping that a new device wouldn't require charging all the time. The Nexus 7 battery life is awesome; I'd have to be playing a graphic intensive game for an entire day to get through a full battery charge.
<br/ >
<br/ >
The third reason; just because, I'm a geek, that's what geeks do.
<br/ >
<br/ >
So after flashing my S3 from stock to CM10, I had a couple of minor issues.
<br/ >
<br/ >
First of all, the <a href="http://galaxys3root.com/galaxy-s3-root/how-to-root-galaxy-s3-on-linuxubuntu/">Ubuntu walkthrough</a> didn't work for me, so I had to go down the <a href="https://plus.google.com/u/0/101582135773317791268/posts/Y6kk85sMmUL">Windows</a> root, sorry route. (Good, concise, clear tutorial; thanks @galaxys3root)
<br/ >
<br/ >
Secondly of all, the flashing of clockworkmod didn't seem to stick, though this seemed to remedy itself on the 3rd flash. I'm not sure why it stuck on the 3rd attempt tbh. I used Odin to root the phone, which worked first time. I also used Odin to replace Android System Recovery with CWM, the settings were the same each time flashed CWM, and it was only the 3rd time that it stuck. The settings I used were pretty much default but I deselected auto-reboot; so ensure all of the Options are deselected - this is so the phone doesn't reboot straight away and you can pull the battery.
<br/ >
<br/ >
Third of all, there was an issue was with the external SD card. CWM couldn't see it. The card would not format in the phone. None of my Ubuntu machines recognised the card, and Windows wanted to reformat it. "You need to format the disk in drive E: before you can use it."
<br/ >
<br/ >
I got a bit #LazyWeb about this and asked about it on the <a href="http://forum.cyanogenmod.org/topic/63884-gt-i9300-clockworkmod-cyanogen-10-nightly-external-sd-not-mounting/">cyanogenmod forum</a> and on <a href="http://android.stackexchange.com/questions/35891/gt-i9300-clockworkmod-cyanogen-10-nightly-external-sd-not-mounting">android.stackexchange</a> but didn't get too much back tbh.
<br/ >
<br/ >
Here's a tip for you if you still have data on there that you want to retrieve. On a Windows box you can run:
<br/ >
<br/ >
<code>
Start -> cmd -> chkdsk e: /f
</code>
<br/ >
<br/ >
Windows will then recognised it, and pop up the AutoPlay applet. You can then back up any data that you may have missed prior to flashing the phone.
<br/ >
<br/ >
I formatted the card as ExtFat. It reports 59.4GB even though it's a 64GB card.
That didn't work at and my progress bar was still at 0%
<br/ >
<br/ >
Let's try formatting on Ubuntu...
Dang! Ubuntu doesn't have support for ExFat; you'll need to
<br/ >
<br/ >
<code>
sudo add-apt-repository ppa:relan/exfat
<br/ >
sudo apt-get update && sudo apt-get install exfat-utils fuse-exfat
</code>
<br/ >
<br/ >
Let's have a look at the partitioning. You'll need gparted for this:
<br/ >
<br/ >
<code>
sudo apt-get update
<br/ >
sudo apt-get install gparted
</code>
<br/ >
<br/ >
Pop the card into the adapter.
<br/ >
Start up gparted.
<br/ >
unmount the card.
<br/ >
Format as ext4
<br/ >
Pop it back into the phone.
<br/ >
<br/ >
I didn't get an error this time and the phone shows the message that it is preparing the card, but it doesn't mount.
<br/ >
You still get the same error through CWM too.
<br/ >
<br/ >
Pop the card into the adapter.
<br/ >
Start up gparted.
<br/ >
unmount the card.
<br/ >
Format as ntfs
<br/ >
Pop it back into the phone.
<br/ >
<br/ >
I didn't get an error but the phone still shows the message that it is preparing the card, but doesn't mount
<br/ >
<br/ >
Ah, well. Looks like I'm going to have to settle for 32GB.
<br/ >
<br/ >
Pop the card back into the adapter.
<br/ >
Start up gparted.
<br/ >
unmount the card.
<br/ >
Format as fat32; gparted show the partition as 29.23GB.
<br/ >
Pop it back into the phone.
<br/ >
I didn't get an error. Wait... Wait... Wait... It's mounted!
<br/ >
Go to ICS SD Binder set the mapping to the desired path, and hit save and reboot.
<br/ >
When the phone restarts, you'll see this:
<br/ >
Total space 59.43GB
<br/ >
Available space 59.43GB
<br/ >
<br/ >
I didn't really expect that tbh. It looks like it has taken the 29.23GB partition and automagically repartitioned it as 59.43GB. Awesome!
<br/ >
<br/ >
It's still very early days, but CM10 is looking good so far.
<br/ >
<br/ >
The biggest two battery offenders are still the screen and the OS. I have some spare batteries on order, and with the Samsung gumpf now off my phone, the battery is my only gripe with the S3.
<br/ >
<br/ >
I'm going to end with a question. What does the Samsung layer give that I can't get from CM10 and the app store?
<br/ >
<br/ >
For example: AllShare... I've replaced with BubbleUPnP
<br/ >
<br/ >
<b>Useful Links:</b>
<br/ >
<br/ >
<a href="http://goo.im/gapps">Google Apps</a>
<br/ >
<br/ >
<a href="https://play.google.com/store/apps/details?id=com.paolinoalessandro.cmdownloader&feature=search_result#?t=W251bGwsMSwyLDEsImNvbS5wYW9saW5vYWxlc3NhbmRyby5jbWRvd25sb2FkZXIiXQ..">CM10 Updater App</a>Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com8tag:blogger.com,1999:blog-3132785414967539357.post-10027057072658038272012-11-09T15:05:00.000+00:002012-11-14T09:48:19.423+00:00Integrating with Quality CentreI've recently had to go elbow deep to resolve an issue where a product developed by a company does not play well with a second product developed by the same company via a plugin, also developed by the same company.
<br/>
<br/>
As is my way, I'm posting the issue and the solution in a bid to reduce the average amount of pain felt.
<br/>
<br/>
<pre>
WARNING: HP Quality Center Plugin exception
com4j.ComException: 800413ed (Unknown error) : Parameter Type is Invalid : .\invoke.cpp:517
</pre>
<br/>
The fix for this turned out to be replacing the empty String with a com4j.Variant.Type.VT_NULL
<br/>
<pre>
import com4j.Variant;
</pre>
<br/>
Then replace
<br/>
<pre>
Com4jObject bugObject = bugFactory.addItem("");
</pre>
With
<br/>
<pre>
Com4jObject bugObject = bugFactory.addItem(new Variant(Variant.Type.VT_NULL));
</pre>
<br/>
Another tip is formatting the date.
<br/>
<pre>
WARNING: HP Quality Center Plugin exception
com4j.ComException: 80040519 (Unknown error) : Invalid date field value. : .\invoke.cpp:517
</pre>
<br/>
<pre>
DateFormat dateFormat = new SimpleDateFormat("dd/MM/yy");
Date date = new Date();
bug.field("BG_DETECTION_DATE", dateFormat.format(date));
</pre>
<br/>
The final tip is ensuring you use a user that exists on the system.
<br/>
<pre>
WARNING: HP Quality Center Plugin exception
com4j.ComException: 800403ff (Unknown error) : Dookie is not in the users list : .\invoke.cpp:517
</pre>
<br/>
<pre>
bug.field("BG_DETECTED_BY", "this has to be a valid user");
</pre>
<br/>
It's not all good news. For the life of me I couldn't find a solution to the this one and didn't want to get too deeply into the whole DLL thing. No matter how I specified the DLL; jvm params, properties et al of how I registered them or indeed which version I used, I could not overcome the following error. If oyu have a fix, let me know please. The only way to circumnavigate it was to downgrade from a 64-bit to a 32-bit version of the application. So I can only assume there is an issue with com4j-amd64.dll or com4j-x64.dll
<br/>
<pre>
com4j.ExecutionException: com4j.ComException: 80040154 CoCreateInstance failed : Class not registered : .\com4j.cpp:153
</pre>Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com0tag:blogger.com,1999:blog-3132785414967539357.post-27824536737284984422012-07-27T14:11:00.000+01:002012-07-27T15:05:51.067+01:00CouchBase: Permission deniedIt's 1400hrs, I have an hour or so to kill, so I going take some blog notes as I play with <a href="http://www.couchbase.com/">CouchBase</a>. I must be having one of the those weeks where nothing is going to go well; thankfully we're almost at the end of the week.
<br/>
<br/>
I'm finally up and running with an instance of CouchBase (though I had to change OS and grab the latest build), and I can see buckets and documents. Yay! The next logical step for me is to connect programmatically, and maybe create a prototype app. <br/>
<br/>
Let's watch the <a href="http://www.couchbase.com/develop/java/current">video</a> as it will only take 5 minutes. I love the fact this it titled "Get Started in 5 Minutes" but the video is 11 minutes long...<br/>
<br/>
To save you some time, download the <a href="http://packages.couchbase.com/clients/java/1.0.2/Couchbase-Java-Client-1.0.2.zip">driver</a> and add it to your classpath; saved you 10 minutes and 32 seconds.<br/>
<br/>
I believe a <a href="http://en.wikipedia.org/wiki/Hello_world_program">'Hello, World!'</a> program should demonstrate the simplest thing possible, so I just wanted to connect to the instance.<br/>
<br/>
<pre>
public static void main(String args[])
throws Exception {
URI server = new URI("http://192.168.0.8:8091/pools");
ArrayList<URI> servers = new ArrayList<URI>();
servers.add(server);
CouchbaseClient client = new CouchbaseClient(servers, "default", "");
}
</pre>
<br/>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmacWcil-NZuC61c1Z4YoMyOr_TffKv8-py2sRvE7u1MJi6OQteBRoc9IvbFl7WuGNC4Fv2YU8fNd7cAh3tYLUy5h9mAiAYA6EuBPZyJH8b0gsL5sUwOMYmlyiqTDP9xTxtp4C-RxEdbU/s1600/cb2-0.PNG" imageanchor="1" style=""><img border="0" height="320" width="309" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmacWcil-NZuC61c1Z4YoMyOr_TffKv8-py2sRvE7u1MJi6OQteBRoc9IvbFl7WuGNC4Fv2YU8fNd7cAh3tYLUy5h9mAiAYA6EuBPZyJH8b0gsL5sUwOMYmlyiqTDP9xTxtp4C-RxEdbU/s320/cb2-0.PNG" /></a>
<br/>
<br/>
It looks to be failing due to a "Permission denied" exception. FFS! I'm starting to get slightly miffed. Running a quick wget indicates there should be no such permission issues.
<br/>
<br/>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheHqa0Mnw-rQfTHHF6uJd-muP8l1MGrdAX7gDZ5w_LllnTKxHIqjKReePKoce6liC2wO_He-FjCbJgt5JOXHqBGw27BJGc_SKBPyUO_BAvLKEy2GdeKgp7qYswzf-0U0cfByM8QK009cw/s1600/cb2-1.PNG" imageanchor="1" style=""><img border="0" height="154" width="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheHqa0Mnw-rQfTHHF6uJd-muP8l1MGrdAX7gDZ5w_LllnTKxHIqjKReePKoce6liC2wO_He-FjCbJgt5JOXHqBGw27BJGc_SKBPyUO_BAvLKEy2GdeKgp7qYswzf-0U0cfByM8QK009cw/s320/cb2-1.PNG" /></a>
<br/><br />
Ah, ha! Fortunately I've seen this <a href="http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7077696">before</a>, it just took me moment for the old brain cells to remember. To fix the issue you need to add the follow as a VM param:<br/>
<pre>
-Djava.net.preferIPv4Stack=true
</pre>
<br/>
Now that I can finally connect, let's get the KVPs from a bucket.
It looks like <a href="http://www.couchbase.com/forums/thread/get-all-keys-bucket">TAP</a> is the way to go.<br/>
<br/>
<pre>
TapClient tc = new TapClient(servers, "default", "");
tc.tapDump("some id here");
</pre>
<br/>
tapDump returns a TapStream, but I can't find the TapStream dependency. <a href="https://groups.google.com/forum/#!msg/couchbase/PyUxHsUFUA8/rf47X2J1hXYJ">This dude</a> managed to find it, but looks like he may have other issues.<br/>
<br/>
I'm getting bored, and my head is getting sore with all this brick wall hitting. I know it's early, but I fancy a nice cold <a href="http://www.brewdog.com/product/never-mind-the-anabolics">Never Mind The Anabolics</a>, it's in a bottle so I don't have to worry about tap streams...
<br/>
<br/>
Oh, and it looks like my issue with installing on <a href="https://groups.google.com/forum/?fromgroups#!topic/couchbase/N9u6WSqcFQs">Win7 64-bit</a> has been around a while.Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com4tag:blogger.com,1999:blog-3132785414967539357.post-38079497949089287862012-07-26T10:01:00.001+01:002012-07-27T09:09:35.779+01:00CouchBase: SIMPLESo, yesterday we had the <a href="http://www.couchbase.com/">CouchBase</a> guys in the office, I think it was some kind of retaliatory strike to me having brought in the <a href="http://www.mongodb.org/">MongoDB</a> guys a few weeks prior.
<br />
<br />
In the interest of being able to form a balanced view I thought I'd test out the "SIMPLE" part of the CouchBase "SIMPLE, FAST, ELASTIC" tag line by downloading, installing, creating and query some data. I like <a href="http://dictionary.reference.com/browse/simple">simple</a> so this looks like it was made for me.
<br />
<br />
It takes me only a minute or so to update the version of MongoDB I run on my Ubuntu servers; it is very simple, but I've done it few times, so some of the ease of update is simply down to practice. This isn't really going to be a 100% fair and direct comparison if I'm completely honest. However, for the purposes of comparison we can use a <a href="http://docs.mongodb.org/manual/tutorial/install-mongodb-on-windows/">quick guide</a> to downloading, installing, creating and query some data. All the information you need is on one, simple page.
<br />
<br />
<b>MongDB</b><br />
Download: 27 seconds<br />
Unzip package: 4 seconds<br />
Create data directory: 9 seconds<br />
Start server: 19 seconds<br />
Start shell: 16 seconds<br />
Create a document: 16 seconds<br />
Query collection: 8 seconds<br />
<br />
Total time: About 2 minutes.
<br />
<br />
<b>CouchBase</b><br />
Download: 2 minutes, 31 seconds<br />
Install: (via the Wizard) 2 minutes 52 seconds + 3 minutes (Had to run this a seconds time as I was shown a 1618 error saying install was already in progress, which it wasn't. Also, after the wizard said it had finished the install there was no CouchBase directory in the default location and a browser didn't pop up pointing to the Admin UI, as per the install guide)
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDbYFkW2Ukq061a5fRdU-csuIkHiIa9UD9-53tkCeoqQuDtmOlX-2lpwNFlnTh_f-Ho0cLgmcybhUeq_Vtb1vfjgFLtHQAWDz8yrdDQp01bNzaGPWKXHSxpIMNRqJxM4Yoww_8iTfOaME/s1600/cb1.PNG" imageanchor="1"><img border="0" height="244" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDbYFkW2Ukq061a5fRdU-csuIkHiIa9UD9-53tkCeoqQuDtmOlX-2lpwNFlnTh_f-Ho0cLgmcybhUeq_Vtb1vfjgFLtHQAWDz8yrdDQp01bNzaGPWKXHSxpIMNRqJxM4Yoww_8iTfOaME/s320/cb1.PNG" width="320" /></a>
<br />
<br />
Configure server: 1 minute. I wasn't sure how much RAM I should be dedicating to CouchBase, nor was I completely sure which bucket type I should be choosing. The other niggly things with the install were the fact I had to complete the product registration before I could complete the configuration.<br />
<br />
I'm not sure what it's doing here tbh...<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1Due4hTBVGk5bVYZ4KlnieTRc_e81qqS4I8JtL7hCDvLghyFKSFwJstW0VCCtSKS_dKw29Xjvm6Ws_nXyPmyA_JQ029lyRPbEkCCUGuaBxycuDgiPfxxqNv-lPL_Raw3VoeRn4GcciqA/s1600/cb2.PNG" imageanchor="1"><img border="0" height="130" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1Due4hTBVGk5bVYZ4KlnieTRc_e81qqS4I8JtL7hCDvLghyFKSFwJstW0VCCtSKS_dKw29Xjvm6Ws_nXyPmyA_JQ029lyRPbEkCCUGuaBxycuDgiPfxxqNv-lPL_Raw3VoeRn4GcciqA/s320/cb2.PNG" width="320" /></a>
<br />
<br />
Start shell: No time recorded. I couldn't see a quick start, simple, single guide anywhere. I had to google "couchbase quick start guide" which initially took me to a page where I could request quotes around pricing and duration. Gave up after 10 minutes. Looks like I may have to do this via the Admin UI.
<br />
<br />
Insert data: No time recorded. As I couldn't find how to connect to the server via a shell, I hunted around and found this <a href="http://tugdualgrall.blogspot.co.uk/2012/07/couchbase-101-install-store-and-query.html">posting</a>; it looks simple. I couldn't find the 'Document button' that was being referred to though. I hunted around for another 15-20 minutes until I got frustrated and bored. I gave up.
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHQQYM7i1FFXfliuM3hIa1zN9QRy8RDPKqzsUhgG0f-jPdpjGxTK7X3iy2VM-Di-8HMvvAYyVcuITOPsQnenK1_Q1UpRr_LsVr0eWV918rfhyKB2Xzv_-G0kYQApfJaoNejstxZpFSJsM/s1600/cb3.PNG" imageanchor="1"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHQQYM7i1FFXfliuM3hIa1zN9QRy8RDPKqzsUhgG0f-jPdpjGxTK7X3iy2VM-Di-8HMvvAYyVcuITOPsQnenK1_Q1UpRr_LsVr0eWV918rfhyKB2Xzv_-G0kYQApfJaoNejstxZpFSJsM/s320/cb3.PNG" width="320" /></a>
<br />
<br />
Total time: DNQ
<br />
<br />
I'm afraid CouchBase has failed my definition of simple. Maybe they use a different dictionary, or maybe I'm just too simple. If anyone can point me towards a one-pager, simple, 101, 'get started quickly guide', then I'd appreciate it.
<br />
<br />
Let's give it another. So many people are using this that this must be a one off. So, I uninstall and restart my box. Then install afresh. I continue through the default settings. This time when I get to the Console I am met with this...
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4awb8PwmpA3naoifKtbbZsyYRDLcnn3ar6ylklCRPWiB09GuP4AA-utmdbLMEuYVk6iWqJpwXWW8sui-WMZx7n4PNVrsryshtap6xgS8bE068XleVAPrMNvTnTdgHd4lj2_OMSAXA7l4/s1600/cb4.PNG" imageanchor="1"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4awb8PwmpA3naoifKtbbZsyYRDLcnn3ar6ylklCRPWiB09GuP4AA-utmdbLMEuYVk6iWqJpwXWW8sui-WMZx7n4PNVrsryshtap6xgS8bE068XleVAPrMNvTnTdgHd4lj2_OMSAXA7l4/s320/cb4.PNG" width="320" /></a>
<br />
<br />
Not only can I not create a Document, I can barely use the Admin UI because of the modal alert that keeps popping up... FFS!
<br />
<br />
To get over this I do the following:<br />
cd %COUCHBASE_HOME%\Server\bin<br />
service_stop.bat<br />
service_unregister.bat<br />
Edit the script service_register.bat<br />
Replace NS_NAME=ns_1@%IP_ADDR% with the 127.0.0.1.<br />
service_register.bat<br />
service_start.bat<br />
<br />
Final effort. I uninstalled yet again and downloaded the latest version 2.0.0-dev-preview-4 and ran through the install process. Everything came unstuck when the wizard hit the 'Computing space requirements'; we basically hung at this stage. I wasn't able to cancel the install, so I had to kill the process. I tried once more and one last time - same result. There is a saying, 'If at first you don't succeed, try once more and give up. <br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxlhU9QP5mi43kLv6zhQ9-35EH_lz46MBh0D0qEult5OsI0wAnqXzSNxUaloH3dwhVoVcSjtSeJX0NcxgtxqvWds695IU0YCvoGc0y81fonrkZEVI2dDgyE8zTyMPnXjmGRR1FbMJAgAA/s1600/cb5.PNG" imageanchor="1"><img border="0" height="245" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxlhU9QP5mi43kLv6zhQ9-35EH_lz46MBh0D0qEult5OsI0wAnqXzSNxUaloH3dwhVoVcSjtSeJX0NcxgtxqvWds695IU0YCvoGc0y81fonrkZEVI2dDgyE8zTyMPnXjmGRR1FbMJAgAA/s320/cb5.PNG" width="320" /></a>
<br />
<br />
<b>UPDATE:</b> 2012 Olympic Opening Day<br />
$wget http://packages.couchbase.com/builds/1495/couchbase-server-community_x86_64_2.0.0-1495-rel.deb<br />
$sudo dpkg -i couchbase-server-community_x86_64_2.0.0-1495-rel.deb<br />
<br />
dependency error<br />
<br />
$sudo apt-get install libssl0.9.8<br />
$sudo dpkg -i couchbase-server-community_x86_64_2.0.0-1495-rel.deb<br />
<br />
<br />
Now I can hit the Admin UI. I tap up 192.168.0.8:8091<br />
We're getting there; I want to install some samples; I chose the beer sample, no surprise really. I was thrown back an error.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitHXyXjbrBkh9cDQkdBk-bVhNG_7NB5vpGGmpAUkHBEAqpDlyJgM3Z2xB-6ROResZJ1ulkDF6l1Uv87cyyAaDQqIC2op31_k00xFH1xOFhL9Y4UiNzjCOwsI1EqyDMAkUINv5aMdRN4LQ/s1600/cb6.PNG" imageanchor="1"><img border="0" height="258" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitHXyXjbrBkh9cDQkdBk-bVhNG_7NB5vpGGmpAUkHBEAqpDlyJgM3Z2xB-6ROResZJ1ulkDF6l1Uv87cyyAaDQqIC2op31_k00xFH1xOFhL9Y4UiNzjCOwsI1EqyDMAkUINv5aMdRN4LQ/s320/cb6.PNG" width="320" /></a>
<br />
<br />
I hit next again and it said the sample was already installed. OK. So I unseleted the sample and hit next again. Success. So, semi-SIMPLE, it took less than 5 minutes; all I had to do was be careful about my choice of OS and grab the very latest build (-; <br />
<br />
<br />
Thanks to <b>Frank Weigel</b> who pointed me at the latest build.<br />
<br />
Initial thoughts are:
<ol>
<li>it's weird not having the shell and doing everything via a web UI</li>
<li>from the brief look at the beer sample, it looks to my uneducated eye that we are defining the functions by_key_value, by_location and brewery_beers in an almost utility <i>helper class</i> document.</li>
</ol>
<br />
Now may the time to go and find some 101 documentation (-;Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com4tag:blogger.com,1999:blog-3132785414967539357.post-50726117362147448532012-07-19T16:47:00.001+01:002012-07-19T17:01:07.610+01:00shell/mongo.js:91<br />
Today I found some time to play with the new MongoDB aggregation framework. It's not an official, stable release yet, but I'm not thinking about putting it into production just yet.<br />
<br />
I wanted to keep the existing instance on MongoDB intact, so I installed it in it's own directory and created a new data directory to mirror this need. These are the exact steps, if you're interested.<br />
<br />
$wget http://www.mongodb.org/dr/fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.1.2.tgz/download<br />
$mv download mongodb-linux-x86_64-2.1.2.tgz<br />
$tar -zxvf mongodb-linux-x86_64-2.1.2.tgz<br />
$sudo mkdir -p /usr/lib/mongodb/2.1.2<br />
$sudo mv mongodb-linux-x86_64-2.1.2/* /usr/lib/mongodb/2.1.2/<br />
$rm -r mongodb-linux-x86_64-2.1.2<br />
$sudo mkdir -p /data/db-2.1.2<br />
$sudo chown `id -un` /data/db-2.1.2<br />
$/usr/lib/mongodb/2.1.2/bin/mongod --dbpath /data/db-2.1.2 --logpath /data/db-2.1.2/mongod.log<br />
<br />
Now crack open a new terminal so you can connect.<br />
<br />
$cd /usr/lib/mongodb/2.1.2/bin<br />
$./mongo<br />
MongoDB shell version: 2.1.2<br />
connecting to: test<br />
Thu Jul 19 11:40:14 Error: couldn't connect to server 127.0.0.1:27017 src/mongo/ shell/mongo.js:91<br />
exception: connect failed<br />
<br />
Oh dear. But this shows mongod is running:<br />
<br />
$ps -ef | grep mongod<br />
<span style="background-color: white;">/usr/lib/mongodb/2.1.2/bin/mongod --dbpath /data/db-2.1.2 --logpath /data/db-2.1.2/mongod.log</span><br />
<div>
<br /></div>
<div>
What does the log say?</div>
<div>
<br /></div>
$less /data/db-2.1.2/mongod.log<br />
<br />
...<br />
Thu Jul 19 11:40:35 [initandlisten] waiting for connections on port 27017<br />
...<br />
<br />
Hmmm. Let's try again.<br />
<br />
$cd /usr/lib/mongodb/2.1.2/bin<br />
$./mongo<br />
<br />
Hmmm, now it's connecting. Well, I'm off to play with the aggregation framework now and I'll check this anomaly later.<br />
<br />
The time now is 1625hrs and it looks like 2.2.0-rc0 has just been released. Let's give that a go. <br />
<br />
$wget http://www.mongodb.org/dr/fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.2.0-rc0.tgz/download<br />
$mv download mongodb-linux-x86_64-2.2.0-rc0.tgz<br />
$tar -zxvf mongodb-linux-x86_64-2.2.0-rc0.tgz<br />
$sudo mkdir -p /usr/lib/mongodb/2.2.0-rc0<br />
$sudo mv mongodb-linux-x86_64-2.2.0-rc0/* /usr/lib/mongodb/2.2.0-rc0/<br />
$rm -r mongodb-linux-x86_64-2.2.0-rc0<br />
$sudo mkdir -p /data/db-2.2.0-rc0<br />
$sudo chown `id -un` /data/db-2.2.0-rc0<br />
$/usr/lib/mongodb/2.2.0-rc0/bin/mongod --dbpath /data/db-2.2.0-rc0 --logpath /data/db-2.2.0-rc0/mongod.log<br />
<br />
Crack open a new terminal so you can connect.<br />
<br />
$cd /usr/lib/mongodb/2.2.0-rc0/bin<br />
$./mongo<br />
<span style="background-color: white;">MongoDB shell version: 2.2.0-rc0</span><br />
connecting to: test<br />
Thu Jul 19 16:33:41 Error: couldn't connect to server 127.0.0.1:27017 src/mongo/shell/mongo.js:91<br />
exception: connect failed<br />
<div>
<br /></div>
<div>
Is it running?</div>
<div>
<br /></div>
<div>
$ps -ef | grep mongod
</div>
<div>
<div>
/usr/lib/mongodb/2.2.0-rc0/bin/mongod --dbpath /data/db-2.2.0-rc0 --logpath /data/db-2.2.0-rc0/mongod.log</div>
</div>
<div>
<br /></div>
<div>
Yes, it is. Bugger, I can't connect. Let's take a look at the log again.</div>
<div>
<br /></div>
<div>
$less /data/db-2.2.0-rc0/mongod.log
</div>
<br />
...<br />
Thu Jul 19 16:33:29 [initandlisten] journal dir=/data/db-2.2.0-rc0/journal<br />
Thu Jul 19 16:33:29 [initandlisten] recover : no journal files present, no recovery needed<br />
Thu Jul 19 16:33:31 [initandlisten] preallocateIsFaster=true 31.64<br />
Thu Jul 19 16:33:33 [initandlisten] preallocateIsFaster=true 30.5<br />
Thu Jul 19 16:33:37 [initandlisten] preallocateIsFaster=true 30.52<br />
Thu Jul 19 16:33:37 [initandlisten] preallocateIsFaster check took 8.333 secs<br />
Thu Jul 19 16:33:37 [initandlisten] preallocating a journal file /data/db-2.2.0-rc0/journal/prealloc.0<br />
Thu Jul 19 16:33:40 [initandlisten] 304087040/1073741824 28%<br />
Thu Jul 19 16:33:43 [initandlisten] 482344960/1073741824 44%<br />
Thu Jul 19 16:33:46 [initandlisten] 650117120/1073741824 60%<br />
Thu Jul 19 16:33:49 [initandlisten] 817889280/1073741824 76%<br />
Thu Jul 19 16:33:52 [initandlisten] 985661440/1073741824 91%<br />
Thu Jul 19 16:33:56 [initandlisten] preallocating a journal file /data/db-2.2.0-rc0/journal/prealloc.1<br />
Thu Jul 19 16:33:59 [initandlisten] 272629760/1073741824 25%<br />
Thu Jul 19 16:34:02 [initandlisten] 398458880/1073741824 37%<br />
Thu Jul 19 16:34:05 [initandlisten] 555745280/1073741824 51%<br />
Thu Jul 19 16:34:08 [initandlisten] 723517440/1073741824 67%<br />
Thu Jul 19 16:34:11 [initandlisten] 912261120/1073741824 84%<br />
Thu Jul 19 16:34:17 [initandlisten] preallocating a journal file /data/db-2.2.0-rc0/journal/prealloc.2<br />
Thu Jul 19 16:34:20 [initandlisten] 304087040/1073741824 28%<br />
Thu Jul 19 16:34:23 [initandlisten] 471859200/1073741824 43%<br />
Thu Jul 19 16:34:26 [initandlisten] 629145600/1073741824 58%<br />
Thu Jul 19 16:34:29 [initandlisten] 796917760/1073741824 74%<br />
Thu Jul 19 16:34:32 [initandlisten] 964689920/1073741824 89%<br />
Thu Jul 19 16:34:37 [initandlisten] waiting for connections on port 27017<br />
Thu Jul 19 16:34:37 [websvr] admin web console waiting for connections on port 28017<br />
Thu Jul 19 16:34:56 [initandlisten] connection accepted from 127.0.0.1:48357 #1 (1 connection now open)<br />
Thu Jul 19 16:35:30 [conn1] end connection 127.0.0.1:48357 (0 connections now open)<br />
<div>
<br />
Well, there you have it. It's pretty clear. It takes just over a minute between mongod starting up and it being ready for connections. <br />
<br />
<b>Top tip</b><br />
Be patient young Jedi. Mongo is quick, but give it a moment to get out of bed and put on it's dressing gown between ringing the doorbell and banging on the door.<br />
<br /></div>Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com1tag:blogger.com,1999:blog-3132785414967539357.post-21835845399549500482012-06-25T15:35:00.002+01:002012-06-26T13:53:22.803+01:00Advanced Search on GitHubToday I was cruising the <a href="https://github.com/mongodb/mongo-java-driver">MongoDB Java driver</a> GitHub repo. I was interested in the implementation of the <code>eval()</code> method, as I wanted to ensure I cater for all returned types within <a href="https://github.com/JanPaulEttles/mongometer">mongometer</a>.
<br />
<br />
Seemed simple enough, I thought.
<br />
<br />
I went straight to <a href="https://github.com/mongodb/mongo-java-driver/blob/master/src/main/com/mongodb/DB.java">DB.java</a> and saw that we're calling <code>command()</code> and extracting an object keyed by <code>retval</code>. Interesting, to see <code>retval</code>, a potentially project-wide constant, defined as a <code>String</code> rather than as an <code>Enum</code>. Anyhoo, this isn't a critique of the driver code, so I'll park that for now, I just wanted to find what could possibly be returned by <code>eval()</code>.
<br />
<br />
An easy way to do this is to <code><a href="http://gitref.org/remotes/#fetch">fetch</a></code> the branch and search it locally. But I wouldn't really want to do this for every single project that I ever want to cruise? No way, Pedro! So, let's use the online GitHub Search.
<br />
<br />
A good place to start: <code>https://github.com/search</code><br />
Advanced Search : <code>retval repo:mongodb/mongo-java-driver</code><br />
Search for: <code>Code</code><br />
Search Language: <code>Java</code><br />
<br />
That all seems sane enough. Right?
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPaW0KTwQvpfc6nAzcyyGSjc_ou_18ndmtN5Mnq1O16XHupntPmRUsBlPI5QWqffho9HQlrSCxRIZDlXmvusx1niQxvzb2DicT643BkSrRnleramk3FyIsrPgycy7LiB_mroLDCnQaHpI/s1600/1.PNG" imageanchor="1"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPaW0KTwQvpfc6nAzcyyGSjc_ou_18ndmtN5Mnq1O16XHupntPmRUsBlPI5QWqffho9HQlrSCxRIZDlXmvusx1niQxvzb2DicT643BkSrRnleramk3FyIsrPgycy7LiB_mroLDCnQaHpI/s320/1.PNG" width="276" /></a>
<br />
<br />
Wow! That was unexpected. I haven't been returned the results limited to the filetype of <code>Java</code>, I've been returned a list of files that contain the term <code>java</code>. Let's have a quick look at the querystring.
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjk0MVn8t5UB2YQ9cgkywuYfnw8zrYrDwd7aBaV_mej0T6xPyYAPFLlVxSWY5dhFt7vvfMhM4Ft5xmVjmEbe2hun-tTd0vRAHER-d2Xla31K7fjLqTAQYH0EBZ4-nW1w5oCSeTBwehu50I/s1600/2.PNG" imageanchor="1"><img border="0" height="9" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjk0MVn8t5UB2YQ9cgkywuYfnw8zrYrDwd7aBaV_mej0T6xPyYAPFLlVxSWY5dhFt7vvfMhM4Ft5xmVjmEbe2hun-tTd0vRAHER-d2Xla31K7fjLqTAQYH0EBZ4-nW1w5oCSeTBwehu50I/s320/2.PNG" width="320" /></a>
<br />
<br />
It seems to be searching for <code>Java</code>, so let's swap out <code>Java</code> for <code>retval</code>, our <i>actual</i> search term.
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiO-Ulk2WhT4NFxm7uXpdM1b_iabpO1AUWtEXjsNFJBCWM__WL9WEAw5lMbEd-5Dyp25OnzIBzvpFEB8EXj94EtQm61jC09Lizi8QYd4oHLwOcL6D5hCKJ2FlnDgeKRi3UXuNvbtib0qOI/s1600/3.PNG" imageanchor="1"><img border="0" height="9" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiO-Ulk2WhT4NFxm7uXpdM1b_iabpO1AUWtEXjsNFJBCWM__WL9WEAw5lMbEd-5Dyp25OnzIBzvpFEB8EXj94EtQm61jC09Lizi8QYd4oHLwOcL6D5hCKJ2FlnDgeKRi3UXuNvbtib0qOI/s320/3.PNG" width="320" /></a>
<br />
<br />
Now you get the results for <code>retval</code>. We have an unknown number of matches for <code>retval</code> from within the Java driver code base. But is seems we have been returned results for every version of the file that the search term is found in. Let's park that and come back to it later.
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvMVhAqliG9j1YCGpA3oVGrw5Is1j9ND1XQZLkrAHkIn4cUofZb2lmz-HYlsN0rp-4UI0snhtB0rr4OxGf2MeQ_8xZmzfFc9O4U0e97BsSs0w19z_9vHAJk_rHUeBqkyfhkL8yT6R_5IU/s1600/4.PNG" imageanchor="1"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvMVhAqliG9j1YCGpA3oVGrw5Is1j9ND1XQZLkrAHkIn4cUofZb2lmz-HYlsN0rp-4UI0snhtB0rr4OxGf2MeQ_8xZmzfFc9O4U0e97BsSs0w19z_9vHAJk_rHUeBqkyfhkL8yT6R_5IU/s320/4.PNG" width="307" /></a>
<br />
<br />
You get the same results when you completely remove the <code>language</code> from the querystring. Let's remove it and leave it off as it reverts back to using <code>Java</code> as the search term.
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfxJpP247CSZCgxTOhqocDL4QH4nLgb-WJo4Pt_m8QlGqejnHlC7dicmpSk1XkvcT41o0QrFgrJI9gVJ9mgAjRrkbWP0M9LNlPu-Oc2K96X-5xkpBLBrscevc1E7gcBTEmaFhbou_U4mc/s1600/2.1.PNG" imageanchor="1" style=""><img border="0" height="10" width="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfxJpP247CSZCgxTOhqocDL4QH4nLgb-WJo4Pt_m8QlGqejnHlC7dicmpSk1XkvcT41o0QrFgrJI9gVJ9mgAjRrkbWP0M9LNlPu-Oc2K96X-5xkpBLBrscevc1E7gcBTEmaFhbou_U4mc/s320/2.1.PNG" /></a>
<br />
<br />
It might not seem like it, but we're getting somewhere. Notice there is a <code>repo</code> parameter on the querystring. Let's pull the <code>repo:mongodb/mongo-java-driver</code> out of the <code>q</code> term and stick it in the <code>repo</code> parameter.
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigDtuKCM1xqJGsBmhkjnq63Jtcmw5pJfiHT40FuXHfLhAE6UxRF-oKAi5JNPEmersqHzVGy9ykvR9LGXcVOksualrEb_55lBBFRIis4c6jAqAJiX2Oz8IY5VifsNh4620_nec_EJE12m0/s1600/3.1.PNG" imageanchor="1" style=""><img border="0" height="12" width="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigDtuKCM1xqJGsBmhkjnq63Jtcmw5pJfiHT40FuXHfLhAE6UxRF-oKAi5JNPEmersqHzVGy9ykvR9LGXcVOksualrEb_55lBBFRIis4c6jAqAJiX2Oz8IY5VifsNh4620_nec_EJE12m0/s320/3.1.PNG" /></a>
<br />
<br />
Now on the search form we have a separate input field where you can specify the <code>repo</code>.
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4MsZuit9r-nMfYiEYA66Xg-W5RAtqHKqCKYy0mKfSd46csNU5-jR4HBFlbCNPHl8XG2v9XA5ZGVfMSUfem5O0lubCvLca0OEYC1g5_GmUlFeD7iV9dzc3hrOH1wxoO2VUudXhkvlHghk/s1600/5.PNG" imageanchor="1" style=""><img border="0" height="320" width="309" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4MsZuit9r-nMfYiEYA66Xg-W5RAtqHKqCKYy0mKfSd46csNU5-jR4HBFlbCNPHl8XG2v9XA5ZGVfMSUfem5O0lubCvLca0OEYC1g5_GmUlFeD7iV9dzc3hrOH1wxoO2VUudXhkvlHghk/s320/5.PNG" /></a>
<br />
<br />
So, let's try limiting it to a single version of each file in the repo. Hmmm, not sure how to do this. Anyone got any ideas? I must be missing something as I'd have thought that search is fundamental to any website these days. Anything I try seems to result in with the same error message.
<br />
<br />
<code>Invalid search query. Try quoting it.</code>
<br />
<br />
All I want to do is search files for a given string, without having to fetch the entire repo.
<br />
<br />
I'd look through the <a href="https://github.com/search?q=github.com&type=Everything&repo=&langOverride=&start_value=1">github.com</a> repo to investigate further, but I don't seem to be able to find github on github.
<br />
<br />
To be continued...Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com10tag:blogger.com,1999:blog-3132785414967539357.post-52874055282425853982012-06-21T18:32:00.001+01:002012-06-21T19:23:21.932+01:00Key Stretching; an exampleI've had quite a few questions about my previous post (from June 2010) on <a href="http://exceptionallyexceptionalexceptions.blogspot.co.uk/2010/06/passwords.html">Passwords</a> since I recently reposted it.
<br />
<br />
More specifically the questions and comments were around <a href="http://en.wikipedia.org/wiki/Key_stretching">key stretching</a>.
<br />
<br />
<b>Q.</b> But doesn't looping that many times slow the password verification step down?<br />
<b>A.</b> Well yes. That's kind of the point. The user may experience negligible latency during the authentication process, this can be tuned to have no or little effect on the user experience. This same delay is multiplied by the number of brute force attempts made by the attacker.<br />
<br />
<b>Q.</b> The attacker doesn't need the salt or the algorithm if they are going through the front door?<br />
<b>A.</b> Agreed. Which is why you'd have some controls and triggers at the front door to alert when abnormal or atypical behaviour is detected.<br />
<br />
<b>Q.</b> If an attack gains access to a system, surely it's game over for user PII?<br />
<b>A.</b> The thing I like about key stretching is that a hacker needs to have access to the salt, the hashed password and stretching algorithm. So, if you store the salt in a different table or even DB than the hashed password, the attacker would need to get their mitts on both tables or both DBs. Gaining visibility of the algorithm means that the attacker would also need to have access to, or knowledge about the specific implementation details; ie the source code. Typical environment configurations ensure that there is no way to get to a development environment from a production environment (and vice versa), making it difficult to gain access to the source.<br />
<br />
This is a quick and dirty example of what key stretching would look implemented in Java. <br />
<br />
<script src="https://gist.github.com/2966947.js?file=keystretching.java"></script>
<br />
This runs in around 0.75 seconds on my local box. Which means every iteration of a dictionary attack or a brute force attack would take the same time.
<br />
<br />
Caveat: this is purely an example of how you could implement stretching.<br />Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com0tag:blogger.com,1999:blog-3132785414967539357.post-8446481290035721462012-06-19T16:58:00.000+01:002012-06-19T16:58:45.928+01:00mongometer UnhackedWhen I first hacked <a href="https://github.com/JanPaulEttles/mongometer">mongometer</a> together, I didn't think that I'd need to spend more than that <a href="http://exceptionallyexceptionalexceptions.blogspot.co.uk/2012/04/performance-testing-mongodb.html">one hour</a> on both coding and posting on the hack. However, the more I used it, the more I found it useful a way of comparing the relative performance of my scripts, and the more I found it useful, the more I thought that others may find it useful. <br />
<br />
(I also found a few shortcomings/annoyances, which is great, because that gives me some incentive to improve the hack.)<br />
<br />
<ol>
<li>If I needed to change the hosts, the database, the credentials, you needed to rebuild and redeploy the jar.<br />
<li>There was no way to tweak the options [http://api.mongodb.org/java/2.7.2/com/mongodb/MongoOptions.html].<br />
<li>Connections didn't seem to be getting recycled. You had to shutdown jmeter before the open connections were tried up.<br />
<li>Wanted to ensure that the connection didn't need to be created every time there was an iteration, and that the connection pooling could be utilised<br />
</ol>
<br />
1. Easy enough to the change this, I added the fields that had been frustrating me.
<br />
<br />
2. Again, easy enough to add these additional properties, though I did come up against a slight <a href="http://exceptionallyexceptionalexceptions.blogspot.co.uk/2012/06/jmeter-bug-bean.html">frustration</a>
<br />
<br />
3. This was also slightly frustrating. JMeter didn't seem to be release any of the resources tied up in the Sampler. This was eventually resolved by a bit of experimentation and a bit of 4.
<br />
<br />
4. This turned out to be a lesson in RTFM. Implementing <a href="http://jmeter.apache.org/api/org/apache/jmeter/testelement/TestListener.html">TestListener</a> allows you to connect to MongoDB at the start of the test run, use the connection pooling during the iterations, and tidy up resources at the end of the run.<br />
<br />
<b>Moral</b><br />
I don't really do 'and the moral of the story is...', but in this case, RTFM is reasonably apt. And, it's quick and easy to hack together something that works, it's not so easy to put together something usable in the longer term; but we all know that.<br />
<br />
<b>Request For Comments</b><br />
If you found this useful, please leave a comment.<br />
If there is a missing feature, please let me know and I'll try and add it.<br />Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com0tag:blogger.com,1999:blog-3132785414967539357.post-3474885875675753342012-06-19T13:18:00.000+01:002012-06-19T13:24:11.353+01:00JMeter Bug BeanI came across a potential bug in <a href="http://svn.apache.org/repos/asf/jmeter/trunk/src/core/org/apache/jmeter/testbeans/BeanInfoSupport.java">JMeter</a> when adding support for <a href="http://api.mongodb.org/java/2.7.2/com/mongodb/MongoOptions.html">MongoOptions</a> within <a href="https://github.com/JanPaulEttles/mongometer">mongometer</a>. It's not a big deal in itself as I worked around it, but it was slightly frustrating at the time.<br />
<br />
Implementing the MongoOptions seemed rather innocuous at the time; after all it's just another bunch of properties the user can alter. I ended up just adding them all at once. Imagine my disappointment when the extension failed to load in Jmeter as intended. To keep a long story short, the offender was <a href="http://api.mongodb.org/java/2.7.2/com/mongodb/MongoOptions.html#wtimeout">wTimeout</a>. Now, wtimeout (is one of a group a rather poorly named variables, others include <a href="http://api.mongodb.org/java/2.7.2/com/mongodb/MongoOptions.html#j">j</a> and <a href="http://api.mongodb.org/java/2.7.2/com/mongodb/MongoOptions.html#w">w</a> ), but why was there an issue with wtimeout?
<br />
<br />
<b>The error of my ways</b><br />
<br />
This is the error that was generated when starting up JMeter:<br />
<br />
<script src="https://gist.github.com/2953772.js?file=jme1.log">
</script>
<br />
This is how it was defined, firstly in ScriptSamplerBeanInfo.java<br />
<br />
<script src="https://gist.github.com/2953782.js?file=jme2.java">
</script>
<br />
Next, in ScriptSamplerResource.properties<br />
<br />
<script src="https://gist.github.com/2953789.js?file=jme3.properties">
</script>
<br />
And finally, in ScriptSampler.java<br />
<br />
<script src="https://gist.github.com/2953793.js?file=jme4.java">
</script>
<br />
I checked and double checked the spelling. It didn't really make sense, it should've been fine. Out of all the properties that were defined, the wTimeout property was the odd one out, in that it was the only one that had the pattern of lowercase{1}uppercase{N}. All the others were either lowercase{1}, lowercase{N} or lowercase{N}uppercase{N}. So long story, short; switching from wTimeout to doubleuTimeout resolved the issue.
<br />
<br />
<script src="https://gist.github.com/2953794.js?file=jme5.java">
</script>
<br />
Next, in ScriptSamplerResource.properties<br />
<br />
<script src="https://gist.github.com/2953798.js?file=jme6.properties">
</script>
<br />
And finally, in ScriptSampler.java<br />
<br />
<script src="https://gist.github.com/2953804.js?file=jme7.java">
</script>
<br />
It seems that when performing the creation of the property, if there is only a single lowercase character followed by an uppercase character, we get an error.
<br />
<br />
<b>The moral of the story</b><br />
<br />
Hmmmm. I don't really do morals, but if this is a bug, then it needs to be fixed. Either way j, w and wtimeout are really rather poor names for variables.Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com1tag:blogger.com,1999:blog-3132785414967539357.post-81791653359193291002012-06-18T11:15:00.001+01:002012-06-18T11:19:28.634+01:00Creating your project with IntelliJ<span style="background-color: white;">Now we have a development environment <a href="http://exceptionallyexceptionalexceptions.blogspot.co.uk/2012/06/configuring-mongometer-development.html">configured</a>, lets create the project and start taking it to the next level.</span><br />
<br />
<b>Creating a project in IntelliJ</b><br />
<br />
New Project<br />
New Project from Scratch<br />
Enter the name: mongometer<br />
Select Maven Module<br />
Next<br />
Select Create from archetype (I chose maven j2ee simple)<br />
<br />
<b>Arrrrg!</b> Don't do this! Man, IntelliJ sat spinning the tyres for an age. I ended up killing it and starting again.<br />
<br />
To delete a project, close intellij and remove the project folder<br />
projects are in IdeaProjects<br />
$rm -r ~/IdeaProjects/mongometer<br />
<br />
Second time around<br />
New Project<br />
New Project from Scratch<br />
Enter the name: mongometer<br />
Select Maven Module<br />
Next<br />
Deselect the create from archetype<br />
<br />
Create a package<br />
Create the classes and the properties file<br />
<br />
Fix imports<br />
Add dependencies to the pom.xml<br />
<br />
This is what I need for mongometer<br />
<br />
<ul>
<li>junit4.10<br />
</li>
<li>mongodb 2.7.2<br />
</li>
<li>ApacheJMeter_core 2.6<br />
</li>
<li>jorphan 2.6<br />
</li>
</ul>
<br />
<b>Push to GitHub</b><br />
<br />
$cd ~/IdeaProjects/mongometer<br />
$git init<br />
$git add *.java<br />
$git add *.properties<br />
$git add *.xml<br />
$git commit -m 'Initial upload of the project'<br />
$git remote add origin ssh://github.com/yourname/mongometer.git<br />
$git push -u origin +master<br />
<br />
Voilà! You can go to your repo, and the project files are there. Happy days.<br />
<br />
<b>Points of interest</b><br />
<ul>
<li>The + performs a force push to github. I'm guessing this happened because when I created a new repository on GitHub I opted to create the README file. I won't do this again.<br />
</li>
<li>IntelliJ complained about setting up the repo outside of IntelliJ. No problem, you just add the repo inside IntelliJ. I've been an Eclipse person for years (though we've decided that we'd like spend some time apart) so I'm not 100% sure of how to do this inside IntelliJ just yet.<br />
</li>
<li>This is a great site for listing dependencies http://mvnrepository.com/ for setting up your pom.xml
</li>
</ul>
<br />Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com0tag:blogger.com,1999:blog-3132785414967539357.post-63399500779581021712012-06-18T10:31:00.000+01:002012-06-19T16:41:08.487+01:00Configuring the mongometer Development EnvironmentAs I've already set up a git repository on Ubuntu and published the step-by-step, command-by-command <a href="http://exceptionallyexceptionalexceptions.blogspot.co.uk/2012/06/install-git-on-ubuntu.html">instructions</a> for doing this, I thought it would be worth publishing the same guides for setting up the entire development environment on Ubuntu.<br />
<br />
<b>Set up your JDK</b><br />
<br />
Download<br />
http://www.oracle.com/technetwork/java/javase/downloads/jdk-7u4-downloads-1591156.html<br />
<br />
Configure<br />
$tar -xvf ~/Downloads/jdk-7u4-linux-x64.tar.gz<br />
$sudo mkdir -p /usr/lib/jvm/jdk1.7.0<br />
$sudo mv jdk1.7.0_04/* /usr/lib/jvm/jdk1.7.0/<br />
$rm -r jdk1.7.0_04<br />
$sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.7.0/bin/java" 1<br />
$sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/lib/jvm/jdk1.7.0/bin/javac" 1<br />
$sudo update-alternatives --install "/usr/bin/javaws" "javaws" "/usr/lib/jvm/jdk1.7.0/bin/javaws" 1<br />
$mkdir ~/.mozilla/plugins/<br />
$ln -s /usr/lib/jvm/jdk1.7.0/jre/lib/amd64/libnpjp2.so ~/.mozilla/plugins/<br />
<br />
<b>Set up MongoDB</b><br />
<br />
Download<br />
http://www.mongodb.org/dr/fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.0.6.tgz/download<br />
$md5sum ~/Downloads/mongodb-linux-x86_64-2.0.6.tgz<br />
69eece640fcb1684190a4585f31df954<br />
<br />
Configure<br />
$tar -zxvf ~/Downloads/mongodb-linux-x86_64-2.0.6.tgz<br />
$sudo mkdir -p /usr/lib/mongodb/2.0.6<br />
$sudo mv mongodb-linux-x86_64-2.0.6/* /usr/lib/mongodb/2.0.6/<br />
$rm -r mongodb-linux-x86_64-2.0.6<br />
$sudo mkdir -p /data/db<br />
$sudo chown `id -un` /data/db<br />
$/usr/lib/mongodb/2.0.6/bin/mongod --dbpath /data/db --logpath /data/db/mongod.log<br />
$mongod --config /etc/mongod.conf<br />
<br />
(from a new terminal start the shell)<br />
$cd /usr/lib/mongodb/2.0.6/bin/<br />
$./mongo<br />
> db.test.save( { a: 1 } )<br />
> db.test.find()<br />
<br />
<b>Set up Maven</b><br />
<br />
Download<br />
http://www.apache.org/dyn/closer.cgi/maven/binaries/apache-maven-3.0.4-bin.tar.gz<br />
$md5sum ~/Downloads/apache-maven-3.0.4-bin.tar.gz<br />
e513740978238cb9e4d482103751f6b7<br />
<br />
Configure<br />
$tar -xzvf ~/Downloads/apache-maven-3.0.4-bin.tar.gz<br />
$sudo mkdir -p /usr/local/maven/3.0.4<br />
$sudo mv apache-maven-3.0.4/* /usr/local/maven/3.0.4<br />
$rm -r apache-maven-3.0.4<br />
<br />
$sudo gedit /etc/environment<br />
<br />
JAVA_HOME="/usr/lib/jvm/jdk1.7.0"<br />
M2_HOME="/usr/local/maven/3.0.4"<br />
MAVEN_HOME="/usr/local/maven/3.0.4"<br />
M2="/usr/local/maven/3.0.4/bin"<br />
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/maven/3.0.4/bin"<br />
<br />
Verifiy the configuration<br />
Logout and login<br />
<br />
$mvn -version<br />
$mkdir ~/.m2<br />
$sudo chown `id -un` -R ~/.m2<br />
<br />
<b>Set up IntelliJ</b><br />
<br />
Download<br />
http://www.jetbrains.com/idea/download/<br />
<br />
$tar -xvf ~/Downloads/ideaIC-11.1.2.tar.gz<br />
$cd ~/idea-IC-117.418/bin<br />
$chmod +x idea.sh<br />
$./idea.sh<br />
<br />
(this is the initial install of intellij, so take the second option)<br />
File -> Project Structure<br />
Platform Settings -> SDKs<br />
Add -> JSDK<br />
/usr/lib/jvm/jdk1.7.0/<br />
Select -> OK<br />
<br />
<b>Set up JMeter</b><br />
<br />
Download<br />
http://mirror.lividpenguin.com/pub/apache//jmeter/binaries/apache-jmeter-2.7.tgz<br />
$md5sum ~/Downloads/apache-jmeter-2.7.tgz<br />
73435baa6ed99c528dacfa36c7e1f119<br />
<br />
Configure<br />
$tar -zxvf ~/Downloads/apache-jmeter-2.7.tgz<br />
$sudo mkdir -p /usr/lib/jmeter/2.7<br />
$sudo mv apache-jmeter-2.7/* /usr/lib/jmeter/2.7/<br />
$rm -r apache-jmeter-2.7<br />
$/usr/lib/jmeter/2.7/bin/jmeter.sh<br />Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com7tag:blogger.com,1999:blog-3132785414967539357.post-57761457800711374272012-06-16T22:14:00.000+01:002012-06-18T09:12:06.722+01:00Install git on ubuntuThis is a brief step by step guide to install git and hook up to github using ssh.</br>
</br>
<b>Get git</b></br>
$sudo apt-get install git-core</br>
</br>
<b>Check the install</b></br>
$which git</br>
$git --version</br>
</br>
<b>Configure git</b></br>
$git config --global user.name "Firstname Lastname"</br>
$git config --global user.email you@domain.com</br>
</br>
<b>Set up your ssh key</b></br>
$ssh-keygen -t rsa -C "you@domain.com"</br>
[hit enter]</br>
set your key password</br>
Your identification has been saved in /home/username/.ssh/id_rsa.</br>
Your public key has been saved in /home/username/.ssh/id_rsa.pub.</br>
The key fingerprint is:</br>
fingerprint you@domain.com </br>
</br>
<b>Add the key to github</b></br>
$less ~/.ssh/id_rsa.pub</br>
Copy the key to your clipboard</br>
</br>
Go to your github Account Settings</br>
</br>
Click "SSH Keys" in the left sidebar</br>
Click "Add SSH key"</br>
Paste your key into the "Key" field</br>
Click "Add key"</br>
Confirm the action by entering your GitHub password</br>
</br>
<b>Verify the key</b></br>
$ssh -T git@github.com</br>
Y</br>
enter the key password</br>
</br>
You should get a success, "but GitHub does not provide shell access".</br>
</br>
<b>Set up a test repo</b></br>
$mkdir repo-name</br>
$cd repo-name</br>
$git init</br>
$touch README</br>
$git add README</br>
$git commit -m 'first commit'</br>
$git remote add origin https://github.com/gitusername/repo-name.git</br>
$git push -u origin master</br>
</br>
you'll get a 403 error</br>
</br>
<b>Getting past the 403 error</b></br>
$vi .git/config</br>
change https for ssh</br>
$git push -u origin master</br>
</br>
you'll get a Permission denied (publickey) error</br>
</br>
hmmm</br>
</br>
$ssh-add -l shows the same fingerprint as the account settings on github</br>
$ssh -vT git@github.com comes back saying I have successfully authenticated...</br>
</br>
<b>Getting past the Permission denied error</b></br>
$cd ~/.ssh</br>
$vi config</br>
</br>
Host github.com</br>
User git </br>
Port 22</br>
Hostname github.com </br>
IdentityFile ~/.ssh/id_rsa</br>
TCPKeepAlive yes </br>
IdentitiesOnly yes</br>
</br>
$git push -u origin master</br>
</br>
Happy days! You should now see the README file on github</br>Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com1tag:blogger.com,1999:blog-3132785414967539357.post-82895923497869918122012-05-01T16:06:00.001+01:002012-05-01T16:06:28.260+01:00A life ban from Stackoverflow<br />
<span style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">So, today I decided I'll spend an hour looking at StackOverflow and helping out where I could. It has come to the rescue on a few occasions in the past so I thought I'd return the favour.</span><br />
<br />
<span style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">I signed up and started looking at the > 3 million questions. Bound to be something in there I can help with. I looked at the newest questions and clicked through. Many of the questions are poorly phrased, some aren't even asking a question and others just want the code; they don't want to do any thinking, they don't want to write the code, they just want the code sent to them.</span><br />
<br />
<span style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">So I tackled a few questions, some on mongodb, some on jquery, some of java and some javascript.</span><br />
<br />
<span style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">Some question gets answered so quickly, so you have to be quick. The reason is your post can be deleted if it is deemed a duplicate answer and deletion count against you. There were a couple of occasions where there was less than a minute between my answer and a similar answer were posted and one of them being deleted as a duplicate. </span><br />
<br />
<span style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">As a new user I cannot post comments. I didn't think anything of it at the time, so I put my requests for further information in the answer boxes. I had a few responses and started to earn a few points. Then I started to get lambasted for not answering the question and just asking more questions. I've always thought it is difficult to answer a question if it is unclear what is being asked. Eventually, whilst posting an answer to a mongodb question and the 16M BSON limit I received the message stating my account was now banned from posting and would never be reinstated; a life ban.</span><br />
<br />
<span style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">So basically, I had to earn 50 points before I can comment. But I can't earn any points until I understand the question or there is at least enough information in the question for it to be answerable, or at least until it is actually a question.</span>
<span style="font-family: arial, sans-serif; font-size: x-small;"><span style="line-height: 18px;"><br /></span></span><br />
<span style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">Apparently this has already been discussed <a href="http://meta.stackoverflow.com/questions/51926/new-users-cant-ask-for-clarifications-except-as-answers" target="_blank">here</a></span><br />
<br />
<span style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">I guess that is the good deed of the day shot down in flames. I'll think twice before I attempt to help again.</span><br />
<br />
So, should we boycott Stack Overflow?
<br />
<br />
Hell no! It has saved my life on a couple of occasions. They could/should however make easier for people (esp. those new and/or under 50 points) to ask for clarity in poorly phrased questions, or in some instances ask the asker what it is they are actually asking.Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com13tag:blogger.com,1999:blog-3132785414967539357.post-30206539220191358512012-04-26T11:14:00.000+01:002012-05-11T19:10:05.886+01:00Performance testing MongoDBSo, this morning I was hacking around in the mongo shell. I had come up with three different ways to aggregate the data I wanted, but wasn't sure about which one I should subsequently port to code to use within my application.<br />
<br />
So how would I decide on which method to implement? Well, lets just choose the one that performs the best. Ok, how do I do that? Hmmm. I could download and install some of the tools out there, or I could just wrap the shell code in a function and add some timings. OR, I could use the same tool that I use to performance test everything else; JMeter. To me it was a no brainer.<br />
<br />
So how do we do it?<br />
<br />
There is a full tutorial <a href="http://www.jajakarta.org/jmeter/1.7/en/extending/JMeter%20Extension%20Scenario.html">here</a>.<br />
<br />
Simply put, you need to do the following:<br />
<ol>
<li>Create a Sampler class. </li>
<li>Create a BeanInfo class. </li>
<li>Create a properties file. </li>
<li>Bundle up into a jar and drop into the apache-jmeter-X.X\lib\ext folder </li>
<li>Update search_paths=../lib/ext/mongodb.jar in jmeter.properties if you place the jar anywhere else. </li>
</ol>
<br />
<b>How I did it</b><br />
<br />
I tend to have a scratch pad project set up in my IDE, so I decided just to go with that. Just to be on the safe side, I imported all the dependencies from:
<br />
<ul>
<li>apache-jmeter-X.X\lib </li>
<li>apache-jmeter-X.X\lib\ext </li>
<li>apache-jmeter-X.X\lib\junit</li>
</ul>
I then created the two classes and the properties file.
<br />
<br />
<script src="https://gist.github.com/2498420.js?file=Sampler.java">
</script>
<br />
<script src="https://gist.github.com/2498398.js?file=SamplerBeanInfo.java">
</script>
<br />
<script src="https://gist.github.com/2498404.js?file=SamplerResources.properties">
</script>
I then exported the jar to apache-jmeter-X.X\lib\ext, and fired up jmeter.
<br />
<br />
Go through the normal steps to set the test plan up:<br />
<ol>
<li>Right click Test Plan and add a Thread Group. </li>
<li>Right click the Thread Group and add a Sampler, in this case a MongoDB Script Sampler. </li>
<li>Add your script to the textarea; db.YOUR_COLLECTION_NAME.insert({"jan" : "thinks he is great"}) </li>
<li>Run the test</li>
</ol>
<br />
Happy days. You can then use JMeter as you would for any other sampler.<br />
<b><br /></b><br />
<b>Future enhancements</b><br />
<br />
This is just a hack that took me 37 minutes to get running, plus 24 minutes if you include this post. This can certainly be extended to allow you to enter the replicaset config details for instance and to pull the creation of the connection out so we're not initiating this each time run a test.Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com49tag:blogger.com,1999:blog-3132785414967539357.post-31613406320115887942012-03-28T16:28:00.000+01:002012-03-29T17:25:42.366+01:00Store numbers as numbersHave you ever had your inner voice say to you 'you idiot' or 'you tool' or just 'Bravo' whilst clapping sarcastically (Homer: <a href="http://www.snpp.com/episodes/1F07.html">The Last Temptation of Homer</a>)? Yeah, me too, twice this week in fact. The second instance was particularly embarrassing, so I think I should share it. It was more embarrassing as I was just complaining that morning about how the MongoDB drivers should probably implement basic functions such as this.
<br/>
<br/>
Now the eagle-eyed among you will spot this straight away, so please don't shout out the answer and spoil it for everyone else.
<br/>
<br/>
Imagine you have a collection - I'll keep it as small as possible so that the error stands out more - and you want to find the maximum value of "someNumber". Simples; I've done this a thousand times, well hundreds of times. I think for the first five times I had to find the max I implemented a different solution each time; back when I first started with Mongo.
<br/>
<br/>
<script src="https://gist.github.com/2227128.js?file=jandefence"></script>
I'm going to show only one way of finding the maximum value, I'm choosing this particular method for no other reason than it's probably the simplest and easiest to understand.
<br/>
<br/>
In this slightly contrived example I only have 10 documents in the collection and in my defence the collection I was working on had a few hundred thousand - not much of a defence granted, but henceforth I'll refer to it as the <b>Jan Defence</b>.
So, initially I ran this query:
<br/>
<br/>
<script src="https://gist.github.com/2227033.js?file=jandefence1"></script>
Worked first time I thought. Rock on. Hang on... that number seems a bit low. Let's put in some logging:
<br/>
<br/>
<script src="https://gist.github.com/2227053.js?file=jandefence2"></script>
Instead of the 10 represented here, replace that with a number as per the <b>Jan Defence</b>. Lets put in some more logging.
<br/>
<br/>
<script src="https://gist.github.com/2227066.js?file=jandefence3"></script>
Again, remember the <b>Jan Defence</b>. Surely 50 isn't bigger than 200.
<br/>
<br/>
<penny drops/>
<br/>
<br/>
It is, if it is a String. 'You tool'.
<br/>
<br/>
<script src="https://gist.github.com/2227085.js?file=jandefence4"></script>
Et voilà!
<br/>
<br/>
Fortunately I didn't burn too much time with this.
<br/>
<br/>
The moral of the story? You should store your numbers as numbers.
<br/>
<br/>
If anyone would like to see more implementations of finding the max and min values, I'm happy to share.Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com0tag:blogger.com,1999:blog-3132785414967539357.post-18571827088623710572012-03-12T12:33:00.000+00:002012-03-12T12:33:08.853+00:00Mongodb MapReduce scope variablesI recently had a requirement for conditional emission from a map function. Essentially I only wanted to emit where the date was within a given range.
<br/>
<br/>
In SQL, grouping a count for a given time granularity within a date range would look <i>something</i> like this:
<br/>
<br/>
<script src="https://gist.github.com/2021381.js?file=scope.sql"></script>
I wasn't 100% sure about the 'right way' to achieve this in NOSql/MongoDB. So this is <i>a</i> solution.
<br/>
It requires you to know about scope variables. Problem is, I found that scope variables are not very well documented. You can find more about scope variables in the MongoDB documentation <a href="http://www.mongodb.org/display/DOCS/MapReduce#MapReduce-Overview">MapReduce-Overview</a>. The relevant parts are:
<br/>
<br/>
<span style="font-family: Courier, monospace">[, scope : <object where fields go into javascript global scope >]</span>
<br/>
and
<br/>
<span style="font-family: Courier, monospace">scope - can pass in variables that can be access from map/reduce/finalize. </span>
<br/>
<br/>
Back to this example. First, let's define some data:
<br/>
<br/>
<script src="https://gist.github.com/2021365.js?file=scope2.json"></script>
Before we implement this, lets get it working at the command-line, in our mongo shell:
<br/>
<br/>
<script src="https://gist.github.com/2021370.js?file=scope2.js"></script>
What is happening here?
<br/>
<br/>
Well, we're selecting the sub-document of this collection where the value of "meh" is "meh". Then we've defined two dates; <b>from</b> and <b>to</b> to represent the boundaries of the date range, we're including these within the MapReduce function call. Basically what this means is that we can use what ever is defined here in the Map function (btw, we can also use them in the Reduce and Finalize functions).
<br/>
<br/>
Once we have this working from the shell, it is straight forward to implement it. This is the very same implemented in Java.
<br/>
<br/>
<script src="https://gist.github.com/2021377.js?file=scope3.java"></script>
Caveat: This is an <i>example</i> of how to use <b>scope variables</b>, I'm sure if you go to any of the <a href="http://www.mongodb.org/display/DOCS/Events">Events</a> or <a href="http://gmane.org/info.php?group=gmane.comp.db.mongodb.user">gmane.comp.db.mongodb.user group</a> you'll get some advice straight from the 10Gen guys.Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com4tag:blogger.com,1999:blog-3132785414967539357.post-66789787236983366202012-03-09T15:29:00.000+00:002012-03-09T15:51:00.228+00:00Changing date types; from JavaScript UTC to Mongo ISODateThe scenario is this.
<br/>
<br/>
You have an HTML form that allows your customer to add an arbitrary amount of <b>stuff</b> to <b>something</b>. <b>stuff</b> is a partial JSON document which is a list of String/String and String/Date NVPs contained within <b>something</b>. Mongo will by default store the date as an ISODate. With that said, the <b>something</b> collection looks something like this:
<br/>
<br/>
<script src="https://gist.github.com/2006884.js?file=datetypes1.sh"></script>
<br/>
<script src="https://gist.github.com/2006950.js?file=datetypes2.json"></script>
<br/>
The simple solution to the format issue is to display the Date as an ISODate on the form, thus when the form is submitted, the date can be treated just like any other string (though you still need to tell mongo to treat it as a Date and not a String, so you need to 'find' it and change the type; more on that later).
<br/>
<br/>
<script src="https://gist.github.com/2006955.js?file=datetypes3.js"></script>
Would display: 2011-11-01T11:51:46.000Z
<br/>
<br/>
But that would be way too easy; in the real world, easy is considered as rare as unicorn herders. The Use Case/User Story/Whatever acceptance criteria is that we display the Date in the full UTC form. OK then.
<br/>
<br/>
<script src="https://gist.github.com/2006958.js?file=datetypes4.js"></script>
Which would display: Tue Nov 01 2011 11:51:46 GMT+0000 (GMT Standard Time)
<br/>
<br/>
This is easily done, in fact, it is easier than displaying it as an ISODate, but now we have to parse it and convert to an ISODate, which involves finding it in the JSON document first.
<br/>
<br/>
Parsing a UTC date into an ISO is trivial but should have been easier than this - I needed to faff around with the DateFormat string before I got rid of the ParseExceptions.
<br/>
<br/>
This is roughly how I did it. Doesn't look too pretty tbh.
<br/>
<br/>
<script src="https://gist.github.com/2007014.js?file=datetypes4.java"></script>
<br/>
It's amazing how simple this should be. You set it up as a date, you pass in a date, yet you still need to explicitly convert it from initially from a String to Date, and then from UTC to ISO.Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com2tag:blogger.com,1999:blog-3132785414967539357.post-39407098691675227462012-03-06T19:24:00.000+00:002012-03-08T19:06:29.454+00:00Neo4j: Mechanically SympatheticAbout a month ago I was 5 minutes late for a <a href="http://skillsmatter.com/">#skillsmatter</a> event, a talk on <a href="http://skillsmatter.com/podcast/home/neo4j-internals/js-3349">#neo4j and mechanical sympathy</a>. The talk was given by a #neo4j employee, thus given by someone in-the-know.
<br/>
<br/>
Because of my tardiness I missed why neo4j is mechanically sympathetic; the whole point of the talk. I should have had a chat with the presenter after the talk to clear up my lack of understanding, however the Slaughtered Lamb is not as conducive to conversation as it once was...
<br/>
<br/>
Are we saying that neo4j is mechanically sympathetic in terms of the records being powers of 2, or that the talk allows us to understand how neo4j works in the same way that Jackie Stewart knew how his cars worked because he had previously worked as a mechanic? I'm guessing the later as the <a href="http://3.bp.blogspot.com/__Sn-iXmVbEI/TLDLADnUwbI/AAAAAAAAADU/WoqsZHQNuCA/s1600/NeoRecordsCheatsheet.png">neo4j cheat sheet</a> shows records as 5, 9, 25 and 33 bytes, not exactly powers of 2.
<br/>
<br/>
A question came up during the session about the how many nodes you could have; the maximum NodeRecordIdSize. The response was that it wasn't 4 bytes as shown on the cheat sheet, but 4 bytes, plus 7 bits. The suggestion was that you could use the 7 'spare' bits of the 'in use' byte as only 1 bit of this byte is actually used.
<br/>
<br/>
With that said, my questions is, how would that affect the 4 byte reference allocation in, for instance, the RelationshipRecord, which is only allocated 4 bytes? Where are the 7 'spare bits' stored?Anonymoushttp://www.blogger.com/profile/14335719314813410020noreply@blogger.com0