Showing posts with label MongoDB. Show all posts
Showing posts with label MongoDB. Show all posts

Saturday, 2 February 2013

MongoDB Authentication

I recently updated mongometer to make it a bit more flexible. Shortly after releasing the new version, one of the users fed back an issue via a comment on the post. I booted up my machine, opened up my IDE, found the issue and had pushed the fix out to github within half-an-hour.

This isn't a quick turn-around, success story post. It quickly dawned on me that if I was going to do anything in the future with mongometer, I should really know a little more about how a user authenticates against a database within MongoDB. (I don't want to spend more than an hour or so on this as I've just cracked open a bottle of Nyetimber Classic Cuvee - I'm also cooking a chicken pie (ping me if you want the recipe) and I'd rather be finished this post before I finish the bottle.) Before diving into any documentation that may exist around MongoDB Security, I'll start with a few observations. So in typical man style, let's kick the tyres and then if required, RTFM.

Start up a mongod instance.

$ /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --fork --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log
$ ./mongo --port 27001


Create an admin user

> use admin
> db.addUser("mongouser","mongopass")
1


Restart mongod

$ sudo kill -15 $(ps -ef | grep mongo | grep -v grep | cut -f8 -d" ")
$ /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --fork --auth --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log
$ ./mongo --port 27001


Authenticate to admin

> use admin
switched to db admin
> db.aut("mongouser","mongopass")
Thu Jan 31 13:53:31.271 javascript execution failed (shell):1 TypeError: Property 'aut' of object admin is not a function
db.aut("mongouser","mongopass")
^

> db.aut("mongouser","mongopass")


Ooops. Fat-fingered it. Hang on, I think I've found Issue #1

Issue #1
If an admin user mistypes the auth command and not the credentials, then the actual credentials stay in the shell history, which persists across sessions. Any other user could potentially come along and view the shell history and pick the credentials up.

On the other hand, if the command is correct and either the username or password or both are incorrect, or indeed if the authentication attempts succeeds, then the command is not kept in the history. (The command history for the mongo shell is available in the same way as on a linux box - using the up arrow)

> db.auth("mongouser","mongopass0")
{ ok: 0.0, errmsg: "auth fails" }
0
> db.auth("mongouser0","mongopass0")
{ ok: 0.0, errmsg: "auth fails" }
0
> db.auth("mongouser0","mongopass")
{ ok: 0.0, errmsg: "auth fails" }
0


Ok. Let's authenticate against admin and continue.

> use admin
switched to db admin
> db.auth("mongouser","mongopass")
1


Oooops. I almost missed one there.

Issue #2
Until the mongod instance is restarted, any user can...

> use admin
switched to db admin
> db.system.users.find()
{ "_id" : ObjectId("510a58c6de50e136190f9ed7"), "user" : "mongouser", "readOnly" : false, "pwd" : "c49caa1cb6b287ff6b1deaeeb8f4d149" }


...grab the usernames and hashes.

So, now that I've restarted the mongod instance, any user is going to have to authenticate against admin to be able to view the contents of system.users.

Now, continuing on from entering incorrect credentials, I'm going to launch a dictionary attack and see what happens. Oh dear. Found another issue.

Issue #3
There is no lock-out. I wrote a quick hack to connect to the mongod instance, to switch over to admin and attempt to log in. Using a rather large dictionary (with "mongopass" tacked on at the end) I attempted to log in over a million times. This was only a crude single-threaded attempt that took around 17 seconds to complete, but it shows that there is no account lock out. I'm confident I could put together a multi-threaded brute-forcer if required. I'll need to look into this further to see if there is any brute forcing/dictionary attack alerting that can be configured or whether there is a lock-out policy that can be applied. I'm not ready to RTFM just yet.

Let's take a closer look at the format of the password in system.users.

c49caa1cb6b287ff6b1deaeeb8f4d149

That looks like an MD5 to me. Let's take a look in the code, which is available to cruise on github.

Wow! I got luck straight off-the-bat. db.js has the following method:

function _hashPassword(username, password) {
    return hex_md5(username + ":mongo:" + password);
}


With hex_md5 then referencing native_hex_md5 within utils.cpp:

void installGlobalUtils( Scope& scope ) {
    scope.injectNative( "hex_md5" , native_hex_md5 );
    scope.injectNative( "version" , native_version );
    scope.injectNative( "sleep" , native_sleep );
    installBenchmarkSystem( scope );
}

static BSONObj native_hex_md5( const BSONObj& args, void* data ) {
    uassert( 10261, "hex_md5 takes a single string argument -- hex_md5(string)",
    args.nFields() == 1 && args.firstElement().type() == String );
    const char * s = args.firstElement().valuestrsafe();

    md5digest d;
    md5_state_t st;
    md5_init(&st);
    md5_append( &st , (const md5_byte_t*)s , strlen( s ) );
    md5_finish(&st, d);

    return BSON( "" << digestToString( d ) );
}


Time for a quick recap. Just in case you missed anything:
  1. the hashing algorithm is MD5; my least favourite hashing algorithm.
  2. the string to be hashed is in the form username + ":mongo:" + password; using the same "salt" is non-optimal...
  3. the string :mongo: is global; I'm not really sure why it's there at all tbh.
I think this is probably enough to go with for now, else this will turn into a tl;dr and I may exceed my self imposed time constraints.

Thinking back to any discussions I had with regards to MongoDB, the same statements always arose within the context of Security.
  1. Authentication is off by default.
  2. MongoDB was always meant to be deployed in a trusted environment
I have to say that even with authentication on, we still have some gnarly issues. Further, I don't think a trusted environment exists.

Right then, time to RTFM with regards to Security. I'm hoping to find a roadmap defined that will deal with the issues stated above or there are already some mitigating steps that can be taken.

So, there are some Authentication features coming out in the near future. It looks like the new authentication features are only available under the MongoDB Subscriber Edition, I'm not sure what that means tbh... I also came across this know issue, which forms the basis for...

Issue #4
"if a user has the same password in multiple databases, the hash will be the same on all database. A malicious user could exploit this to gain access on a second database use a different users’ credentials." [sic]

Let's break that down.

"if a user has the same password in multiple databases, the hash will be the same on all database."

Yes. Correct. Same username, same password and same "salt" (ie the ":mongo:" string") equals same hash. OK, cool, let's move on.

"A malicious user could exploit this to gain access on a second database use a different users’ credentials." [sic]

A malicious user could exploit this if, and only if they have a non-readonly user on both databases involved.

If they only have readonly access, then they cannot list the system.users collection. In which case they will never see that the hashes are the same across different databases in the first place.

If they are not readonly, then they could list the system.users collection and take the hashed passwords offline to crack.

You're going to have to move into cracking territory if the hashes don't match across databases, in summary:
  1. the user attribute would have be the same. The odds of different users on different databases having the user could be high.
  2. the pwd attribute would have be the same. The odds of different users creating the same pwd is probably quite high.
  3. the "salt" is the same, so it has no real relevance here.
So the problem here is that a user (that is not readonly) can pull all the password hashes for a given database and take them offline to crack. The malicious user already has the user name and the "salt", all they have to find is the password.

Conclusions

Issue #1
This one is a bit of a pain tbh. When the command is entered correctly (ignoring whether the credentials are correct or not) the command is not shown in the history. When the command is not entered correctly, then it is difficult to know what to exclude from the command history. I guess you could retrospectively remove commands that resulted in errors (ie invalid commands) that preceded the authentication. That is not a solution...

Issue #2
There may be an argument that once the admin user is created in system.users in the admin database that a restart should be forced.

Issue #3
A no-brainer. I've written password policies on multiple occasions (what a fun life I live, eh?), account lock-out is password 101.

Issue #4
It seems that creating a "salt" (":mongo:") per database would resolve the issue. Looking at the code, it looks like the implementation is a doddle, a quick and easy win. Adding the option to manually set it would be grand. Implementing a unique "salt" under the covers such that users didn't have to think about it would be equally grand.

So, Nyetimber finished, post finished.

I'm not saying that there is anything in this post that is new or clever, it's a cursory glance. I'm not having a go; everything I've mentioned is merely observation. I install mongo on almost a daily basis because it's a great product, I do however like having a balanced view and identifying any elephants in the room. I'd be interested in any feedback.

Monday, 21 January 2013

SpiderMonkey to V8, mongometer and the Aggregation Framework

I previously posted a comparison that covered running some simple queries against versions 2.2.2 and 2.3.2 of MongoDB. They were pretty basic examples, I just wanted to demonstrate one of the uses of mongometer; comparing the relative performance of MongoDB releases and the MongoDB scripts you write to run on them.

Now I'm going to knock the complexity up a notch and perform a comparison between the different releases of MongoDB, their underlying JavaScript engines using another relatively new feature (added in version 2.1), the Aggregation Framework. I'm going to use the example they have in the documentation, mainly so I don't have to make something up.

{
title : "this is my title" ,
author : "bob" ,
posted : new Date () ,
pageViews : 5 ,
tags : [ "fun" , "good" , "fun" ] ,
comments : [
{ author :"joe" , text : "this is cool" } ,
{ author :"sam" , text : "this is bad" }
],
other : { foo : 5 }
}


db.articles.aggregate(
{ $project : {
author : 1,
tags : 1,
} },
{ $unwind : "$tags" },
{ $group : {
_id : { tags : "$tags" },
authors : { $addToSet : "$author" }
} }
);


I'm populating the collection with data in the form as described above. The only thing I'm adding is that I'm using the JMeter CSV Data Set to populate the author attribute.

Version 2.2.2
So let's make sure we're starting from a clean slate.

$ /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log

$ ps -ef | grep mongo
4974 /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log

$ ./mongo --port 27000

> show dbs
local 0.078125GB
test (empty)




> show dbs
aggregation 0.203125GB
local 0.078125GB
test (empty)

> use aggregation
switched to db aggregation
> db.dropDatabase()
{ "dropped" : "aggregation", "ok" : 1 }

$ sudo kill -15 4974


Version 2.3.2
Let's ensure we have that same clean slate as with Version 2.2.2

$ /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log

$ ps -ef | grep mongo
1463 /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log

$ ./mongo --port 27001

> show dbs
local 0.078125GB
test (empty)




> show dbs
aggregation 0.203125GB
local 0.078125GB
test (empty)

> use aggregation
switched to db aggregation
> db.dropDatabase()
{ "dropped" : "aggregation", "ok" : 1 }

$ sudo kill -15 1463


Conclusions
I ran this a few times and the results were consistent. I'll knock it up another notch over time, and hopefully draw out some useful conclusions, until then, you can draw your own.

Suggestions and comments welcome.

Sunday, 20 January 2013

SpiderMonkey to V8 and mongometer

With 10gen switching the default JavaScript engine for MongoDB 2.3/2.4 from SpiderMonkey to V8 I thought I'd take the opportunity to compare the relative performances of the releases using mongometer. Being a Security bod, I really should have looked at the Additional Authentication Features first... Hey ho.

I'll document the steps taken during the comparison, including the set up, so this can be repeated and validated - just in case anyone is interested - but mainly so I can remind myself of what I did; memory, sieve.

The set up
I'm going to install 2.2.2 and 2.3.2 side-by-side on a dedicated machine. I'll then use the latest version of the Java driver with mongometer.

$ wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.3.2.tgz
$ wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.3.2.tgz.md5

I got a 403 response for this request...

$ wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.2.2.tgz
$ wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.2.2.tgz.md5

$ md5sum -c mongodb-linux-x86_64-2.2.2.tgz.md5
md5sum: mongodb-linux-x86_64-2.2.2.tgz.md5: no properly formatted MD5 checksum lines found


Grrr. An md5 file is supposed to be the checksum (then x2 spaces) and then the filename of the file being checksummed. I'll have to eyeball them instead, well, eyeball the one that I could actually download...

$ md5sum mongodb-linux-x86_64-2.2.2.tgz
be0f5969b0ca23a0a383e4ca2ce50a39 mongodb-linux-x86_64-2.2.2.tgz

$ cat mongodb-linux-x86_64-2.2.2.tgz.md5
be0f5969b0ca23a0a383e4ca2ce50a39


Configure
$ tar -zxvf ~/mongodb-linux-x86_64-2.2.2.tgz
$ sudo mkdir -p /usr/lib/mongodb/2.2.2
$ sudo mv mongodb-linux-x86_64-2.2.2/* /usr/lib/mongodb/2.2.2/
$ rm -r mongodb-linux-x86_64-2.2.2
$ sudo mkdir -p /data/db/2.2.2
$ sudo chown `id -un` /data/db/2.2.2
$ /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log

$ tar -zxvf ~/mongodb-linux-x86_64-2.3.2.tgz
$ sudo mkdir -p /usr/lib/mongodb/2.3.2
$ sudo mv mongodb-linux-x86_64-2.3.2/* /usr/lib/mongodb/2.3.2/
$ rm -r mongodb-linux-x86_64-2.3.2
$ sudo mkdir -p /data/db/2.3.2
$ sudo chown `id -un` /data/db/2.3.2
$ /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log

Let's check they are running.

$ ps -ef | grep mongod
1795 /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log
2059 /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log


Now, let's kill one (gracefully) and move on to the interesting stuff.

$ sudo kill -15 2059
$ ps -ef | grep mongod
1795 /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log


Now I'm jumping on to another box.

$ wget https://github.com/downloads/mongodb/mongo-java-driver/mongo-2.10.1.jar
$ cp mongo-2.10.1.jar /usr/lib/jmeter/2.8/lib/ext
$ cp ~/IdeaProjects/mongometer/out/artifacts/mongometer_jar/mongometer.jar /usr/lib/jmeter/2.8/lib/ext
$ /usr/lib/jmeter/2.8/bin/jmeter.sh


The tests
The tests are really rather basic; I'll perform an insert into two different databases, and perform finds against those databases.

Version 2.2.2

> show dbs
local 0.078125GB













> show dbs
jmeter 0.203125GB
jmeter2 0.203125GB
local 0.078125GB

> use jmeter
> db.jmeter.find().count()
1000
> db.dropDatabase()

> use jmeter2
> db.jmeter.find().count()
1000
> db.dropDatabase()

$ ps -ef | grep mongo
2690 /usr/lib/mongodb/2.2.2/bin/mongod --port 27000 --dbpath /data/db/2.2.2 --logpath /data/db/2.2.2/mongod.log

$ sudo kill -15 2690
$ ps -ef | grep mongo

Nothing. Let's get the 2.3.2 instance up and running.

$ /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log

$ ps -ef | grep mongo 2947 /usr/lib/mongodb/2.3.2/bin/mongod --port 27001 --dbpath /data/db/2.3.2 --logpath /data/db/2.3.2/mongod.log


Version 2.3.2

> show dbs
local 0.078125GB













> show dbs
jmeter 0.203125GB
jmeter2 0.203125GB
local 0.078125GB

> use jmeter
> db.jmeter.find().count()
1000
> db.dropDatabase()

> use jmeter2
> db.jmeter.find().count()
1000
> db.dropDatabase()


Conclusions
I guess you should draw your own. I ran this a couple of times and am considering scripting it so the environments are cleaned down prior to each run, I could probably add more complex queries too. Perhaps if I find some time next weekend then I will.

If you have any suggestions, please leave a comment.

Saturday, 19 January 2013

mongometer v2.0

A while back I knocked up mongometer to compare the relative performance of MongoDB scripts. I then made some minor changes, and since then - and only recently - made additional changes based on feedback.

I've now made slightly more significant changes. I'll now cover them briefly.

The minimum JMeter version is now 2.8, this is because it is dependent on a change introduce in JMeter 2.8.

I've created a MongoSourceElement which is accessible from the context menu; Add -> Config Element -> MongoDB Source Config

This pulls all the MongoDB connection details up a level and allows you to share it between multiple MongoScriptSamplers. This means you only need to define the connection once and associate a source name with the instance. When you create a MongoScriptSampler you can then reference the source name as defined in MongoSourceElement.

MongoScriptSampler now only includes the fields that you need. It allows you to specify the mongo source, the database, the database credentials and of course the script to run.

The way I'm currently using it is to create a MongoSourceElement.
Under that I create a ThreadGroup.
And under that I add the MongoDB Script v2.0, View Results Tree, and Graph Results.

If I have multiple script to run, I use the same grouping of, MongoDB Script v2.0, View Results Tree, and Graph Results. See the image below for an example.



There is the added MongoUtils class which has a static method which allows you to specify multiple hosts (if you want) and breaks it out into each host and port, returning an ServerAddress, which to be honest, I'm surprised isn't available under the Java MongoDB driver.

The other semi-useful addition is the QuickEnvironment. It basically allows you to start up mongod, mongo, jmeter and a tail of the jmeter log. You could do this in a script, but hey ho, I did it in a helper class.

The latest version is available on github.

Please have a play, try and break it, make suggestions, give me feedback.

Thursday, 26 July 2012

CouchBase: SIMPLE

So, yesterday we had the CouchBase guys in the office, I think it was some kind of retaliatory strike to me having brought in the MongoDB guys a few weeks prior.

In the interest of being able to form a balanced view I thought I'd test out the "SIMPLE" part of the CouchBase "SIMPLE, FAST, ELASTIC" tag line by downloading, installing, creating and query some data. I like simple so this looks like it was made for me.

It takes me only a minute or so to update the version of MongoDB I run on my Ubuntu servers; it is very simple, but I've done it few times, so some of the ease of update is simply down to practice. This isn't really going to be a 100% fair and direct comparison if I'm completely honest. However, for the purposes of comparison we can use a quick guide to downloading, installing, creating and query some data. All the information you need is on one, simple page.

MongDB
Download: 27 seconds
Unzip package: 4 seconds
Create data directory: 9 seconds
Start server: 19 seconds
Start shell: 16 seconds
Create a document: 16 seconds
Query collection: 8 seconds

Total time: About 2 minutes.

CouchBase
Download: 2 minutes, 31 seconds
Install: (via the Wizard) 2 minutes 52 seconds + 3 minutes (Had to run this a seconds time as I was shown a 1618 error saying install was already in progress, which it wasn't. Also, after the wizard said it had finished the install there was no CouchBase directory in the default location and a browser didn't pop up pointing to the Admin UI, as per the install guide)



Configure server: 1 minute. I wasn't sure how much RAM I should be dedicating to CouchBase, nor was I completely sure which bucket type I should be choosing. The other niggly things with the install were the fact I had to complete the product registration before I could complete the configuration.

I'm not sure what it's doing here tbh...



Start shell: No time recorded. I couldn't see a quick start, simple, single guide anywhere. I had to google "couchbase quick start guide" which initially took me to a page where I could request quotes around pricing and duration. Gave up after 10 minutes. Looks like I may have to do this via the Admin UI.

Insert data: No time recorded. As I couldn't find how to connect to the server via a shell, I hunted around and found this posting; it looks simple. I couldn't find the 'Document button' that was being referred to though. I hunted around for another 15-20 minutes until I got frustrated and bored. I gave up.



Total time: DNQ

I'm afraid CouchBase has failed my definition of simple. Maybe they use a different dictionary, or maybe I'm just too simple. If anyone can point me towards a one-pager, simple, 101, 'get started quickly guide', then I'd appreciate it.

Let's give it another. So many people are using this that this must be a one off. So, I uninstall and restart my box. Then install afresh. I continue through the default settings. This time when I get to the Console I am met with this...



Not only can I not create a Document, I can barely use the Admin UI because of the modal alert that keeps popping up... FFS!

To get over this I do the following:
cd %COUCHBASE_HOME%\Server\bin
service_stop.bat
service_unregister.bat
Edit the script service_register.bat
Replace NS_NAME=ns_1@%IP_ADDR% with the 127.0.0.1.
service_register.bat
service_start.bat

Final effort. I uninstalled yet again and downloaded the latest version 2.0.0-dev-preview-4 and ran through the install process. Everything came unstuck when the wizard hit the 'Computing space requirements'; we basically hung at this stage. I wasn't able to cancel the install, so I had to kill the process. I tried once more and one last time - same result. There is a saying, 'If at first you don't succeed, try once more and give up.



UPDATE: 2012 Olympic Opening Day
$wget http://packages.couchbase.com/builds/1495/couchbase-server-community_x86_64_2.0.0-1495-rel.deb
$sudo dpkg -i couchbase-server-community_x86_64_2.0.0-1495-rel.deb

dependency error

$sudo apt-get install libssl0.9.8
$sudo dpkg -i couchbase-server-community_x86_64_2.0.0-1495-rel.deb


Now I can hit the Admin UI. I tap up 192.168.0.8:8091
We're getting there; I want to install some samples; I chose the beer sample, no surprise really. I was thrown back an error.



I hit next again and it said the sample was already installed. OK. So I unseleted the sample and hit next again. Success. So, semi-SIMPLE, it took less than 5 minutes; all I had to do was be careful about my choice of OS and grab the very latest build (-;


Thanks to Frank Weigel who pointed me at the latest build.

Initial thoughts are:
  1. it's weird not having the shell and doing everything via a web UI
  2. from the brief look at the beer sample, it looks to my uneducated eye that we are defining the functions by_key_value, by_location and brewery_beers in an almost utility helper class document.

Now may the time to go and find some 101 documentation (-;

Thursday, 19 July 2012

shell/mongo.js:91


Today I found some time to play with the new MongoDB aggregation framework.  It's not an official, stable release yet, but I'm not thinking about putting it into production just yet.

I wanted to keep the existing instance on MongoDB intact, so I installed it in it's own directory and created a new data directory to mirror this need.  These are the exact steps, if you're interested.

$wget http://www.mongodb.org/dr/fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.1.2.tgz/download
$mv download mongodb-linux-x86_64-2.1.2.tgz
$tar -zxvf mongodb-linux-x86_64-2.1.2.tgz
$sudo mkdir -p /usr/lib/mongodb/2.1.2
$sudo mv mongodb-linux-x86_64-2.1.2/* /usr/lib/mongodb/2.1.2/
$rm -r mongodb-linux-x86_64-2.1.2
$sudo mkdir -p /data/db-2.1.2
$sudo chown `id -un` /data/db-2.1.2
$/usr/lib/mongodb/2.1.2/bin/mongod --dbpath /data/db-2.1.2 --logpath /data/db-2.1.2/mongod.log

Now crack open a new terminal so you can connect.

$cd /usr/lib/mongodb/2.1.2/bin
$./mongo
MongoDB shell version: 2.1.2
connecting to: test
Thu Jul 19 11:40:14 Error: couldn't connect to server 127.0.0.1:27017 src/mongo/ shell/mongo.js:91
exception: connect failed

Oh dear.  But this shows mongod is running:

$ps -ef | grep mongod
/usr/lib/mongodb/2.1.2/bin/mongod --dbpath /data/db-2.1.2 --logpath /data/db-2.1.2/mongod.log

What does the log say?

$less /data/db-2.1.2/mongod.log

...
Thu Jul 19 11:40:35 [initandlisten] waiting for connections on port 27017
...

Hmmm.  Let's try again.

$cd /usr/lib/mongodb/2.1.2/bin
$./mongo

Hmmm, now it's connecting.  Well, I'm off to play with the aggregation framework now and I'll check this anomaly later.

The time now is 1625hrs and it looks like 2.2.0-rc0 has just been released.  Let's give that a go.

$wget http://www.mongodb.org/dr/fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.2.0-rc0.tgz/download
$mv download mongodb-linux-x86_64-2.2.0-rc0.tgz
$tar -zxvf mongodb-linux-x86_64-2.2.0-rc0.tgz
$sudo mkdir -p /usr/lib/mongodb/2.2.0-rc0
$sudo mv mongodb-linux-x86_64-2.2.0-rc0/* /usr/lib/mongodb/2.2.0-rc0/
$rm -r mongodb-linux-x86_64-2.2.0-rc0
$sudo mkdir -p /data/db-2.2.0-rc0
$sudo chown `id -un` /data/db-2.2.0-rc0
$/usr/lib/mongodb/2.2.0-rc0/bin/mongod --dbpath /data/db-2.2.0-rc0 --logpath /data/db-2.2.0-rc0/mongod.log

Crack open a new terminal so you can connect.

$cd /usr/lib/mongodb/2.2.0-rc0/bin
$./mongo
MongoDB shell version: 2.2.0-rc0
connecting to: test
Thu Jul 19 16:33:41 Error: couldn't connect to server 127.0.0.1:27017 src/mongo/shell/mongo.js:91
exception: connect failed

Is it running?

$ps -ef | grep mongod
/usr/lib/mongodb/2.2.0-rc0/bin/mongod --dbpath /data/db-2.2.0-rc0 --logpath /data/db-2.2.0-rc0/mongod.log

Yes, it is.  Bugger, I can't connect.  Let's take a look at the log again.

$less /data/db-2.2.0-rc0/mongod.log

...
Thu Jul 19 16:33:29 [initandlisten] journal dir=/data/db-2.2.0-rc0/journal
Thu Jul 19 16:33:29 [initandlisten] recover : no journal files present, no recovery needed
Thu Jul 19 16:33:31 [initandlisten] preallocateIsFaster=true 31.64
Thu Jul 19 16:33:33 [initandlisten] preallocateIsFaster=true 30.5
Thu Jul 19 16:33:37 [initandlisten] preallocateIsFaster=true 30.52
Thu Jul 19 16:33:37 [initandlisten] preallocateIsFaster check took 8.333 secs
Thu Jul 19 16:33:37 [initandlisten] preallocating a journal file /data/db-2.2.0-rc0/journal/prealloc.0
Thu Jul 19 16:33:40 [initandlisten]             304087040/1073741824    28%
Thu Jul 19 16:33:43 [initandlisten]             482344960/1073741824    44%
Thu Jul 19 16:33:46 [initandlisten]             650117120/1073741824    60%
Thu Jul 19 16:33:49 [initandlisten]             817889280/1073741824    76%
Thu Jul 19 16:33:52 [initandlisten]             985661440/1073741824    91%
Thu Jul 19 16:33:56 [initandlisten] preallocating a journal file /data/db-2.2.0-rc0/journal/prealloc.1
Thu Jul 19 16:33:59 [initandlisten]             272629760/1073741824    25%
Thu Jul 19 16:34:02 [initandlisten]             398458880/1073741824    37%
Thu Jul 19 16:34:05 [initandlisten]             555745280/1073741824    51%
Thu Jul 19 16:34:08 [initandlisten]             723517440/1073741824    67%
Thu Jul 19 16:34:11 [initandlisten]             912261120/1073741824    84%
Thu Jul 19 16:34:17 [initandlisten] preallocating a journal file /data/db-2.2.0-rc0/journal/prealloc.2
Thu Jul 19 16:34:20 [initandlisten]             304087040/1073741824    28%
Thu Jul 19 16:34:23 [initandlisten]             471859200/1073741824    43%
Thu Jul 19 16:34:26 [initandlisten]             629145600/1073741824    58%
Thu Jul 19 16:34:29 [initandlisten]             796917760/1073741824    74%
Thu Jul 19 16:34:32 [initandlisten]             964689920/1073741824    89%
Thu Jul 19 16:34:37 [initandlisten] waiting for connections on port 27017
Thu Jul 19 16:34:37 [websvr] admin web console waiting for connections on port 28017
Thu Jul 19 16:34:56 [initandlisten] connection accepted from 127.0.0.1:48357 #1 (1 connection now open)
Thu Jul 19 16:35:30 [conn1] end connection 127.0.0.1:48357 (0 connections now open)

Well, there you have it.  It's pretty clear.  It takes just over a minute between mongod starting up and it being ready for connections.

Top tip
Be patient young Jedi.  Mongo is quick, but give it a moment to get out of bed and put on it's dressing gown between ringing the doorbell and banging on the door.

Monday, 25 June 2012

Advanced Search on GitHub

Today I was cruising the MongoDB Java driver GitHub repo. I was interested in the implementation of the eval() method, as I wanted to ensure I cater for all returned types within mongometer.

Seemed simple enough, I thought.

I went straight to DB.java and saw that we're calling command() and extracting an object keyed by retval. Interesting, to see retval, a potentially project-wide constant, defined as a String rather than as an Enum. Anyhoo, this isn't a critique of the driver code, so I'll park that for now, I just wanted to find what could possibly be returned by eval().

An easy way to do this is to fetch the branch and search it locally. But I wouldn't really want to do this for every single project that I ever want to cruise? No way, Pedro! So, let's use the online GitHub Search.

    A good place to start: https://github.com/search
    Advanced Search : retval repo:mongodb/mongo-java-driver
    Search for: Code
    Search Language: Java

That all seems sane enough. Right?



Wow! That was unexpected. I haven't been returned the results limited to the filetype of Java, I've been returned a list of files that contain the term java. Let's have a quick look at the querystring.



It seems to be searching for Java, so let's swap out Java for retval, our actual search term.



Now you get the results for retval. We have an unknown number of matches for retval from within the Java driver code base. But is seems we have been returned results for every version of the file that the search term is found in. Let's park that and come back to it later.



You get the same results when you completely remove the language from the querystring. Let's remove it and leave it off as it reverts back to using Java as the search term.



It might not seem like it, but we're getting somewhere. Notice there is a repo parameter on the querystring. Let's pull the repo:mongodb/mongo-java-driver out of the q term and stick it in the repo parameter.



Now on the search form we have a separate input field where you can specify the repo.



So, let's try limiting it to a single version of each file in the repo. Hmmm, not sure how to do this. Anyone got any ideas? I must be missing something as I'd have thought that search is fundamental to any website these days. Anything I try seems to result in with the same error message.

Invalid search query. Try quoting it.

All I want to do is search files for a given string, without having to fetch the entire repo.

I'd look through the github.com repo to investigate further, but I don't seem to be able to find github on github.

To be continued...