Integrating with Quality Centre

I've recently had to go elbow deep to resolve an issue where a product developed by a company does not play well with a second product developed by the same company via a plugin, also developed by the same company.

As is my way, I'm posting the issue and the solution in a bid to reduce the average amount of pain felt.

WARNING: HP Quality Center Plugin exception
com4j.ComException: 800413ed (Unknown error) : Parameter Type is Invalid : .\invoke.cpp:517

The fix for this turned out to be replacing the empty String with a com4j.Variant.Type.VT_NULL

import com4j.Variant;

Then replace

Com4jObject bugObject = bugFactory.addItem("");

With

Com4jObject bugObject = bugFactory.addItem(new Variant(Variant.Type.VT_NULL));

Another tip is formatting the date.

WARNING: HP Quality Center Plugin exception
com4j.ComException: 80040519 (Unknown error) : Invalid date field value. : .\invoke.cpp:517

DateFormat dateFormat = new SimpleDateFormat("dd/MM/yy");
Date date = new Date();
bug.field("BG_DETECTION_DATE", dateFormat.format(date));

The final tip is ensuring you use a user that exists on the system.

WARNING: HP Quality Center Plugin exception
com4j.ComException: 800403ff (Unknown error) : Dookie is not in the users list : .\invoke.cpp:517

bug.field("BG_DETECTED_BY", "this has to be a valid user");

It's not all good news. For the life of me I couldn't find a solution to the this one and didn't want to get too deeply into the whole DLL thing. No matter how I specified the DLL; jvm params, properties et al of how I registered them or indeed which version I used, I could not overcome the following error. If oyu have a fix, let me know please. The only way to circumnavigate it was to downgrade from a 64-bit to a 32-bit version of the application. So I can only assume there is an issue with com4j-amd64.dll or com4j-x64.dll

com4j.ExecutionException: com4j.ComException: 80040154 CoCreateInstance failed : Class not registered : .\com4j.cpp:153

CouchBase: Permission denied

It's 1400hrs, I have an hour or so to kill, so I going take some blog notes as I play with CouchBase. I must be having one of the those weeks where nothing is going to go well; thankfully we're almost at the end of the week.

I'm finally up and running with an instance of CouchBase (though I had to change OS and grab the latest build), and I can see buckets and documents.  Yay!  The next logical step for me is to connect programmatically, and maybe create a prototype app.

Let's watch the video as it will only take 5 minutes. I love the fact this it titled "Get Started in 5 Minutes" but the video is 11 minutes long...

To save you some time, download the driver and add it to your classpath; saved you 10 minutes and 32 seconds.

I believe a 'Hello, World!' program should demonstrate the simplest thing possible, so I just wanted to connect to the instance.

public static void main(String args[]) 
 throws Exception {

 URI server = new URI("http://192.168.0.8:8091/pools");
 ArrayList servers = new ArrayList();
 servers.add(server);
        
 CouchbaseClient client = new CouchbaseClient(servers, "default", "");
}

It looks to be failing due to a "Permission denied" exception.  FFS!  I'm starting to get slightly miffed. Running a quick wget indicates there should be no such permission issues.



Ah, ha!  Fortunately I've seen this before, it just took me moment for the old brain cells to remember. To fix the issue you need to add the follow as a VM param:

-Djava.net.preferIPv4Stack=true

Now that I can finally connect, let's get the KVPs from a bucket.   It looks like TAP is the way to go.

TapClient tc = new TapClient(servers, "default", "");
tc.tapDump("some id here");

tapDump returns a TapStream, but I can't find the TapStream dependency. This dude managed to find it, but looks like he may have other issues.

I'm getting bored, and my head is getting sore with all this brick wall hitting.  I know it's early, but I fancy a nice cold Never Mind The Anabolics, it's in a bottle so I don't have to worry about tap streams...

Oh, and it looks like my issue with installing on Win7 64-bit has been around a while.

CouchBase: SIMPLE

So, yesterday we had the CouchBase guys in the office, I think it was some kind of retaliatory strike to me having brought in the MongoDB guys a few weeks prior.

In the interest of being able to form a balanced view I thought I'd test out the "SIMPLE" part of the CouchBase "SIMPLE, FAST, ELASTIC" tag line by downloading, installing, creating and query some data. I like simple so this looks like it was made for me.

It takes me only a minute or so to update the version of MongoDB I run on my Ubuntu servers; it is very simple, but I've done it few times, so some of the ease of update is simply down to practice. This isn't really going to be a 100% fair and direct comparison if I'm completely honest. However, for the purposes of comparison we can use a quick guide to downloading, installing, creating and query some data. All the information you need is on one, simple page.

MongDB
Download: 27 seconds
Unzip package: 4 seconds
Create data directory: 9 seconds
Start server: 19 seconds
Start shell: 16 seconds
Create a document: 16 seconds
Query collection: 8 seconds

Total time: About 2 minutes.

CouchBase
Download: 2 minutes, 31 seconds
Install: (via the Wizard) 2 minutes 52 seconds + 3 minutes (Had to run this a seconds time as I was shown a 1618 error saying install was already in progress, which it wasn't. Also, after the wizard said it had finished the install there was no CouchBase directory in the default location and a browser didn't pop up pointing to the Admin UI, as per the install guide)



Configure server: 1 minute. I wasn't sure how much RAM I should be dedicating to CouchBase, nor was I completely sure which bucket type I should be choosing. The other niggly things with the install were the fact I had to complete the product registration before I could complete the configuration.

I'm not sure what it's doing here tbh...



Start shell: No time recorded. I couldn't see a quick start, simple, single guide anywhere. I had to google "couchbase quick start guide" which initially took me to a page where I could request quotes around pricing and duration. Gave up after 10 minutes. Looks like I may have to do this via the Admin UI.

Insert data: No time recorded. As I couldn't find how to connect to the server via a shell, I hunted around and found this posting; it looks simple. I couldn't find the 'Document button' that was being referred to though. I hunted around for another 15-20 minutes until I got frustrated and bored. I gave up.



Total time: DNQ

I'm afraid CouchBase has failed my definition of simple. Maybe they use a different dictionary, or maybe I'm just too simple. If anyone can point me towards a one-pager, simple, 101, 'get started quickly guide', then I'd appreciate it.

Let's give it another. So many people are using this that this must be a one off. So, I uninstall and restart my box. Then install afresh. I continue through the default settings. This time when I get to the Console I am met with this...



Not only can I not create a Document, I can barely use the Admin UI because of the modal alert that keeps popping up... FFS!

To get over this I do the following:
cd %COUCHBASE_HOME%\Server\bin
service_stop.bat
service_unregister.bat
Edit the script service_register.bat
Replace NS_NAME=ns_1@%IP_ADDR% with the 127.0.0.1.
service_register.bat
service_start.bat

Final effort. I uninstalled yet again and downloaded the latest version 2.0.0-dev-preview-4 and ran through the install process. Everything came unstuck when the wizard hit the 'Computing space requirements'; we basically hung at this stage. I wasn't able to cancel the install, so I had to kill the process. I tried once more and one last time - same result. There is a saying, 'If at first you don't succeed, try once more and give up.



UPDATE: 2012 Olympic Opening Day
$wget http://packages.couchbase.com/builds/1495/couchbase-server-community_x86_64_2.0.0-1495-rel.deb
$sudo dpkg -i couchbase-server-community_x86_64_2.0.0-1495-rel.deb

dependency error

$sudo apt-get install libssl0.9.8
$sudo dpkg -i couchbase-server-community_x86_64_2.0.0-1495-rel.deb


Now I can hit the Admin UI. I tap up 192.168.0.8:8091
We're getting there; I want to install some samples; I chose the beer sample, no surprise really. I was thrown back an error.



I hit next again and it said the sample was already installed. OK. So I unseleted the sample and hit next again. Success. So, semi-SIMPLE, it took less than 5 minutes; all I had to do was be careful about my choice of OS and grab the very latest build (-;


Thanks to Frank Weigel who pointed me at the latest build.

Initial thoughts are:

1. it's weird not having the shell and doing everything via a web UI
2. from the brief look at the beer sample, it looks to my uneducated eye that we are defining the functions by_key_value, by_location and brewery_beers in an almost utility helper class document.

Now may the time to go and find some 101 documentation (-;

shell/mongo.js:91

Today I found some time to play with the new MongoDB aggregation framework.  It's not an official, stable release yet, but I'm not thinking about putting it into production just yet.

I wanted to keep the existing instance on MongoDB intact, so I installed it in it's own directory and created a new data directory to mirror this need.  These are the exact steps, if you're interested.

$wget http://www.mongodb.org/dr/fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.1.2.tgz/download
$mv download mongodb-linux-x86_64-2.1.2.tgz
$tar -zxvf mongodb-linux-x86_64-2.1.2.tgz
$sudo mkdir -p /usr/lib/mongodb/2.1.2
$sudo mv mongodb-linux-x86_64-2.1.2/* /usr/lib/mongodb/2.1.2/
$rm -r mongodb-linux-x86_64-2.1.2
$sudo mkdir -p /data/db-2.1.2
$sudo chown `id -un` /data/db-2.1.2
$/usr/lib/mongodb/2.1.2/bin/mongod --dbpath /data/db-2.1.2 --logpath /data/db-2.1.2/mongod.log

Now crack open a new terminal so you can connect.

$cd /usr/lib/mongodb/2.1.2/bin
$./mongo
MongoDB shell version: 2.1.2
connecting to: test
Thu Jul 19 11:40:14 Error: couldn't connect to server 127.0.0.1:27017 src/mongo/ shell/mongo.js:91
exception: connect failed

Oh dear.  But this shows mongod is running:

$ps -ef | grep mongod
/usr/lib/mongodb/2.1.2/bin/mongod --dbpath /data/db-2.1.2 --logpath /data/db-2.1.2/mongod.log

What does the log say?

$less /data/db-2.1.2/mongod.log

...
Thu Jul 19 11:40:35 [initandlisten] waiting for connections on port 27017
...

Hmmm.  Let's try again.

$cd /usr/lib/mongodb/2.1.2/bin
$./mongo

Hmmm, now it's connecting.  Well, I'm off to play with the aggregation framework now and I'll check this anomaly later.

The time now is 1625hrs and it looks like 2.2.0-rc0 has just been released.  Let's give that a go.

$wget http://www.mongodb.org/dr/fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.2.0-rc0.tgz/download
$mv download mongodb-linux-x86_64-2.2.0-rc0.tgz
$tar -zxvf mongodb-linux-x86_64-2.2.0-rc0.tgz
$sudo mkdir -p /usr/lib/mongodb/2.2.0-rc0
$sudo mv mongodb-linux-x86_64-2.2.0-rc0/* /usr/lib/mongodb/2.2.0-rc0/
$rm -r mongodb-linux-x86_64-2.2.0-rc0
$sudo mkdir -p /data/db-2.2.0-rc0
$sudo chown `id -un` /data/db-2.2.0-rc0
$/usr/lib/mongodb/2.2.0-rc0/bin/mongod --dbpath /data/db-2.2.0-rc0 --logpath /data/db-2.2.0-rc0/mongod.log

Crack open a new terminal so you can connect.

$cd /usr/lib/mongodb/2.2.0-rc0/bin
$./mongo
MongoDB shell version: 2.2.0-rc0
connecting to: test
Thu Jul 19 16:33:41 Error: couldn't connect to server 127.0.0.1:27017 src/mongo/shell/mongo.js:91
exception: connect failed

Is it running?

$ps -ef | grep mongod

/usr/lib/mongodb/2.2.0-rc0/bin/mongod --dbpath /data/db-2.2.0-rc0 --logpath /data/db-2.2.0-rc0/mongod.log

Yes, it is.  Bugger, I can't connect.  Let's take a look at the log again.

$less /data/db-2.2.0-rc0/mongod.log

...
Thu Jul 19 16:33:29 [initandlisten] journal dir=/data/db-2.2.0-rc0/journal
Thu Jul 19 16:33:29 [initandlisten] recover : no journal files present, no recovery needed
Thu Jul 19 16:33:31 [initandlisten] preallocateIsFaster=true 31.64
Thu Jul 19 16:33:33 [initandlisten] preallocateIsFaster=true 30.5
Thu Jul 19 16:33:37 [initandlisten] preallocateIsFaster=true 30.52
Thu Jul 19 16:33:37 [initandlisten] preallocateIsFaster check took 8.333 secs
Thu Jul 19 16:33:37 [initandlisten] preallocating a journal file /data/db-2.2.0-rc0/journal/prealloc.0
Thu Jul 19 16:33:40 [initandlisten]             304087040/1073741824    28%
Thu Jul 19 16:33:43 [initandlisten]             482344960/1073741824    44%
Thu Jul 19 16:33:46 [initandlisten]             650117120/1073741824    60%
Thu Jul 19 16:33:49 [initandlisten]             817889280/1073741824    76%
Thu Jul 19 16:33:52 [initandlisten]             985661440/1073741824    91%
Thu Jul 19 16:33:56 [initandlisten] preallocating a journal file /data/db-2.2.0-rc0/journal/prealloc.1
Thu Jul 19 16:33:59 [initandlisten]             272629760/1073741824    25%
Thu Jul 19 16:34:02 [initandlisten]             398458880/1073741824    37%
Thu Jul 19 16:34:05 [initandlisten]             555745280/1073741824    51%
Thu Jul 19 16:34:08 [initandlisten]             723517440/1073741824    67%
Thu Jul 19 16:34:11 [initandlisten]             912261120/1073741824    84%
Thu Jul 19 16:34:17 [initandlisten] preallocating a journal file /data/db-2.2.0-rc0/journal/prealloc.2
Thu Jul 19 16:34:20 [initandlisten]             304087040/1073741824    28%
Thu Jul 19 16:34:23 [initandlisten]             471859200/1073741824    43%
Thu Jul 19 16:34:26 [initandlisten]             629145600/1073741824    58%
Thu Jul 19 16:34:29 [initandlisten]             796917760/1073741824    74%
Thu Jul 19 16:34:32 [initandlisten]             964689920/1073741824    89%
Thu Jul 19 16:34:37 [initandlisten] waiting for connections on port 27017
Thu Jul 19 16:34:37 [websvr] admin web console waiting for connections on port 28017
Thu Jul 19 16:34:56 [initandlisten] connection accepted from 127.0.0.1:48357 #1 (1 connection now open)
Thu Jul 19 16:35:30 [conn1] end connection 127.0.0.1:48357 (0 connections now open)

Well, there you have it.  It's pretty clear.  It takes just over a minute between mongod starting up and it being ready for connections.

Top tip
Be patient young Jedi.  Mongo is quick, but give it a moment to get out of bed and put on it's dressing gown between ringing the doorbell and banging on the door.

Advanced Search on GitHub

Today I was cruising the MongoDB Java driver GitHub repo. I was interested in the implementation of the eval() method, as I wanted to ensure I cater for all returned types within mongometer.

Seemed simple enough, I thought.

I went straight to DB.java and saw that we're calling command() and extracting an object keyed by retval. Interesting, to see retval, a potentially project-wide constant, defined as a String rather than as an Enum. Anyhoo, this isn't a critique of the driver code, so I'll park that for now, I just wanted to find what could possibly be returned by eval().

An easy way to do this is to fetch the branch and search it locally. But I wouldn't really want to do this for every single project that I ever want to cruise? No way, Pedro! So, let's use the online GitHub Search.

    A good place to start: https://github.com/search
    Advanced Search : retval repo:mongodb/mongo-java-driver
    Search for: Code
    Search Language: Java

That all seems sane enough. Right?



Wow! That was unexpected. I haven't been returned the results limited to the filetype of Java, I've been returned a list of files that contain the term java. Let's have a quick look at the querystring.



It seems to be searching for Java, so let's swap out Java for retval, our actual search term.



Now you get the results for retval. We have an unknown number of matches for retval from within the Java driver code base. But is seems we have been returned results for every version of the file that the search term is found in. Let's park that and come back to it later.



You get the same results when you completely remove the language from the querystring. Let's remove it and leave it off as it reverts back to using Java as the search term.



It might not seem like it, but we're getting somewhere. Notice there is a repo parameter on the querystring. Let's pull the repo:mongodb/mongo-java-driver out of the q term and stick it in the repo parameter.



Now on the search form we have a separate input field where you can specify the repo.



So, let's try limiting it to a single version of each file in the repo. Hmmm, not sure how to do this. Anyone got any ideas? I must be missing something as I'd have thought that search is fundamental to any website these days. Anything I try seems to result in with the same error message.

Invalid search query. Try quoting it.

All I want to do is search files for a given string, without having to fetch the entire repo.

I'd look through the github.com repo to investigate further, but I don't seem to be able to find github on github.

To be continued...

Key Stretching; an example

I've had quite a few questions about my previous post (from June 2010) on Passwords since I recently reposted it.

More specifically the questions and comments were around key stretching.

Q.  But doesn't looping that many times slow the password verification step down?
A.  Well yes.  That's kind of the point.  The user may experience negligible latency during the authentication process, this can be tuned to have no or little effect on the user experience.  This same delay is multiplied by the number of brute force attempts made by the attacker.

Q.  The attacker doesn't need the salt or the algorithm if they are going through the front door?
A.  Agreed.  Which is why you'd have some controls and triggers at the front door to alert when abnormal or atypical behaviour is detected.

Q.  If an attack gains access to a system, surely it's game over for user PII?
A.  The thing I like about key stretching is that a hacker needs to have access to the salt, the hashed password and stretching algorithm.  So, if you store the salt in a different table or even DB than the hashed password, the attacker would need to get their mitts on both tables or both DBs. Gaining visibility of the algorithm means that the attacker would also need to have access to, or knowledge about the specific implementation details; ie the source code. Typical environment configurations ensure that there is no way to get to a development environment from a production environment (and vice versa), making it difficult to gain access to the source.

This is a quick and dirty example of what key stretching would look implemented in Java.


This runs in around 0.75 seconds on my local box. Which means every iteration of a dictionary attack or a brute force attack would take the same time.

Caveat: this is purely an example of how you could implement stretching.

mongometer Unhacked

When I first hacked mongometer together, I didn't think that I'd need to spend more than that one hour on both coding and posting on the hack. However, the more I used it, the more I found it useful a way of comparing the relative performance of my scripts, and the more I found it useful, the more I thought that others may find it useful.

(I also found a few shortcomings/annoyances, which is great, because that gives me some incentive to improve the hack.)

1. If I needed to change the hosts, the database, the credentials, you needed to rebuild and redeploy the jar.
   
2. There was no way to tweak the options [http://api.mongodb.org/java/2.7.2/com/mongodb/MongoOptions.html].
   
3. Connections didn't seem to be getting recycled.  You had to shutdown jmeter before the open connections were tried up.
   
4. Wanted to ensure that the connection didn't need to be created every time there was an iteration, and that the connection pooling could be utilised
   

1. Easy enough to the change this, I added the fields that had been frustrating me.

2. Again, easy enough to add these additional properties, though I did come up against a slight frustration

3. This was also slightly frustrating. JMeter didn't seem to be release any of the resources tied up in the Sampler. This was eventually resolved by a bit of experimentation and a bit of 4.

4. This turned out to be a lesson in RTFM. Implementing TestListener allows you to connect to MongoDB at the start of the test run, use the connection pooling during the iterations, and tidy up resources at the end of the run.

Moral
I don't really do 'and the moral of the story is...', but in this case, RTFM is reasonably apt. And, it's quick and easy to hack together something that works, it's not so easy to put together something usable in the longer term; but we all know that.

Request For Comments
If you found this useful, please leave a comment.
If there is a missing feature, please let me know and I'll try and add it.