So we found a tricky bug in the Dave's original NN code. The line had assumed that vectors of weights and inputs were of identical length when one was shorter - leading to iterating past the end.
So this was why problems kept coming and going again. My main thought after all this (very boring) debugging effort is how we identify and locate these type of errors more easily. Reading the code it was not obvious that the last weight was the output weight (at least at the location of the bug).
So in ones own code it helps to follow the principle of least astonishment - but in other peoples code? Is unit testing the only way? After all the confusing debugging this I'm inclined to say yes. I don't think static testing would have found this and valgrind didn't comment as the memory area was valid. I feel like I still have a lot to learn esp in the QA department!

Spent last night writing my grad symposium talk, I'm not sure its that great. But it just has to be good enough I guess.
All the pieces are in place for the system test, the files, erik is soon to commit the True Flavour code. Just the damn convergence with the old offline code. Hoping to solve this by narrowing down the differences tonight.
While reseaching my talk I found this gem: http://vlhc.org/Limon_seminar.pdf
These guys are the most funding optimistic physicists I've ever seen! 200km tunnel!

Fixed the tag processor - note to self: always be careful copying objects,
I managed to nuke the stack (but in a subtle way) corrupting the inputs.
On the other upside I did a full 20000 event Mokka run at the Z peak (CSF farm),
can't wait to see what the performance will be like. Almost there......

Some weirdness in either the plot processor or sgv, sigh. This better be the last time.
On the upside I ran a 200 event Mokka run (10x200min on ppslgen) and got this from my patented comparing widget:

Mok 0 is subset of: 0
Mok 1 is subset of: 1
Mok 2 is subset of: 2 66
Mok 3 is subset of: 3
Mok 4 is subset of: 4 5 47 98 162 184
Mok 5 is subset of: 5
Mok 6 is subset of: 6
Mok 7 is subset of: 7
Mok 8 is subset of: 8 86 184
Mok 9 is subset of: 9 155
Mok 10 is subset of: 10 31 54 85 101 111 125 129 133 144 188 191 194
Mok 11 is subset of: 11
Mok 12 is subset of: 12
Mok 13 is subset of: 13
Mok 14 is subset of: 14

w00t!

Heres the FINAL todo list:
True Jet Flavour - Erik
High level docs - Sonja
Finish class level docs - All
Sys test - (me???)
Fix p-e plot problem - me

Almost there.....

Geant4/Mokka

Decided it was time I installed Geant4 and Mokka to give them a spin. Installed Mokka 06-03-p01 with Geant 4.8.1.p02 and CLHEP 2.0.2.3.
Were a few issues:

Run make in $G4SYSTEM/physics_lists/hadronic as its not built by the main G4 make (Mokka complains of missing -lFTFC)

Edit MokkaGear.cc to remove the gear namespace (Not in header file)

Edit Mokka/src/Kernel/GNUMakefile to always add -lG4geomdivision (Mokka complains of undefined G4Division)

I'm getting about 1 event for 17min on my 3GHz p4. Looks like I need to get the grid going to use this in anger.

Lame

I stopped updating when I had no internet (note to self don't pick a dodgy cheap isp) and never picked up again. Lame. I'll try again.