Several years ago, when testing the first version of pplacer, I built a Rube-Goldbergian system of shell scripts to test the effect of the various parameters on accuracy, and another system of scripts to collect the results. After getting it all set up, which was a pain, I still had a delicate and inflexible system depending on filename conventions. Once that paper was done, I vowed never to have to go through that annoyance again.

What I came up with was the first version of nestly. To keep things organized, it made a nested set of directories, with each level in the hierarchy corresponding to a parameter choice, and with JSON files at the tips of the directory tree with all of the relevant parameters. That first version was not very elegant and completely opaque.

Connor and Aaron then got a hold of it and made it a thing of beauty, with both a simple way of using it, and very powerful SCons integration that does incremental builds. Along with a script for actually running your code in the various settings, there is also a script and an API for collating results from the different runs. It has abundant documentation and examples and as well as continuous integration on Travis.

It is no exaggeration that this small package has become completely essential in our workflow, for simulations and analysis alike. The short paper description just came out (a PDF is here).