I've been stabilizing our build process at work, and have found some oddities in using nant and nunit.  The issue's roots seem to come from handling of config files.

Here is what's happening.  We maintain a “program” solution, and a “test” solution.  The program solution has its own config file, and we needed a way to have the test solutions also carry a config file.  We tried a number of different ways, but ended up using a test.config, which the testfixtures copy to assemblyname.test.dll.config for automated testing.

The nunit2 task in nant supports providing a assembly, and a config file.  So I went from using a fileset including **.test.dll, to using a number of <test> tasks, each individually specifying the assembly and the config file.  This seemed to get close to working, but would still intermittently throw odd errors that I couldn't track down.  The part that gets me, is calling nunit-gui or nunit-console on the same assemblies would not throw an error.  Looking into the nant code, we saw that nant creates a new appdomain, and that’s why when using the nant task you must specify the config file, but it doesn’t seem to work, or at least not reliably.

After hours of mucking around with config files, and tasks in nant, I finally broke down and added an <exec> task calling nunit-console for each assembly we wanted to test.  It is not as dynamic as I was shooting for, but now it seems to work very reliably, so I am willing to sacrifice flexibility to reliability in this instance.

My question, though, is who else has seen behavior like this?  We have the nunit2 task working for assemblies without config files, but with them it just doesn’t seem to fly.  Anyone else notice this?