Experimental Design with Nimrod/E

The techniques of formal experimental design and analysis are powerful tools for scientists and engineers. However, these techniques are currently underused for experiments conducted with computer models. This has motivated the incorporation of experimental design functionality into the Nimrod tool chain. Nimrod has been extensively used for exploration of the response of models to their input parameters; the addition of experimental design tools will combine the efficiency of carefully designed experiments with the power of distributed execution.

For researchers exploring the effect of the inputs on the results produced by an experiment, a traditonal approach has been to vary one parameter at a time, holding the rest fixed. This may give useful information but will give a misleading picture if the outputs are strongly dependent on interactions of the input parameters. At the other extreme the researcher may select certain values for each parameter and then run all possible combinations of these, a full factorial design. This should give a fuller picture of the response, but becomes impractical when there are more than a few inputs of interest, as the number of runs generated becomes huge. Experimental design techniques show how to select a subset of the runs to the best effect.

The design method implemented in Nimrod/E is fractional factorial design. This was developed by R. A. Fisher working on agricultural experiments in the 1920s. He observed that any full factorial design may be expressed as a sum of terms each dependent on a single input parameter, plus terms each representing two-way interactions of parameters, plus terms for the three-way interactions, and so on. Fisher found that, usually, the higher order interactions were negligible; good approximations could be obtained with only single and two-way interactions. When an experiment has many parameters, this greatly reduces the number of runs required. For example, an experiment with 80 inputs,each using two values, generates over 10²⁴ runs. A fractional factorial design, neglecting interactions of three or more inputs, needs only 65 536 runs, a practical number for a Nimrod controlled experiment.

The Nimrod/E user decides the values of interest for each parameter, which interactions will be ignored and which will be estimated. Nimrod/E will produce a fractional factorial design, passes the jobs to Nimrod/G for execution, and then produce analyses of the results for each output of interest.