NImrod/K (Kepler) - Work Flow Engine for the Nimrod Toolkit

A challenge for Grid computing is the difficulty in developing software that is parallel, distributed and highly dynamic. Whilst there have been many general purpose mechanisms developed over the years, Grid programming still remains a low level, error prone task. Scientific workflow engines can double as programming environments, and allow a user to compose ‘virtual’ Grid applications from pre-existing components. Whilst existing workflow can specify arbitrary parallel programs, (where components use message passing) they are typically not effective with large and variable parallelism.

Over the years we have developed expertise in massively parallel parameter sweep workflows, using the Nimrod family of tools. Nimrod is a system that makes it very easy to build parameter sweep and search applications using the Grid. Users can run an arbitrary executable image, varying a number of parameters. Nimrod contains tools that perform a complete parameter sweep across all possible combinations (Nimrod/G), or search using non-linear optimization algorithms (Nimrod/O) or experimental design techniques (Nimrod/E).

However, Nimrod was not designed to execute arbitrary workflows of the type discussed above. Thus, it is difficult to run sweeps over workflows, and workflows containing sweeps. Likewise, as illustrated, most workflow systems do not support the parallel execution of tasks that is supported in Nimrod, and are not well suited to parameter sweeps and searches.

We have implemented a prototype tool called Nimrod/K that demonstrates the new mechanisms. Nimrod/K is built on Kepler using Kepler’s runtime engine (Ptolemy). It uses a dataflow execution model that was originally developed for highly parallel dataflow computers in the 1980’s, and this provides an extremely rich execution mechanism. It leverages a number of the techniques developed in the earlier Nimrod tools for distributing tasks to the Grid.