Introduction

Example Screen Shots

The Nimrod portal experiment page showing the Nimrod plan file and status of the test experiment.

The Nimrod job viewer applet showing the progress of jobs from the test experiment executing on a GT4 resource and Amazon EC2.

Output of a job that ran in the cloud as displayed from the Nimrod portal.

In late 2008 MeSsAGE Lab ran a student project which involved prototyping a Nimrod/G interface to Amazon's Elastic Compute Cloud (EC2). A press release and further details are available here: Nimrod soars into the clouds. Since then researchers in the lab have been working to re-engineer and stabilise the interface and it is now available in the Nimrod daily build available off the downloads page.

Features

• The interface makes use of the boto client library. This means it will work with any EC2 compatible interface, for example Eucalyptus clouds.
• Cloud resources can be used alongside clusters or computational grid resources, this allows users to seamlessly mix heterogeneous resources in a single computational experiment.
• A specialised scheduler manages requesting of agents (remote job execution managers) for the cloud, users can nominate conservative or aggressive scheduling with regard to the number of instances launched in a scheduling interval.
• The cloud interface makes full use of Nimrod's meta-scheduling capabilities allowing cloud instances to be utilised efficiently by running multiple jobs or parameter sets on a single instance. This can occur both sequentially and in parallel across machine instances, and also within multi-core machine instances if desired.
• Users provide credentials, service endpoints (these default to the EC2 US-East-1 region), and the type and ID of instances desired (limits on the number of machine instances running in parallel can also be specified).
• Coupled with Nimrod's existing economic scheduling capabilities this interface can be used to help a user meet deadlines by scheduling overflow jobs, from e.g. local free of charge resources, onto a commercial cloud such as EC2.

Basic Example

The example illustrated by the images on the right demonstrates a Nimrod/G experiment running across a GT4 resource and the Amazon Elastic Compute Cloud. The EC2 resource has been added using the default parameters plus specifying an instance limit of four, for the command line interface this is:
nimrod resource add ec2 --key=./ec2id --secret=./ec2secret --instlimit 4

Coming Soon

• A Nimrod release featuring this functionality.
• User documentation of the new features.

Possible Future Additions

• Use S3 or a dynamic EBS volume to cache remote components - depending on interface consistency with Eucalyptus resources.
• Use S3 to store meta-data regarding managed instances so that running instances can be re-used be different invocations.

Want More Information?

Please contact:
blair.bethwaite@infotech.monash.edu.au, and/or david.abramson@infotech.monash.edu.au