Render farms, cluster nodes and other types of distributed computing often require using a lot of machines that don’t have a lot of stuff running on them and are only needed during certain times. Such is the life of a compute cluster, which is what EC2 is there for. Because cluster nodes are so homogenous by nature you can deploy them en masse. Picking up where I left off with deploying EC2 via the command line we’re going to look at spinning up let’s say 100 virtual machines with the large designation, from a pricing standpoint.
As with the previous example, we’re going to use ami-767676 (although you’ll more than likely want to choose an image of your own using the ec2-describe-images command) as the AMI name and use a predefined key called my-keypair to access these hosts. The command would then be:
ec2-run-instances -n 100 -t m1.large -k my-keypair ami-767676
The -n flag told ec2-run-instances how many instances to fire up and the -t flag told it what type of images. We can go a step further and start to automate the setup of software on these instances. When you use the ec2-run-instances you can specify a file to be copied to the instance. For example, -f MYSCRIPT.zip would copy a zipped script to the instances. You can also prebuild security groups and then use the -g flag to assign all of the instances you create to a security group that provides you with SSH access.
Once you have a key, a local script and SSH access to the hosts it is straight forward to loop through the instances initiating the script. When you run the ec2-run-instance you will get a line similar to the following:
INSTANCE i-767676 ami-767676 pending
The second position here (i-767676 for the example) is the name of the instance, which can be obtained using >awk ‘{printf $2}’with the output, grabbing only the names. For some of mine I use a script stored within the cluster and not made available outside and have the script update itself using a method I described last week. There are tons of other ways of doing that kind of thing as well.
Overall, the ability to provision servers for compute power has never been easier. No wait times for massive quantities of iron to roll in on pallets and then having the n00b rack mount it all. Now, just pay Amazon Web Services by the hour if/when you need a little firepower. Finally, it is worth noting that if you have a need where Amazon has provided an existing API to hook into EC2 (eg – SOAP), then that’s often a far more elegant (and therefore efficient) use of resources. Before you begin provisioning en masse, make sure it’s the right use for the right situation!