CAMBRIDGE, Mass -- Amazon Web Services Spot Instances can save companies substantial amounts of money on their cloud computing bills, as long as those applications are fault-tolerant and can programmatically take advantage of the instance type.
Spot Instances are spare capacity Amazon sells through a public bid process for up to 90% off the on-demand instance price. The highest bidder gets to use this spare capacity at the set spot price, but the instance can be turned off at any time, without notification, usually because the spot price has exceeded the user's bid.
Applications running on spot instances must be able to cope with this, said an experienced Amazon Web Services (AWS) spot instances customer who spoke here this week at an AWS Meetup.
"There are certain ways to develop your product that can take advantage of Spot Instances in better ways and save a lot of money," said Ed Abrams, principal software architect for SynapDx Corp., a Lexington, Mass., company that's developing a blood test for early detection of Autism Spectrum Disorders.
SynapDx makes the most of AWS Spot Instances
When Abrams first joined SynapDx eight months ago, gene sequencing runs could cost up to $40 per blood sample, but the company has reduced that to $1 per sample in part by using AWS Spot Instances.
The company's internally developed application relies on a software-based infrastructure and uses a distributed workflow pattern that doesn’t require a centralized job scheduler. This eliminates a single point of failure and makes it fault-tolerant enough to handle Spot Instance interruptions. Workflows can also run autonomously, without any dependencies on each other.
"A workload can fail, and then be picked up by a completely [different] node, and resume where the work left off," Abrams said.
The company's infrastructure is also entirely automated.
"TeamCity is also producing distributables, which we do in Python; and so our distributables are ready to go at any given time for a machine that's been launched by CloudFormation to … install the appropriate software on it,” Abrams said.
Once software is installed on an instance, two more applications, CloudInit and Chef, configure that machine. This machine is in turn placed into an AutoScaling Group with an alarm trigger so it scales automatically according to demand.
"We don’t want to launch 500 machines and then discover that in the middle of processing, [part] of our software is going to fail," Abrams said. "We'd rather launch 10 of them, see that and fix it before we launch the other 490."
Spot Instances pricing is built into SynapDx's CloudFormation template, so the company can detect when a price for Spot Instances is available and automatically spin it up. Spot prices are also baked in at the command-line level, allowing an operator to mention a bid price and let the application make it a spot instance.
Once instances have been instantiated, genetic processing begins. When job queues are empty, the machines can be shut down immediately and the process starts again.
AWS Spot Instances tips from the trenches
"We learned a bunch of very important things about bidding strategy," Abrams said. As a bidding strategy, Abrams advises companies to abstract away the notion of instance size and which generation of AWS instances it belongs to.
"For example, we're on c3.8xlarges, but it turned out the c2.2xlarges were really cheap a few months ago, and it was a tenth of the price and half as efficient in terms of time," Abrams said. "That's a net win for us, and we just ended up using those."
Be flexible about which AWS Availability Zone Spot Instances are spun up in as well -- contention for one Availability Zone can result in price spikes that, in some cases, drive Spot Instances prices up above the on-demand rate.
Apps like Hadoop or Elastic MapReduce, grid computing applications and apps that work off job queues are best suited for Spot Instances, according to Stephen Elliott, senior product manager for AWS EC2 Spot.
Elliott also offered tips on what not to do when it comes to bidding, chief among them is making a very high bid in the hopes of avoiding interruption.
People try this anyway, Elliott said.
"There are some people who just really cannot tolerate interruption at all and they've misunderstood spot and think that no one else would bid that high," he said. "Until someone else bids that high and they’re interrupted."