Amazon ClusterK buy could boost AWS Spot Instances

Amazon acquired ClusterK, a startup that claims to have built a better mousetrap for orchestrating AWS Spot Instances and making them more reliable.

Amazon has bought a Silicon Valley startup that orchestrates AWS Spot Instances for greater reliability than is offered natively.

Amazon confirmed that it acquired ClusterK Inc. based in Palo Alto, Calif., for an undisclosed amount. ClusterK's software inserts a layer called a Balancer between Amazon Web Services (AWS) Spot Instances and the applications that use them. The Balancer algorithmically predicts the Spot price based on historical data, as well as Spot price spikes that could shut down instances.

The Balancer can also spin up new AWS Spot Instances before failures occur, to ensure capacity doesn't drop below a user-set minimum. It also continually tries to provision a user-designated optimum number of Spot Instance cores that span multiple instance types.

One beta tester of the product, 6Sense, a predictive analytics software maker based in San Francisco, has used the product successfully in production for a few months and would like to see the ClusterK Balancer become a permanent part of AWS Spot Instance product development.

"We're still seeing a ton of savings," said Premal Shah, co-founder and software engineer for 6Sense. Shah estimates that the ClusterK software with Spot Instances has saved his company more than $500 a day on AWS instance costs.

AWS customers had questions about whether changes are needed to AWS governance and orchestration tools to take advantage of ClusterK's software. Others questioned just how much more reliable ClusterK's software can make Spot Instances.

"Anything that makes Spot Instances more attractive is great – they're underutilized," said Brian Tarbox, a lead engineer working for a stealth startup in the Northeast, who has used Spot Instances extensively in previous jobs and is implementing them at his current company.

However, Spot Instances still disappear when the Spot price goes above a user's bid, Tarbox said. While ClusterK's software spins up new instances to replace them, Tarbox wondered if the work those instances were doing would still be lost.

"What does [ClusterK’s software] do if the Spot price exceeds On-Demand?" Tarbox said, adding that he'd seen this exact pattern in Spot prices in some US-East region Availability Zones over the past three months.

Neither Amazon nor ClusterK officials were immediately reachable for comment on these concerns.

This move marks the third acquisition for AWS in the last six months (following Annapurna Labs and 2lemetry), which could be a sign of a strategic shift for Amazon, according to Carl Brooks, analyst with 451 Research, based in New York.

"In six months, Amazon has acquired more companies for AWS than in the last seven years," Brooks said. "Either they are now moving faster than they can handle on adoption and development or there has been a significant change in strategic thinking about outside [intellectual property] assets, or both.

If Amazon has changed its mind about outside intellectual property (IP), it's probably because the ubiquity of AWS has made it a native platform for outside developers and thus they're more compatible with internal AWS development than older technologies were, Brooks said.

"Amazon seems to be willing to buy IP that would be difficult to develop themselves, or would take too long to develop themselves, as long as the price is right," said Lydia Leong, analyst with Gartner Inc. based in Stamford Conn.

 "Amazon would love to drive more utilization of Spot Instances, and anything that makes it more difficult for other providers to compete on price is to their advantage," she added.  

Beth Pariseau is senior news writer for SearchAWS. Write to her at [email protected] or follow @PariseauTT on Twitter.  

Dig Deeper on AWS management