News Stay informed about the latest enterprise technology news and product updates.

Amazon ClusterK buy could boost AWS Spot Instances

Amazon acquired ClusterK, a startup that claims to have built a better mousetrap for orchestrating AWS Spot Instances and making them more reliable.

Amazon has bought a Silicon Valley startup that orchestrates AWS Spot Instances for greater reliability than is offered natively.

Amazon confirmed that it acquired ClusterK Inc. based in Palo Alto, Calif., for an undisclosed amount. ClusterK's software inserts a layer called a Balancer between Amazon Web Services (AWS) Spot Instances and the applications that use them. The Balancer algorithmically predicts the Spot price based on historical data, as well as Spot price spikes that could shut down instances.

The Balancer can also spin up new AWS Spot Instances before failures occur, to ensure capacity doesn't drop below a user-set minimum. It also continually tries to provision a user-designated optimum number of Spot Instance cores that span multiple instance types.

One beta tester of the product, 6Sense, a predictive analytics software maker based in San Francisco, has used the product successfully in production for a few months and would like to see the ClusterK Balancer become a permanent part of AWS Spot Instance product development.

"We're still seeing a ton of savings," said Premal Shah, co-founder and software engineer for 6Sense. Shah estimates that the ClusterK software with Spot Instances has saved his company more than $500 a day on AWS instance costs.

AWS customers had questions about whether changes are needed to AWS governance and orchestration tools to take advantage of ClusterK's software. Others questioned just how much more reliable ClusterK's software can make Spot Instances.

"Anything that makes Spot Instances more attractive is great – they're underutilized," said Brian Tarbox, a lead engineer working for a stealth startup in the Northeast, who has used Spot Instances extensively in previous jobs and is implementing them at his current company.

However, Spot Instances still disappear when the Spot price goes above a user's bid, Tarbox said. While ClusterK's software spins up new instances to replace them, Tarbox wondered if the work those instances were doing would still be lost.

"What does [ClusterK’s software] do if the Spot price exceeds On-Demand?" Tarbox said, adding that he'd seen this exact pattern in Spot prices in some US-East region Availability Zones over the past three months.

Neither Amazon nor ClusterK officials were immediately reachable for comment on these concerns.

This move marks the third acquisition for AWS in the last six months (following Annapurna Labs and 2lemetry), which could be a sign of a strategic shift for Amazon, according to Carl Brooks, analyst with 451 Research, based in New York.

"In six months, Amazon has acquired more companies for AWS than in the last seven years," Brooks said. "Either they are now moving faster than they can handle on adoption and development or there has been a significant change in strategic thinking about outside [intellectual property] assets, or both.

If Amazon has changed its mind about outside intellectual property (IP), it's probably because the ubiquity of AWS has made it a native platform for outside developers and thus they're more compatible with internal AWS development than older technologies were, Brooks said.

"Amazon seems to be willing to buy IP that would be difficult to develop themselves, or would take too long to develop themselves, as long as the price is right," said Lydia Leong, analyst with Gartner Inc. based in Stamford Conn.

 "Amazon would love to drive more utilization of Spot Instances, and anything that makes it more difficult for other providers to compete on price is to their advantage," she added.  

Beth Pariseau is senior news writer for SearchAWS. Write to her at or follow @PariseauTT on Twitter.  

Dig Deeper on AWS management

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

Several things that balancer does to prevent instance shutdown by spot price spikes:
1. predict them, use markets which will not be spiking soon
2. in case spike happens anyway, provision capacity in other markets before decommissioning the expensive spots (you can bid higher than OD price so the instances will continue working meanwhile).
3. even for the most sudden and high spikes, before the spot terminates it receives an advance notification. We can rebalance to the more reliable spot or on-demand markets in that time, and instance could try and complete the work it was doing.

As for work (potentially) being lost, yes, even on demand instances can die without warning due to problems with underlying HW/SW. We have no problem running several hour long tasks using spots.
This really highlights one of the things that Simon Wardley constantly mentions about AWS - they have a much better "early warning system" about customer usage then traditional vendors. Very few people had heard of ClusterK before the announcement. They obviously had and have a sense of how the acquisition can either help them maintain higher margins (shut it down) or help better capture a segment of their market that was under-optimized for certain use-cases.