News Stay informed about the latest enterprise technology news and product updates.

Kinesis could push AWS billing to near-real time

Amazon reps piqued attendees' interest at an AWS Roadshow this week when they detailed how Kinesis could be used to enhance the AWS billing process.

CAMBRIDGE, Mass. -- Amazon Web Services could use its Kinesis data streaming service to enhance its billing process, an idea customers hope will come to fruition.

Amazon Web Services (AWS) customers say a billing system based on Kinesis would be a good move for the company, following a data analytics roadshow by one company official who suggested a possible internal use for the data streaming service.

Amazon's streaming data analytics service, Kinesis, could refine AWS' billing and metering system, which involves billions of records generated per hour, said Eric Heikkila, business development manager for data analytics for AWS. The AWS billing system currently uses Elastic MapReduce (EMR), better suited to batch jobs, so bills are current up to 48 hours ahead, since it takes about 24 hours to aggregate data and run the EMR batch processing. Using Kinesis and running custom code on "a massive fleet" of Elastic Compute Cloud (EC2) instances could cut that time down to as little as ten seconds, Heikkila said.

Customers in attendance at the meeting whose businesses are based entirely on AWS were enthusiastic about this idea.

"Being able to manage your AWS environment based on more accurate cost data, knowing where you are in terms of your current bill and being able to marry it up with the price on EC2 at a given point in time, being able to spin up EC2 instances [in response], [and] doing all that while managing your AWS budget is pretty powerful," said Arturo Fagundo, founder of, a startup based in Marlborough, Mass., that performs secure backups between AWS accounts for its customers.

There would still be some work to do to take advantage of finer-grained billing data, Fagundo said.

Another session attendee echoed Fagundo's enthusiasm.

"We're constantly developing and integrating new data sources into our products, so seeing near-real-time costs of our efforts would be helpful rather than a 48-hour delay," said Tanya Cashorali, director of analytics at a Boston-based firm specializing in telecommunications data analysis. "When working in such an agile environment it can be painstaking to go back and check all AWS bills and try to figure out which experiment or process incurred which costs."

EC2 instance tagging advances wanted

Depending on the company, speedier AWS billing is not necessarily critical, according to James Staten, analyst with Forrester Research, based in Cambridge.

"A lot of enterprises are miles away from thinking this way, so this is not something that's necessarily an enterprise value proposition," he said. "But if you're a startup, or a digital gaming company, with a 100% cloud-based service or value proposition, there's significant appeal to this."

Google already provides billing information in 10-minute increments, but Amazon can offer better granularity in its billing with its tagging system, according to Staten.

Amazon still needs to work out some kinks with tagging and could benefit from including automatic tagging or policy-based tagging, added Staten.

Reached for comment after the meeting, an AWS spokesperson declined to say whether Kinesis-based billing is on the company's roadmap.

Beth Pariseau is senior news writer for SearchAWS. Write to her at or follow @PariseauTT on Twitter.

Dig Deeper on AWS streaming and file transfer services

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

What else should AWS do to improve its billing system?
One of the issues we have found is the discrepancy in the resource platform descriptions. When you log into AWS console, and you right-click and EC2 resource, you get a particular platform description. Then when you look at the billing CSV file from the S3 billing bucket, there is another platform description that mostly matches up, but not always. When you query the resource using the AWS you get yet a third description. 
When you have an automated cloud financial intelligence platform like that automates all this, it can be a challenge to auto-map reserved instances to lower-level account hierarchies because of this. As you probably know, the RI must match the size, region, AZ and the platform. It is this platform issue that causes so many headaches as the master-data approach to this is questionable at best.
I heard a similar comment / use-case about the new SignalFX service at VelocityConf last week. They had customers who were sending the transaction logs to them because they could process the streams faster and do better/more-accurate billing than the customer could themselves.