AWS cost management tools wrangle unwieldy expenses

How costs can spiral out of control

Anyone who remembers the early days of cellular plans, which charged for every text message or minute of usage, knows that a usage-based pricing model isn't always efficient or cheaper than an unlimited plan. AWS' usage model is similar, as resources are so simple to provision -- yet it's easy to forget to scale them back when you no longer need them.

Over time, as cloud usage progresses from niche to broad deployments, most enterprises discover -- usually after receiving their first six-figure monthly bill -- the critical importance of AWS cost management.

Thus, the two biggest sources of AWS cost overruns are related:

• unattended, unmonitored instances that run idle or with light workloads, all while the cost meter continues to run
• resource sprawl that leads to a growing collection of instances that outlive the applications they initially supported.

Additionally, other factors exacerbate the wastage. Overprovisioned resources can often start out too large and quickly become too expensive for the job at hand. Also, little or no governance over multiple accounts can lead to resource redundancy and no accountability over usage.

It's also important to understand the relatively complex pricing and discount models for AWS virtual infrastructure. For example, EC2 has four different pricing plans:

• On-Demand -- the most familiar -- in which you provision resources as you need them;
• Spot Instances, which offer deeply discounted spare capacity but aren't always available;
• Reserved Instances (RI), which provide three different payment plans that can become confusing; and
• Dedicated Hosts, with which a customer rents the entire capacity of a physical system.

Not only do the byzantine, non-uniform pricing models of AWS and its competitors make comparison shopping nearly impossible, they also require customers to match individual workload characteristics with the appropriate pricing plan.

For example, Dropbox, a company built on AWS and an enormous user of cloud resources, saved a net of almost $75 million over two years when it moved storage off of Amazon S3 and onto internal, custom-designed infrastructure. While most enterprises don't have million-dollar monthly AWS bills, nor Dropbox's need for capacity, this example shows how blind usage of AWS can result in massive overspending.

AWS-native tools

In addition to detailed cost and usage reports, which provide granular, log file-level detail, AWS offers six tools to assist customers in cost assessment and management:

• The Billing dashboard is a summary view available in AWS Management Console that includes charts with a spending summary, month-to-date relative spending by service and month-to-date actual spending by services -- both sorted by usage. These are summaries of historical records and do not forecast future spending.
• AWS Cost Explorer is another graphical tool that analyzes spending. Cost Explorer enables customers to view the last 13 months of spending and statistically forecast the next three months. Based on usage patterns, Cost Explorer recommends On-Demand Instances or RI, including instance sizes. It also includes greater control over visualizations to illustrate patterns by service over different time periods. These visualizations can use one of three types of reports: cost and usage, RI utilization and RI coverage. Users can query and automate Cost Explorer via AWS Cost Management APIs.
• RI Reporting is a set of Cost Explorer reports that provide details about RI utilization, with targets showing underutilized instances. These reports also detail RI coverage, displaying the number of instance hours for RI-capable services, such as EC2, Redshift, Relational Database Service (RDS) and ElastiCache. The services enable customers to adjust RI utilization or coverage targets so that they trigger alerts when usage drops below these instance hours. Its recommendation algorithm does not predict future usage but uses data from a single account over the last seven, 30 or 60 days and then calculates the percentage cost savings and total annual dollar amount of using RI instead of On-Demand instances. Users can change the RI parameters to customize recommendations.
• AWS Budgets enables customers to set cost- or usage-based targets by month, quarter or year. Users can filter budgets by service type, linked account, tag, instance type (On-Demand or RI), availability zone or API operation, such as S3 bucket creation. The AWS budgeting tool supports up to 20,000 budgets per account, which facilitates multiple targets for stakeholders who track spending at various levels of the organization. If spending exceeds set targets, AWS Budgets can send an email or Amazon Simple Notification Service (SNS) message to trigger manual or automated remediation. For example, SNS could trigger a Lambda function that automatically reduces the number of EC2 instances in an Auto Scaling group to cut costs.
• AWS Trusted Advisor automatically scans an AWS account and makes recommendations to improve performance, security, reliability and fault tolerance, as well as reduce cost. While AWS provides a basic set of free security checks, the full suite, which includes cost checks, requires a Business or Enterprise support plan. AWS Trusted Advisor makes recommendations on the following parameters for cost optimizations:
  • EC2 conversion from On-Demand to partial upfront RI;
  • RI leases recently expired or set to in the next 30 days;
  • low-utilization EC2 instances of less than 10%;
  • idle load balancers;
  • underutilized Elastic Block Store volumes or Redshift clusters;
  • idle RDS instances; and
  • unassociated Elastic IP addresses not linked to an EC2 instance.
• Cost management APIs enable users to programmatically access cost, usage, pricing and budget data for use in external applications or spreadsheets. These APIs enable third-party AWS cost management applications to access data to generate more detailed reports and recommendations.

The case for third-party vendors

Organizations should start with AWS-native tools before they spend money elsewhere for AWS cost management. Even if AWS tools prove inadequate for their cost assessment needs, organizations can learn which features they will need from a third-party product.

Third-party products can fill gaps in AWS-native tools for organizations that deploy advanced cloud environments, such as a multi-cloud architecture. AWS has historically taken a dim view of the concept of hybrid or multi-cloud -- until recently. As multi-cloud deployments grow in popularity, enterprises need cost management tools that can work across platforms, unlike native AWS tools.

Additionally, AWS tools provide a basic set of reports, charts and forecasting algorithms, but there's room for improvement. For example, even an automatic tool like Trusted Advisor can't analyze an entire fleet of EC2 instances, storage volumes and workloads to suggest optimal sizing and workload placement strategies. AWS could stand to improve its forecasting methods, which use simple linear extrapolations of past trends, not powerful machine learning algorithms, which can automatically establish normal usage baselines, detect temporal versus long-term trends and make more accurate usage forecasts. Also, AWS visualizations and dashboards have limited customization options, particularly for services other than EC2.

Through its cost management APIs, AWS has provided the hooks for third-party software and services to access cost, usage and pricing data. These third-party AWS cost management tools generally simplify the configuration process, but it gets more complicated when you want to use them to pull data from multiple environments. In this situation, the level of integration might not be as tight, and on-premises systems might even require additional software or plug-ins.

Third-party products might also lack support for the latest Amazon cloud services, such as Lambda, Aurora or AI products. Add-on products also include additional expenses, not just for licensing, but also training and administrative overhead.