Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Log API calls and monitor AWS apps with CloudTrail

Are you trying to understand role-use patterns or track what AWS users with admin privileges are doing? CloudTrail service monitors API call logs.

Building your own applications gives you full control over logging. With cloud computing, you're limited to the...

service provider's logging options. CloudTrail is a monitoring tool that provides application programming interface call logging across a broad range of Amazon cloud services.

Still in beta, CloudTrail allows admins to set up a trail, or the option to log a particular set of application programming interface (API) calls, giving them insight into AWS cloud resources. A log entry for an API call includes the identity of the API function caller, the time the API function was called, the source IP address of the caller, the parameters in the API call and the response from AWS. Entries are collected in a log file, which is delivered to the AWS Simple Storage Service (S3) bucket you specify.

Because CloudTrail log files are stored in an S3 bucket, you can aggregate the files from multiple accounts in a single bucket. You will have to configure the S3 bucket policy to grant CloudTrail permission to write log files from multiple accounts. AWS offers documentation to do so. You can also collect log files from multiple regions in a single bucket.

The monitoring service  creates a notification that the Simple Notification Service delivers, and it allows you to monitor when log files are created. If you are monitoring several services or otherwise have log files that are created frequently, consider handling notifications programmatically to avoid a large stream of them. CloudTrail information can be used for security monitoring, compliance reporting and resource management. A log of API calls can be especially helpful for security monitoring. 

Log entries include:

  • a user identity element that specifies the type of caller, such as root, Identity Access Management (IAM) user, AWS service, etc.;
  • an indication if temporary credentials were used and how they were obtained;
  • a notice if the API call was made from the AWS management console;
  • an element indicating if Web identity federation was used to authenticate the caller.

This information can help admins understand typical role-use patterns. For example, you might find that most AWS end users with administrative privileges make a common set of calls to Elastic Compute Cloud (EC2), IAM and Relational Database Service (RDS). This can become a baseline to identify anomalous behavior, such as an administrator who is running an unusually large number of IAM calls. This administrator may be updating identity management information as part of a normal day or it could indicate unauthorized activities.

What CloudTrail won't tell you 
CloudTrail does not provide performance or health monitoring information -- you should use Amazon CloudWatch for that. When combined, the two tools give admins deeper insights into application performance.

When CloudWatch indicates a problem or unusual performance issues, you can look at CloudTrail log entries from the same time period to possibly identify the root cause of the problem. CloudWatch might indicate poor performance on an EC2 instance that correlates with a large number of calls to the RDS API function DescribeEvents, for example. Further investigation could detect a bug in the application code that called the API function within a loop -- when it should have been called outside the loop.

Manually scrubbing over log files or writing scripts to aggregate data from multiple log files is time-consuming. Many admins would prefer to hand these tasks off to a third party. Amazon has partnered with 12 vendors to provide log analysis services, including Splunk, Loggly and CloudCheckr.

CloudTrail is available for AWS EC2, Elastic Block Storage (EBS), Elastic MapReduce, RDS and Elastic Load Balancing as well as newer services such as Amazon Kinesis. In total, CloudTrail can monitor 17 AWS services. It's available in the Northern Virginia, Oregon, Northern California, Ireland and Sydney regions. Even though CloudTrail is a free service, customers are charged for Amazon S3 storage and SNS notifications.

About the author:
Dan Sullivan
holds a Master of Science degree and is an author, systems architect and consultant with more than 20 years of IT experience. He has had engagements in advanced analytics, systems architecture, database design, enterprise security and business intelligence. He has worked in a broad range of industries, including financial services, manufacturing, pharmaceuticals, software development, government, retail and education. Dan has written extensively about topics that range from data warehousing, cloud computing and advanced analytics to security management, collaboration and text mining.

Dig Deeper on AWS architecture and design

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.