Cloud denialists are decreasing in numbers, but some enterprises still cite loss of control, security issues and...
a lack of visibility as primary reasons to avoid public cloud services. Cloud services and independent analysts have addressed and debunked these issues, and the "folded arms gang" of resistors is on the wane as cloud services prove their value and integrity. Still, IT lives by the Cold War adage "trust, but verify," and no organization should blindly deploy applications on a cloud service without having monitoring and auditing programs in place.
Amazon Web Services (AWS) offers several tools to help administrators monitor and audit security. Many AWS logging services are complete with APIs, scriptable command-line interfaces and management consoles, making each automatable and extensible. When you have the right tools, it's easy to automate AWS security monitoring and auditing.
The foundation of every security audit or forensic analysis is a log trail of activity. All major AWS applications include logging features, but the most important items to log, collect and analyze include the following:
CloudTrail management activity. CloudTrail records all AWS API calls, making it useful for monitoring access to the management console, CLI usage and programmatic access to other Amazon services. CloudTrail also provides key input for security audits, recording all administrator activity such as policy changes on an S3 bucket, starts and stops on Amazon Elastic Compute Cloud (EC2) instances and changes to user groups or roles.
CloudFront access. CloudFront is the AWS content delivery network and can be configured to log detailed information about every user request. This is useful for certain content, but could lead to information overload.
Amazon Relational Database Service. This includes Relational Database Service logs console, CLI and API activity, including query errors and performance.
Amazon Simple Storage Service (S3) server access and bucket policies. S3 can record changes to bucket and object policies. Details of every access request are also logged, including requester, bucket name, request time, action taken, response status and error code, if any. This can also log object expiration and scheduled removal.
AWS logging -- from events to configurations
AWS logging provides a detailed record of all admin activity; it's also helpful to have a comprehensive summary and history of your AWS resources and configurations. AWS Config provides a detailed inventory of EC2 instances, configurations and associated block (Elastic Block Store) and network (Virtual Private Cloud) resources. AWS Config records changes, sends notifications via Simple Notification Service and can display the state of AWS infrastructure at any time.
Combining AWS Config with CloudTrail and other logging services enables auditors to correlate configuration changes. Such adjustments include access policies for an instance or storage bucket, with specific events, including details such as username, source IP and other actions that happened around the same time. The following example illustrates how AWS Config and CloudTrail combine in the forensic analysis of AWS systems.
AWS Config aggregates configuration and change management records, provides AWS resource inventory, records configuration history and triggers configuration change notifications.
Track and monitor events with AWS CloudWatch
Collecting relevant data isn't enough, admins also need a way to automatically monitor, measure, act on and visualize the data. That's where CloudWatch comes in; it's the monitoring and reporting engine for AWS resources and log files.
CloudWatch is programmable using an API/ SDK and CLI; it can be used to trigger both real-time alerts, such as resource utilization over a set threshold, or chart historical metrics, such as CPU use. CloudTrail and other logs can feed CloudWatch, allowing admins to track events from the component alongside those from the operating system, applications and other services sent to those CloudWatch logs.
Although AWS security and event monitoring tools differ from those used on-premises, the system design strategy is the same. The goal is to aggregate log data into a single repository, then use software to monitor, flag anomalies, measure and chart metrics, while aiding in forensic, post hoc analysis. AWS CloudTrail, CloudWatch and the logging capabilities of each AWS component form the data input. S3 is typically used for persistent storage while CloudWatch and third-party software handle data analysis.
AWS and third-party software -- better together
CloudTrail provides a good set of basic features, but it can't match the sophistication of dedicated log and operational analysis software. Popular products like Log Manager from Alert Logic, Logentries, Loggly and Splunk mirror the features of on-premises counterparts and are available through the AWS Marketplace.
Admins can deploy these tools using a plug-in service running on AWS. For example, the Splunk add-on collects events, alerts, performance metrics, configuration snapshots and billing information from CloudWatch, CloudTrail and AWS Config, along with generic log data stored in S3 buckets. The service then feeds Splunk Enterprise, which can be deployed as a self-managed service on AWS using Splunk-supplied Amazon machine instances or software.
Although AWS and third-party providers found in the AWS Marketplace offer an ample tool set for building automated cloud monitoring and auditing systems, putting a system together still requires some effort and expertise. AWS documentation, white papers and re:Invent presentations provide useful monitoring and auditing details. If your enterprise does not have the skills or time for a do-it-yourself project, consider a managed service such as 2nd Watch or Datapipe that design and operate complex AWS infrastructures.
Parsing AWS CloudTrail logs for useful information
What Amazon CloudWatch logs can and can't do
The business case for AWS' three monitoring tools