Manage Learn to apply best practices and optimize your operations.

AWS logging tools provide extra security

Those apprehensive of public cloud often cite a lack of security as a concern, but monitoring AWS with native and third-party tools can ease your mind.

Cloud denialists are decreasing in numbers, but some enterprises still cite loss of control, security issues and...

a lack of visibility as primary reasons to avoid public cloud services. Cloud services and independent analysts have addressed and debunked these issues, and the "folded arms gang" of resistors is on the wane as cloud services prove their value and integrity. Still, IT lives by the Cold War adage "trust, but verify," and no organization should blindly deploy applications on a cloud service without having monitoring and auditing programs in place.

Amazon Web Services (AWS) offers several tools to help administrators monitor and audit security. Many AWS logging services are complete with APIs, scriptable command-line interfaces and management consoles, making each automatable and extensible. When you have the right tools, it's easy to automate AWS security monitoring and auditing.

The foundation of every security audit or forensic analysis is a log trail of activity. All major AWS applications include logging features, but the most important items to log, collect and analyze include the following:

CloudTrail management activity. CloudTrail records all AWS API calls, making it useful for monitoring access to the management console, CLI usage and programmatic access to other Amazon services. CloudTrail also provides key input for security audits, recording all administrator activity such as policy changes on an S3 bucket, starts and stops on Amazon Elastic Compute Cloud (EC2) instances and changes to user groups or roles.

CloudFront access. CloudFront is the AWS content delivery network and can be configured to log detailed information about every user request. This is useful for certain content, but could lead to information overload.

Amazon Relational Database Service. This includes Relational Database Service logs console, CLI and API activity, including query errors and performance.

Amazon Simple Storage Service (S3) server access and bucket policies. S3 can record changes to bucket and object policies. Details of every access request are also logged, including requester, bucket name, request time, action taken, response status and error code, if any. This can also log object expiration and scheduled removal.

AWS logging -- from events to configurations

AWS logging provides a detailed record of all admin activity; it's also helpful to have a comprehensive summary and history of your AWS resources and configurations. AWS Config provides a detailed inventory of EC2 instances, configurations and associated block (Elastic Block Store) and network (Virtual Private Cloud) resources. AWS Config records changes, sends notifications via Simple Notification Service and can display the state of AWS infrastructure at any time.

Combining AWS Config with CloudTrail and other logging services enables auditors to correlate configuration changes. Such adjustments include access policies for an instance or storage bucket, with specific events, including details such as username, source IP and other actions that happened around the same time. The following example illustrates how AWS Config and CloudTrail combine in the forensic analysis of AWS systems.

AWS Config aggregates configuration and change management records, provides AWS resource inventory, records configuration history and triggers configuration change notifications.

Track and monitor events with AWS CloudWatch

Collecting relevant data isn't enough, admins also need a way to automatically monitor, measure, act on and visualize the data. That's where CloudWatch comes in; it's the monitoring and reporting engine for AWS resources and log files.

CloudWatch is programmable using an API/ SDK and CLI; it can be used to trigger both real-time alerts, such as resource utilization over a set threshold, or chart historical metrics, such as CPU use. CloudTrail and other logs can feed CloudWatch, allowing admins to track events from the component alongside those from the operating system, applications and other services sent to those CloudWatch logs.

Although AWS security and event monitoring tools differ from those used on-premises, the system design strategy is the same. The goal is to aggregate log data into a single repository, then use software to monitor, flag anomalies, measure and chart metrics, while aiding in forensic, post hoc analysis. AWS CloudTrail, CloudWatch and the logging capabilities of each AWS component form the data input. S3 is typically used for persistent storage while CloudWatch and third-party software handle data analysis.

AWS and third-party software -- better together

CloudTrail provides a good set of basic features, but it can't match the sophistication of dedicated log and operational analysis software. Popular products like Log Manager from Alert Logic, Logentries, Loggly and Splunk mirror the features of on-premises counterparts and are available through the AWS Marketplace.

Admins can deploy these tools using a plug-in service running on AWS. For example, the Splunk add-on collects events, alerts, performance metrics, configuration snapshots and billing information from CloudWatch, CloudTrail and AWS Config, along with generic log data stored in S3 buckets. The service then feeds Splunk Enterprise, which can be deployed as a self-managed service on AWS using Splunk-supplied Amazon machine instances or software.

Although AWS and third-party providers found in the AWS Marketplace offer an ample tool set for building automated cloud monitoring and auditing systems, putting a system together still requires some effort and expertise. AWS documentation, white papers and re:Invent presentations provide useful monitoring and auditing details. If your enterprise does not have the skills or time for a do-it-yourself project, consider a managed service such as 2nd Watch or Datapipe that design and operate complex AWS infrastructures.

Next Steps

Parsing AWS CloudTrail logs for useful information

What Amazon CloudWatch logs can and can't do

The business case for AWS' three monitoring tools

This was last published in May 2015

Dig Deeper on AWS security

Join the conversation

4 comments

Send me notifications when other members comment.

By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Please create a username to comment.

What monitoring and security auditing tools do you use within AWS?
Cancel
We use Evident to secure our AWS. We used to do manual audits, and it was a huge pain as we grew from a handful of accounts to almost 100 AWS accounts. The great thing is since it is continuous and we can write our own security checks inside the platform, that we now know each team is following corporate security policy and any regulatory requirements they may be obliged to follow.

We've also used security monkey, but it was painful to setup and manage, and didn't scale (plus lacked a ton of useful security checks). A lot of the Netflix tools ended up being brittle for us so we had to look for commercial tools that would just work.
Cancel
Security - Evident
Logging - Loggly

The AWS native tools are OK, but these SaaS-based tools are more advanced and their core focused. Better in-depth information for the operations and application teams. Plus they natively integrate with a bunch of the other tools people use (eg. PagerDuty, etc.)
Cancel
There’s not really a lack of security (The AWS Shared Responsibility Model is actually pretty good: https://aws.amazon.com/compliance/shared-responsibility-model/). There is, however, a lack of good security practices on the part of the cloud users.
Cancel

-ADS BY GOOGLE

SearchCloudApplications

TheServerSide.com

SearchSoftwareQuality

SearchCloudComputing

Close