everythingpossible - Fotolia


New CloudTrail features help verify API calls, AWS usage

Updates to AWS CloudTrail allow admins and developers to more quickly analyze calls to Amazon cloud services and act on invalid access attempts.

Until recently, the only way to deal with the mountain of information that AWS CloudTrail provided was to manually browse its logs for useful information -- an inefficient and exhaustive process.

At AWS re:Invent 2015, the cloud provider announced several improved ways to turn this raw AWS usage data info into valuable information. The simplest way to use the logged information is via the AWS CloudTrail console.

The AWS CloudTrail console is free with CloudTrail and allows its user base -- which can include security admins, DevOps and developers -- to filter AWS usage events by user name, event name, resource type or resource name and date range. The CloudTrail console allows cloud admins to easily view any events, such as those related to inbound security rules. This approach is helpful for an ad-hoc investigation of the set of API calls being made against particular resources and at particular times.

The AWS CloudTrail console displays recent API activity.
The AWS CloudTrail console displays recent API activity.

The CloudTrail command line interface (CLI) now supports a set of filter options similar to the console. These options are still more useful for investigations rather than ongoing analysis, but they take a step in that direction.

An interesting twist on the use of the AWS CLI is the pair of parameters "—generate-cli-skeleton" and "—cli-input-json." Many of the CLI commands in the AWS universe take a large number of optional parameters, which can make it difficult to build a valid command. The first of the new parameters creates a skeleton JSON file of the options relevant to that particular command. You can then edit the file and pass it to the "—cli-input-json" parameter. This eases the burden of creating and maintaining a valid parameter set.

For example, you could enter the command: "aws cloudtrail lookup-events --generate-cli-skeleton > ct.json," to create a JSON file with defaulted values for the various CloudTrail filter options that you could then edit and reuse.

The CloudTrail Processing Library can be used to develop a Java program to analyze the AWS usage event stream as it is generated. Fundamentally, all a CloudTrail user has to do is write an event processor method. This method is called for each CloudTrail event and receives the event; developers can then do whatever they want with the event. By default, an event processor will receive all CloudTrail events, which is probably not desirable; developers can add a filter to receive only the subset of events needed for a particular scenario.

public class SampleEventsProcessor implements EventsProcessor {


    public void process(List<CloudTrailEvent> events) {

        int i = 0;

        for (CloudTrailEvent event : events) {

            System.out.println(String.format("Process event %d : %s", i++, event.getEventData()));




To take CloudTrail's AWS usage analysis capabilities a step further, load the data into resilient distributed datasets (RDDs), where Spark can perform sophisticated analyses and data mining.

The high-level view of the process is that the user pulls the CloudTrail event stream into RDDs, creates Spark DataFrames for each event and then registers those DataFrames as Spark tables. That grants the user the full power of Spark for analyzing the entire body of an event stream. It's not a trivial exercise to set this up, and the example uses Scala -- a programming language that may be unfamiliar to many IT professionals -- but the payoffs in terms of mining AWS usage data may be worth it.

There are 43 AWS products that support CloudTrail in a total of nine worldwide regions, which means events about API calls can be received and processed nearly anywhere in the world.

Next Steps

CloudTrail polices the AWS cloud

AWS log management tools sort out data clutter

Logging tools provide security boost

Dig Deeper on AWS CloudWatch and application performance monitoring