This content is part of the Essential Guide: An admin's guide to AWS data management

AWS log management tools make sense of gobs of data

Amazon services produce an avalanche of log data. How can IT pros sort that data into actionable information?

AWS customers now have a glut of log data at their fingertips to help manage cloud infrastructure, but making sense of it all may require a tool from an Amazon partner.

Amazon Web Services (AWS) provides a number of services which generate logs, but it is often unreasonable for customers to manage these manually, especially as the number of log-generating services such as CloudTrail and AWS Config grow. And while Amazon provides the logs, it doesn't offer a native log analysis tool.

For Citymaps, a social mapping app maker in New York, Amazon CloudFront is a prime example of this. CloudFront can export logs to the Simple Storage Service (S3), but it's up to the user to analyze them. Citymaps needed a way to track API calls from its application to the CloudFront content delivery network.

Enter Loggly, Inc., a cloud-based AWS log management and analytics service provider in San Francisco. Loggly's application provides agentless log collection, automated log parsing, search and data visualization tools.

"We like Loggly's aggregation, reporting and notifications," said Bob Matsuoka, CTO for Citymaps. "We're able to do cross-reporting, which we couldn't do if they were separate systems."

Loggly recently added a feature called Derived Fields, which allows users to write their own adapters to parse logs not supported out of the box. CloudFront logs fall into this category.

"By using those Derived Fields we're able to take a log file and parse it and then index the fields that we're looking for," Matsuoka said.

Eventually, it would be nice to have the CloudFront log files supported natively, Matsuoka said, as there's a bit of a pipeline required to process a large number of small files CloudFront sends to S3.

"The ideal would be if we could have CloudFront go right to Loggly and then have them aggregated and maybe some sort of map/reduce to just get the active data," Matsuoka said. Building a billing and reporting system based on Loggly's parsing of CloudFront data would probably require deeper integration, he said.

Logentries makes sense of CloudTrail audit data

The data produced by native AWS cloud management services such as AWS CloudTrail and the systems logs generated by thousands of AWS instances led another AWS customer, AdRoll, to AWS partner Logentries.

AdRoll, based in San Francisco, provides re-targeting products for marketers. It has more than 5,000 instances in the Elastic Compute Cloud -- far too many to manage and parse system logs manually. AdRoll looked into building its own log management system using Elasticsearch, but chose an off-the-shelf product instead.

Some AWS customers have run into difficulty hosting Elasticsearch on AWS, but, ultimately, AdRoll chose Logentries for AWS log management because the open source data visualization tool Kibana, often paired with Elasticsearch, doesn't have alerting capabilities.

"We would have had to build them," said Ilya Kalinin, senior DevOps engineer for AdRoll. Logentries also can display data from the JSON documents produced by CloudTrail, another feature which attracted AdRoll to the service.

Logentries has only been on the job a few months but has already identified the source of a problem with an HBase server that became unreachable.

A tool is not required to parse CloudTrail logs, but it definitely makes life easier, Kalinin said. With CloudTrail's native tools, "you have to click a couple of extra buttons for every event," he said.

On the wish list for Logentries is the ability to hide instances that are no longer active or have been terminated -- when there's a production issue, these logs can get in the way, according to Kalinin. Logentries officials say this is on the roadmap.

Logentries is priced starting at $29 per month for a starter plan with 14-day retention of up to 30 GB per month in log data. It competes with other log analysis tools such as Loggly and Sumo Logic. Loggly starts at $499 a month for an enterprise plan.

Beth Pariseau is senior news writer for SearchAWS. Write to her at [email protected] or follow @PariseauTT on Twitter.

Dig Deeper on AWS big data and data analytics