sommai - Fotolia

New Amazon Elasticsearch service eases setup, with exceptions

AWS shops are eager for relief setting up Elasticsearch clusters, even if integration with one important tool still requires some manual intervention.

AWS customers are excited to have most of the Elasticsearch management burden taken off their hands with a new Amazon Elasticsearch service dubbed Amazon ES.

As reported by SearchAWS in March, the Amazon Elasticsearch service takes the "undifferentiated heavy lifting" out of setting up a cluster to support the open-source search engine, which Amazon Web Services (AWS) users have found tricky to do on their own in the Elastic Compute Cloud (EC2).

Sonian, Inc., an email archiving cloud service provider located in Dedham, Mass., and an early participant in the Amazon ES beta program, has one of the largest Elasticsearch environments in the world, with hundreds of nodes indexing petabytes of data, and people dedicated to its care and feeding, according to its CTO Greg Arnette.

"The fact that we're going to be able to offload that to AWS and purchase Elasticsearch as a service and pay for what we consume is the overriding positive there," Arnette said.

Now, instead of having to agonize over setting up redundant Elasticsearch clusters, IT pros can click one checkbox labeled "Enable Zone Awareness" in the Amazon ES interface and have Elasticsearch nodes distributed across Availability Zones for resiliency. Enabling a dedicated master node for the cluster is a similar process, and a click on a link brings up the Kibana data visualization tool in a separate tab in the console, according to the AWS blog.

One company that uses another search engine built on top of Apache Lucene, Amazon CloudSearch, will start over with Elasticsearch because of the "Enable Zone Awareness" feature.

"That's probably hours of work being replaced with a click," said Mario Cruz, director at Watsco Ventures, an HVAC distributor in Miami. "That sold me right there."

Logstash integration limited in first Elasticsearch release

While Elasticsearch clusters and the Kibana plugin can be accessed with a few clicks, the "L" in the Elasticsearch / Logstash / Kibana (ELK) stack will still take some doing for Amazon ES users.

A Logstash plugin must be built, downloaded and installed into DynamoDB by the user, according to Amazon's documentation. The service also supports only one Logstash output plugin, according to Amazon's Developer Guide.

However, this isn't a deal-breaker for Arnette or Cruz, who both said they expect Logstash integration to deepen with further releases of the Amazon Elasticsearch service.

Another unsupported feature -- dynamic scripting -- is enough to keep at least one customer away from Amazon ES.

"They don't support custom scoring scripts … the small pieces of code to adjust hit 'scores' or the sorting value returned [on a search] … our search 'special sauce,' so to speak," said Bob Matsuoka, CTO at CityMaps, Inc., a social map app maker in New York.

Amazon declined to comment on this story.

Beth Pariseau is senior news writer for SearchAWS. Write to her at [email protected] or follow @PariseauTT on Twitter.  

Dig Deeper on AWS instances strategy and setup