This content is part of the Essential Guide: AWS Lambda architecture brings serverless to enterprise cloud

SQS queue workarounds for AWS Lambda functions

AWS Lambda integrates with many services under the AWS umbrella -- but not with Simple Queue Service. There are several bypasses admins can take to receive asynchronous messages from the service.

Containers and microservices are two of the hottest -- and potentially disruptive -- concepts in IT. Together they...

make it easier to piece together applications out of lean, feature-specific components that use fewer resources, are faster to develop and can be more densely packed on virtual infrastructure. But what if you could build an application service without the server?

That's where AWS Lambda comes into play. AWS Lambda is a service that runs code on-demand without the need to worry about instantiating, sizing, scaling and decommissioning server instances. Like all new AWS products, Lambda is still evolving, and while it is already quite versatile, it's missing one feature that could turn off developers using asynchronous message queues, like Amazon Simple Queue Service (SQS). Why not allow an SQS queue to trigger a microservice Lambda function instead of a full-blown application?

Lambda works by listening for triggers from other AWS products; it currently supports events from the following services:

  • Amazon Simple Storage Service (S3): triggered from object creation or deletion in select buckets.
  • DynamoDB: item updates in a NoSQL table.
  • Kinesis: triggered when data is added to a particular stream.
  • Simple Notification Service (SNS): launches Lambda functions from event-driven notifications to a particular topic.
  • Simple Email Service (SES): invoke asynchronously to control mail flow.
  • Cognito: triggered based on mobile back-end data sync events.
  • CloudWatch: triggered based on events in real-time log feeds and state changes.
  • CloudFormation (service templates): specifies a Lambda function as a custom resource to run any custom commands as part of stack creation.
  • Scheduled events (powered by CloudWatch events): allows regular invocation of Lambda code much like a CRON job. Events run at a specific time or recur, use CRON syntax with five-minute granularity and can poll from various AWS data sources.
  • Amazon API Gateway: used to define a custom REST API and endpoint that invokes a Lambda function.
  • AWS IoT: configure rules to route sensor data to other services.

But Amazon SQS is conspicuously absent from this list, which is notable because SQS is designed to safely, asynchronously pass messages and move data between distributed applications. The omission was a source of much angst on AWS discussion forums last year, and while a company representative said the feature "is on our radar," Lambda integration with the SQS queue hasn't been implemented. But when it comes to software development, there is more than one way to skin a cat.

Sample Lambda-SQS use cases

Application middleware offers a typical scenario in which integrating Lambda and SQS would be handy. For example, a front-end client -- Web, mobile, smart device -- might create data that needs further processing before being passed along to a back-end database. Conversely, the client might send a request to pull data from the back end that needs preprocessing for display on the client. In either situation, serverless Lambda functions are ideal for the middleware tier.

If the event flow involves messages -- not data -- Lambda can be directly invoked by using SNS, SES or Cognito, for mobile clients, by sending events triggering Lambda functions. For example, these functions can write data to DynamoDB or an S3 bucket or they can retrieve data or images to the client for display. However, if the application event involves data and the architecture uses message queues, invoking a Lambda function requires a bit more work.

Polling the SQS queue with scheduled events

The goal of message queues is to decouple application functions from direct dependence and communication, making the entire end-to-end application flow more robust and reliable. One component can send data to the SQS queue, and others can watch the queue and act on the data independently of the sender.

Lambda can't be directly triggered by SQS activity, so the only option is to periodically poll the queue. Until recently, this required a hack -- using an app running on a separate EC2 instance to poll or watch SQS queues. Of course, this defeated the point of using Lambda in the first place, because the app instance could just as easily implement the required Lambda functionality. But AWS added scheduled events to allow regular invocation of Lambda functions.

Scheduled events are like Lambda CRON jobs; they are configured using standard CRON syntax. Any Lambda function that needs to trigger from a message queue can be set up as a recurring event; the first thing the function does is poll the SQS queue. If there's new activity, the function retrieves the information and processes it, which could also include writing another a message back on the queue. If there are no new messages, the function sleeps until the next scheduled time, which can be as frequent as every five minutes.

If Lambda functions don't need to fire on the sequential order of events, a message queue probably isn't necessary. Lambda functions can execute in parallel, meaning you could use an SNS topic as the event hub and subscribe Lambda functions to the appropriate topic.

AWS Lambda isn't suited for every situation, but it is ideal for event-driven services that are relatively compact and complete in a relatively short amount of time. Lambda functions can be written in Node.js (JavaScript), Java or Python, must use no more than 512 MB of scratch file space, carry payloads of 6 MB or less and complete in less than 300 seconds.

Next Steps

Scalable apps benefit from Amazon SQS

Monitor SQS using CloudWatch

Take the Amazon SQS quiz

Dig Deeper on AWS Lambda