Containers and microservices are two of the hottest -- and potentially disruptive -- concepts in IT. Together they...
make it easier to piece together applications out of lean, feature-specific components that use fewer resources, are faster to develop and can be more densely packed on virtual infrastructure. But what if you could build an application service without the server?
That's where AWS Lambda comes into play. AWS Lambda is a service that runs code on-demand without the need to worry about instantiating, sizing, scaling and decommissioning server instances. Like all new AWS products, Lambda is still evolving, and while it is already quite versatile, it's missing one feature that could turn off developers using asynchronous message queues, like Amazon Simple Queue Service (SQS). Why not allow an SQS queue to trigger a microservice Lambda function instead of a full-blown application?
Lambda works by listening for triggers from other AWS products; it currently supports events from the following services:
- Amazon Simple Storage Service (S3): triggered from object creation or deletion in select buckets.
- DynamoDB: item updates in a NoSQL table.
- Kinesis: triggered when data is added to a particular stream.
- Simple Notification Service (SNS): launches Lambda functions from event-driven notifications to a particular topic.
- Simple Email Service (SES): invoke asynchronously to control mail flow.
- Cognito: triggered based on mobile back-end data sync events.
- CloudWatch: triggered based on events in real-time log feeds and state changes.
- CloudFormation (service templates): specifies a Lambda function as a custom resource to run any custom commands as part of stack creation.
- Scheduled events (powered by CloudWatch events): allows regular invocation of Lambda code much like a CRON job. Events run at a specific time or recur, use CRON syntax with five-minute granularity and can poll from various AWS data sources.
- Amazon API Gateway: used to define a custom REST API and endpoint that invokes a Lambda function.
- AWS IoT: configure rules to route sensor data to other services.
But Amazon SQS is conspicuously absent from this list, which is notable because SQS is designed to safely, asynchronously pass messages and move data between distributed applications. The omission was a source of much angst on AWS discussion forums last year, and while a company representative said the feature "is on our radar," Lambda integration with the SQS queue hasn't been implemented. But when it comes to software development, there is more than one way to skin a cat.
Sample Lambda-SQS use cases
Application middleware offers a typical scenario in which integrating Lambda and SQS would be handy. For example, a front-end client -- Web, mobile, smart device -- might create data that needs further processing before being passed along to a back-end database. Conversely, the client might send a request to pull data from the back end that needs preprocessing for display on the client. In either situation, serverless Lambda functions are ideal for the middleware tier.
If the event flow involves messages -- not data -- Lambda can be directly invoked by using SNS, SES or Cognito, for mobile clients, by sending events triggering Lambda functions. For example, these functions can write data to DynamoDB or an S3 bucket or they can retrieve data or images to the client for display. However, if the application event involves data and the architecture uses message queues, invoking a Lambda function requires a bit more work.
Polling the SQS queue with scheduled events
The goal of message queues is to decouple application functions from direct dependence and communication, making the entire end-to-end application flow more robust and reliable. One component can send data to the SQS queue, and others can watch the queue and act on the data independently of the sender.
Lambda can't be directly triggered by SQS activity, so the only option is to periodically poll the queue. Until recently, this required a hack -- using an app running on a separate EC2 instance to poll or watch SQS queues. Of course, this defeated the point of using Lambda in the first place, because the app instance could just as easily implement the required Lambda functionality. But AWS added scheduled events to allow regular invocation of Lambda functions.
Scheduled events are like Lambda CRON jobs; they are configured using standard CRON syntax. Any Lambda function that needs to trigger from a message queue can be set up as a recurring event; the first thing the function does is poll the SQS queue. If there's new activity, the function retrieves the information and processes it, which could also include writing another a message back on the queue. If there are no new messages, the function sleeps until the next scheduled time, which can be as frequent as every five minutes.
If Lambda functions don't need to fire on the sequential order of events, a message queue probably isn't necessary. Lambda functions can execute in parallel, meaning you could use an SNS topic as the event hub and subscribe Lambda functions to the appropriate topic.
Scalable apps benefit from Amazon SQS
Monitor SQS using CloudWatch
Take the Amazon SQS quiz