WavebreakmediaMicro - Fotolia


Using Amazon SQS for scalable applications

AWS SQS has a few advantages over other message queuing systems. The service supports decoupled environments and eliminates queuing management tasks.

Amazon Simple Queue Service is a scalable, managed, on-demand message queuing service that allows data to communicate...

between machines and be accessed from any computer. Messages are buffered until another machine is ready to process them. But that's not its biggest advantage.

Amazon Simple Queue Service (SQS) can support decoupled environments. It allows developers to focus on application development, rather than creating their own sophisticated message-based applications. It also eliminates queuing management tasks, such as storage. SQS also works with AWS resources, so you can use it to make reliable and scalable applications on top of an AWS infrastructure.

Suppose an application receives millions of requests per day. That application is required to follow one of the following steps to process data, as shown in Figure 1:

  • If an input request is in an audio format, create an .mp3 file.
  • If an input request is in an image format, create a .bmp file.
  • If an input request is in a text format, create an .html file.
  • If an input request is in a video format, create an .mp4 file.
  • Encrypt data.
  • Store data.
Traditional message queuing
Figure 1. An application that receives millions of requests per day must follow one of these paths to process data.

But there are a few problems with this:

  • It's not scalable.
  • What happens if one of the applications (video, HTML, etc.) is down?
  • Bottlenecks may occur -- if a video process takes too long and data is not sent back, SQS will keep waiting for it and will not process new files. This can create a longer queue.

Because SQS works as a buffer between the components that produce, process and save data, it resolves issues that arise if work is being produced faster than it can be processed. The same is true if the producer or consumer is only intermittently connected to the network.

This is beneficial because a distributed architecture is better with SQS. In addition, if one of the processes takes longer to complete, the main application does not need to wait for it to finish, as shown in Figure 2. Scaling a single cluster (video/HTML/image) is easy because all clusters fetch data from SQS.

Amazon SQS message queuing
Figure 2. The Amazon SQS queue resolves issues that crop up if work is being produced faster than it can be processed.

SQS removes the dependency associated with message queuing, so applications do not need to wait for one another, and it provides reliable message delivery.

Amazon SQS vs. other message queuing systems

So, what does AWS SQS have that other queuing systems, such as Apache ActiveMQ, Rabbitmq or Redis, don't have?

SQS provides a managed queue, so a developer doesn't need to set up or manage queuing. He simply creates a queue using the AWS console, API or SDK to begin consuming resources. It's scalable and can process millions of messages per day. AWS SQS' pay-as-you-go model is also cost-effective.

For security, SQS can be created securely with AWS Identity and Access Management (IAM) using cryptographic methods to authenticate a requestor’s identity upon access -- either through the use of your secret access key ID or through an X.509 certificate. Amazon SQS ensures delivery of each message at least once, and supports multiple reader and writer interactions within the same queue. In addition, it's highly reliable: All messages are stored on multiple data centers in the same region for high availability and fault tolerance.

About the author:
Ofir Nachmani is a business technology advisor, blogger and lecturer. Nachmani's extensive experience in the world of business technology has made his critically acclaimed blog, IamOnDemand.com, the go-to guide for modern technology startups and developers in the world of cloud computing. Today he advises organizations, leading them through new IT market modifications, while building and executing a modern go-to-market strategy.

Dig Deeper on AWS streaming and file transfer services