Combat a common error with AWS EC2 instances

RequestLimitExceeded is a common EC2 error that occurs when you make too many calls to AWS instances. But how many are too many?

Amazon Web Services lets enterprises believe they have their own private resources, but there are times when the

shared nature of the cloud system comes back to bite unsuspecting enterprises. An error seen in Amazon Web Services EC2 instances -- RequestLimitExceeded -- is one of those times.

Most often, workloads run on a virtual machine oblivious to the fact the Amazon Web Services (AWS) Elastic Compute Cloud (EC2) instance is a subsystem of a hypervisor that's using resources from an underlying physical computer. To put it another way, there are probably dozens or hundreds of micro instances running on each actual piece of Amazon hardware.

If you operate on your own instance, you can't really affect other AWS EC2 instances -- until you start making requests, or calls, to the AWS EC2 application programming interface (API). Whenever you make an AWS call, you're placing a small demand on the system that all instances share. If you create an infinite loop that asks for the list of running instances, you could create a denial of service situation against other users trying to make AWS calls.

Therefore, if you make too many AWS calls, your call can fail with the RequestLimitExceeded error. However, AWS does not specify how many calls are "too many," likely because it's a complex, undisclosed algorithm that AWS plans to further evolve. But this means there's no way to predict when the error might happen.

Coding around AWS EC2 errors

There is a method you can use to avoid running into this error.

First, think about AWS as a constrained resource with an expensive round trip. Just as you don't read a file one byte at a time, don't interrogate AWS in small chunks.

If you want to learn something about each instance you're running, you could make a separate AWS call for each instance or you could use the flexibility of the API to make a single request for information about a list of AWS EC2 instances. The first approach is far more likely to cause issues.

A second option is to consider how often you need to refresh whatever AWS data you're requesting. Perhaps you're collecting instance data to make a manual scaling decision. Frequently refreshing data increases your accuracy -- until you get a request denied notice and have to exponentially back off. Strike a balance on how often you ask for data.

After doing this, however, ­­­­­all AWS calls need to defend against the request limit exception and you have to decide how to handle that. Some calls can simply fail and you can try again based on logic in the caller; other calls need to be retried locally until they succeed.

For that second case many administrators write code similar to this:

int backOffFactor = 0;
while(true) {
  try {
    amazonClient.someCall();
    break;
  } catch(AmazonServiceException e) {
    if(e.getErrorCode().equalsIgnoreCase("RequestLimitExceeded")) {
       quietSleepSeconds(++backOffFactor);
    }
  }

This code does a progressive back off by sleeping for longer periods between retries using a utility function all admins have written to sleep and catch/ignore an InterruptedException until the "too many" condition is gone. You can adjust how quickly the sleep time increases and you can create an upper limit to the sleep time.

This isn't the prettiest code and it could be handled as a lambda expression in those languages supporting closures, but it conveys the basic intent: Assume the possibility of failure and slow down until the system stops complaining.

About the author:
Brian Tarbox has been doing mission-critical programming since he created a timing-and-scoring program for the Head of the Connecticut Regatta back in 1981. Though primarily an Amazon Java programmer, Brian is a firm believer that engineers should be polylingual and use the best language for the problem. Brian holds patents in the fields of UX and VideoOnDemand with several more in process. His Log4JFugue open source project won the 2010 Duke's Choice award for the most innovative use of Java; he also won a JavaOne best speaker Rock Star award as well as a Most Innovative Use of Jira award from Atlassian in 2010.Brian has published several dozen technical papers and is a regular speaker at local Meetups.

This was first published in July 2014

Dig deeper on Amazon EC2 (Elastic Compute Cloud) management

Pro+

Features

Enjoy the benefits of Pro+ membership, learn more and join.

0 comments

Oldest 

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

SearchCloudApplications

SearchSOA

TheServerSide

SearchSoftwareQuality

SearchCloudComputing

Close