ra2 studio - Fotolia

Which EC2 instance types are better for regression tests?

When performing regression tests in the cloud, should we choose to run tests on one AWS EC2 instance versus another?

Regression testing is the practice of running tests on existing software after a patch, update or change. Regression...

tests are designed to detect if and how any new additions to software affect the program. A common regression testing method is to run the same tests used on previous software versions to see if results differ in any way.

A variety of regression testing software is available from commercial vendors and open source projects. This software allows developers to automate testing procedures. Combining this software with the cloud's on-demand computing and storage capabilities offers the components of a low-cost, high-availability test platform.

Amazon Web Services (AWS) offers three different EC2 instance types -- On-Demand, Spot Instances and Reserved Instances. However, not all EC2 instance types are suitable for such tests. Spot Instances are the best fit for regression tests because they provide a relatively inexpensive, stable environment. Spot instances are extra EC2 instance types that are priced according to the current supply and demand of EC2 instances. When companies purchase Spot Instances, they can request Availability Zone, number of instances needed and the max price they intend to pay.

AWS offers the last 90 days of Spot Instance pricing. Use the ec2-describe-spot-price-history command to automatically collect details about price history and to calculate an appropriate bid (e.g. average price in past 24 hours * 1.5).

One potential problem with using AWS Spot Instances for regression tests is that you will lose your instance if the bid price drops below the current price. In automation scripts, test for the existence of results files or other verification that the regression test has completed. If results do not exist, you should re-run the test. If you find that you repeatedly lose Spot Instances because of low bids, consider increasing your bid. You will only be charged the current Spot Instance price -- even if your bid is higher. Therefore, there is no risk in spending more than the current spot price.

Next Steps

Knowing when to use AWS Spot Instances

How to implement regression testing

Which EC2 instance type offers the highest storage capacity?

Dig Deeper on AWS instances strategy and setup