When Google and Amazon Web Services launched new solid-state drive-backed instances nearly simultaneously this...
summer, it was unclear which offered the better price-performance deal.
But independent storage performance benchmark tests, performed in September and November by benchmarking firm CloudHarmony, Inc., based in Laguna Beach, Calif., raise questions about both vendors' advertised performance specs.
Analysts aren't surprised the benchmark results differ from what's advertised.
"You have to take any performance claim with a grain of salt," said Greg Schulz, founder and analyst with the Server and StorageIO Group, based in Stillwater, Minn. "You have to apply the proper metric and have the proper context."
The same goes for the benchmark tests themselves.
"You have to make sure you're comparing apples to apples, and then make sure, are we talking about Granny Smiths or Macintoshes?" Schulz said. "Welcome to the world of benchmark testing and specmanship."
Google and Amazon SSD performance results
Google Compute Engine (GCE)'s Persistent Disk SSD performance is bound to volume size, according to Jason Read, founder of CloudHarmony. Thus testing was done using multiple volume size permutations.
At small volume sizes, GCE's SSD IOPS are consistent with what's advertised. SSD persistent disks of 10 gigabytes (GB) perform at approximately 300 IOPS for both reads and writes; 50 GB volumes can achieve approximately 1,500 IOPS for both reads and writes and 100 GB volumes can achieve approximately 3,000 IOPS for both reads and writes, according to CloudHarmony's test results.
For 200 GB volumes and larger, however, CloudHarmony's IOPS results were lower than what's advertised on the Google product page. In addition, 200 GB volumes are advertised as 6,000 IOPS, but benchmark testing results showed them at 3,800 to 4,500 IOPS, except for on the largest possible instance, the n1-standard-16, where it achieved just over 6,000 IOPS.
The 333 GB volumes are advertised as 10,000 IOPS, while the actual performance was between 3,000 and 4,500 IOPS in the benchmark tests, again with the exception of the n1-standard-16 instance, where 333 GB volumes came close to the advertised numbers at approximately 9,000 IOPS. Similarly, 500 GB volumes are advertised as 10,000 read and 15,000 write IOPS, but the actual performance was approximately 9,000 IOPS for both reads and writes in the benchmark tests.
Finally, in previous benchmark tests performed in September, 1 TB volumes (1,000 GB) are advertised as 10,000 read and 15,000 write IOPS, but the actual performance was 8,500 read and 7,400 write IOPS.
The discrepancies are "due to restrictive thread counts and queue depth parameters wherein advertised rates can be achieved," Read said. In other words, Google's performance claims can be achieved, but only under a strict set of circumstances*, which include large volume sizes on high-CPU instances. Whether an I/O or queue-based scheduler is used also can impact the results, Read said.
Amazon's Elastic Block Store (EBS) is more lenient in this regard, so users will be able to achieve advertised IOPS under more broad workload I/O characteristics, Read said.
AWS SSDs offer wide variety of performance options
AWS performance results were also not consistent with advertised rates -- they were actually, in some cases, higher, depending on volume and instance size.
On general purpose EBS volumes, smaller instances and volume sizes performed at a base rate slower than larger instances and volume sizes. For example, the t2.medium instance with a 256 GB volume yielded 790.6 read IOPS and 768.4 write IOPS, whereas the m3.2xlarge instance with eight CPUs attached to 16 one TB volumes performed at 26,913 IOPS for reads and 30,347 IOPS for writes.
AWS' EBS provisioned IOPS also corresponded to instance size during the November tests, with an advertised cap of 4,000 read/write IOPS for each volume. Users can also increase total IOPS capacity on a single virtual machine by attaching up to 16 such volumes to larger Elastic Compute Cloud instances, for an advertised maximum of up to 48,000 IOPS read/write.
The CloudHarmony benchmark tests revealed provisioned IOPS volumes can actually perform at higher IOPS rates than the 48,000 IOPS advertised on Amazon's website; on large instances where up to 16 volumes can be striped together, such as the c3.8xlarge, the mean provisioned IOPS performance in the benchmark tests was as high as 66,000 IOPS for reads and 56,000 IOPS for writes.
While Amazon has a wider range of options for high-performance workloads, Google's approach of expanding single volumes to scale up IOPS is a simpler approach, Read said. The CloudHarmony tests also didn't take bursting into account, which allows even higher performance on general purpose volumes for up to 30 minutes as needed.
Latency is another important measure of storage performance. In the November tests, AWS instances achieved lower latencies, but also showed wider variability in latencies between instances than Google. Google's lowest latency was 0.67 milliseconds on the n1-standard-1 attached to a 100 GB volume, and its highest was 3.31 milliseconds on a g1-small attached to a 10 GB volume. Amazon, meanwhile, ranged from 0.32 milliseconds latency on the m3.large attached to a 512 GB volume to 5.13 seconds on the same instance type -- the m3.large -- attached to a 64 GB volume.
Another difference between Google and Amazon's SSD offerings is price; GCE's SSD capacity costs more at $0.17 per GB compared to Amazon's $0.10 per GB for general purpose volumes. Provisioned IOPS cost slightly more than EBS general purpose SSDs at $.125 per GB per month plus $.065 per provisioned IOPS.
"It's roughly 75% more expensive to go with Google's SSDs vs. EBS," Read said. Provisioned IOPS are more of a premium service, he said.
AWS EBS also offers bursting and burst credits to further soup up performance. "Because of the duration and intensity of our block storage tests, most of EBS bursting characteristics are not reflected in the data," Read said. "Users with less intensive or more bursty IO workloads will likely obtain higher IOPS on EBS." Google and Amazon declined to provide comment for this article.
Full storage performance results will be posted at https://cloudharmony.com/benchmarks later this month.
*Further testing in January according to new best practices documentation from Google achieved advertised IOPS for all but the 10,000 GB (10 TB) volume size when one CPU core was allocated per 2500 IOPS, and the I/O scheduler was changed to noop, a type of Linux scheduler with low overhead.
*information added after initial publication