Recently, I found an analysis paper about cloud server performance conducted by an independent cloud performance metrics company, Cloud Spectator.

This post is to summarize the paper and I definitely encourage you to read the full report over here: http://www.iqcloud.net/wp-content/uploads/2013/07/Cloud-Computing-Performance-A-Comparative-Analysis-of-5-Large-Cloud-IaaS….pdf

Objective of analysis study

The objective of the paper is determine the price-performance value of the cloud providers. Providing some valuable insight for customer when selecting their prefer cloud vendor.

Figure 1 – Principle of value proposition [figure from the paper]

Who are being compared

The study (done in June 2013) compared five large IaaS providers in the industry:

• Amazon AWS EC2
• Rackspace OpenStack Cloud
• HP Cloud
• SoftLayer CloudLayer Compute
• Windows Azure

Methodology

Timeframe

The tests were run for 3 times in 5 consecutive days: May 25, 2013 – May 29, 2013.

VM Size

The most common size for cloud server, Medium Size (or equivalent / similar setup) was chosen from the 5 cloud vendors:

Figure 2 – Medium VM Spec [figure from the paper]

Benchmark

The tests used Unixbench 5.1.3 as benchmarking the performance of Linux OPS running on virtualized infrastructure, producing rating out of 10 stars. Details of Unixbench can be found here: https://code.google.com/p/byte-unixbench/

Information

Two important pieces of info are collected:

• Performance: how well the provider scores on Unixbench, and how consistent the scores are.
• Price-Performance: after performance scores are established, we factor in cost to understand how much performance a user can expect on return for every amount of money spent, i.e., the value.

The Results

Performance Only

The performance result shows that Windows Azure provides the best performance and notably 3 times higher than AWS EC2 on average!

Figure 3 – Performance Only Result [figure from the paper]

Figure 4 – Average Unixbench Score, derived from Figure 3 [figure from the paper]

Price-Performance=Value

Retail hourly price of the cloud providers are captured (pay-as-you-go) basis on date of experiment.

Figure 5 – Pay-per-hour price [figure from the paper]

By taking each score and dividing by the price, we can get a relative price-to-performance score for each provider. And here are the score (The higher the score, the better):

Figure 6 – Price-Performance Result [figure from the paper]

CloudSpecs Score

CloudSpecs score is a further normalized value from Figure 6, taking the highest value to 100. And here’re the scores:

With the cloudspecs score, the ratio of each of the providers are formed as following

Conclusion

While acknowledging that Unixbench is just one test, customers may always consider other factors when selecting their cloud vendor.

To conclude, Amazon EC2 and Windows Azure offers the lowest price at $0.12 per hour. However, Windows Azure performs much better than EC2 in this experiment (approximately 3 times). The experiment also shows that Rackspace scores worst in term of price-performance.