Pitting Cloud against Cloud

Tools that benchmark performance promise to reveal the strengths and weaknesses of competing cloud providers.

New software developed to measure the performance of different cloud computing platforms could make it easier for prospective users to figure out which of these increasingly popular services is right for them.

Right now, developers have little means of comparing cloud providers, which lease access to computing power based in vast and distant data centers. Until actually migrating their software to a cloud service, they can’t know exactly how fast that service will perform calculations, retrieve data, or respond to sudden spikes in demand. But Duke University computer scientist Xiaowei Yang and her colleague Ang Li are trying to make the cloud market more like the car market, where, as Yang says, “you can compare specifications like engine size or top speed.”

Working with Srikanth Kandula and Ming Zhang of Microsoft Research in Redmond, WA, Yang and Li have developed a suite of benchmarking tools that make it possible to compare the performance of different cloud platforms without moving applications between them. These tools use algorithms to measure the speed of computation, and shuttle data around to test the speed at which new copies of an application are created, the speed at which data can be stored and retrieved, the speed at which it can be shuttled between applications inside the same cloud, and the responsiveness of a cloud to network requests from distant places. The researchers used the software to test the services offered by six providers: Amazon, Microsoft, Google, GoGrid, RackSpace and CloudSites. Results of those tests were combined with the providers’ pricing models to allow for quick comparisons.

The results are among the first attempts to compare the performance of several clouds platforms, says Yang. “We found that it’s very hard to find a provider that is best in all metrics,” she says. “Some are twice as fast for just 10 percent extra cost, which is a very good deal, but at the same time their storage service is actually very slow and has a lot of latency variation.” Another provider showed good computation speeds but was less quick at spawning new instances of an application–something that might be necessary for a service that experiences peaks in demand, as a video site does when some of its content goes viral. “It seems like in today’s market it is hard to pick a provider that is good at everything,” says Yang.

Continue Reading at MIT