TechMan: Supercomputers in the cloud

Sunday, 08 January 2012 22:25

Written by Ced Kurtz

You've probably heard a lot of talk about the cloud -- software and services that live on computers somewhere else but can be used by you over the Internet.

Well there are also supercomputers in the cloud, brought to you by that retail giant Amazon. In 2006 Amazon debuted its Elastic Compute Cloud (EC2).

What does a company that sells books and CDs and electronics have to do with supercomputing?

Well, besides having warehouses full of goods, Amazon has buildings full of computer servers. And some of these machines are not occupied full time taking your orders. So Amazon has started to sell supercomputing time, and at strikingly reasonable prices.

These days a supercomputer is often made up of thousands of off-the-shelf computers working together to solve complex problems.

Implementing the software that creates these supercomputers in the cloud from machines all over the world is where Jason Stowe comes in. Mr. Stowe is a jovial and enthusiastic man who attended Carnegie Mellon University at age 16 and Cornell, who has worked for his father's software company and for Microsoft, and who has helped to make the Disney computer-animated movie "The Wild."

In 2005, he began Cycle Computing, which helps its clients assemble and use cloud-based supercomputers.

It is a concept known as utility supercomputing, and Mr. Stowe describes it as "the ability to create supercomputers in the cloud that people can use when they need them and turn off when they're done."

So you can rent supercomputing time by the hour, sort of like you can rent a room at a hot sheet joint.

But, to be serious, the possibilities are impressive.

Mr. Stowe's company got some publicity recently when it assembled a 30,000-core (a core is a computer processor, and Mr. Stowe said graphics processor chips also can be employed) supercomputer using Amazon servers for a top-five pharmaceutical company. The power of the virtual machine ranked it 42nd among the world's most powerful supercomputers.

Total cost including Amazon and Cycle fees -- $1,279 an hour.

That's a bargain.

"That puts it within the reach of the individual researcher who couldn't spend the $10 million to $15 million" to acquire a 30,000-core cluster or "wait six to 12 months" for their jobs to be scheduled and run on an existing supercomputer.

Once Cycle has set up a virtual computer, it automates the process so that it is available again on short notice. Mr. Stowe said it is possible to have a job running within 15 minutes. "You click Start and start it," he said.

There are several ways to get data to the cluster. Terabytes can be shipped on a hard disk drive or two. "Never underestimate the bandwidth of the FedEx truck," Mr. Stowe said.

So what do Cycle's customers use the supercomputer time for? Gene sequencing, insurance and finance tasks such as calculating annuity risks, fluid dynamics such as simulating the flow of liquids around propeller blades as well as basic research.

"What's really important is that we can totally change the way researchers do science. Now people can ask the crazy big questions that they never would have considered because they wouldn't have run in a reasonable time on the 256 cores they have at their small department," Mr. Stowe said.

Mr. Stowe said there have been examples of "grid environments," where a university, for example, might try to yoke together multiple computers on its network to build a powerful computer.

The problem with these, he said, is that the component computers have different software and libraries loaded and are often being used for other work.

In the cloud-based system, he said, there are processors "whose sole purpose is to do the computation you want to do."

Obviously Mr. Stowe has high hopes for utility supercomputing.

To encourage researchers to use utility supercomputing, Cycle has instituted the Big Science Challenge, in which it gives eight hours of 30,000-core cluster time free to a researcher with a project "that might benefit humanity."

"What's really awesome," Mr. Stowe said, "is that someone is going to take one of those clusters, and they're going to cure cancer or create a new material that will produce solar power much more efficiently.

"Steve Jobs once said the computer is the bicycle for the mind," he said. "This is a Ferrari."