Ethernet's Gig In Supercomputers
By JEFF CARUSO
When is a network of workstations not a network? When it's a supercomputer.
Advances in LAN switching speeds are giving rise to a new kind of computing platform: relatively cheap super-computers composed of microprocessor-based workstations linked by Gigabit Ethernet or Fast Ethernet switches.
Though these so-called Beowulf-class machines have been in research facilities since 1994, higher LAN speeds are making them more versatile and prevalent.
"The higher bandwidth and lower latency are enabling a broader class of application" that they can handle, said Thomas Sterling, principal scientist at the California Institute of Technology and creator of the first Beowulf.
With more efficient communications among components, these supercomputers now can be used to simulate fusion reactors, or forecast Earth's weather by taking into account the oceans and the atmosphere together. Caltech's Beowulf is studying the formation of galaxies.
But more down-to-earth applications suitable for commercial IT shops--such as Web servers, search engines or transaction processors--could easily be implemented, Sterling said.
Traditional supercomputers have been out of reach for the vast majority of businesses and educational institutions. But putting together a Beowulf-class computer is less expensive than buying a traditional vector supercomputer. The performance of a Beowulf system is estimated at 10 to 20 times better than a comparably priced supercomputer such as an IBM SP2, according to the users interviewed for this story.
Los Alamos National Laboratory recently created the 315th-fastest computer in the world, called Avalon, for about $150,000. The supercomputer is rated at nearly 20 billion floating-point operations per second, or 20 gigaflops.
The National Institutes of Health (NIH) built a supercomputer last year for about $180,000 out of 64 dual-processor Pentium Pro machines. The computers are arranged in a ring, each with three Fast Ethernet cards. One goes directly to the next computer in the ring, another is linked to the previous computer, and the third goes to a Foundry Networks Fast Ethernet switch.
"The communications are what slow you down," said Eric Billings, staff scientist at NIH. He is starting to evaluate technologies for the next version of LoBoS (Lots of Boxes on Shelves).
The limiting factor of the current LoBoS is each PC's 32-bit PCI bus, which can push data through only at about 200 Mbps, Billings said. The next LoBoS will use dual-processor, 400-MHz Pentium II machines with 64-bit PCI.
The new bus will be better able to take advantage of Gigabit Ethernet. The NIH also is investigating an alternate technology being developed specifically for this kind of application, Billings said. The NIH hopes to keep the price tag under $4,500 per node.
The Caltech Beowulf uses two Fast Ethernet switches from Lucent Technologies Inc., with multiple Gigabit Ethernet connections between the Fast Ethernet switches. Caltech needed a nonblocking switch to connect the 155 Pentium Pro nodes, and the 46-Gbps backplane of the Lucent P550 Cajun switch provided the necessary headroom. Caltech believes it can scale Beowulf up to 250 nodes.
The Beowulf computers typically cut some costs by using Linux, a free Unix operating system. NIH developed software for dividing up calculations among the various processors and will make that software available to whomever wants it, Billings said.
The hardest part of coordinating the processors' activities is the initialization of processing, said Caltech's Sterling. But once the program is started, one processor spawns work on several other processors, and those in turn pass jobs to other processors, and so on down the line. There is no one central processor that could be a bottleneck, Sterling said.
Part of the strength of Beowulf-class computers is that they can rapidly take advantage of processing advances. As faster microprocessors hit the shelves, users can add them to the supercomputer quickly and immediately increase its overall performance.
But monitoring so many machines can be somewhat of a hassle.
In the NIH system, each processor checks in with the master nodes once a minute, passing information about whether they and their communications links are active. These master nodes also keep track of the job queue.
If one node fails in a Beowulf system, the job is flushed and the other nodes working on that segment of the problem immediately become available for other jobs. Previous attempts at supercomputing through clusters required that all of the tightly coupled nodes be reinitialized, Billings said.
