The Horus chipset takes a different approach. Instead of creating one giant communications ring structure on which all of the processors and their cache memories are linked to each other, Newisys has decided to adopt a four-way cell board architecture based on standard Opteron chips and then use the Horus chipset as an intermediary. Each cell board uses HyperTransport to cluster four Opterons and keep their caches coherent and also uses the Horus chipset to keep track of the state of caches on remote cell boards in the systems.
Conceptually, this is very similar to the architectures IBM is using in its Summit xSeries and various Power machines, and bears some resemblance to the means Unisys uses in its ES7000s and indeed in most modern Unix architectures. Horus is a ring that can support up to 32 sockets, which means up to eight four-socket cell boards. Today, Advanced Micro Devices only sells single-core Opterons, but when AMD jumps to dual-core Opterons in 2005, the Horus servers will scale to 64 cores. This will be as big of a box as any other server vendor can put on the market.
The trick to any NUMA-like architecture, says Oehler, is keeping the latency between the cache memory on the cell boards down. To keep from having to rewrite an operating system and its applications, Oehler says a server design has to have a 3:1 ratio or less between the time it takes for a CPU to reach into the cache of a cell board on the other side of the server compared to the time it takes for that CPU to reach into the local cache memory on its own cell board.
Oehler won't say how low the Horus designs will go, but he has said that Newisys has added a 64MB L3 cache to the Opteron architecture - Opterons include a main memory controller as well as L1 and L2 caches on chip - that it uses as a remote data cache to keep track of what CPU is using what cache lines. The ring of Horus ASICs are basically reading this cache very quickly and allowing the call boards to work through it to reach the cache in adjacent cell boards.
The Horus ASIC also has a remote directory, which keeps track of what cache lines are being accessed and controlled by cells outside of a given cell board. He also adds that AMD helped Newisys minimize the cache coherency traffic, which further reduced latencies. Since the Horus chipset creates another cache hierarchy above that built into the Opteron chips, Newisys will be calling it the Extended Scale Architecture when it becomes a product next year.
Oehler says that the Horus chipset has taped out, and that Newisys expects to get ASICs back from the foundry in early 2005 and into systems for OEM customers to examine by the middle of 2005. If all goes well, server makers could OEM the product and have it for sale by the end of next year. "My first job is to get a server built," says Oehler.
"It will be straightforward for us to get to eight, 12, and even 16 processors, but getting to 32 processors will be more of a challenge because of software. The hardware always leads the software in the right direction," he adds, and he knows a thing or two
