Earlier this week, NVidia and NextIO announced that NextIO is taking over the Tesla S-series product. If you’re not familiar with the S-Series Tesla, it’s the rack-mount product that carries 4 Tesla cards in a 1U chassis, used by several GPU-computing clusters. It uses a small half-height PCIe Expansion card to connect to the external box, and allows you to put GPU compute capability in the popular CPU-dense cluster designs from the major manufacturers. NextIO will be taking over the product and they’re using Nvidia’s existing supply chain and their own, and they’ve rebranded it the “vCORE Express”. They’re here at GTC talking about the product, their plans, and showing off some of their other options.
NextIO is known in the industry for their impressive PCI expansion and virtualization technology. You can read my SC09 report, but to sum it up, their appliances enable:
- Connecting multiple PCIe devices up to multiple servers,
- Assigning any combination of these devices to any combination of the servers.
- Device virtualization for SR-IOV and MR-IOV devices (mainly Network interfaces)
- Hot-swap capability for any PCIe Device
- Full GUI & commandline control of everything.
It’s a great device for large datacenters that can solve many of the power, heating, and reliability issues of things like GPUs and Network interfaces by putting them all in a separate unit, making it easier to maintain and reducing downtime. Imagine if the next time you lost a network interface or fried a GPU, you didn’t have to reboot the node to replace it?
So here at GTC, they’ve announced the vCORE Express product. Currently, it’s an exact copy of the existing NVidia Tesla-S series product, offering 4 Tesla cards in a 1U Chassis connected to 2 servers. Currently, it offers none of the “secret sauce” that makes NextIO attractive, but that’s a temporary situation. NextIO is already looking to add more to the device, but what’s even better is the upcoming “vCORE Extreme” product.
The vCORE Extreme will ramp up the power with 16 GPUs connected to 8 systems (via PCIe Gen2), but sacrifices 3U for the priviledge. Still no IOV or HotPlug support, but a nice boost in horsepower for the “very” demanding among us. Conveniently, it’s expected to be out in only a few months, but if you really must have the “gold standard” of configurability and flexibility, you’ll want to hold out for the “vCORE Flex”, coming in 2011.
The vCORE Flex will come with all of the features you’ve come to know and love in the vNET product line. 16 GPUs connected to 8 servers (via PCIe Gen3) in a 4U rack, but that extra 1U gives you all of the hotplug and IOV support you could want. In the end, NextIO will be the provider for a whole suite of products that offer everything from the “quick and easy” vCORE Express to the die-hard Master Suite VCore Flex.
One powerful demo that NextIO loves to show is their dynamic reallocation of GPU resources. In the demo, they have 3 or 4 nodes connected up to the vCORE Flex system, each one using a GPU. Most of the nodes are, obviously, idle. While running a Monte Carlo simulation on one node, they get tired of waiting for it to complete. They connect to the control panel, remove a GPU from a node that isn’t currently using it, and attach it to the working node. Restart the Monte Carlo sim, and voila it’s running on 2 GPU’s now. Nothing had to reboot, not even a cable moved, nothing more than a drag and drop on the web-based control panel and it’s done.
Of course, NextIO offers more than just a GPU Chassis. Their other products, the vNET line and the vSTOR line, offer similar capabilities with other devices. The vSTOR is optimized for containing storage devices like the FusionIO cards, while the vNET is the generic “do anything” box that you can fill to your own specs. In fact, if your pockets are deep enough, you can load up the vNET or vSTOR appliance with 14 Terabytes of SSD storage (with plenty of FusionIO) cards, and then divvy it up amongst your rack.
In fact, one particularly interesting use case comes up, and NextIO tells me they actually have a few major studios doing this now. Imagine you work in a big visual effects studio. The camera department pulls the footage off equipment and throws it up on the big server. The “big server” has a large drive available which is actually a giant SSD array inside a NextIO box. Then, instead of copying all the data over to the Color Grading department so they can work on it, someone simply reconfigures the NextIO box to unmount it from the Camera Department server, and remount it on the Color Grading server. Same thing happens when it moves to the VFX department, then to the Compositing, and then to the final Editing department. No more data duplication, no more sneakernet, no more lengthy network file copies. Just a simple drag and drop in a Web interface, and viola it appears on the new system with all the files intact.
That is just one of the beautiful and amazing things you can do with the NextIO system. That’s just SSD’s, but imagine having the same flexibility with GPUS (Dynamically allocating GPU resources within a cluster via your Queueing system), network interfaces (putting ALL of your network interfaces in 1 box and sharing them amongst the cluster), or any other PCIe device.
NextIO is also very active in the PCI-SIG, and was instrumental in the creation of the SR-IOV and MR-IOV standards. While currently only network interfaces really support the standards, they expect that GPUs and PCI-SSD’s will be compliant in the upcoming year. If so, then that means you’ll be able to share GPU’s across multiple nodes. While this may not be a terribly desirable thing to do in most cases (who wants to divvy up a computational resource anyway?), it’ll be great for people who have GPU’s that are only useful in a tiny portion of their computation, making utilization “bursty”.
Probably most important of all, this isn’t just smoke and mirrors. It’s actually used in production on large systems. In fact, if you hit the NextIO website you can read a case study from the Rocky Mountain Supercomputing Center (View Online via GDocs) that is using NextIO vCORE systems with GPUs allocated dynamically via the MOAB scheduler in an IBM BladeCenter system.
It’s real, and it works.
