A high-end quad-GPU workstation

This machine is very similar to a workstation we used to perform a complete reprocessing of the 12TB EMPIAR 10061 beta-galactosidase dataset in just 115h on a single workstation (all the way from micrographs to final 2.2Å structure).

  • You need a case that can fit quad GPUs. For our machines we used the same case as NVIDIA used for their “DIGITS DevBox”, Corsair Carbide AIR 540 ($129), but you might also be lucky with Fractal Design XL R2.case
  • To push enough air through this case, get a couple of extra 140mm case fans. A silent fan might be $13. Note that it is important to mount them so you push-pull air from the front, over the GPUs, to the back. To get even better airflow, cover the holes on the top of the case.

extra_fans

  • A motherboard with support for four double-wide PCIe x16 GPU slots. We originally got an ASUS X99-E WS board (now $520), but today we would likely try a cheaper Gigabyte X99P-SLI for $249.
    Update: We have had one colleague who tried the Gigabyte card and there might be some problems with quad GPUs. Until we know for sure, you might not want to risk this.
  • An 8-core CPU. We used Core i7 5960X (3GHz, now $1000). Today Intel has released the next-generation CPU, which ~10% faster, but also more expensive. A new 8-core Core i7-6900K will be $1200. Don’t bother splashing out for the super-expensive 10-core version; it’s not worth it.
  • 64 GB DDR4 memory. Use 4 sticks of 16GB each, so you have the possibility of upgrading to 128 GB with another four later. Any brand will work, right now it’s $229 at Newegg.
  • The power supply (PSU) is probably the most complex item for quad-GPU systems. Right now the fastest consumer cards draw between 150W (GeForce 1070) and 180W (GeForce 1080), but the fastest cards they release (like 980Ti) will draw up to 250W each.
    • If you want to be able to upgrade to 250W cards later, get a 1500W PSU. We got a Corsair AX1500i ($399 now), but there are also cheaper options that will probably be fine (EnerMax EMR1500EWT, $250). Just make sure that it has plenty of PCI power connectors for the GPUs.
    • For the GeForce 1080 cards, you can likely get by with a 1000W PSU since we never run the cards and GPU at 100%, but to have a bit of margin for disks and later improvements we would recommend a 1200W PSU, e.g. Corsair HX1200i for $240 at Newegg.
    • If you will only use the 150W GeForce 1070 cards, a 1000W PSU is definitely enough. Corsair RM1000X is $159.
  • At the higest end, some NVIDIA GPUs can draw up to 250W per card (e.g. the 980Ti). For that you need a 1500W power supply. However, right now the fastest card you can get is the GeForce GTX 1080 (180W), and the best value for money is probably the GTX 1070 (150W). GPUs evolve fast, so I suspect you will want to update. Based on history, I would expect NVIDIA to release an even faster 250W GeForce GTX 1080Ti in early 2017. If you want to upgrade to that, get the larger PSU already now.
  • A fast SSD to cache data, say Samsung 850 EVO 512GB for $219. 
    UPDATE 2016-09-13: We’ve noticed that the SSD cache despite its speed amazingly enough can become the bottleneck in this machine. However, this is easy to work around. Instead of a single large SSD, get two smaller ones (e.g. Samsung 850 EVO 256GB) and configure them as RAID0 to double the effective bandwidth.
    You might also want to consider the slightly more expensive longer-life professional drives (e.g. Samsung 850 Pro) if you use the machine a lot and put valuable data like your home directory on the SSD.
    All SSDs will eventually wear out, and the consumer ones wear out faster.
    Some modern motherboards have support for an M.2 slot, but this will use 2-4 PCIe lanes and reduce bandwidth to one of the GPUs – so for quad-GPU machines you want to stay away from it.
  • A larger mechanical hard drive. It’s worth getting a 7200rpm drive for better speed. Newegg has a Toshiba X300 for $140, but any brand will do. If you have lots of data you can add multiple 6TB drives as well as multiple SSDs in these cases.
  • A quiet CPU fan. Not absolutely necessary, but it will make your office a nicer place at very low cost. The Cooler Master Hyper 212 EVO ($29) is what we usually get, simply because we know it fits in our cases.
  • Four GPUs. We initially had GeForce GTX 980Ti cards in this machine, so with a 1500W power supply it can definitely handle four cards each drawing 250W. Today we would get GeForce GTX 1080 cards ($699) to optimize for performance, or GeForce GTX 1070 cards ($449) to optimize for value.
    Update 2016-10-21: The “Founder’s edition” models are still fine, but by now they have mostly been replaced with vendor models, and can be hard to get. We’ve been quite happy with the even-cheaper ASUS TURBO-GTX1080 ($650) or ASUS TURBO-GTX1070 ($395). However, here we would recommend to be a bit careful! A lot of vendors (including ASUS – avoid their “dual” models) offer overclocked cards that provide even better performance. That can work great when you only put a single card in the machine, but they achieve improved cooling by using bulkier cooling solutions and by blowing the air transversally instead of longitudinally over the card. This can be really bad with a quad-GPU config, since you then just rotate air inside the machine of blowing it through the machine.  You can identify these cards with name suffixes such as “OC”, that they have several fans, or by the direction of the fins on the coolers. Do yourself a favor and stay away from all such overclocked cards.

quad2

IMPORTANT: For a quad-GPU machine it is critical that you go with reference cards (“Founder’s Edition”). We are well aware that you have just seen the lower-priced “gamer’s edition” that scores higher on all the review sites, and that it is said to have better cooling. However, these cards achieve this better cooling by having the air flow the short way across the card instead. This is great when you only have a single card in your case, but in a quad-GPU configuration that would now blow out all the hot air inside the case, and the cards will throttle when they get too hot.
Update 2016-09-13: We have been happy with the few ASUS Turbo GTX 1080 ($650) and 1070 ($400) cards we’ve gotten, although we haven’t tested dozens of them. These are the black models with longitudinal cooling. Stay away from an model that says “OC” (overclock) or the “twin” models with two fans, but vertical cooling.

Don’t be stupid and cause yourself a bunch of trouble by trying to save the last $100 – get either the reference model cards, or at least cards that are not overclocked and have the air flowing in the long direction of the card. If the card has some fancy/special cooling solution you might not even be able to mount four of them next to each other.

Mounting all this stuff in the case is straightforward if you have ever built any PC, but there are a couple of tricks to use that will improve your cooling – which will cover in the next post.

back

 

This will get you an insanely fast personal RELION-2 workstation – think of having the equivalent of half-a-rack of cluster nodes under your desk. With GeForce GTX 1080 cards it will be $5000. Since RELION-2 still does not max out performance of the GPUs (i.e., use them to 100%), the performance difference if you go with 1070 cards instead will only be roughly 10% – and then the cost will be just under $4000 per machine.

However, there is one caveat: Since RELION can now be more than an order-of-magnitude faster, it is also extremely data-hungry. You will need an SSD just to cache particles, and even when you have data connected on a fast external drive that will be a bottleneck. Both the low-end and high-end workstations described here have the problem that they only come with 1Gbit network interfaces, and for the high-end workstation there is not even an option of adding a 10Gbit network card – since we just filled up all slots with GPUs.

There are a couple of ways around this:

  1. Some vendors sell workstations that have room both four four GPUs and some extra cards, for instance Dell T630 or Supermicro SYS-7048GR-TR. They are typically equipped with dual sockets, which doesn’t hurt, but it will cost you a bit extra.
  2. There might be some single-socket quad-GPU motherboards on the market that have built-in 10Gbit ethernet.
  3. Go with rackmount nodes that have built-in 10Gbit ethernet or infiniband. This will be covered in the next post.

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *