4 |
5 |
2 |
3 |
4 |
5 |
10 | Instructions to build your own research GPU box for deep learning AND get it set up with tensorflow, pytorch and CUDA drivers. 11 |
12 | 13 | 14 | 15 | 16 | #### After following these instructions you'll have: 17 | 1. A sick GPU box with multuple GPUs set up. 18 | 2. A functional OS ready for deep learning. 19 | 20 | **Note: I've built 4 of these already and they are amazing. Total cost is around $6,000.** 21 | 22 | ## Parts 23 | 24 | | Part | Brand / Link | Purpose | Quantity | Optional? | 25 | |---|---|---|---|---| 26 | | SSD | [Samsung 850 evo (250 GB)](https://www.amazon.com/gp/product/B00OAJ412U/ref=oh_aui_detailpage_o08_s01?ie=UTF8&psc=1) | OS + working dir | 1 | N| 27 | | SSD | [Samsung 850 evo (1 TB)](https://www.amazon.com/gp/product/B00OBRFFAS/ref=oh_aui_detailpage_o08_s01?ie=UTF8&th=1) | fast data loading. Keep datasets not in use on the SSHD | 1+ | N | 28 | | SSHD | [Seagate FireCuda (2TB)](https://www.amazon.com/gp/product/B01IEKG2HM/ref=oh_aui_detailpage_o05_s00?ie=UTF8&psc=1) | data storage, long term model storage | 1+ | Y | 29 | | Processor | [Intel i7 (6+ cores)](https://www.amazon.com/gp/product/B01FJLAITC/ref=oh_aui_detailpage_o08_s00?ie=UTF8&psc=1) | At least 1 core per GPU | 1 | N | 30 | | Power source | [Corsair 1500 W](https://www.amazon.com/gp/product/B00MFJ4OBA/ref=oh_aui_detailpage_o07_s03?ie=UTF8&psc=1) | Enough for 4 gpus | 1 | N | 31 | | Motherboard | [Asus X99-E WS](https://www.amazon.com/gp/product/B00XUDLXJG/ref=oh_aui_detailpage_o07_s00?ie=UTF8&psc=1) | Big enough for 4 GPUs | 1 | N | 32 | | RAM | [Crucial (16GB+ each)](https://www.amazon.com/gp/product/B019FRBCQE/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1) | You need at least as much RAM as GPU RAM | 4 | N | 33 | | Tower | [Phanteks Enthoo Pro](https://www.amazon.com/dp/B00K6S1B3Q/ref=dp_prsubs_2) | Big enough for 4 GPUs | 1 | N | 34 | | GPUs | [NVIDIA 1080Ti](https://www.amazon.com/gp/product/B06Y13N2B6/ref=oh_aui_detailpage_o01_s00?ie=UTF8&psc=1) | Go with 12 GB RAM. Best price/teraflops+RAM out there. (Optional jet fans vs open fan. I use both) | 4 | N | 35 | | CPU Water Cooler | [Corsair CW-9060027-WW Hydro Series H115i](https://www.amazon.com/gp/product/B019955RNQ/ref=oh_aui_detailpage_o07_s02?ie=UTF8&psc=1) | Try to keep the overall temp in the box down or GPUs will throttle. (Optional but HIGHLY recommended) | 1 | Y | 36 | | Small Fans | [Corsair ML120 Pro LED](https://www.amazon.com/gp/product/B01G5I6MUW/ref=oh_aui_detailpage_o06_s00?ie=UTF8&psc=1) | Replace front and bottom pannel fans. These are more quiet than the stock fans | 3 | Y | 37 | | Large Fans | [Corsair ML140 Pro LED](https://www.amazon.com/gp/product/B01G5I6Q94/ref=oh_aui_detailpage_o01_s00?ie=UTF8&psc=1) | Assuming you get rid of the tower stock fans. (Highly recommended) | 4 | Y | 38 | | FAN connectors | [4Pin PWM to 3Pin](https://www.amazon.com/gp/product/B01H0OZC9W/ref=oh_aui_detailpage_o01_s00?ie=UTF8&psc=1) | You'll run out of fan connectors with new fans | 1 | Y | 39 | 40 | 41 | ### Considerations 42 | 1. Temperature 43 | - If your machine gets too hot, the GPUs will auto-throttle down. 44 | 45 | 2. Bottlenecks 46 | - With deep learnning, the biggest bottleneck is not the GPU but the DATA TRANSFER to the GPUs. 47 | - This is why the Motherboard needs to be fast enough and should have at least 40 PCI lanes. 48 | - The drives need to be really fast. 49 | - Use the SSD to feed data directly to model. Use SSHD for long term storage that won't go into the model directly. 50 | 51 | 3. RAM 52 | - Have at least as much RAM as you have GPU RAM. 53 | 54 | 4. CPU 55 | - Have at least 1 core per GPU. 56 | - Water cooling can help keep the overall temperature low. 57 | 58 | ### Assembly / Install instructions 59 | 60 | I'll add more details later, but in order you should: 61 | 1. Install fans. 62 | 2. install motherboard. 63 | 3. Install power source (but don't screw in yet). 64 | 4. Connect all the fans to motherboard. 65 | 5. Install drives. 66 | 6. Connect drives to motherboard. 67 | 7. Connect motherboard to power supply. 68 | 8. Connect power button, usb, etc... to motherboard. 69 | 9. Install GPUs. 70 | 10. Connect GPUs to power supply. 71 | 11. Install RAM. 72 | 12. Screw in powersource. 73 | 13. Fix all the cables neatly. 74 | 14. Insert an ubuntu live USB. 75 | 15. Turn on. 76 | 16. Pray. 77 | 16. Boot into BIOS and set the USB as the priority drive. 78 | 17. Install Ubuntu (or your OS). 79 | 18. Follow [these instructions](https://github.com/williamFalcon/tensorflow-gpu-install-ubuntu-16.04) to set up your system with tensorflow, pytorch and cuda drivers. 80 | 19. Learn deeply. 81 | 82 | 83 | ### Some deep learning tips 84 | 1. Log your experiments and parallelize hyperparameter search using [the python library test tube](https://github.com/williamFalcon/test-tube). 85 | 86 | -------------------------------------------------------------------------------- /gpu_box.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/williamFalcon/deep-learning-gpu-box-build-instructions/ff33cb9101d945bfcbfa3c7af9b29ddf08f305d5/gpu_box.jpeg --------------------------------------------------------------------------------