├── .gitignore ├── README.md ├── bootstrap ├── bootstrap.sh └── files │ ├── etc │ ├── environment │ ├── grub.d │ │ └── 50_blacklist_nouveau.conf │ ├── ld.so.conf.d │ │ └── cuda.conf │ └── modprobe.d │ │ └── blacklist-cuda.conf │ └── home │ └── ubuntu │ ├── .emacs.d │ └── init.el │ ├── .screenrc │ ├── .theanorc │ ├── check_theano.py │ └── check_theano.sh ├── build.yaml ├── fabfile.py └── gpu.yaml /.gitignore: -------------------------------------------------------------------------------- 1 | *~ 2 | *.pyc 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Simple AWS GPU setup 2 | ==================== 3 | 4 | 1. Create your AWS account 5 | -------------------------- 6 | 7 | If you haven't got an account with Amazon AWS yet, you'll need to create one. Go to http://aws.amazon.com/, click Sign Up and follow the instructions. You'll probably have to receive a phone call from them to confirm your identity. 8 | 9 | [Create a new EC2 key pair](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#KeyPairs:) and place your downloaded `.pem` file in your `~/.ssh` directory. 10 | 11 | Go to [Security Credentials](https://console.aws.amazon.com/iam/home?#security_credential), click *Access Keys*, and *Create New Access Key*. Copy the secret access key and put it in a temporary file somewhere. You can only see it when you create it, but you can create as many as you want. 12 | 13 | Define the following environment variables with the information you obtained in steps 2 and 3 (put these lines in `~/.bash_profile` on OSX or `~/.bashrc` on Linux): 14 | 15 | export AWS_KEYPAIR_NAME=... # The name of the key pair you created in step 2 16 | export AWS_SSH_KEY_FILENAME=... # The path where you put your .pem file in step 2 (e.g. ~/.ssh/foo.pem) 17 | export AWS_ACCESS_KEY_ID=... # Found under Security Credentials -> Access Keys 18 | export AWS_SECRET_ACCESS_KEY=... # The secret key you copied in step 3 19 | 20 | Open a new terminal window or reload `.bash_profile`/`.bashrc` 21 | 22 | 2. Install software 23 | ------------------- 24 | 25 | [Install pip](http://pip.readthedocs.org/en/latest/installing.html) (a Python package manager) if you don't have it installed already. 26 | 27 | Then (on the command line), install [headintheclouds](http://headintheclouds.readthedocs.org): 28 | 29 | pip install headintheclouds 30 | 31 | Download this repository: 32 | 33 | git clone https://github.com/andreasjansson/simple-aws-gpu-setup.git 34 | 35 | Cd into the repo root: 36 | 37 | cd simple-aws-gpu-setup 38 | 39 | If everything is installed properly, and all your environment variables from step 4 are correct, you should be able to type `fab pricing` and get a table of EC2 spot instance and hourly costs. 40 | 41 | 3. Create your cheap GPU spot instance 42 | -------------------------------------- 43 | 44 | On your terminal, type 45 | 46 | fab ensemble.up:gpu 47 | 48 | You should get a prompt like this: 49 | 50 | Calculating changes... 51 | The following servers will be created: 52 | gpu 53 | Do you wish to continue? [Y/n] 54 | 55 | Answer `Y` 56 | 57 | Now a spot instance will be created, based on the configuration in `gpu.yaml`. This will take around five minutes. If for some reason the current spot price is above the bid in `gpu.yaml`, edit that file such that `bid` is higher than the current asking price (or wait for a few hours for the asking price to go down again). 58 | 59 | The image defined in `gpu.yaml` (ami-1ca6f874) is based on a stock Ubuntu 14.04 image ([ami-8827efe0](http://thecloudmarket.com/image/ami-8827efe0--ubuntu-images-hvm-ssd-ubuntu-trusty-14-04-amd64-server-20140724)) and is bootstrapped with Theano, NumPy, SciPy, Emacs, and some other goodies. The actual bootstrap script is included in the `bootstrap` directory. 60 | 61 | Once that's done you should see your server running if you type 62 | 63 | fab nodes 64 | 65 | 4. Write some Theano code 66 | ------------------------- 67 | 68 | Log in to the new server 69 | 70 | fab ssh 71 | 72 | On the remote machine, start an IPython notebook on port 80 73 | 74 | sudo ipython notebook --ip=0.0.0.0 --port=80 --no-browser 75 | 76 | You can access the notebook in a web browser using the `public_ip` address from the `fab nodes` command. 77 | 78 | Since this notebook is world readable, anyone can access your machine as root. That's a terrible idea. So shut down the notebook by hitting `ctrl-c`. 79 | 80 | On your local machine, figure out your public IP address (either just google "what is my ip" or `curl ifconfig.me`). Replace `80: "*"` in the firewall config of `gpu.yaml` with `80: YOUR_PUBLIC_IP`. For the firewall to be applied, once again run 81 | 82 | fab ensemble.up:gpu 83 | 84 | Now you can safely start the IPython notebook again. 85 | 86 | I added a `check_theano` notebook to the image. Open the IPython notebook web interface in a browser again and click the `check_theano` notebook. Once inside it, click the first cell and hit Run (or `ctrl-return`). It should say "Used the gpu". 87 | 88 | **Don't forget to back up your work regularly to S3 or some other persistent storage, since spot instances can be terminated at any point in time** 89 | 90 | When you're done with your instance you can terminate it by typing 91 | 92 | fab -R gpu terminate 93 | 94 | 95 | ----------- 96 | 97 | 98 | Building the image from scratch 99 | ------------------------------- 100 | 101 | The scripts and files I used to build the image are in the `bootstrap` directory. To build with headintheclouds: 102 | 103 | fab ensemble.up:build 104 | fab -R build bootstrap 105 | 106 | Or you can just copy the files over from `bootstrap/files` and run the `bootstrap/bootstrap.sh` script yourself. 107 | -------------------------------------------------------------------------------- /bootstrap/bootstrap.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -eux 4 | 5 | sudo aptitude update 6 | 7 | sudo DEBIAN_FRONTEND=noninteractive aptitude install -y -o Dpkg::Options::="--force-confnew" -o Dpkg::Options::="--force-confold" \ 8 | build-essential \ 9 | openssh-server \ 10 | emacs24-nox \ 11 | wget \ 12 | libglu-dev \ 13 | linux-headers-`uname -r` \ 14 | linux-image-`uname -r` \ 15 | linux-image-extra-`uname -r` \ 16 | python-pip \ 17 | python-dev \ 18 | gfortran \ 19 | python-matplotlib \ 20 | libblas-dev \ 21 | liblapack-dev \ 22 | uuid-dev \ 23 | libzmq-dev \ 24 | python-nose \ 25 | git \ 26 | 27 | export CUDA_BUILD=/mnt/cuda 28 | export CUDA_HOME=/opt/cuda 29 | sudo mkdir -p $CUDA_BUILD 30 | sudo chmod 777 $CUDA_BUILD 31 | pushd $CUDA_BUILD 32 | sudo wget http://developer.download.nvidia.com/compute/cuda/6_0/rel/installers/cuda_6.0.37_linux_64.run 33 | sudo chmod +x cuda_6.0.37_linux_64.run 34 | sudo ./cuda_6.0.37_linux_64.run \ 35 | -silent \ 36 | -driver \ 37 | -toolkit \ 38 | -toolkitpath=$CUDA_HOME/toolkit \ 39 | -samples \ 40 | -samplespath=$CUDA_HOME/samples \ 41 | -verbose \ 42 | -override \ 43 | -kernel-source-path=/usr/src/linux-headers-`uname -r` 44 | popd 45 | 46 | sudo update-initramfs -u 47 | 48 | sudo ldconfig 49 | 50 | sudo pip install \ 51 | pylint==0.28.0 \ 52 | ipython[all]==2.4.0 \ 53 | theano 54 | -------------------------------------------------------------------------------- /bootstrap/files/etc/environment: -------------------------------------------------------------------------------- 1 | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/cuda/toolkit/bin 2 | C_INCLUDE_PATH=/opt/cuda/include 3 | CPLUS_INCLUDE_PATH=/opt/cuda/include 4 | -------------------------------------------------------------------------------- /bootstrap/files/etc/grub.d/50_blacklist_nouveau.conf: -------------------------------------------------------------------------------- 1 | rdblacklist=nouveau -------------------------------------------------------------------------------- /bootstrap/files/etc/ld.so.conf.d/cuda.conf: -------------------------------------------------------------------------------- 1 | /opt/cuda/toolkit/lib 2 | -------------------------------------------------------------------------------- /bootstrap/files/etc/modprobe.d/blacklist-cuda.conf: -------------------------------------------------------------------------------- 1 | blacklist amd76x_edac 2 | blacklist vga16fb 3 | blacklist nouveau 4 | blacklist lbm-nouveau 5 | blacklist rivafb 6 | blacklist nvidiafb 7 | blacklist rivatv 8 | -------------------------------------------------------------------------------- /bootstrap/files/home/ubuntu/.emacs.d/init.el: -------------------------------------------------------------------------------- 1 | (blink-cursor-mode 0) 2 | (setq backup-inhibited t) 3 | (setq auto-save-default nil) 4 | (recentf-mode t) 5 | (setq disabled-command-function nil) 6 | (setq-default indent-tabs-mode nil) 7 | (add-to-list 'load-path "~/.emacs.d") 8 | (define-coding-system-alias 'UTF-8 'utf-8) 9 | (global-set-key [(control x) (k)] 'kill-this-buffer) 10 | 11 | (require 'package) 12 | (setq package-archives '(("ELPA" . "http://tromey.com/elpa/") 13 | ("gnu" . "http://elpa.gnu.org/packages/") 14 | ("marmalade" . "http://marmalade-repo.org/packages/") 15 | ("melpa" . "http://melpa.milkbox.net/packages/"))) 16 | (setq package-list ' 17 | ( 18 | ace-jump-mode 19 | color-theme 20 | browse-kill-ring 21 | )) 22 | (package-initialize) 23 | 24 | ; fetch the list of packages available 25 | (unless package-archive-contents 26 | (package-refresh-contents)) 27 | 28 | ; install the missing packages 29 | (dolist (package package-list) 30 | (unless (package-installed-p package) 31 | (package-install package))) 32 | 33 | ;; colours 34 | (require 'color-theme) 35 | (eval-after-load "color-theme" 36 | '(progn 37 | (color-theme-initialize) 38 | (color-theme-hober))) 39 | 40 | ;; ido 41 | ;(ido-mode) 42 | (iswitchb-mode t) 43 | 44 | ;; ace-jump 45 | (define-key global-map (kbd "C-c SPC") 'ace-jump-mode) 46 | 47 | ;; python 48 | (defun insert-ipdb-set-trace () 49 | "Insert 'import ipdb; ipdb.set_trace()' here." 50 | (interactive) 51 | (insert "import ipdb; ipdb.set_trace()")) 52 | 53 | (add-hook 'python-mode-hook 54 | '(lambda () 55 | (subword-mode t) 56 | (flymake-mode t) 57 | (local-set-key (kbd "C-c p") 'insert-ipdb-set-trace) 58 | (local-set-key (kbd "C-c f n") 'flymake-goto-next-error) 59 | (local-set-key (kbd "C-c f p") 'flymake-goto-prev-error) 60 | (local-set-key (kbd "C-c f SPC") 'flymake-display-err-menu-for-current-line) 61 | )) 62 | (setq 63 | python-shell-interpreter "ipython" 64 | python-shell-interpreter-args "--pylab" 65 | python-shell-prompt-regexp "In \\[[0-9]+\\]: " 66 | python-shell-prompt-output-regexp "Out\\[[0-9]+\\]: " 67 | python-shell-completion-setup-code 68 | "from IPython.core.completerlib import module_completion" 69 | python-shell-completion-module-string-code 70 | "';'.join(module_completion('''%s'''))\n" 71 | python-shell-completion-string-code 72 | "';'.join(get_ipython().Completer.all_completions('''%s'''))\n") 73 | 74 | ;; Configure flymake for Python 75 | (when (load "flymake" t) 76 | (defun flymake-pylint-init () 77 | (let* ((temp-file (flymake-init-create-temp-buffer-copy 78 | 'flymake-create-temp-inplace)) 79 | (local-file (file-relative-name 80 | temp-file 81 | (file-name-directory buffer-file-name)))) 82 | (list "epylint" (list local-file)))) 83 | (add-to-list 'flymake-allowed-file-name-masks 84 | '("\\.py\\'" flymake-pylint-init))) 85 | 86 | ;; server 87 | (server-mode t) 88 | 89 | (require 'uniquify) 90 | 91 | (custom-set-variables 92 | ;; custom-set-variables was added by Custom. 93 | ;; If you edit it by hand, you could mess it up, so be careful. 94 | ;; Your init file should contain only one such instance. 95 | ;; If there is more than one, they won't work right. 96 | '(c-basic-offset 4) 97 | '(create-lockfiles nil)) 98 | (custom-set-faces 99 | ;; custom-set-faces was added by Custom. 100 | ;; If you edit it by hand, you could mess it up, so be careful. 101 | ;; Your init file should contain only one such instance. 102 | ;; If there is more than one, they won't work right. 103 | ) 104 | -------------------------------------------------------------------------------- /bootstrap/files/home/ubuntu/.screenrc: -------------------------------------------------------------------------------- 1 | escape ^~~ 2 | startup_message off 3 | vbell off 4 | -------------------------------------------------------------------------------- /bootstrap/files/home/ubuntu/.theanorc: -------------------------------------------------------------------------------- 1 | [global] 2 | device = gpu 3 | floatX = float32 4 | warn_float_64 = warn 5 | cast_policy = numpy+floatX 6 | 7 | [cuda] 8 | root = /opt/cuda/toolkit 9 | -------------------------------------------------------------------------------- /bootstrap/files/home/ubuntu/check_theano.py: -------------------------------------------------------------------------------- 1 | from theano import function, config, shared, sandbox 2 | import theano.tensor as T 3 | import numpy 4 | import time 5 | 6 | vlen = 10 * 30 * 768 # 10 x #cores x # threads per core 7 | iters = 1000 8 | 9 | rng = numpy.random.RandomState(22) 10 | x = shared(numpy.asarray(rng.rand(vlen), config.floatX)) 11 | f = function([], T.exp(x)) 12 | print f.maker.fgraph.toposort() 13 | t0 = time.time() 14 | for i in xrange(iters): 15 | r = f() 16 | t1 = time.time() 17 | print 'Looping %d times took' % iters, t1 - t0, 'seconds' 18 | print 'Result is', r 19 | if numpy.any([isinstance(x.op, T.Elemwise) for x in f.maker.fgraph.toposort()]): 20 | print 'Used the cpu' 21 | else: 22 | print 'Used the gpu' 23 | -------------------------------------------------------------------------------- /bootstrap/files/home/ubuntu/check_theano.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | echo 4 | echo '------------------' 5 | echo 'TESTING THEANO CPU' 6 | echo '------------------' 7 | THEANO_FLAGS=mode=FAST_RUN,device=cpu,floatX=float32 python check_theano.py 8 | 9 | echo 10 | echo '------------------' 11 | echo 'TESTING THEANO GPU' 12 | echo '------------------' 13 | THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python check_theano.py 14 | 15 | echo 16 | -------------------------------------------------------------------------------- /build.yaml: -------------------------------------------------------------------------------- 1 | build: 2 | provider: ec2 3 | image: ami-8827efe0 4 | size: g2.2xlarge 5 | -------------------------------------------------------------------------------- /fabfile.py: -------------------------------------------------------------------------------- 1 | from headintheclouds.tasks import * 2 | from headintheclouds import ec2 3 | from headintheclouds import ensemble 4 | from headintheclouds import docker 5 | -------------------------------------------------------------------------------- /gpu.yaml: -------------------------------------------------------------------------------- 1 | gpu: 2 | provider: ec2 3 | image: ami-1ca6f874 4 | size: g2.2xlarge 5 | bid: 0.1 6 | firewall: 7 | 80: "*" 8 | --------------------------------------------------------------------------------