├── .gitignore ├── assets ├── cat.jpg ├── map.png ├── nn.jpg ├── wb.jpeg ├── bing.jpg ├── cveg.jpeg ├── cvplot.png ├── knn.jpeg ├── margin.jpg ├── nneg.jpeg ├── sine.png ├── svm1d.png ├── classify.png ├── cnn │ ├── cnn.jpeg │ ├── pool.jpeg │ ├── stride.jpeg │ ├── convnet.jpeg │ ├── depthcol.jpeg │ ├── maxpool.jpeg │ └── weights.jpeg ├── imagemap.jpg ├── key-pair.png ├── nn3 │ ├── opt1.gif │ ├── opt2.gif │ ├── loss.jpeg │ ├── values.jpeg │ ├── nesterov.jpeg │ ├── updates.jpeg │ ├── weights.jpeg │ ├── accuracies.jpeg │ ├── cnnweights.jpg │ ├── gridsearchbad.jpeg │ └── learningrates.jpeg ├── samenorm.png ├── stepsize.jpg ├── svm_all.jpg ├── svm_one.jpg ├── svmbowl.png ├── trainset.jpg ├── aws-signin.png ├── aws-signup.png ├── cat_tinted.jpg ├── challenges.jpeg ├── crossval.jpeg ├── dataflow.jpeg ├── nn1 │ ├── neuron.png │ ├── relu.jpeg │ ├── tanh.jpeg │ ├── alexplot.jpeg │ ├── sigmoid.jpeg │ ├── layer_sizes.jpeg │ ├── neural_net.jpeg │ ├── neural_net2.jpeg │ ├── neuron_model.jpeg │ └── reg_strengths.jpeg ├── pixelspace.jpeg ├── sine_cosine.png ├── templates.jpg ├── terminal-my.jpg ├── ami-selection.png ├── aws-homepage.png ├── classifydemo.jpeg ├── cnnvis │ ├── act1.jpeg │ ├── act2.jpeg │ ├── filt1.jpeg │ ├── filt2.jpeg │ ├── tsne.jpeg │ ├── occlude.jpeg │ └── pool5max.jpeg ├── ec2-dashboard.png ├── eg │ ├── spiral_net.png │ ├── spiral_raw.png │ └── spiral_linear.png ├── launch-screen.png ├── nn2 │ ├── dropout.jpeg │ ├── prepro1.jpeg │ ├── prepro2.jpeg │ └── cifar10pca.jpeg ├── svmvssoftmax.png ├── cloud-for-free.png ├── community-AMIs.png ├── instances-page.png ├── jupyter-screen.png ├── key-pair-create.png ├── launching-screen.png ├── terminal-shared.jpg ├── cat_tinted_imshow.png ├── cloud-external-ip.png ├── instance-selection.png ├── sadpuppy_nocredits.png ├── pixels_embed_cifar10.jpg ├── sine_cosine_subplot.png ├── terminal-coursework.jpg ├── terminal-development.jpg ├── cloud-dashboard-screen.png ├── cloud-instance-started.png ├── cloud-launching-screen.png ├── create-instance-screen.png ├── google-cloud-instance-gpus.png ├── google-cloud-quotas-form.png ├── google-cloud-quotas-screen.png ├── pixels_embed_cifar10_big.jpg ├── cloud-create-instance-screen.png ├── cloud-networking-external-ip.png ├── ipython-tutorial │ ├── notebook-1.png │ ├── notebook-2.png │ ├── notebook-3.png │ ├── file-browser.png │ ├── notebook-error.png │ └── save-notebook.png ├── cloud-instance-dashboard-screen.png ├── cloud-networking-firewall-rule.png ├── google-cloud-select-cloud-storage.png ├── google-cloud-select-custom-image.png ├── cloud-networking-external-ip-address.png ├── cloud-networking-external-ip-naming.png ├── google-cloud-create-image-screenshot.png ├── cloud-networking-firewall-rule-create.png └── conv-demo │ ├── utils.js │ └── index.html ├── create-instance-screen.png ├── Readme.md ├── _includes ├── header.html ├── head.html └── footer.html ├── _layouts ├── page.html ├── post.html └── default.html ├── overview.md ├── _config.yml ├── convnet-tips.md ├── LICENSE ├── ipython-tutorial.md ├── terminal-tutorial.md ├── assignments ├── 2015 │ ├── assignment2.md │ ├── assignment3.md │ └── assignment1.md ├── 2016 │ ├── assignment1.md │ ├── assignment2.md │ └── assignment3.md └── 2017 │ ├── assignment1.md │ ├── assignment2.md │ └── assignment3.md ├── aws-tutorial.md ├── transfer-learning.md ├── index.html ├── understanding-cnn.md ├── assignment2_google_cloud.md ├── css └── main.css ├── google_cloud_tutorial.md └── neural-networks-case-study.md /.gitignore: -------------------------------------------------------------------------------- 1 | _site 2 | .DS_Store 3 | *.swp 4 | -------------------------------------------------------------------------------- /assets/cat.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cat.jpg -------------------------------------------------------------------------------- /assets/map.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/map.png -------------------------------------------------------------------------------- /assets/nn.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn.jpg -------------------------------------------------------------------------------- /assets/wb.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/wb.jpeg -------------------------------------------------------------------------------- /assets/bing.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/bing.jpg -------------------------------------------------------------------------------- /assets/cveg.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cveg.jpeg -------------------------------------------------------------------------------- /assets/cvplot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cvplot.png -------------------------------------------------------------------------------- /assets/knn.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/knn.jpeg -------------------------------------------------------------------------------- /assets/margin.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/margin.jpg -------------------------------------------------------------------------------- /assets/nneg.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nneg.jpeg -------------------------------------------------------------------------------- /assets/sine.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/sine.png -------------------------------------------------------------------------------- /assets/svm1d.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/svm1d.png -------------------------------------------------------------------------------- /assets/classify.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/classify.png -------------------------------------------------------------------------------- /assets/cnn/cnn.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cnn/cnn.jpeg -------------------------------------------------------------------------------- /assets/imagemap.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/imagemap.jpg -------------------------------------------------------------------------------- /assets/key-pair.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/key-pair.png -------------------------------------------------------------------------------- /assets/nn3/opt1.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn3/opt1.gif -------------------------------------------------------------------------------- /assets/nn3/opt2.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn3/opt2.gif -------------------------------------------------------------------------------- /assets/samenorm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/samenorm.png -------------------------------------------------------------------------------- /assets/stepsize.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/stepsize.jpg -------------------------------------------------------------------------------- /assets/svm_all.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/svm_all.jpg -------------------------------------------------------------------------------- /assets/svm_one.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/svm_one.jpg -------------------------------------------------------------------------------- /assets/svmbowl.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/svmbowl.png -------------------------------------------------------------------------------- /assets/trainset.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/trainset.jpg -------------------------------------------------------------------------------- /assets/aws-signin.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/aws-signin.png -------------------------------------------------------------------------------- /assets/aws-signup.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/aws-signup.png -------------------------------------------------------------------------------- /assets/cat_tinted.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cat_tinted.jpg -------------------------------------------------------------------------------- /assets/challenges.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/challenges.jpeg -------------------------------------------------------------------------------- /assets/cnn/pool.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cnn/pool.jpeg -------------------------------------------------------------------------------- /assets/cnn/stride.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cnn/stride.jpeg -------------------------------------------------------------------------------- /assets/crossval.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/crossval.jpeg -------------------------------------------------------------------------------- /assets/dataflow.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/dataflow.jpeg -------------------------------------------------------------------------------- /assets/nn1/neuron.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn1/neuron.png -------------------------------------------------------------------------------- /assets/nn1/relu.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn1/relu.jpeg -------------------------------------------------------------------------------- /assets/nn1/tanh.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn1/tanh.jpeg -------------------------------------------------------------------------------- /assets/nn3/loss.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn3/loss.jpeg -------------------------------------------------------------------------------- /assets/nn3/values.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn3/values.jpeg -------------------------------------------------------------------------------- /assets/pixelspace.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/pixelspace.jpeg -------------------------------------------------------------------------------- /assets/sine_cosine.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/sine_cosine.png -------------------------------------------------------------------------------- /assets/templates.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/templates.jpg -------------------------------------------------------------------------------- /assets/terminal-my.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/terminal-my.jpg -------------------------------------------------------------------------------- /assets/ami-selection.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/ami-selection.png -------------------------------------------------------------------------------- /assets/aws-homepage.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/aws-homepage.png -------------------------------------------------------------------------------- /assets/classifydemo.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/classifydemo.jpeg -------------------------------------------------------------------------------- /assets/cnn/convnet.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cnn/convnet.jpeg -------------------------------------------------------------------------------- /assets/cnn/depthcol.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cnn/depthcol.jpeg -------------------------------------------------------------------------------- /assets/cnn/maxpool.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cnn/maxpool.jpeg -------------------------------------------------------------------------------- /assets/cnn/weights.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cnn/weights.jpeg -------------------------------------------------------------------------------- /assets/cnnvis/act1.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cnnvis/act1.jpeg -------------------------------------------------------------------------------- /assets/cnnvis/act2.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cnnvis/act2.jpeg -------------------------------------------------------------------------------- /assets/cnnvis/filt1.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cnnvis/filt1.jpeg -------------------------------------------------------------------------------- /assets/cnnvis/filt2.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cnnvis/filt2.jpeg -------------------------------------------------------------------------------- /assets/cnnvis/tsne.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cnnvis/tsne.jpeg -------------------------------------------------------------------------------- /assets/ec2-dashboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/ec2-dashboard.png -------------------------------------------------------------------------------- /assets/eg/spiral_net.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/eg/spiral_net.png -------------------------------------------------------------------------------- /assets/eg/spiral_raw.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/eg/spiral_raw.png -------------------------------------------------------------------------------- /assets/launch-screen.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/launch-screen.png -------------------------------------------------------------------------------- /assets/nn1/alexplot.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn1/alexplot.jpeg -------------------------------------------------------------------------------- /assets/nn1/sigmoid.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn1/sigmoid.jpeg -------------------------------------------------------------------------------- /assets/nn2/dropout.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn2/dropout.jpeg -------------------------------------------------------------------------------- /assets/nn2/prepro1.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn2/prepro1.jpeg -------------------------------------------------------------------------------- /assets/nn2/prepro2.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn2/prepro2.jpeg -------------------------------------------------------------------------------- /assets/nn3/nesterov.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn3/nesterov.jpeg -------------------------------------------------------------------------------- /assets/nn3/updates.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn3/updates.jpeg -------------------------------------------------------------------------------- /assets/nn3/weights.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn3/weights.jpeg -------------------------------------------------------------------------------- /assets/svmvssoftmax.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/svmvssoftmax.png -------------------------------------------------------------------------------- /assets/cloud-for-free.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cloud-for-free.png -------------------------------------------------------------------------------- /assets/cnnvis/occlude.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cnnvis/occlude.jpeg -------------------------------------------------------------------------------- /assets/cnnvis/pool5max.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cnnvis/pool5max.jpeg -------------------------------------------------------------------------------- /assets/community-AMIs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/community-AMIs.png -------------------------------------------------------------------------------- /assets/eg/spiral_linear.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/eg/spiral_linear.png -------------------------------------------------------------------------------- /assets/instances-page.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/instances-page.png -------------------------------------------------------------------------------- /assets/jupyter-screen.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/jupyter-screen.png -------------------------------------------------------------------------------- /assets/key-pair-create.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/key-pair-create.png -------------------------------------------------------------------------------- /assets/launching-screen.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/launching-screen.png -------------------------------------------------------------------------------- /assets/nn1/layer_sizes.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn1/layer_sizes.jpeg -------------------------------------------------------------------------------- /assets/nn1/neural_net.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn1/neural_net.jpeg -------------------------------------------------------------------------------- /assets/nn1/neural_net2.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn1/neural_net2.jpeg -------------------------------------------------------------------------------- /assets/nn2/cifar10pca.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn2/cifar10pca.jpeg -------------------------------------------------------------------------------- /assets/nn3/accuracies.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn3/accuracies.jpeg -------------------------------------------------------------------------------- /assets/nn3/cnnweights.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn3/cnnweights.jpg -------------------------------------------------------------------------------- /assets/terminal-shared.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/terminal-shared.jpg -------------------------------------------------------------------------------- /create-instance-screen.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/create-instance-screen.png -------------------------------------------------------------------------------- /assets/cat_tinted_imshow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cat_tinted_imshow.png -------------------------------------------------------------------------------- /assets/cloud-external-ip.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cloud-external-ip.png -------------------------------------------------------------------------------- /assets/instance-selection.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/instance-selection.png -------------------------------------------------------------------------------- /assets/nn1/neuron_model.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn1/neuron_model.jpeg -------------------------------------------------------------------------------- /assets/nn1/reg_strengths.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn1/reg_strengths.jpeg -------------------------------------------------------------------------------- /assets/nn3/gridsearchbad.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn3/gridsearchbad.jpeg -------------------------------------------------------------------------------- /assets/nn3/learningrates.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/nn3/learningrates.jpeg -------------------------------------------------------------------------------- /assets/sadpuppy_nocredits.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/sadpuppy_nocredits.png -------------------------------------------------------------------------------- /assets/pixels_embed_cifar10.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/pixels_embed_cifar10.jpg -------------------------------------------------------------------------------- /assets/sine_cosine_subplot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/sine_cosine_subplot.png -------------------------------------------------------------------------------- /assets/terminal-coursework.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/terminal-coursework.jpg -------------------------------------------------------------------------------- /assets/terminal-development.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/terminal-development.jpg -------------------------------------------------------------------------------- /assets/cloud-dashboard-screen.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cloud-dashboard-screen.png -------------------------------------------------------------------------------- /assets/cloud-instance-started.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cloud-instance-started.png -------------------------------------------------------------------------------- /assets/cloud-launching-screen.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cloud-launching-screen.png -------------------------------------------------------------------------------- /assets/create-instance-screen.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/create-instance-screen.png -------------------------------------------------------------------------------- /assets/google-cloud-instance-gpus.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/google-cloud-instance-gpus.png -------------------------------------------------------------------------------- /assets/google-cloud-quotas-form.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/google-cloud-quotas-form.png -------------------------------------------------------------------------------- /assets/google-cloud-quotas-screen.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/google-cloud-quotas-screen.png -------------------------------------------------------------------------------- /assets/pixels_embed_cifar10_big.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/pixels_embed_cifar10_big.jpg -------------------------------------------------------------------------------- /assets/cloud-create-instance-screen.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cloud-create-instance-screen.png -------------------------------------------------------------------------------- /assets/cloud-networking-external-ip.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cloud-networking-external-ip.png -------------------------------------------------------------------------------- /assets/ipython-tutorial/notebook-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/ipython-tutorial/notebook-1.png -------------------------------------------------------------------------------- /assets/ipython-tutorial/notebook-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/ipython-tutorial/notebook-2.png -------------------------------------------------------------------------------- /assets/ipython-tutorial/notebook-3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/ipython-tutorial/notebook-3.png -------------------------------------------------------------------------------- /assets/cloud-instance-dashboard-screen.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cloud-instance-dashboard-screen.png -------------------------------------------------------------------------------- /assets/cloud-networking-firewall-rule.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cloud-networking-firewall-rule.png -------------------------------------------------------------------------------- /assets/ipython-tutorial/file-browser.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/ipython-tutorial/file-browser.png -------------------------------------------------------------------------------- /assets/ipython-tutorial/notebook-error.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/ipython-tutorial/notebook-error.png -------------------------------------------------------------------------------- /assets/ipython-tutorial/save-notebook.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/ipython-tutorial/save-notebook.png -------------------------------------------------------------------------------- /assets/google-cloud-select-cloud-storage.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/google-cloud-select-cloud-storage.png -------------------------------------------------------------------------------- /assets/google-cloud-select-custom-image.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/google-cloud-select-custom-image.png -------------------------------------------------------------------------------- /assets/cloud-networking-external-ip-address.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cloud-networking-external-ip-address.png -------------------------------------------------------------------------------- /assets/cloud-networking-external-ip-naming.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cloud-networking-external-ip-naming.png -------------------------------------------------------------------------------- /assets/google-cloud-create-image-screenshot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/google-cloud-create-image-screenshot.png -------------------------------------------------------------------------------- /assets/cloud-networking-firewall-rule-create.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/yunzhu-li/cs231n.github.io/master/assets/cloud-networking-firewall-rule-create.png -------------------------------------------------------------------------------- /Readme.md: -------------------------------------------------------------------------------- 1 | 2 | Notes and assignments for Stanford CS class [CS231n: Convolutional Neural Networks for Visual Recognition](http://vision.stanford.edu/teaching/cs231n/) 3 | 4 | -------------------------------------------------------------------------------- /_includes/header.html: -------------------------------------------------------------------------------- 1 |
2 | 3 |
4 | {{ site.title }} 5 |
6 | 7 |
8 | -------------------------------------------------------------------------------- /_layouts/page.html: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | --- 4 |
5 | 6 |
7 |

{{ page.title }}

8 |
9 | 10 |
11 | {{ content }} 12 |
13 | 14 |
-------------------------------------------------------------------------------- /overview.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | title: Overview of Computer Vision and Visual Recognition 4 | permalink: /overview/ 5 | --- 6 | 7 | Our introductory lecture covered the history of Computer Vision and many of its current applications, to help you understand the context in which this class is offered. See the [slides](http://vision.stanford.edu/teaching/cs231n/slides/lecture1.pdf) for details. 8 | -------------------------------------------------------------------------------- /_layouts/post.html: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | --- 4 |
5 | 6 |
7 |

{{ page.title }}

8 |

{{ page.date | date: "%b %-d, %Y" }}{% if page.author %} • {{ page.author }}{% endif %}{% if page.meta %} • {{ page.meta }}{% endif %}

9 |
10 | 11 |
12 | {{ content }} 13 |
14 | 15 |
-------------------------------------------------------------------------------- /_layouts/default.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | {% include head.html %} 5 | 6 | 7 | 8 | {% include header.html %} 9 | 10 |
11 |
12 | {{ content }} 13 |
14 |
15 | 16 | {% include footer.html %} 17 | 18 | 19 | 20 | 21 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | # Site settings 2 | title: CS231n Convolutional Neural Networks for Visual Recognition 3 | email: karpathy@cs.stanford.edu 4 | description: "Course materials and notes for Stanford class CS231n: Convolutional Neural Networks for Visual Recognition." 5 | baseurl: "" 6 | url: "http://cs231n.github.io" 7 | twitter_username: cs231n 8 | github_username: cs231n 9 | 10 | # Build settings 11 | markdown: kramdown 12 | permalink: pretty 13 | 14 | highlighter: rouge 15 | kramdown: 16 | input: GFM 17 | auto_ids: true 18 | syntax_highlighter: rouge 19 | 20 | -------------------------------------------------------------------------------- /convnet-tips.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | permalink: /convnet-tips/ 4 | --- 5 | 6 | 7 | ### Addressing Overfitting 8 | 9 | #### Data Augmentation 10 | 11 | - Flip the training images over x-axis 12 | - Sample random crops / scales in the original image 13 | - Jitter the colors 14 | 15 | #### Dropout 16 | 17 | - Dropout is just as effective for Conv layers. Usually people apply less dropout right before early conv layers since there are not that many parameters there compared to later stages of the network (e.g. the fully connected layers). 18 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2015 Andrej Karpathy 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /_includes/head.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | {% if page.title %}{{ page.title }}{% else %}{{ site.title }}{% endif %} 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 26 | 27 | 28 | -------------------------------------------------------------------------------- /ipython-tutorial.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | title: IPython Tutorial 4 | permalink: /ipython-tutorial/ 5 | --- 6 | 7 | ***(Note: some of the screenshots here may be out-of-date. However, this should still prove 8 | useful as a quick intro, and for the general menu layout, etc.)*** 9 | 10 | In this class, we will use IPython notebooks (more recently known as 11 | [Jupyter notebooks](https://jupyter.org/)) for the programming assignments. 12 | An IPython notebook lets you write and execute Python code in your web browser. 13 | IPython notebooks make it very easy to tinker with code and execute it in bits 14 | and pieces; for this reason IPython notebooks are widely used in scientific 15 | computing. 16 | 17 | *(Note: if your virtual environment installed correctly (as per the assignment handouts), 18 | then you shouldn't have to install from the install instructions on the website. Just 19 | remember to run `source .env/bin/activate` in your assignment folder.)* 20 | 21 | 24 | 25 | Once you have it installed, start it with this command: 26 | 27 | ``` 28 | jupyter notebook 29 | ``` 30 | 31 | Once your notebook server is running, point your web browser at http://localhost:8888 to 32 | start using your notebooks. If everything worked correctly, you should 33 | see a screen like this, showing all available IPython notebooks in the current 34 | directory: 35 | 36 |
37 | 38 |
39 | 40 | If you click through to a notebook file, you will see a screen like this: 41 | 42 |
43 | 44 |
45 | 46 | An IPython notebook is made up of a number of **cells**. Each cell can contain 47 | Python code. You can execute a cell by clicking on it and pressing `Shift-Enter`. 48 | When you do so, the code in the cell will run, and the output of the cell 49 | will be displayed beneath the cell. For example, after running the first cell 50 | the notebook looks like this: 51 | 52 |
53 | 54 |
55 | 56 | Global variables are shared between cells. Executing the second cell thus gives 57 | the following result: 58 | 59 |
60 | 61 |
62 | 63 | By convention, IPython notebooks are expected to be run from top to bottom. 64 | Failing to execute some cells or executing cells out of order can result in 65 | errors: 66 | 67 |
68 | 69 |
70 | 71 | After you have modified an IPython notebook for one of the assignments by 72 | modifying or executing some of its cells, remember to **save your changes!** 73 | 74 |
75 | 76 |
77 | 78 | This has only been a brief introduction to IPython notebooks, but it should 79 | be enough to get you up and running on the assignments for this course. 80 | -------------------------------------------------------------------------------- /terminal-tutorial.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | title: Terminal.com Tutorial 4 | permalink: /terminal-tutorial/ 5 | --- 6 | For the assignments, we offer an option to use [Terminal](https://www.stanfordterminalcloud.com) for developing and testing your implementations. Notice that we're not using the main Terminal.com site but a subdomain which has been assigned specifically for this class. [Terminal](https://www.stanfordterminalcloud.com) is an online computing platform that allows us to access pre-configured command line environments. Note that, it's not required to use [Terminal](https://www.stanfordterminalcloud.com) for your assignments; however, it might make life easier with all the required dependencies and development toolkits configured for you. 7 | 8 | This tutorial lists the necessary steps of working on the assignments using Terminal. First of all, [sign up your own account](https://www.stanfordterminalcloud.com/signup). Log in [Terminal](https://www.stanfordterminalcloud.com) with the account that you have just created. 9 | 10 | For each assignment, we will provide you a link to a shared terminal snapshot. These snapshots are pre-configured command line environments with the starter code, where you can write your implementations and execute the code. 11 | 12 | Here's an example of what a snapshot page looked like for an assignment in 2015: 13 | 14 |
15 | 16 |
17 | 18 | Yours will look similar. Click the "Start" button on the lower right corner. This will clone the shared snapshot to your own account. Now you should be able to find the terminal under the [My Terminals](https://www.stanfordterminalcloud.com/terminals) tab. 19 | 20 |
21 | 22 |
23 | 24 | Yours will look similar. You are all set! To work on the assignments, click the link to your terminal (shown in the red box in the above image). This link will open up the user interface layer over an AWS machine. It will look something similar to this: 25 | 26 |
27 | 28 |
29 | 30 | We have set up the Jupyter Notebook and other dependencies in the terminal. Launch a new console window with the small + sign (if you don't already have one), navigate around and look for the assignment folder and code. Launch a Jupyer notebook and work on the assignment. If your're a student enrolled in the class you will submit your assignment through Coursework: 31 | 32 |
33 | 34 |
35 | 36 | For more information about [Terminal](https://www.stanfordterminalcloud.com), check out the [FAQ](https://www.stanfordterminalcloud.com/faq) page. 37 | 38 | **Important Note:** the usage of Terminal is charged on an hourly rate based on the instance type. A medium type instance costs $0.124 per hour. If you are enrolled in the class email Serena Yeung (syyeung@cs.stanford.edu) to request Terminal credits. We will send you $3 the first time around, and you can request more funds on a rolling basis when you run out. Please be responsible with the funds we allocate you. -------------------------------------------------------------------------------- /_includes/footer.html: -------------------------------------------------------------------------------- 1 | 61 | -------------------------------------------------------------------------------- /assignments/2015/assignment2.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | mathjax: true 4 | permalink: /assignments2015/assignment2/ 5 | --- 6 | 7 | In this assignment you will practice writing backpropagation code, and training Neural Networks and Convolutional Neural Networks. The goals of this assignment are as follows: 8 | 9 | - understand **Neural Networks** and how they are arranged in layered architectures 10 | - understand and be able to implement (vectorized) **backpropagation** 11 | - implement the core parameter update loop of **mini-batch gradient descent** 12 | - effectively **cross-validate** and find the best hyperparameters for Neural Network architecture 13 | - understand the architecture of **Convolutional Neural Networks** and train gain experience with training these models on data 14 | 15 | ## Setup 16 | You can work on the assignment in one of two ways: locally on your own machine, or on a virtual machine through [Terminal](https://www.terminal.com/). 17 | 18 | ### Working locally 19 | **Get the code:** 20 | [Download the starter code here](http://cs231n.stanford.edu/assignments/2015/assignment2.zip). 21 | 22 | **[Optional] virtual environment:** 23 | Once you have unzipped the starter code, you might want to create a 24 | [virtual environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/) 25 | for the project. If you choose not to use a virtual environment, it is up to you 26 | to make sure that all dependencies for the code are installed on your machine. 27 | To set up a virtual environment, run the following: 28 | 29 | ```bash 30 | cd assignment2 31 | sudo pip install virtualenv # This may already be installed 32 | virtualenv .env # Create a virtual environment 33 | source .env/bin/activate # Activate the virtual environment 34 | pip install -r requirements.txt # Install dependencies 35 | # Work on the assignment for a while ... 36 | deactivate # Exit the virtual environment 37 | ``` 38 | 39 | You can reuse the virtual environment that you created for the first assignment, 40 | but you will need to run `pip install -r requirements.txt` after activating it 41 | to install additional dependencies required by this assignment. 42 | 43 | **Download data:** 44 | Once you have the starter code, you will need to download the CIFAR-10 dataset. 45 | Run the following from the `assignment2` directory: 46 | 47 | ```bash 48 | cd cs231n/datasets 49 | ./get_datasets.sh 50 | ``` 51 | 52 | **Compile the Cython extension:** Convolutional Neural Networks require a very efficient implementation. We have implemented of the functionality using [Cython](http://cython.org/); you will need to compile the Cython extension before you can run the code. From the `cs231n` directory, run the following command: 53 | 54 | ```bash 55 | python setup.py build_ext --inplace 56 | ``` 57 | 58 | **Start IPython:** 59 | After you have the CIFAR-10 data, you should start the IPython notebook server from the 60 | `assignment2` directory. If you are unfamiliar with IPython, you should read our 61 | [IPython tutorial](/ipython-tutorial). 62 | 63 | 64 | ### Working on Terminal 65 | We have created a Terminal snapshot that is preconfigured for this assignment; 66 | you can [find it here](https://www.terminal.com/tiny/WcL5zSlT0e). Terminal allows you to work on the assignment from your browser. You can find a tutorial on how to use it [here](/terminal-tutorial). 67 | 68 | ### Submitting your work: 69 | Whether you work on the assignment locally or using Terminal, once you are done 70 | working run the `collectSubmission.sh` script; this will produce a file called 71 | `assignment2.zip`. Upload this file to your dropbox on 72 | [the coursework](https://coursework.stanford.edu/portal/site/W15-CS-231N-01/) 73 | page for the course. 74 | 75 | ### Q1: Two-layer Neural Network (30 points) 76 | 77 | The IPython Notebook `two_layer_net.ipynb` will walk you through implementing a two-layer neural network on CIFAR-10. You will write a hard-coded 2-layer Neural Network, implement its backprop pass, and tune its hyperparameters. 78 | 79 | ### Q2: Modular Neural Network (30 points) 80 | 81 | The IPython Notebook `layers.ipynb` will walk you through a modular Neural Network implementation. You will implement the forward and backward passes of many different layer types, including convolution and pooling layers. 82 | 83 | ### Q3: ConvNet on CIFAR-10 (40 points) 84 | 85 | The IPython Notebook `convnet.ipynb` will walk you through the process of training a (shallow) convolutional neural network on CIFAR-10. It wil then be up to you to train the best network that you can. 86 | 87 | ### Q4: Do something extra! (up to +20 points) 88 | 89 | In the process of training your network, you should feel free to implement anything that you want to get better performance. You can modify the solver, implement additional layers, use different types of regularization, use an ensemble of models, or anything else that comes to mind. If you implement these or other ideas not covered in the assignment then you will be awarded some bonus points. 90 | 91 | -------------------------------------------------------------------------------- /assignments/2015/assignment3.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | mathjax: true 4 | permalink: assignments2015/assignment3/ 5 | --- 6 | 7 | In the previous assignment, you implemented and trained your own ConvNets. 8 | In this assignment, we will explore many of the ideas we have discussed in 9 | lectures. Specifically, you will: 10 | 11 | - Reduce overfitting using **dropout** and **data augmentation** 12 | - Combine models into **ensembles** to boost performance 13 | - Use **transfer learning** to adapt a pretrained model to a new dataset 14 | - Use data gradients to visualize **saliency maps** and create **fooling images** 15 | 16 | ## Setup 17 | You can work on the assignment in one of two ways: locally on your own machine, or on a virtual machine through [Terminal](https://www.terminal.com/). 18 | 19 | ### Working locally 20 | **Get the code:** 21 | Download the starter code [here](http://cs231n.stanford.edu/assignments/2015/assignment3.zip). 22 | 23 | 24 | **[Optional] virtual environment:** 25 | Once you have unzipped the starter code, you might want to create a 26 | [virtual environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/) 27 | for the project. If you choose not to use a virtual environment, it is up to you 28 | to make sure that all dependencies for the code are installed on your machine. 29 | To set up a virtual environment, run the following: 30 | 31 | ```bash 32 | cd assignment3 33 | sudo pip install virtualenv # This may already be installed 34 | virtualenv .env # Create a virtual environment 35 | source .env/bin/activate # Activate the virtual environment 36 | pip install -r requirements.txt # Install dependencies 37 | # Work on the assignment for a while ... 38 | deactivate # Exit the virtual environment 39 | ``` 40 | 41 | You can reuse the virtual environment that you created for the first or second 42 | assignment, but you will need to run `pip install -r requirements.txt` after 43 | activating it to install additional dependencies required by this assignment. 44 | 45 | **Download data:** 46 | Once you have the starter code, you will need to download the CIFAR-10 dataset, 47 | the TinyImageNet-100-A and TinyImageNet-100-B datasets, and pretrained models 48 | for the TinyImageNet-100-A dataset. 49 | 50 | Run the following from the `assignment3` directory: 51 | 52 | NOTE: After downloading and unpacking, the data and pretrained models will 53 | take about 900MB of disk space. 54 | 55 | ```bash 56 | cd cs231n/datasets 57 | ./get_datasets.sh 58 | ./get_tiny_imagenet_splits.sh 59 | ./get_pretrained_models.sh 60 | ``` 61 | 62 | **Compile the Cython extension:** Convolutional Neural Networks require a very efficient implementation. We have implemented of the functionality using [Cython](http://cython.org/); you will need to compile the Cython extension before you can run the code. From the `cs231n` directory, run the following command: 63 | 64 | ```bash 65 | python setup.py build_ext --inplace 66 | ``` 67 | 68 | **Start IPython:** 69 | After you have downloaded the data and compiled the Cython extensions, 70 | you should start the IPython notebook server from the 71 | `assignmen3` directory. If you are unfamiliar with IPython, you should read our 72 | [IPython tutorial](/ipython-tutorial). 73 | 74 | ### Working on Terminal 75 | We have created a Terminal snapshot that is preconfigured for this assignment; 76 | you can [find it here](https://www.terminal.com/snapshot/f8e4a56cfde0d0baa519c501dfe24b3394a3124b8b84356d2cd033a229a59bff). Terminal allows you to work on the assignment from your browser. You can find a tutorial on how to use it [here](/terminal-tutorial). 77 | 78 | ### Submitting your work: 79 | Whether you work on the assignment locally or using Terminal, once you are done 80 | working run the `collectSubmission.sh` script; this will produce a file called 81 | `assignment3.zip`. Upload this file to your dropbox on 82 | [the coursework](https://coursework.stanford.edu/portal/site/W15-CS-231N-01/) 83 | page for the course. 84 | 85 | ### Q1: Dropout and Data Augmentation 86 | 87 | In the IPython notebook `q1.ipynb` you will implement dropout and several forms of data augmentation. This will allow you to reduce overfitting when training on a small subset of the CIFAR-10 dataset. 88 | 89 | 90 | ### Q2: TinyImageNet and Model Ensembles 91 | 92 | In the IPython notebook `q2.ipynb` we will introduce the TinyImageNet-100-A 93 | dataset. You will try to classify examples from this dataset by hand, and you 94 | will combine pretrained models into an ensemble to boost your performance on 95 | this dataset. 96 | 97 | ### Q3: Transfer Learning 98 | 99 | In the IPython notebook `q3.ipynb` you will implement several forms of transfer 100 | learning. You will use adapt a pretrained model for TinyImageNet-100-A to achieve 101 | reasonable performanc with minimal training time on the TinyImageNet-100-B 102 | dataset. 103 | 104 | ### Q4: Visualizing and Breaking ConvNets 105 | 106 | In the IPython notebook `q4.ipynb` you will visualize data gradients and you 107 | will generate images to fool a pretrained ConvNet. 108 | 109 | -------------------------------------------------------------------------------- /assignments/2015/assignment1.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | mathjax: true 4 | permalink: assignments2015/assignment1/ 5 | --- 6 | 7 | In this assignment you will practice putting together a simple image classification pipeline, based on the k-Nearest Neighbor or the SVM/Softmax classifier. The goals of this assignment are as follows: 8 | 9 | - understand the basic **Image Classification pipeline** and the data-driven approach (train/predict stages) 10 | - understand the train/val/test **splits** and the use of validation data for **hyperparameter tuning**. 11 | - develop proficiency in writing efficient **vectorized** code with numpy 12 | - implement and apply a k-Nearest Neighbor (**kNN**) classifier 13 | - implement and apply a Multiclass Support Vector Machine (**SVM**) classifier 14 | - implement and apply a **Softmax** classifier 15 | - understand the differences and tradeoffs between these classifiers 16 | - get a basic understanding of performance improvements from using **higher-level representations** than raw pixels (e.g. color histograms, Histogram of Gradient (HOG) features) 17 | 18 | ## Setup 19 | You can work on the assignment in one of two ways: locally on your own machine, or on a virtual machine 20 | through [Terminal](https://www.terminal.com/). 21 | 22 | ### Working locally 23 | **Get the code:** 24 | [Download the starter code here](http://cs231n.stanford.edu/assignments/2015/assignment1.zip). 25 | 26 | **[Optional] virtual environment:** 27 | Once you have unzipped the starter code, you might want to create a 28 | [virtual environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/) 29 | for the project. If you choose not to use a virtual environment, it is up to you 30 | to make sure that all dependencies for the code are installed on your machine. 31 | To set up a virtual environment, run the following: 32 | 33 | ```bash 34 | cd assignment1 35 | sudo pip install virtualenv # This may already be installed 36 | virtualenv .env # Create a virtual environment 37 | source .env/bin/activate # Activate the virtual environment 38 | pip install -r requirements.txt # Install dependencies 39 | # Work on the assignment for a while ... 40 | deactivate # Exit the virtual environment 41 | ``` 42 | 43 | **Download data:** 44 | Once you have the starter code, you will need to download the CIFAR-10 dataset. 45 | Run the following from the `assignment1` directory: 46 | 47 | ```bash 48 | cd cs231n/datasets 49 | ./get_datasets.sh 50 | ``` 51 | 52 | **Start IPython:** 53 | After you have the CIFAR-10 data, you should start the IPython notebook server from the 54 | `assignment1` directory. If you are unfamiliar with IPython, you should read our 55 | [IPython tutorial](/ipython-tutorial). 56 | 57 | ### Working on Terminal 58 | We have created a Terminal snapshot that is preconfigured for this assignment; 59 | you can [find it here](https://www.terminal.com/tiny/hUxP8UTMKa). Terminal allows you to work on the assignment from your browser. You can find a tutorial on how to use it [here](/terminal-tutorial). 60 | 61 | ### Submitting your work: 62 | Whether you work on the assignment locally or using Terminal, once you are done 63 | working run the `collectSubmission.sh` script; this will produce a file called 64 | `assignment1.zip`. Upload this file to your dropbox on 65 | [the coursework](https://coursework.stanford.edu/portal/site/W15-CS-231N-01/) 66 | page for the course. 67 | 68 | ### Q1: k-Nearest Neighbor classifier (30 points) 69 | 70 | The IPython Notebook **knn.ipynb** will walk you through implementing the kNN classifier. 71 | 72 | ### Q2: Training a Support Vector Machine (30 points) 73 | 74 | The IPython Notebook **svm.ipynb** will walk you through implementing the SVM classifier. 75 | 76 | ### Q3: Implement a Softmax classifier (30 points) 77 | 78 | The IPython Notebook **softmax.ipynb** will walk you through implementing the Softmax classifier. 79 | 80 | ### Q4: Higher Level Representations: Image Features (10 points) 81 | 82 | The IPython Notebook **features.ipynb** will walk you through this exercise, in which you will examine the improvements gained by using higher-level representations as opposed to using raw pixel values. 83 | 84 | ### Q5: Bonus: Design your own features! (+10 points) 85 | 86 | In this assignment we provide you with Color Histograms and HOG features. To claim these bonus points, implement your own additional features from scratch, and using only numpy or scipy (no external dependencies). You will have to research different feature types to get ideas for what you might want to implement. Your new feature should help you improve the performance beyond what you got in Q4 if you wish to get these bonus points. If you come up with nice features we'll feature them in the lecture. 87 | 88 | ### Q6: Cool Bonus: Do something extra! (+10 points) 89 | 90 | Implement, investigate or analyze something extra surrounding the topics in this assignment, and using the code you developed. For example, is there some other interesting question we could have asked? Is there any insightful visualization you can plot? Or maybe you can experiment with a spin on the loss function? If you try out something cool we'll give you points and might feature your results in the lecture. 91 | -------------------------------------------------------------------------------- /assignments/2016/assignment1.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | mathjax: true 4 | permalink: /assignments2016/assignment1/ 5 | --- 6 | 7 | In this assignment you will practice putting together a simple image classification pipeline, based on the k-Nearest Neighbor or the SVM/Softmax classifier. The goals of this assignment are as follows: 8 | 9 | - understand the basic **Image Classification pipeline** and the data-driven approach (train/predict stages) 10 | - understand the train/val/test **splits** and the use of validation data for **hyperparameter tuning**. 11 | - develop proficiency in writing efficient **vectorized** code with numpy 12 | - implement and apply a k-Nearest Neighbor (**kNN**) classifier 13 | - implement and apply a Multiclass Support Vector Machine (**SVM**) classifier 14 | - implement and apply a **Softmax** classifier 15 | - implement and apply a **Two layer neural network** classifier 16 | - understand the differences and tradeoffs between these classifiers 17 | - get a basic understanding of performance improvements from using **higher-level representations** than raw pixels (e.g. color histograms, Histogram of Gradient (HOG) features) 18 | 19 | ## Setup 20 | You can work on the assignment in one of two ways: locally on your own machine, or on a virtual machine through Terminal.com. 21 | 22 | ### Working in the cloud on Terminal 23 | 24 | Terminal has created a separate subdomain to serve our class, [www.stanfordterminalcloud.com](https://www.stanfordterminalcloud.com). Register your account there. The Assignment 1 snapshot can then be found [here](https://www.stanfordterminalcloud.com/snapshot/49f5a1ea15dc424aec19155b3398784d57c55045435315ce4f8b96b62819ef65). If you're registered in the class you can contact the TA (see Piazza for more information) to request Terminal credits for use on the assignment. Once you boot up the snapshot everything will be installed for you, and you'll be ready to start on your assignment right away. We've written a small tutorial on Terminal [here](/terminal-tutorial). 25 | 26 | ### Working locally 27 | Get the code as a zip file [here](http://cs231n.stanford.edu/assignments/2016/winter1516_assignment1.zip). As for the dependencies: 28 | 29 | **[Option 1] Use Anaconda:** 30 | The preferred approach for installing all the assignment dependencies is to use [Anaconda](https://www.continuum.io/downloads), which is a Python distribution that includes many of the most popular Python packages for science, math, engineering and data analysis. Once you install it you can skip all mentions of requirements and you're ready to go directly to working on the assignment. 31 | 32 | **[Option 2] Manual install, virtual environment:** 33 | If you'd like to (instead of Anaconda) go with a more manual and risky installation route you will likely want to create a [virtual environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/) for the project. If you choose not to use a virtual environment, it is up to you to make sure that all dependencies for the code are installed globally on your machine. To set up a virtual environment, run the following: 34 | 35 | ```bash 36 | cd assignment1 37 | sudo pip install virtualenv # This may already be installed 38 | virtualenv .env # Create a virtual environment 39 | source .env/bin/activate # Activate the virtual environment 40 | pip install -r requirements.txt # Install dependencies 41 | # Work on the assignment for a while ... 42 | deactivate # Exit the virtual environment 43 | ``` 44 | 45 | **Download data:** 46 | Once you have the starter code, you will need to download the CIFAR-10 dataset. 47 | Run the following from the `assignment1` directory: 48 | 49 | ```bash 50 | cd cs231n/datasets 51 | ./get_datasets.sh 52 | ``` 53 | 54 | **Start IPython:** 55 | After you have the CIFAR-10 data, you should start the IPython notebook server from the 56 | `assignment1` directory. If you are unfamiliar with IPython, you should read our 57 | [IPython tutorial](/ipython-tutorial). 58 | 59 | **NOTE:** If you are working in a virtual environment on OSX, you may encounter 60 | errors with matplotlib due to the [issues described here](http://matplotlib.org/faq/virtualenv_faq.html). You can work around this issue by starting the IPython server using the `start_ipython_osx.sh` script from the `assignment1` directory; the script assumes that your virtual environment is named `.env`. 61 | 62 | ### Submitting your work: 63 | Whether you work on the assignment locally or using Terminal, once you are done 64 | working run the `collectSubmission.sh` script; this will produce a file called 65 | `assignment1.zip`. Upload this file to your dropbox on 66 | [the coursework](https://coursework.stanford.edu/portal/site/W16-CS-231N-01/) 67 | page for the course. 68 | 69 | ### Q1: k-Nearest Neighbor classifier (20 points) 70 | 71 | The IPython Notebook **knn.ipynb** will walk you through implementing the kNN classifier. 72 | 73 | ### Q2: Training a Support Vector Machine (25 points) 74 | 75 | The IPython Notebook **svm.ipynb** will walk you through implementing the SVM classifier. 76 | 77 | ### Q3: Implement a Softmax classifier (20 points) 78 | 79 | The IPython Notebook **softmax.ipynb** will walk you through implementing the Softmax classifier. 80 | 81 | ### Q4: Two-Layer Neural Network (25 points) 82 | The IPython Notebook **two\_layer\_net.ipynb** will walk you through the implementation of a two-layer neural network classifier. 83 | 84 | ### Q5: Higher Level Representations: Image Features (10 points) 85 | 86 | The IPython Notebook **features.ipynb** will walk you through this exercise, in which you will examine the improvements gained by using higher-level representations as opposed to using raw pixel values. 87 | 88 | ### Q6: Cool Bonus: Do something extra! (+10 points) 89 | 90 | Implement, investigate or analyze something extra surrounding the topics in this assignment, and using the code you developed. For example, is there some other interesting question we could have asked? Is there any insightful visualization you can plot? Or anything fun to look at? Or maybe you can experiment with a spin on the loss function? If you try out something cool we'll give you up to 10 extra points and may feature your results in the lecture. 91 | -------------------------------------------------------------------------------- /assignments/2016/assignment2.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | mathjax: true 4 | permalink: assignments2016/assignment2/ 5 | --- 6 | 7 | In this assignment you will practice writing backpropagation code, and training 8 | Neural Networks and Convolutional Neural Networks. The goals of this assignment 9 | are as follows: 10 | 11 | - understand **Neural Networks** and how they are arranged in layered 12 | architectures 13 | - understand and be able to implement (vectorized) **backpropagation** 14 | - implement various **update rules** used to optimize Neural Networks 15 | - implement **batch normalization** for training deep networks 16 | - implement **dropout** to regularize networks 17 | - effectively **cross-validate** and find the best hyperparameters for Neural 18 | Network architecture 19 | - understand the architecture of **Convolutional Neural Networks** and train 20 | gain experience with training these models on data 21 | 22 | ## Setup 23 | You can work on the assignment in one of two ways: locally on your own machine, 24 | or on a virtual machine through Terminal.com. 25 | 26 | ### Working in the cloud on Terminal 27 | 28 | Terminal has created a separate subdomain to serve our class, 29 | [www.stanfordterminalcloud.com](https://www.stanfordterminalcloud.com). Register 30 | your account there. The Assignment 2 snapshot can then be found [HERE](https://www.stanfordterminalcloud.com/snapshot/6c95ca2c9866a962964ede3ea5813d4c2410ba48d92cf8d11a93fbb13e08b76a). If you are 31 | registered in the class you can contact the TA (see Piazza for more information) 32 | to request Terminal credits for use on the assignment. Once you boot up the 33 | snapshot everything will be installed for you, and you will be ready to start on 34 | your assignment right away. We have written a small tutorial on Terminal 35 | [here](/terminal-tutorial). 36 | 37 | ### Working locally 38 | Get the code as a zip file 39 | [here](http://cs231n.stanford.edu/assignments/2016/winter1516_assignment2.zip). 40 | As for the dependencies: 41 | 42 | **[Option 1] Use Anaconda:** 43 | The preferred approach for installing all the assignment dependencies is to use 44 | [Anaconda](https://www.continuum.io/downloads), which is a Python distribution 45 | that includes many of the most popular Python packages for science, math, 46 | engineering and data analysis. Once you install it you can skip all mentions of 47 | requirements and you are ready to go directly to working on the assignment. 48 | 49 | **[Option 2] Manual install, virtual environment:** 50 | If you do not want to use Anaconda and want to go with a more manual and risky 51 | installation route you will likely want to create a 52 | [virtual environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/) 53 | for the project. If you choose not to use a virtual environment, it is up to you 54 | to make sure that all dependencies for the code are installed globally on your 55 | machine. To set up a virtual environment, run the following: 56 | 57 | ```bash 58 | cd assignment2 59 | sudo pip install virtualenv # This may already be installed 60 | virtualenv .env # Create a virtual environment 61 | source .env/bin/activate # Activate the virtual environment 62 | pip install -r requirements.txt # Install dependencies 63 | # Work on the assignment for a while ... 64 | deactivate # Exit the virtual environment 65 | ``` 66 | 67 | **Download data:** 68 | Once you have the starter code, you will need to download the CIFAR-10 dataset. 69 | Run the following from the `assignment2` directory: 70 | 71 | ```bash 72 | cd cs231n/datasets 73 | ./get_datasets.sh 74 | ``` 75 | 76 | **Compile the Cython extension:** Convolutional Neural Networks require a very 77 | efficient implementation. We have implemented of the functionality using 78 | [Cython](http://cython.org/); you will need to compile the Cython extension 79 | before you can run the code. From the `cs231n` directory, run the following 80 | command: 81 | 82 | ```bash 83 | python setup.py build_ext --inplace 84 | ``` 85 | 86 | **Start IPython:** 87 | After you have the CIFAR-10 data, you should start the IPython notebook server 88 | from the `assignment2` directory. If you are unfamiliar with IPython, you should 89 | read our [IPython tutorial](/ipython-tutorial). 90 | 91 | **NOTE:** If you are working in a virtual environment on OSX, you may encounter 92 | errors with matplotlib due to the 93 | [issues described here](http://matplotlib.org/faq/virtualenv_faq.html). 94 | You can work around this issue by starting the IPython server using the 95 | `start_ipython_osx.sh` script from the `assignment2` directory; the script 96 | assumes that your virtual environment is named `.env`. 97 | 98 | 99 | ### Submitting your work: 100 | Whether you work on the assignment locally or using Terminal, once you are done 101 | working run the `collectSubmission.sh` script; this will produce a file called 102 | `assignment2.zip`. Upload this file under the Assignments tab on 103 | [the coursework](https://coursework.stanford.edu/portal/site/W15-CS-231N-01/) 104 | page for the course. 105 | 106 | 107 | ### Q1: Fully-connected Neural Network (30 points) 108 | The IPython notebook `FullyConnectedNets.ipynb` will introduce you to our 109 | modular layer design, and then use those layers to implement fully-connected 110 | networks of arbitrary depth. To optimize these models you will implement several 111 | popular update rules. 112 | 113 | ### Q2: Batch Normalization (30 points) 114 | In the IPython notebook `BatchNormalization.ipynb` you will implement batch 115 | normalization, and use it to train deep fully-connected networks. 116 | 117 | ### Q3: Dropout (10 points) 118 | The IPython notebook `Dropout.ipynb` will help you implement Dropout and explore 119 | its effects on model generalization. 120 | 121 | ### Q4: ConvNet on CIFAR-10 (30 points) 122 | In the IPython Notebook `ConvolutionalNetworks.ipynb` you will implement several 123 | new layers that are commonly used in convolutional networks. You will train a 124 | (shallow) convolutional network on CIFAR-10, and it will then be up to you to 125 | train the best network that you can. 126 | 127 | ### Q5: Do something extra! (up to +10 points) 128 | In the process of training your network, you should feel free to implement 129 | anything that you want to get better performance. You can modify the solver, 130 | implement additional layers, use different types of regularization, use an 131 | ensemble of models, or anything else that comes to mind. If you implement these 132 | or other ideas not covered in the assignment then you will be awarded some bonus 133 | points. 134 | 135 | -------------------------------------------------------------------------------- /assignments/2016/assignment3.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | mathjax: true 4 | permalink: assignments2016/assignment3/ 5 | --- 6 | 7 | In this assignment you will implement recurrent networks, and apply them to image captioning on Microsoft COCO. We will also introduce the TinyImageNet dataset, and use a pretrained model on this dataset to explore different applications of image gradients. 8 | 9 | The goals of this assignment are as follows: 10 | 11 | - Understand the architecture of *recurrent neural networks (RNNs)* and how they operate on sequences by sharing weights over time 12 | - Understand the difference between vanilla RNNs and Long-Short Term Memory (LSTM) RNNs 13 | - Understand how to sample from an RNN at test-time 14 | - Understand how to combine convolutional neural nets and recurrent nets to implement an image captioning system 15 | - Understand how a trained convolutional network can be used to compute gradients with respect to the input image 16 | - Implement and different applications of image gradients, including saliency maps, fooling images, class visualizations, feature inversion, and DeepDream. 17 | 18 | ## Setup 19 | You can work on the assignment in one of two ways: locally on your own machine, 20 | or on a virtual machine through Terminal.com. 21 | 22 | ### Working in the cloud on Terminal 23 | 24 | Terminal has created a separate subdomain to serve our class, 25 | [www.stanfordterminalcloud.com](https://www.stanfordterminalcloud.com). Register 26 | your account there. The Assignment 3 snapshot can then be found [HERE](https://www.stanfordterminalcloud.com/snapshot/29054ca27bc2e8bda888709ba3d9dd07a172cbbf0824152aac49b14a018ffbe5). 27 | If you are registered in the class you can contact the TA (see Piazza for more 28 | information) to request Terminal credits for use on the assignment. Once you 29 | boot up the snapshot everything will be installed for you, and you will be ready to start on your assignment right away. We have written a small tutorial on Terminal [here](/terminal-tutorial). 30 | 31 | ### Working locally 32 | Get the code as a zip file 33 | [here](http://cs231n.stanford.edu/assignments/2016/winter1516_assignment3.zip). 34 | As for the dependencies: 35 | 36 | **[Option 1] Use Anaconda:** 37 | The preferred approach for installing all the assignment dependencies is to use 38 | [Anaconda](https://www.continuum.io/downloads), which is a Python distribution 39 | that includes many of the most popular Python packages for science, math, 40 | engineering and data analysis. Once you install it you can skip all mentions of 41 | requirements and you are ready to go directly to working on the assignment. 42 | 43 | **[Option 2] Manual install, virtual environment:** 44 | If you do not want to use Anaconda and want to go with a more manual and risky 45 | installation route you will likely want to create a 46 | [virtual environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/) 47 | for the project. If you choose not to use a virtual environment, it is up to you 48 | to make sure that all dependencies for the code are installed globally on your 49 | machine. To set up a virtual environment, run the following: 50 | 51 | ```bash 52 | cd assignment3 53 | sudo pip install virtualenv # This may already be installed 54 | virtualenv .env # Create a virtual environment 55 | source .env/bin/activate # Activate the virtual environment 56 | pip install -r requirements.txt # Install dependencies 57 | # Work on the assignment for a while ... 58 | deactivate # Exit the virtual environment 59 | ``` 60 | 61 | **Download data:** 62 | Once you have the starter code, you will need to download the processed MS-COCO dataset, the TinyImageNet dataset, and the pretrained TinyImageNet model. Run the following from the `assignment3` directory: 63 | 64 | ```bash 65 | cd cs231n/datasets 66 | ./get_coco_captioning.sh 67 | ./get_tiny_imagenet_a.sh 68 | ./get_pretrained_model.sh 69 | ``` 70 | 71 | **Compile the Cython extension:** Convolutional Neural Networks require a very 72 | efficient implementation. We have implemented of the functionality using 73 | [Cython](http://cython.org/); you will need to compile the Cython extension 74 | before you can run the code. From the `cs231n` directory, run the following 75 | command: 76 | 77 | ```bash 78 | python setup.py build_ext --inplace 79 | ``` 80 | 81 | **Start IPython:** 82 | After you have the data, you should start the IPython notebook server 83 | from the `assignment3` directory. If you are unfamiliar with IPython, you should 84 | read our [IPython tutorial](/ipython-tutorial). 85 | 86 | **NOTE:** If you are working in a virtual environment on OSX, you may encounter 87 | errors with matplotlib due to the 88 | [issues described here](http://matplotlib.org/faq/virtualenv_faq.html). 89 | You can work around this issue by starting the IPython server using the 90 | `start_ipython_osx.sh` script from the `assignment3` directory; the script 91 | assumes that your virtual environment is named `.env`. 92 | 93 | 94 | ### Submitting your work: 95 | Whether you work on the assignment locally or using Terminal, once you are done 96 | working run the `collectSubmission.sh` script; this will produce a file called 97 | `assignment3.zip`. Upload this file under the Assignments tab on 98 | [the coursework](https://coursework.stanford.edu/portal/site/W15-CS-231N-01/) 99 | page for the course. 100 | 101 | 102 | ### Q1: Image Captioning with Vanilla RNNs (40 points) 103 | The IPython notebook `RNN_Captioning.ipynb` will walk you through the 104 | implementation of an image captioning system on MS-COCO using vanilla recurrent 105 | networks. 106 | 107 | ### Q2: Image Captioning with LSTMs (35 points) 108 | The IPython notebook `LSTM_Captioning.ipynb` will walk you through the 109 | implementation of Long-Short Term Memory (LSTM) RNNs, and apply them to image 110 | captioning on MS-COCO. 111 | 112 | ### Q3: Image Gradients: Saliency maps and Fooling Images (10 points) 113 | The IPython notebook `ImageGradients.ipynb` will introduce the TinyImageNet 114 | dataset. You will use a pretrained model on this dataset to compute gradients 115 | with respect to the image, and use them to produce saliency maps and fooling 116 | images. 117 | 118 | ### Q4: Image Generation: Classes, Inversion, DeepDream (15 points) 119 | In the IPython notebook `ImageGeneration.ipynb` you will use the pretrained 120 | TinyImageNet model to generate images. In particular you will generate 121 | class visualizations and implement feature inversion and DeepDream. 122 | 123 | ### Q5: Do something extra! (up to +10 points) 124 | Given the components of the assignment, try to do something cool. Maybe there is 125 | some way to generate images that we did not implement in the assignment? 126 | 127 | -------------------------------------------------------------------------------- /aws-tutorial.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | title: AWS Tutorial 4 | permalink: /aws-tutorial/ 5 | --- 6 | For GPU instances, we also have an Amazon Machine Image (AMI) that you can use 7 | to launch GPU instances on Amazon EC2. This tutorial goes through how to set up 8 | your own EC2 instance with the provided AMI. **We do not currently 9 | distribute AWS credits to CS231N students but you are welcome to use this 10 | snapshot on your own budget.** 11 | 12 | **TL;DR** for the AWS-savvy: Our image is 13 | `cs231n_caffe_torch7_keras_lasagne_v2`, AMI ID: `ami-125b2c72` in the us-west-1 14 | region. Use a `g2.2xlarge` instance. Caffe, Torch7, Theano, Keras and Lasagne 15 | are pre-installed. Python bindings of caffe are available. It has CUDA 7.5 and 16 | CuDNN v3. 17 | 18 | First, if you don't have an AWS account already, create one by going to the [AWS 19 | homepage](http://aws.amazon.com/), and clicking on the yellow "Sign In to the 20 | Console" button. It will direct you to a signup page which looks like the 21 | following. 22 | 23 |
24 | 25 |
26 | 27 | Select the "I am a new user" checkbox, click the "Sign in using our secure 28 | server" button, and follow the subsequent pages to provide the required details. 29 | They will ask for a credit card information, and also a phone verification, so 30 | have your phone and credit card ready. 31 | 32 | Once you have signed up, go back to the [AWS homepage](http://aws.amazon.com), 33 | click on "Sign In to the Console", and this time sign in using your username and 34 | password. 35 | 36 |
37 | 38 |
39 | 40 | Once you have signed in, you will be greeted by a page like this: 41 | 42 |
43 | 44 |
45 | 46 | Make sure that the region information on the top right is set to N. California. 47 | If it is not, change it to N. California by selecting from the dropdown menu 48 | there. 49 | 50 | (Note that the subsequent steps requires your account to be "Verified" by 51 | Amazon. This may take up to 2 hrs, and you may not be able to launch instances 52 | until your account verification is complete.) 53 | 54 | Next, click on the EC2 link (first link under the Compute category). You will go 55 | to a dashboard page like this: 56 | 57 |
58 | 59 |
60 | 61 | Click the blue "Launch Instance" button, and you will be redirected to a page 62 | like the following: 63 | 64 |
65 | 66 |
67 | 68 | Click on the "Community AMIs" link on the left sidebar, and search for "cs231n" 69 | in the search box. You should be able to see the AMI 70 | `cs231n_caffe_torch7_keras_lasagne_v2` (AMI ID: `ami-125b2c72`). Select that 71 | AMI, and continue to the next step to choose your instance type. 72 | 73 |
74 | 75 |
76 | 77 | Choose the instance type `g2.2xlarge`, and click on "Review and Launch". 78 | 79 |
80 | 81 |
82 | 83 | In the next page, click on Launch. 84 | 85 |
86 | 87 |
88 | 89 | You will be then prompted to create or use an existing key-pair. If you already 90 | use AWS and have a key-pair, you can use that, or alternately you can create a 91 | new one by choosing "Create a new key pair" from the drop-down menu and giving 92 | it some name of your choice. You should then download the key pair, and keep it 93 | somewhere that you won't accidentally delete. Remember that there is **NO WAY** 94 | to get to your instance if you lose your key. 95 | 96 |
97 | 98 |
99 | 100 |
101 | 102 |
103 | 104 | Once you download your key, you should change the permissions of the key to 105 | user-only RW, In Linux/OSX you can do it by: 106 | 107 | ``` 108 | $ chmod 600 PEM_FILENAME 109 | ``` 110 | Here `PEM_FILENAME` is the full file name of the .pem file you just downloaded. 111 | 112 | After this is done, click on "Launch Instances", and you should see a screen 113 | showing that your instances are launching: 114 | 115 |
116 | 117 |
118 | 119 | Click on "View Instances" to see your instance state. It should change to 120 | "Running" and "2/2 status checks passed" as shown below within some time. You 121 | are now ready to ssh into the instance. 122 | 123 |
124 | 125 |
126 | 127 | First, note down the Public IP of the instance from the instance listing. Then, 128 | do: 129 | 130 | ``` 131 | ssh -i PEM_FILENAME ubuntu@PUBLIC_IP 132 | ``` 133 | 134 | Now you should be logged in to the instance. You can check that Caffe is working 135 | by doing: 136 | 137 | ``` 138 | $ cd caffe 139 | $ ./build/tools/caffe time --gpu 0 --model examples/mnist/lenet.prototxt 140 | ``` 141 | 142 | We have Caffe, Theano, Torch7, Keras and Lasagne pre-installed. Caffe python 143 | bindings are also available by default. We have CUDA 7.5 and CuDNN v3 installed. 144 | 145 | If you encounter any error such as 146 | 147 | ``` 148 | Check failed: error == cudaSuccess (77 vs. 0) an illegal memory access was encountered 149 | ``` 150 | 151 | you might want to terminate your instance and start over again. I have observed 152 | this rarely, and I am not sure what causes this. 153 | 154 | About how to use these instances: 155 | 156 | - The root directory is only 12GB, and only ~ 3GB of that is free. 157 | - There should be a 60GB `/mnt` directory that you can use to put your data, 158 | model checkpoints, models etc. 159 | - Remember that the `/mnt` directory won't be persistent across 160 | reboots/terminations. 161 | - Stop your instances when are done for the day to avoid incurring charges. GPU 162 | instances are costly. Use your funds wisely. Terminate them when you are sure 163 | you are done with your instance (disk storage also costs something, and can be 164 | significant if you have a large disk footprint). 165 | - Look into creating custom alarms to automatically stop your instances when 166 | they are not doing anything. 167 | - If you need access to a large dataset and don't want to download it every time 168 | you spin up an instance, the best way to go would be to create an AMI for that 169 | and attach that AMI to your machine when configuring your instance (before 170 | launching but after you have selected the AMI). 171 | -------------------------------------------------------------------------------- /transfer-learning.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | permalink: /transfer-learning/ 4 | --- 5 | 6 | (These notes are currently in draft form and under development) 7 | 8 | Table of Contents: 9 | 10 | - [Transfer Learning](#tf) 11 | - [Additional References](#add) 12 | 13 | 14 | 15 | ## Transfer Learning 16 | 17 | In practice, very few people train an entire Convolutional Network from scratch (with random initialization), because it is relatively rare to have a dataset of sufficient size. Instead, it is common to pretrain a ConvNet on a very large dataset (e.g. ImageNet, which contains 1.2 million images with 1000 categories), and then use the ConvNet either as an initialization or a fixed feature extractor for the task of interest. The three major Transfer Learning scenarios look as follows: 18 | 19 | - **ConvNet as fixed feature extractor**. Take a ConvNet pretrained on ImageNet, remove the last fully-connected layer (this layer's outputs are the 1000 class scores for a different task like ImageNet), then treat the rest of the ConvNet as a fixed feature extractor for the new dataset. In an AlexNet, this would compute a 4096-D vector for every image that contains the activations of the hidden layer immediately before the classifier. We call these features **CNN codes**. It is important for performance that these codes are ReLUd (i.e. thresholded at zero) if they were also thresholded during the training of the ConvNet on ImageNet (as is usually the case). Once you extract the 4096-D codes for all images, train a linear classifier (e.g. Linear SVM or Softmax classifier) for the new dataset. 20 | - **Fine-tuning the ConvNet**. The second strategy is to not only replace and retrain the classifier on top of the ConvNet on the new dataset, but to also fine-tune the weights of the pretrained network by continuing the backpropagation. It is possible to fine-tune all the layers of the ConvNet, or it's possible to keep some of the earlier layers fixed (due to overfitting concerns) and only fine-tune some higher-level portion of the network. This is motivated by the observation that the earlier features of a ConvNet contain more generic features (e.g. edge detectors or color blob detectors) that should be useful to many tasks, but later layers of the ConvNet becomes progressively more specific to the details of the classes contained in the original dataset. In case of ImageNet for example, which contains many dog breeds, a significant portion of the representational power of the ConvNet may be devoted to features that are specific to differentiating between dog breeds. 21 | - **Pretrained models**. Since modern ConvNets take 2-3 weeks to train across multiple GPUs on ImageNet, it is common to see people release their final ConvNet checkpoints for the benefit of others who can use the networks for fine-tuning. For example, the Caffe library has a [Model Zoo](https://github.com/BVLC/caffe/wiki/Model-Zoo) where people share their network weights. 22 | 23 | **When and how to fine-tune?** How do you decide what type of transfer learning you should perform on a new dataset? This is a function of several factors, but the two most important ones are the size of the new dataset (small or big), and its similarity to the original dataset (e.g. ImageNet-like in terms of the content of images and the classes, or very different, such as microscope images). Keeping in mind that ConvNet features are more generic in early layers and more original-dataset-specific in later layers, here are some common rules of thumb for navigating the 4 major scenarios: 24 | 25 | 1. *New dataset is small and similar to original dataset*. Since the data is small, it is not a good idea to fine-tune the ConvNet due to overfitting concerns. Since the data is similar to the original data, we expect higher-level features in the ConvNet to be relevant to this dataset as well. Hence, the best idea might be to train a linear classifier on the CNN codes. 26 | 2. *New dataset is large and similar to the original dataset*. Since we have more data, we can have more confidence that we won't overfit if we were to try to fine-tune through the full network. 27 | 3. *New dataset is small but very different from the original dataset*. Since the data is small, it is likely best to only train a linear classifier. Since the dataset is very different, it might not be best to train the classifier form the top of the network, which contains more dataset-specific features. Instead, it might work better to train the SVM classifier from activations somewhere earlier in the network. 28 | 4. *New dataset is large and very different from the original dataset*. Since the dataset is very large, we may expect that we can afford to train a ConvNet from scratch. However, in practice it is very often still beneficial to initialize with weights from a pretrained model. In this case, we would have enough data and confidence to fine-tune through the entire network. 29 | 30 | **Practical advice**. There are a few additional things to keep in mind when performing Transfer Learning: 31 | 32 | - *Constraints from pretrained models*. Note that if you wish to use a pretrained network, you may be slightly constrained in terms of the architecture you can use for your new dataset. For example, you can't arbitrarily take out Conv layers from the pretrained network. However, some changes are straight-forward: Due to parameter sharing, you can easily run a pretrained network on images of different spatial size. This is clearly evident in the case of Conv/Pool layers because their forward function is independent of the input volume spatial size (as long as the strides "fit"). In case of FC layers, this still holds true because FC layers can be converted to a Convolutional Layer: For example, in an AlexNet, the final pooling volume before the first FC layer is of size [6x6x512]. Therefore, the FC layer looking at this volume is equivalent to having a Convolutional Layer that has receptive field size 6x6, and is applied with padding of 0. 33 | - *Learning rates*. It's common to use a smaller learning rate for ConvNet weights that are being fine-tuned, in comparison to the (randomly-initialized) weights for the new linear classifier that computes the class scores of your new dataset. This is because we expect that the ConvNet weights are relatively good, so we don't wish to distort them too quickly and too much (especially while the new Linear Classifier above them is being trained from random initialization). 34 | 35 | 36 | 37 | ## Additional References 38 | 39 | - [CNN Features off-the-shelf: an Astounding Baseline for Recognition](http://arxiv.org/abs/1403.6382) trains SVMs on features from ImageNet-pretrained ConvNet and reports several state of the art results. 40 | - [DeCAF](http://arxiv.org/abs/1310.1531) reported similar findings in 2013. The framework in this paper (DeCAF) was a Python-based precursor to the C++ Caffe library. 41 | - [How transferable are features in deep neural networks?](http://arxiv.org/abs/1411.1792) studies the transfer learning performance in detail, including some unintuitive findings about layer co-adaptations. 42 | -------------------------------------------------------------------------------- /assignments/2017/assignment1.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | mathjax: true 4 | permalink: /assignments2017/assignment1/ 5 | --- 6 | 7 | In this assignment you will practice putting together a simple image classification pipeline, based on the k-Nearest Neighbor or the SVM/Softmax classifier. The goals of this assignment are as follows: 8 | 9 | - understand the basic **Image Classification pipeline** and the data-driven approach (train/predict stages) 10 | - understand the train/val/test **splits** and the use of validation data for **hyperparameter tuning**. 11 | - develop proficiency in writing efficient **vectorized** code with numpy 12 | - implement and apply a k-Nearest Neighbor (**kNN**) classifier 13 | - implement and apply a Multiclass Support Vector Machine (**SVM**) classifier 14 | - implement and apply a **Softmax** classifier 15 | - implement and apply a **Two layer neural network** classifier 16 | - understand the differences and tradeoffs between these classifiers 17 | - get a basic understanding of performance improvements from using **higher-level representations** than raw pixels (e.g. color histograms, Histogram of Gradient (HOG) features) 18 | 19 | ## Setup 20 | You can work on the assignment in one of two ways: locally on your own machine, or on a virtual machine on Google Cloud. 21 | 22 | ### Working remotely on Google Cloud (Recommended) 23 | 24 | **Note:** after following these instructions, make sure you go to **Download data** below (you can skip the **Working locally** section). 25 | 26 | As part of this course, you can use Google Cloud for your assignments. We recommend this route for anyone who is having trouble with installation set-up, or if you would like to use better CPU/GPU resources than you may have locally. Please see the set-up tutorial [here](http://cs231n.github.io/gce-tutorial/) for more details. :) 27 | 28 | ### Working locally 29 | Get the code as a zip file [here](http://cs231n.stanford.edu/assignments/2017/spring1617_assignment1.zip). As for the dependencies: 30 | 31 | **Installing Python 3.5+:** 32 | To use python3, make sure to install version 3.5 or 3.6 on your local machine. If you are on Mac OS X, you can do this using [Homebrew](https://brew.sh) with `brew install python3`. You can find instructions for Ubuntu [here](https://www.digitalocean.com/community/tutorials/how-to-install-python-3-and-set-up-a-local-programming-environment-on-ubuntu-16-04). 33 | 34 | **Virtual environment:** 35 | If you decide to work locally, we recommend using [virtual environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/) for the project. If you choose not to use a virtual environment, it is up to you to make sure that all dependencies for the code are installed globally on your machine. To set up a virtual environment, run the following: 36 | 37 | ```bash 38 | cd assignment1 39 | sudo pip install virtualenv # This may already be installed 40 | virtualenv -p python3 .env # Create a virtual environment (python3) 41 | # Note: you can also use "virtualenv .env" to use your default python (usually python 2.7) 42 | source .env/bin/activate # Activate the virtual environment 43 | pip install -r requirements.txt # Install dependencies 44 | # Work on the assignment for a while ... 45 | deactivate # Exit the virtual environment 46 | ``` 47 | 48 | Note that every time you want to work on the assignment, you should run `source .env/bin/activate` (from within your `assignment1` folder) to re-activate the virtual environment, and `deactivate` again whenever you are done. 49 | 50 | ### Download data: 51 | Once you have the starter code (regardless of which method you choose above), you will need to download the CIFAR-10 dataset. 52 | Run the following from the `assignment1` directory: 53 | 54 | ```bash 55 | cd cs231n/datasets 56 | ./get_datasets.sh 57 | ``` 58 | 59 | ### Start IPython: 60 | After you have the CIFAR-10 data, you should start the IPython notebook server from the 61 | `assignment1` directory, with the `jupyter notebook` command. (See the [Google Cloud Tutorial](http://cs231n.github.io/gce-tutorial/) for any additional steps you may need to do for setting this up, if you are working remotely) 62 | 63 | If you are unfamiliar with IPython, you can also refer to our 64 | [IPython tutorial](/ipython-tutorial). 65 | 66 | ### Some Notes 67 | **NOTE 1:** This year, the `assignment1` code has been tested to be compatible with python versions `2.7`, `3.5`, `3.6` (it may work with other versions of `3.x`, but we won't be officially supporting them). You will need to make sure that during your `virtualenv` setup that the correct version of `python` is used. You can confirm your python version by (1) activating your virtualenv and (2) running `which python`. 68 | 69 | **NOTE 2:** If you are working in a virtual environment on OSX, you may *potentially* encounter 70 | errors with matplotlib due to the [issues described here](http://matplotlib.org/faq/virtualenv_faq.html). In our testing, it seems that this issue is no longer present with the most recent version of matplotlib, but if you do end up running into this issue you may have to use the `start_ipython_osx.sh` script from the `assignment1` directory (instead of `jupyter notebook` above) to launch your IPython notebook server. Note that you may have to modify some variables within the script to match your version of python/installation directory. The script assumes that your virtual environment is named `.env`. 71 | 72 | ### Submitting your work: 73 | Whether you work on the assignment locally or using Google Cloud, once you are done 74 | working run the `collectSubmission.sh` script; this will produce a file called 75 | `assignment1.zip`. Please submit this file on [Canvas](https://canvas.stanford.edu/courses/66461/). 76 | 77 | ### Q1: k-Nearest Neighbor classifier (20 points) 78 | 79 | The IPython Notebook **knn.ipynb** will walk you through implementing the kNN classifier. 80 | 81 | ### Q2: Training a Support Vector Machine (25 points) 82 | 83 | The IPython Notebook **svm.ipynb** will walk you through implementing the SVM classifier. 84 | 85 | ### Q3: Implement a Softmax classifier (20 points) 86 | 87 | The IPython Notebook **softmax.ipynb** will walk you through implementing the Softmax classifier. 88 | 89 | ### Q4: Two-Layer Neural Network (25 points) 90 | The IPython Notebook **two\_layer\_net.ipynb** will walk you through the implementation of a two-layer neural network classifier. 91 | 92 | ### Q5: Higher Level Representations: Image Features (10 points) 93 | 94 | The IPython Notebook **features.ipynb** will walk you through this exercise, in which you will examine the improvements gained by using higher-level representations as opposed to using raw pixel values. 95 | 96 | ### Q6: Cool Bonus: Do something extra! (+10 points) 97 | 98 | Implement, investigate or analyze something extra surrounding the topics in this assignment, and using the code you developed. For example, is there some other interesting question we could have asked? Is there any insightful visualization you can plot? Or anything fun to look at? Or maybe you can experiment with a spin on the loss function? If you try out something cool we'll give you up to 10 extra points and may feature your results in the lecture. 99 | -------------------------------------------------------------------------------- /assets/conv-demo/utils.js: -------------------------------------------------------------------------------- 1 | var U = {}; 2 | 3 | (function(global) { 4 | "use strict"; 5 | 6 | // Random number utilities 7 | var return_v = false; 8 | var v_val = 0.0; 9 | var gaussRandom = function() { 10 | if(return_v) { 11 | return_v = false; 12 | return v_val; 13 | } 14 | var u = 2*Math.random()-1; 15 | var v = 2*Math.random()-1; 16 | var r = u*u + v*v; 17 | if(r == 0 || r > 1) return gaussRandom(); 18 | var c = Math.sqrt(-2*Math.log(r)/r); 19 | v_val = v*c; // cache this 20 | return_v = true; 21 | return u*c; 22 | } 23 | var randf = function(a, b) { return Math.random()*(b-a)+a; } 24 | var randi = function(a, b) { return Math.floor(Math.random()*(b-a)+a); } 25 | var randn = function(mu, std){ return mu+gaussRandom()*std; } 26 | 27 | // Array utilities 28 | var zeros = function(n) { 29 | if(typeof(n)==='undefined' || isNaN(n)) { return []; } 30 | if(typeof ArrayBuffer === 'undefined') { 31 | // lacking browser support 32 | var arr = new Array(n); 33 | for(var i=0;i maxv) { maxv = w[i]; maxi = i; } 67 | if(w[i] < minv) { minv = w[i]; mini = i; } 68 | } 69 | return {maxi: maxi, maxv: maxv, mini: mini, minv: minv, dv:maxv-minv}; 70 | } 71 | 72 | // create random permutation of numbers, in range [0...n-1] 73 | var randperm = function(n) { 74 | var i = n, 75 | j = 0, 76 | temp; 77 | var array = []; 78 | for(var q=0;q 6 | 7 | These notes accompany the Stanford CS class CS231n: Convolutional Neural Networks for Visual Recognition. 8 |
9 | For questions/concerns/bug reports contact Justin Johnson regarding the assignments, or contact Andrej Karpathy regarding the course notes. You can also submit a pull request directly to our git repo. 10 |
11 | We encourage the use of the hypothes.is extension to annote comments and discuss these notes inline. 12 | 13 | 14 |
15 |
16 |
Spring 2017 Assignments
17 | 18 |
19 | 20 | Assignment #1: Image Classification, kNN, SVM, Softmax, Neural Network 21 | 22 |
23 | 24 |
25 | 26 | Assignment #2: Fully-Connected Nets, Batch Normalization, Dropout, 27 | Convolutional Nets 28 | 29 |
30 | 31 |
32 | 33 | Assignment #3: Image Captioning with Vanilla RNNs, Image Captioning 34 | with LSTMs, Network Visualization, Style Transfer, Generative Adversarial Networks 35 | 36 |
37 | 38 | 61 | 62 | 83 | 84 |
Module 0: Preparation
85 | 86 |
87 | 88 | Python / Numpy Tutorial 89 | 90 |
91 | 92 |
93 | 94 | IPython Notebook Tutorial 95 | 96 |
97 | 104 |
105 | 106 | Google Cloud Tutorial 107 | 108 |
109 | 110 |
111 | 112 | Google Cloud with GPUs Tutorial (for assignment 2 onwards) 113 | 114 |
115 |
116 | 117 | AWS Tutorial 118 | 119 |
120 | 121 | 122 |
Module 1: Neural Networks
123 | 124 |
125 | 126 | Image Classification: Data-driven Approach, k-Nearest Neighbor, train/val/test splits 127 | 128 |
129 | L1/L2 distances, hyperparameter search, cross-validation 130 |
131 |
132 | 133 |
134 | 135 | Linear classification: Support Vector Machine, Softmax 136 | 137 |
138 | parameteric approach, bias trick, hinge loss, cross-entropy loss, L2 regularization, web demo 139 |
140 |
141 | 142 |
143 | 144 | Optimization: Stochastic Gradient Descent 145 | 146 |
147 | optimization landscapes, local search, learning rate, analytic/numerical gradient 148 |
149 |
150 | 151 |
152 | 153 | Backpropagation, Intuitions 154 | 155 |
156 | chain rule interpretation, real-valued circuits, patterns in gradient flow 157 |
158 |
159 | 160 |
161 | 162 | Neural Networks Part 1: Setting up the Architecture 163 | 164 |
165 | model of a biological neuron, activation functions, neural net architecture, representational power 166 |
167 |
168 | 169 |
170 | 171 | Neural Networks Part 2: Setting up the Data and the Loss 172 | 173 |
174 | preprocessing, weight initialization, batch normalization, regularization (L2/dropout), loss functions 175 |
176 |
177 | 178 |
179 | 180 | Neural Networks Part 3: Learning and Evaluation 181 | 182 |
183 | gradient checks, sanity checks, babysitting the learning process, momentum (+nesterov), second-order methods, Adagrad/RMSprop, hyperparameter optimization, model ensembles 184 |
185 |
186 | 187 |
188 | 189 | Putting it together: Minimal Neural Network Case Study 190 | 191 |
192 | minimal 2D toy data example 193 |
194 |
195 | 196 |
Module 2: Convolutional Neural Networks
197 | 198 |
199 | 200 | Convolutional Neural Networks: Architectures, Convolution / Pooling Layers 201 | 202 |
203 | layers, spatial arrangement, layer patterns, layer sizing patterns, AlexNet/ZFNet/VGGNet case studies, computational considerations 204 |
205 |
206 | 207 |
208 | 209 | Understanding and Visualizing Convolutional Neural Networks 210 | 211 |
212 | tSNE embeddings, deconvnets, data gradients, fooling ConvNets, human comparisons 213 |
214 |
215 | 216 |
217 | 218 | Transfer Learning and Fine-tuning Convolutional Neural Networks 219 | 220 |
221 | 222 |
223 |
224 | -------------------------------------------------------------------------------- /assignments/2017/assignment2.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | mathjax: true 4 | permalink: /assignments2017/assignment2/ 5 | --- 6 | 7 | In this assignment you will practice writing backpropagation code, and training 8 | Neural Networks and Convolutional Neural Networks. The goals of this assignment 9 | are as follows: 10 | 11 | - understand **Neural Networks** and how they are arranged in layered 12 | architectures 13 | - understand and be able to implement (vectorized) **backpropagation** 14 | - implement various **update rules** used to optimize Neural Networks 15 | - implement **batch normalization** for training deep networks 16 | - implement **dropout** to regularize networks 17 | - effectively **cross-validate** and find the best hyperparameters for Neural 18 | Network architecture 19 | - understand the architecture of **Convolutional Neural Networks** and train 20 | gain experience with training these models on data 21 | 22 | ## Setup 23 | You can work on the assignment in one of two ways: locally on your own machine, or on a virtual machine on Google Cloud. 24 | 25 | ### Working remotely on Google Cloud (Recommended) 26 | 27 | **Note:** after following these instructions, make sure you go to **Working on the assignment** below (you can skip the **Working locally** section). 28 | 29 | As part of this course, you can use Google Cloud for your assignments. We recommend this route for anyone who is having trouble with installation set-up, or if you would like to use better CPU/GPU resources than you may have locally. 30 | 31 | #### GPU Resources 32 | **A GPU will only help on Question 5**. Please see the Google Cloud GPU set-up tutorial [here](http://cs231n.github.io/gce-tutorial-gpus/) for instructions. The GPU instances are much more expensive, so use them only when needed. 33 | 34 | Once you've got the cloud instance running, make sure to run the following line to enter the virtual environment that we prepared for you (you do **not** need to make your own virtual environment): 35 | 36 | ``` 37 | source /home/cs231n/myVE35/bin/activate 38 | ``` 39 | 40 | We strongly recommend using Google Cloud with GPU support for the **Question 5** of this assignment (the TensorFlow or PyTorch notebooks), since your training will go much, much faster. However, it will not help on any of the other questions. 41 | 42 | ### Working locally 43 | Here's how you install the necessary dependencies: 44 | 45 | **(OPTIONAL) Installing GPU drivers:** 46 | If you choose to work locally, you are at no disadvantage for the first parts of the assignment. For the last question, which is in TensorFlow or PyTorch, however, having a GPU will be a significant advantage. We recommend using a Google Cloud Instance with a GPU, at least for this part. If you have your own NVIDIA GPU, however, and wish to use that, that's fine -- you'll need to install the drivers for your GPU, install CUDA, install cuDNN, and then install either [TensorFlow](https://www.tensorflow.org/install/) or [PyTorch](http://pytorch.org/). You could theoretically do the entire assignment with no GPUs, though this will make training much slower in the last part. However, our reference code runs in 10-15 minutes on a dual-core laptop without a GPU, so it is certainly possible. 47 | 48 | **Installing Python 3.5+:** 49 | To use python3, make sure to install version 3.5 or 3.6 on your local machine. If you are on Mac OS X, you can do this using [Homebrew](https://brew.sh) with `brew install python3`. You can find instructions for Ubuntu [here](https://www.digitalocean.com/community/tutorials/how-to-install-python-3-and-set-up-a-local-programming-environment-on-ubuntu-16-04). 50 | 51 | **Virtual environment:** 52 | If you decide to work locally, we recommend using [virtual environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/) for the project. If you choose not to use a virtual environment, it is up to you to make sure that all dependencies for the code are installed globally on your machine. To set up a virtual environment, run the following: 53 | 54 | ```bash 55 | cd assignment2 56 | sudo pip install virtualenv # This may already be installed 57 | python3 -m venv .env # Create a virtual environment (python3) 58 | source .env/bin/activate # Activate the virtual environment 59 | pip install -r requirements.txt # Install dependencies 60 | # Note that this does NOT install TensorFlow or PyTorch, 61 | # which you need to do yourself. 62 | 63 | # Work on the assignment for a while ... 64 | # ... and when you're done: 65 | deactivate # Exit the virtual environment 66 | ``` 67 | 68 | Note that every time you want to work on the assignment, you should run `source .env/bin/activate` (from within your `assignment2` folder) to re-activate the virtual environment, and `deactivate` again whenever you are done. 69 | 70 | ## Working on the assignment: 71 | Get the code as a zip file [here](http://cs231n.stanford.edu/assignments/2017/spring1617_assignment2.zip). 72 | 73 | ### Download data: 74 | Once you have the starter code (regardless of which method you choose above), you will need to download the CIFAR-10 dataset. 75 | Run the following from the `assignment2` directory: 76 | 77 | ```bash 78 | cd cs231n/datasets 79 | ./get_datasets.sh 80 | ``` 81 | 82 | ### Start IPython: 83 | After you have the CIFAR-10 data, you should start the IPython notebook server from the 84 | `assignment2` directory, with the `jupyter notebook` command. (See the [Google Cloud Tutorial](http://cs231n.github.io/gce-tutorial/) for any additional steps you may need to do for setting this up, if you are working remotely) 85 | 86 | If you are unfamiliar with IPython, you can also refer to our 87 | [IPython tutorial](/ipython-tutorial). 88 | 89 | ### Some Notes 90 | **NOTE 1:** This year, the `assignment2` code has been tested to be compatible with python versions `3.5` and `3.6` (it may work with other versions of `3.x`, but we won't be officially supporting them). For this assignment, we are NOT officially supporting python2. Use it at your own risk. You will need to make sure that during your `virtualenv` setup that the correct version of `python` is used. You can confirm your python version by (1) activating your virtualenv and (2) running `python --version`. 91 | 92 | **NOTE 2:** If you are working in a virtual environment on OSX, you may *potentially* encounter 93 | errors with matplotlib due to the [issues described here](http://matplotlib.org/faq/virtualenv_faq.html). In our testing, it seems that this issue is no longer present with the most recent version of matplotlib, but if you do end up running into this issue you may have to use the `start_ipython_osx.sh` script from the `assignment1` directory (instead of `jupyter notebook` above) to launch your IPython notebook server. Note that you may have to modify some variables within the script to match your version of python/installation directory. The script assumes that your virtual environment is named `.env`. 94 | 95 | ### Submitting your work: 96 | Whether you work on the assignment locally or using Google Cloud, once you are done 97 | working run the `collectSubmission.sh` script; this will produce a file called 98 | `assignment2.zip`. Please submit this file on [Canvas](https://canvas.stanford.edu/courses/66461/). 99 | 100 | ### Q1: Fully-connected Neural Network (25 points) 101 | The IPython notebook `FullyConnectedNets.ipynb` will introduce you to our 102 | modular layer design, and then use those layers to implement fully-connected 103 | networks of arbitrary depth. To optimize these models you will implement several 104 | popular update rules. 105 | 106 | ### Q2: Batch Normalization (25 points) 107 | In the IPython notebook `BatchNormalization.ipynb` you will implement batch 108 | normalization, and use it to train deep fully-connected networks. 109 | 110 | ### Q3: Dropout (10 points) 111 | The IPython notebook `Dropout.ipynb` will help you implement Dropout and explore 112 | its effects on model generalization. 113 | 114 | ### Q4: Convolutional Networks (30 points) 115 | In the IPython Notebook ConvolutionalNetworks.ipynb you will implement several new layers that are commonly used in convolutional networks. 116 | 117 | ### Q5: PyTorch / TensorFlow on CIFAR-10 (10 points) 118 | For this last part, you will be working in either TensorFlow or PyTorch, two popular and powerful deep learning frameworks. **You only need to complete ONE of these two notebooks.** You do NOT need to do both, but a very small amount of extra credit will be awarded to those who do. 119 | 120 | Open up either `PyTorch.ipynb` or `TensorFlow.ipynb`. There, you will learn how the framework works, culminating in training a convolutional network of your own design on CIFAR-10 to get the best performance you can. 121 | 122 | ### Q5: Do something extra! (up to +10 points) 123 | In the process of training your network, you should feel free to implement 124 | anything that you want to get better performance. You can modify the solver, 125 | implement additional layers, use different types of regularization, use an 126 | ensemble of models, or anything else that comes to mind. If you implement these 127 | or other ideas not covered in the assignment then you will be awarded some bonus 128 | points. 129 | -------------------------------------------------------------------------------- /understanding-cnn.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | permalink: /understanding-cnn/ 4 | --- 5 | 6 | 7 | 8 | (this page is currently in draft form) 9 | 10 | ## Visualizing what ConvNets learn 11 | 12 | Several approaches for understanding and visualizing Convolutional Networks have been developed in the literature, partly as a response the common criticism that the learned features in a Neural Network are not interpretable. In this section we briefly survey some of these approaches and related work. 13 | 14 | ### Visualizing the activations and first-layer weights 15 | 16 | **Layer Activations**. The most straight-forward visualization technique is to show the activations of the network during the forward pass. For ReLU networks, the activations usually start out looking relatively blobby and dense, but as the training progresses the activations usually become more sparse and localized. One dangerous pitfall that can be easily noticed with this visualization is that some activation maps may be all zero for many different inputs, which can indicate *dead* filters, and can be a symptom of high learning rates. 17 | 18 |
19 | 20 | 21 |
22 | Typical-looking activations on the first CONV layer (left), and the 5th CONV layer (right) of a trained AlexNet looking at a picture of a cat. Every box shows an activation map corresponding to some filter. Notice that the activations are sparse (most values are zero, in this visualization shown in black) and mostly local. 23 |
24 |
25 | 26 | **Conv/FC Filters.** The second common strategy is to visualize the weights. These are usually most interpretable on the first CONV layer which is looking directly at the raw pixel data, but it is possible to also show the filter weights deeper in the network. The weights are useful to visualize because well-trained networks usually display nice and smooth filters without any noisy patterns. Noisy patterns can be an indicator of a network that hasn't been trained for long enough, or possibly a very low regularization strength that may have led to overfitting. 27 | 28 |
29 | 30 | 31 |
32 | Typical-looking filters on the first CONV layer (left), and the 2nd CONV layer (right) of a trained AlexNet. Notice that the first-layer weights are very nice and smooth, indicating nicely converged network. The color/grayscale features are clustered because the AlexNet contains two separate streams of processing, and an apparent consequence of this architecture is that one stream develops high-frequency grayscale features and the other low-frequency color features. The 2nd CONV layer weights are not as interpretable, but it is apparent that they are still smooth, well-formed, and absent of noisy patterns. 33 |
34 |
35 | 36 | ### Retrieving images that maximally activate a neuron 37 | 38 | Another visualization technique is to take a large dataset of images, feed them through the network and keep track of which images maximally activate some neuron. We can then visualize the images to get an understanding of what the neuron is looking for in its receptive field. One such visualization (among others) is shown in [Rich feature hierarchies for accurate object detection and semantic segmentation](http://arxiv.org/abs/1311.2524) by Ross Girshick et al.: 39 | 40 |
41 | 42 |
43 | Maximally activating images for some POOL5 (5th pool layer) neurons of an AlexNet. The activation values and the receptive field of the particular neuron are shown in white. (In particular, note that the POOL5 neurons are a function of a relatively large portion of the input image!) It can be seen that some neurons are responsive to upper bodies, text, or specular highlights. 44 |
45 |
46 | 47 | One problem with this approach is that ReLU neurons do not necessarily have any semantic meaning by themselves. Rather, it is more appropriate to think of multiple ReLU neurons as the basis vectors of some space that represents in image patches. In other words, the visualization is showing the patches at the edge of the cloud of representations, along the (arbitrary) axes that correspond to the filter weights. This can also be seen by the fact that neurons in a ConvNet operate linearly over the input space, so any arbitrary rotation of that space is a no-op. This point was further argued in [Intriguing properties of neural networks](http://arxiv.org/abs/1312.6199) by Szegedy et al., where they perform a similar visualization along arbitrary directions in the representation space. 48 | 49 | ### Embedding the codes with t-SNE 50 | 51 | ConvNets can be interpreted as gradually transforming the images into a representation in which the classes are separable by a linear classifier. We can get a rough idea about the topology of this space by embedding images into two dimensions so that their low-dimensional representation has approximately equal distances than their high-dimensional representation. There are many embedding methods that have been developed with the intuition of embedding high-dimensional vectors in a low-dimensional space while preserving the pairwise distances of the points. Among these, [t-SNE](http://lvdmaaten.github.io/tsne/) is one of the best-known methods that consistently produces visually-pleasing results. 52 | 53 | To produce an embedding, we can take a set of images and use the ConvNet to extract the CNN codes (e.g. in AlexNet the 4096-dimensional vector right before the classifier, and crucially, including the ReLU non-linearity). We can then plug these into t-SNE and get 2-dimensional vector for each image. The corresponding images can them be visualized in a grid: 54 | 55 |
56 | 57 |
58 | t-SNE embedding of a set of images based on their CNN codes. Images that are nearby each other are also close in the CNN representation space, which implies that the CNN "sees" them as being very similar. Notice that the similarities are more often class-based and semantic rather than pixel and color-based. For more details on how this visualization was produced the associated code, and more related visualizations at different scales refer to t-SNE visualization of CNN codes. 59 |
60 |
61 | 62 | ### Occluding parts of the image 63 | 64 | Suppose that a ConvNet classifies an image as a dog. How can we be certain that it's actually picking up on the dog in the image as opposed to some contextual cues from the background or some other miscellaneous object? One way of investigating which part of the image some classification prediction is coming from is by plotting the probability of the class of interest (e.g. dog class) as a function of the position of an occluder object. That is, we iterate over regions of the image, set a patch of the image to be all zero, and look at the probability of the class. We can visualize the probability as a 2-dimensional heat map. This approach has been used in Matthew Zeiler's [Visualizing and Understanding Convolutional Networks](http://arxiv.org/abs/1311.2901): 65 | 66 |
67 | 68 |
69 | Three input images (top). Notice that the occluder region is shown in grey. As we slide the occluder over the image we record the probability of the correct class and then visualize it as a heatmap (shown below each image). For instance, in the left-most image we see that the probability of Pomeranian plummets when the occluder covers the face of the dog, giving us some level of confidence that the dog's face is primarily responsible for the high classification score. Conversely, zeroing out other parts of the image is seen to have relatively negligible impact. 70 |
71 |
72 | 73 | ### Visualizing the data gradient and friends 74 | 75 | **Data Gradient**. 76 | 77 | [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](http://arxiv.org/abs/1312.6034) 78 | 79 | **DeconvNet**. 80 | 81 | [Visualizing and Understanding Convolutional Networks](http://arxiv.org/abs/1311.2901) 82 | 83 | **Guided Backpropagation**. 84 | 85 | [Striving for Simplicity: The All Convolutional Net](http://arxiv.org/abs/1412.6806) 86 | 87 | ### Reconstructing original images based on CNN Codes 88 | 89 | [Understanding Deep Image Representations by Inverting Them](http://arxiv.org/abs/1412.0035) 90 | 91 | ### How much spatial information is preserved? 92 | 93 | [Do ConvNets Learn Correspondence?](http://papers.nips.cc/paper/5420-do-convnets-learn-correspondence.pdf) (tldr: yes) 94 | 95 | ### Plotting performance as a function of image attributes 96 | 97 | [ImageNet Large Scale Visual Recognition Challenge](http://arxiv.org/abs/1409.0575) 98 | 99 | ## Fooling ConvNets 100 | 101 | [Explaining and Harnessing Adversarial Examples](http://arxiv.org/abs/1412.6572) 102 | 103 | ## Comparing ConvNets to Human labelers 104 | 105 | [What I learned from competing against a ConvNet on ImageNet](http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/) 106 | -------------------------------------------------------------------------------- /assets/conv-demo/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | Convolution demo 5 | 6 | 7 | 8 | 9 | 15 | 16 | 315 | 316 | 317 | 318 | 319 | 320 | 321 | 322 |
323 |
324 | 325 | 326 | -------------------------------------------------------------------------------- /assignment2_google_cloud.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | title: Google Cloud Tutorial Part 2 (with GPUs) 4 | permalink: /gce-tutorial-gpus/ 5 | --- 6 | # Google Cloud Tutorial (Part 2 With GPUs) # 7 | This tutorial assumes that you have already gone through the first Google Cloud tutorial for assignment 1 [here](http://cs231n.github.io/gce-tutorial/ "title"). The first tutorial takes you through the process of setting up a Google Cloud account, launching a VM instance, accessing Jupyter Notebook from your local computer, working on assignment 1 on your VM instance, and transferring files to your local computer. While you created a VM instance without a GPU in the first tutorial, this one walks you through the necessary steps to create an instance with a GPU, and use our provided disk images to work on assignment 2. If you haven't already done so, we advise you to go through the first tutorial to be comfortable with the process of creating an instance with the right configurations and accessing Jupyter Notebook from your local computer. 8 | 9 | ## Changing your Billing Account ## 10 | Everyone enrolled in the class should have received $100 Google Cloud credits by now. In order to use GPUs, you have to use these coupons instead of your free trial credits. To do this, follow the instructions on [this website](https://support.google.com/cloud/answer/6293499?hl=en "Title") to change the billing address associated with your project to **CS 231n- Convolutional Neural Netwks for Visual Recog-Set 1**. 11 | 12 | ## Requesting GPU Quota Increase ## 13 | To start an instance with a GPU (for the first time) you first need to request an increase in the number of GPUs you can use. To do this, go to your console, click on the **Computer Engine** button and select the **Quotas** menu. You will see a page that looks like the one below. 14 | 15 |
16 | 17 |
18 | 19 | Click on the blue **Request Increase** button. This opens a new tab with a long form titled **Google Compute Engine Quota Change Request Form** (see screenshot below). 20 |
21 | 22 |
23 | 24 | Fill out the required portions of the form and put a value of **1** in the **Total Number of GPU dies** section. You only need to do this for the **us-west1** region... don't request GPUs anywhere else, since all your instances should also live in us-west1. 25 | 26 | Once you have your quota increase you can just use GPUs (without requesting a quota increase). After you submit the form, you should receive an email approving your quota increase shortly after (I received my email within a minute). If you don't receive your approval within 2 business days, please inform the course staff. Once you have received your quota increase, you can start an instance with a GPU. To do this, while launching a virtual instance as described in the **Launch a Virtual Instance** section [here](http://cs231n.github.io/gce-tutorial/ "title"), select the number of GPUs to be 1 (or 2 if you requested for a quota increase of 2 and you really really need to use 2). As a reminder, you can only use up to the number of GPUs allowed by your quota. 27 | 28 | **NOTE: Use your GPUs sparingly because they are expensive. See the pricing [here](https://cloud.google.com/compute/pricing#gpus "title").** For example, in assignment 2, you only need to have a GPU instance for question 5. So you can work on all other parts of the assignment with a CPU instance that does *not* have any GPUs. Once you are done with the questions that do not require a GPU, you can transfer your assignment zip file to your local computer as discussed in the **Transferring Files to Your Local Computer** section below. When you get to question 5, start a GPU instance and transfer your zip file from your local computer to your instance. Refer to [this page](https://cloud.google.com/compute/docs/instances/transfer-files "title") for details on transferring files to Google Cloud. 29 | 30 | ## Starting Your Instance With Our Provided Disk ## 31 | For the remaining assignments and the project, we provide you with disks containing the necessary software for the assignments and commonly used frameworks for the project. To use our disk, you first need to create your own custom image using our file, and use this custom image as the boot disk for your new VM instance. 32 | 33 | ### Creating a Custom Image Using Our Disk ### 34 | To create your custom image using our provided disk, go to **Compute Engine**, then **Images** and click on the blue **Create Image** button at the top of the page. See the screenshot below. 35 |
36 | 37 |
38 | 39 | Enter your preferred name in the **Name** field. Mine is called **final-cs231n**. Select cloud storage file for **Source**, enter **cs231n-files/cs231n_image.tar.gz** as the **Cloud Storage file** and click on the blue **Create** button. See the screenshot below. It will take a few minutes for your image to be created (about 10-15 in our experience, though your mileage may vary). 40 | 41 |
42 | 43 |
44 | 45 | ### Starting Your Instance with Your Custom Image ### 46 | To start your instance using our provided disk, go to **VM Instances** and click on **Create Instance** like you have done before. Make sure you start the instance in **us-west1-b**. Follow the same procedure that you have used to create an instance as detailed [here](http://cs231n.github.io/gce-tutorial/ "title") but with the following differences: 47 | 48 | Make sure to provision 1 GPU to your instance by clicking **Customize** in the **Machine Type** box, as in the screenshot below: 49 | 50 |
51 | 52 |
53 | 54 | Instead of selecting an entry in **OS images** for **Boot disk**, select **Custom images** and the custom image that you created above. Mine is **final-cs231n**. See the screenshot below. 55 | 56 |
57 | 58 |
59 | 60 | It should take about 5 minutes for the instance to get created. You should now be able to launch your instance with our custom image. The custom disk is 40GB and uses Ubuntu 16.04 LTS. 61 | The default python version in the system is Python 2.7.2 and there is a virtual environment in **/home/cs231n/myVE35** with version 3.5.2. 62 | 63 | **NOTE:** Some students have reported GPU instances whose drivers disappear upon restarts. And there is strong evidence to suggest that the issue happens when Ubuntu auto-installs security updates upon booting up. To disable auto-updating, run 64 | 65 | ``` 66 | sudo apt-get remove unattended-upgrades 67 | ``` 68 | 69 | after logging into your instance for the first time. 70 | 71 | ### Load the virtual environment ### 72 | You **don't need to create a new virtual environment** for this assignment -- we are providing you with one, with all the Python packages you need for the assignment already installed. So, unlike the previous assignment, you do not need to run any commands to create the virtual environment (and install packages, etc), just one command to activate it. To use this virtual environment, run the following command (you may have to do this every time you start your instance up, if you don't see your bash prompt prefaced by "(myVE35)"): 73 | 74 | ``` 75 | source /home/cs231n/myVE35/bin/activate 76 | ``` 77 | 78 | Here's what you should see after running that to confirm you're in the virtual environment: 79 | 80 | ``` 81 | username@instance-2:~$ source /home/cs231n/myVE35/bin/activate 82 | (myVE35) username@instance-2:~$ which python 83 | /home/cs231n/myVE35/bin/python 84 | (myVE35) username@instance-2:~$ 85 | ``` 86 | 87 | The disk should also have Jupyter 1.0.0, CUDA 8.0, CUDNN 5.1, Pytorch 0.1.11_5 and TensorFlow 1.0.1. GPU support should be automatically enabled for PyTorch and TensorFlow. 88 | 89 | ### Getting started on Assignment 2 ### 90 | As in assignment 1, you can download the assignment zip file by running: 91 | 92 | ``` 93 | wget http://cs231n.stanford.edu/assignments/2017/spring1617_assignment2.zip 94 | ``` 95 | 96 | You can unzip the assignment zip file by running: 97 | 98 | ``` 99 | sudo apt-get install zip #For when you need to zip the file to submit it. 100 | sudo apt-get install unzip 101 | unzip spring1617_assignment2.zip 102 | ``` 103 | 104 | Get back to the Assignment 2 instructions [here](http://cs231n.github.io/assignments2017/assignment2/). Follow the instructions starting from the **Download data** section. 105 | 106 | **NOTE: Some students have seen errors saying that an NVIDIA driver is not installed.** If you see this error, follow the instructions [here](https://cloud.google.com/compute/docs/gpus/add-gpus#install-driver-script "title") to run the script for **Ubuntu 16.04 LTS or 16.10 - CUDA 8 with latest driver** under **Installing GPU drivers**. I.e. copy and paste the script into a file, lets call the file install_cuda.sh. And then run 107 | 108 | ``` 109 | sudo bash install_cuda.sh 110 | ``` 111 | 112 | Once you run the script above, run the commands 113 | ``` 114 | nvidia-smi 115 | nvcc --version 116 | ``` 117 | 118 | to ensure that the drivers are installed. 119 | 120 | ## Transferring Files to Your Local Computer ## 121 | After following assignment 2 instructions to run the submission script and create assignment2.zip, you can download that file directly from Jupyter. To do this, go to Jupyter Notebook and click on the zip file (in this case assignment2.zip). The file will be downloaded to your local computer. You can also use the **gcloud compute copy-files** command to transfer files as discussed in the **Submission: Transferring Files From Your Instance To Your Computer** section in [the first GCE tutorial](http://cs231n.github.io/gce-tutorial/ "title"). 122 | 123 | # BIG REMINDER: Make sure you stop your instances! # 124 | 125 | Don't forget to stop your instance when you are done (**by clicking on the stop button at the top of the page showing your instances**). You can restart your instance and the downloaded software will still be available. We have already had some students who left their instances running for many days and have ran out of credits. You will be charged per hour when your instance is running. This includes code development time. We encourage you to read up on Google Cloud, regularly keep track of your credits and not solely rely on our tutorials. 126 | -------------------------------------------------------------------------------- /assignments/2017/assignment3.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | mathjax: true 4 | permalink: /assignments2017/assignment3/ 5 | --- 6 | 7 | In this assignment you will implement recurrent networks, and apply them to image captioning on Microsoft COCO. You will also explore methods for visualizing the features of a pretrained model on ImageNet, and also this model to implement Style Transfer. Finally, you will train a generative adversarial network to generate images that look like a training dataset! 8 | 9 | The goals of this assignment are as follows: 10 | 11 | - Understand the architecture of *recurrent neural networks (RNNs)* and how they operate on sequences by sharing weights over time 12 | - Understand and implement both Vanilla RNNs and Long-Short Term Memory (LSTM) RNNs 13 | - Understand how to sample from an RNN language model at test-time 14 | - Understand how to combine convolutional neural nets and recurrent nets to implement an image captioning system 15 | - Understand how a trained convolutional network can be used to compute gradients with respect to the input image 16 | - Implement and different applications of image gradients, including saliency maps, fooling images, class visualizations. 17 | - Understand and implement style transfer. 18 | - Understand how to train and implement a generative adversarial network (GAN) to produce images that look like a dataset. 19 | 20 | ## Setup 21 | You can work on the assignment in one of two ways: locally on your own machine, or on a virtual machine on Google Cloud. 22 | 23 | ### Working remotely on Google Cloud (Recommended) 24 | 25 | **Note:** after following these instructions, make sure you go to **Working on the assignment** below (you can skip the **Working locally** section). 26 | 27 | As part of this course, you can use Google Cloud for your assignments. We recommend this route for anyone who is having trouble with installation set-up, or if you would like to use better CPU/GPU resources than you may have locally. 28 | 29 | #### GPU Resources 30 | GPUs are **not required** for this assignment, but will help to speed up training and processing time for questions 3-5. Please see the Google Cloud GPU set-up tutorial [here](http://cs231n.github.io/gce-tutorial-gpus/) for instructions. The GPU instances are much more expensive, so use them only when needed. **We only recommend a GPU for Q5 (GANs)**, Q3 and Q4 run in under a minute even on CPU, while Q5 can take 30+ minutes for your model to converge on a CPU (versus under a minute on the Google Cloud GPUs). 31 | 32 | Once you've got the cloud instance running, make sure to run the following line to enter the virtual environment that we prepared for you (you do **not** need to make your own virtual environment): 33 | 34 | ```bash 35 | source /home/cs231n/myVE35/bin/activate 36 | ``` 37 | 38 | We recommend using Google Cloud with GPU support for the question 5 of this assignment (the GAN notebook), since your training will go much, much faster. However, it will not help at all for questions 1 and 2 (RNN and LSTM), and questions 3 and 4 are still fast on CPU (these notebook should run in a few minutes). 39 | 40 | #### What do I do if my Google Cloud GPUs disappeared? 41 | You might note that sometimes, your GPUs are no longer accessible on your Google Cloud instance after you restart it. If this happens, please run the following commands in your assignment3 directory: 42 | 43 | ```bash 44 | sudo apt-get remove unattended-upgrades 45 | chmod u+x where_are_my_drivers.sh 46 | ./where_are_my_drivers.sh 47 | ``` 48 | 49 | If this isn't working, you can find more detailed instructions and manual ways of fixing this [here](https://cloud.google.com/compute/docs/gpus/add-gpus#install-driver-script). You should follow the "Ubuntu 16.04" instructions. 50 | 51 | ### Working locally 52 | Here's how you install the necessary dependencies: 53 | 54 | **(OPTIONAL) Installing GPU drivers:** 55 | If you choose to work locally, you are at no disadvantage for the first parts of the assignment. For the last question, which is in TensorFlow or PyTorch, however, having a GPU will be a significant advantage. We recommend using a Google Cloud Instance with a GPU, at least for this part. If you have your own NVIDIA GPU, however, and wish to use that, that's fine -- you'll need to install the drivers for your GPU, install CUDA, install cuDNN, and then install either [TensorFlow](https://www.tensorflow.org/install/) or [PyTorch](http://pytorch.org/). You could theoretically do the entire assignment with no GPUs, though this will make training much slower in the last part. However, our reference code runs in 10-15 minutes on a dual-core laptop without a GPU, so it is certainly possible. 56 | 57 | **Installing Python 3.5+:** 58 | To use python3, make sure to install version 3.5 or 3.6 on your local machine. If you are on Mac OS X, you can do this using [Homebrew](https://brew.sh) with `brew install python3`. You can find instructions for Ubuntu [here](https://www.digitalocean.com/community/tutorials/how-to-install-python-3-and-set-up-a-local-programming-environment-on-ubuntu-16-04). 59 | 60 | **Virtual environment:** 61 | If you decide to work locally, we recommend using [virtual environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/) for the project. If you choose not to use a virtual environment, it is up to you to make sure that all dependencies for the code are installed globally on your machine. To set up a virtual environment, run the following: 62 | 63 | ```bash 64 | cd assignment2 65 | sudo pip install virtualenv # This may already be installed 66 | virtualenv -p python3 .env # Create a virtual environment (python3) 67 | source .env/bin/activate # Activate the virtual environment 68 | pip install -r requirements.txt # Install dependencies 69 | # Note that this does NOT install TensorFlow or PyTorch, 70 | # which you need to do yourself. 71 | 72 | # Work on the assignment for a while ... 73 | # ... and when you're done: 74 | deactivate # Exit the virtual environment 75 | ``` 76 | 77 | Note that every time you want to work on the assignment, you should run `source .env/bin/activate` (from within your `assignment3` folder) to re-activate the virtual environment, and `deactivate` again whenever you are done. 78 | 79 | ## Working on the assignment: 80 | 81 | ### Get the code as a zip file [here](http://cs231n.stanford.edu/assignments/2017/spring1617_assignment3_v3.zip). 82 | 83 | ### Download data: 84 | Once you have the starter code (regardless of which method you choose above), you will need to download the COCO captioning data, pretrained SqueezeNet model (TensorFlow-only), and a few ImageNet validation images. 85 | Run the following from the `assignment3` directory: 86 | 87 | ```bash 88 | cd cs231n/datasets 89 | ./get_assignment3_data.sh 90 | ``` 91 | 92 | ### Start IPython: 93 | After you have downloaded the data, you should start the IPython notebook server from the 94 | `assignment3` directory, with the `jupyter notebook` command. (See the [Google Cloud Tutorial](http://cs231n.github.io/gce-tutorial/) for any additional steps you may need to do for setting this up, if you are working remotely) 95 | 96 | If you are unfamiliar with IPython, you can also refer to our 97 | [IPython tutorial](/ipython-tutorial). 98 | 99 | ### Some Notes 100 | **NOTE 1:** This year, the `assignment3` code has been tested to be compatible with python versions `3.5` and `3.6` (it may work with other versions of `3.x`, but we won't be officially supporting them). For this assignment, we are NOT officially supporting python2. Use it at your own risk. You will need to make sure that during your `virtualenv` setup that the correct version of `python` is used. You can confirm your python version by (1) activating your virtualenv and (2) running `python --version`. 101 | 102 | **NOTE 2:** If you are working in a virtual environment on OSX, you may *potentially* encounter 103 | errors with matplotlib due to the [issues described here](http://matplotlib.org/faq/virtualenv_faq.html). In our testing, it seems that this issue is no longer present with the most recent version of matplotlib, but if you do end up running into this issue you may have to use the `start_ipython_osx.sh` script from the `assignment3` directory (instead of `jupyter notebook` above) to launch your IPython notebook server. Note that you may have to modify some variables within the script to match your version of python/installation directory. The script assumes that your virtual environment is named `.env`. 104 | 105 | ### Submitting your work: 106 | Whether you work on the assignment locally or using Google Cloud, once you are done 107 | working run the `collectSubmission.sh` script; this will produce a file called 108 | `assignment3.zip`. Please submit this file on [Canvas](https://canvas.stanford.edu/courses/66461/). 109 | 110 | #### You can do Questions 3, 4, and 5 in TensorFlow or PyTorch. There are two versions of each notebook, with suffixes -TensorFlow or -PyTorch. No extra credit will be awarded if you do a question in both TensorFlow and PyTorch. 111 | 112 | ### Q1: Image Captioning with Vanilla RNNs (25 points) 113 | The Jupyter notebook `RNN_Captioning.ipynb` will walk you through the 114 | implementation of an image captioning system on MS-COCO using vanilla recurrent 115 | networks. 116 | 117 | ### Q2: Image Captioning with LSTMs (30 points) 118 | The Jupyter notebook `LSTM_Captioning.ipynb` will walk you through the 119 | implementation of Long-Short Term Memory (LSTM) RNNs, and apply them to image 120 | captioning on MS-COCO. 121 | 122 | ### Q3: Network Visualization: Saliency maps, Class Visualization, and Fooling Images (15 points) 123 | The Jupyter notebooks `NetworkVisualization-TensorFlow.ipynb` /`NetworkVisualization-PyTorch.ipynb` will introduce the pretrained SqueezeNet model, compute gradients 124 | with respect to images, and use them to produce saliency maps and fooling 125 | images. Please complete only one of the notebooks (TensorFlow or PyTorch). No extra credit will be awardeded if you complete both notebooks. 126 | 127 | ### Q4: Style Transfer (15 points) 128 | In the Jupyter notebooks `StyleTransfer-TensorFlow.ipynb`/`StyleTransfer-PyTorch.ipynb` you will learn how to create images with the content of one image but the style of another. Please complete only one of the notebooks (TensorFlow or PyTorch). No extra credit will be awardeded if you complete both notebooks. 129 | 130 | ### Q5: Generative Adversarial Networks (15 points) 131 | In the Jupyter notebooks `GANs-TensorFlow.ipynb`/`GANs-PyTorch.ipynb` you will learn how to generate images that match a training dataset, and use these models to improve classifier performance when training on a large amount of unlabeled data and a small amount of labeled data. Please complete only one of the notebooks (TensorFlow or PyTorch). No extra credit will be awarded if you complete both notebooks. 132 | -------------------------------------------------------------------------------- /css/main.css: -------------------------------------------------------------------------------- 1 | /* Base */ 2 | /* ----------------------------------------------------------*/ 3 | 4 | * { 5 | margin: 0; 6 | padding: 0; 7 | } 8 | 9 | html, body { height: 100%; } 10 | 11 | body { 12 | /* font-family: Helvetica, Arial, sans-serif; */ 13 | font-family: 'Roboto', sans-serif; 14 | font-size: 16px; 15 | line-height: 1.5; 16 | font-weight: 300; 17 | background-color: #fdfdfd; 18 | } 19 | 20 | h1, h2, h3, h4, h5, h6 { font-size: 100%; font-weight: 400; } 21 | 22 | a { color: #2a7ae2; text-decoration: none; } 23 | a:hover { color: #000; text-decoration: underline; } 24 | a:visited { color: #205caa; } 25 | 26 | /* Utility */ 27 | 28 | .wrap:before, 29 | .wrap:after { content:""; display:table; } 30 | .wrap:after { clear: both; } 31 | .wrap { 32 | max-width: 800px; 33 | padding: 0 30px; 34 | margin: 0 auto; 35 | zoom: 1; 36 | } 37 | 38 | 39 | /* Layout Styles */ 40 | /* ----------------------------------------------------------*/ 41 | 42 | /* Custom CSS rules for index page */ 43 | 44 | .module-header { 45 | font-size: 24px; 46 | color: #8C1515; 47 | margin-top: 20px; 48 | margin-bottom: 5px; 49 | } 50 | 51 | .materials-wrap { 52 | font-size: 18px; 53 | } 54 | .materials-item a{ 55 | color: #333; 56 | display: block; 57 | padding: 3px; 58 | } 59 | .materials-item { 60 | border-bottom: 1px solid #CCC; 61 | } 62 | .materials-item:nth-child(even) { 63 | background-color: #f7f6f1; 64 | } 65 | 66 | /* Custom CSS rules for content */ 67 | 68 | .embedded-video { 69 | text-align: center; 70 | background-color: #f7f6f1; 71 | padding: 20px 0px 20px 0px; 72 | border-bottom: 1px solid #999; 73 | border-top: 1px solid #999; 74 | } 75 | 76 | /* Custom CSS for pages */ 77 | .figcenter { 78 | text-align: center; 79 | } 80 | .fig img { 81 | max-width: 98%; 82 | } 83 | .figleft img { 84 | max-width: 50%; 85 | float: left; 86 | margin-right: 20px; 87 | } 88 | .figleft svg { 89 | float: left; 90 | margin-right: 20px; 91 | } 92 | .fighighlight { 93 | /* background-color: #f7f6f1; */ 94 | padding: 20px 4px 20px 4px; 95 | border-bottom: 1px solid #999; 96 | border-top: 1px solid #999; 97 | /* box-shadow: 0px 2px 5px #CCC, 0px -2px 5px #CCC; */ 98 | } 99 | .figcaption { 100 | font-weight: 400; 101 | font-size: 14px; 102 | color: #575651; 103 | text-align: justify; 104 | } 105 | 106 | .post-content p { 107 | text-align: justify; 108 | } 109 | 110 | .kw { 111 | font-size: 16px; 112 | color: #08A; 113 | margin-left: 10px; 114 | } 115 | 116 | .notyet { 117 | background-color: #FEE !important; 118 | } 119 | 120 | /* Site header */ 121 | 122 | .title-wrap { 123 | text-align: center; 124 | } 125 | 126 | .site-header { 127 | border-bottom: 1px solid #e8e8e8; 128 | min-height: 56px; 129 | background-color: #8C1515; 130 | } 131 | 132 | .site-title, 133 | .site-title:hover, 134 | .site-title:visited { 135 | display: block; 136 | font-size: 26px; 137 | letter-spacing: -1px; 138 | line-height: 56px; 139 | position: relative; 140 | z-index: 1; 141 | color: #FFF; 142 | font-weight: 100; 143 | } 144 | 145 | .site-nav { 146 | float: right; 147 | line-height: 56px; 148 | } 149 | 150 | .site-nav .menu-icon { display: none; } 151 | 152 | .site-nav .page-link { 153 | margin-left: 20px; 154 | color: #727272; 155 | letter-spacing: -.5px; 156 | } 157 | 158 | /* Site footer */ 159 | 160 | .site-footer { 161 | border-top: 1px solid #e8e8e8; 162 | padding: 30px 0; 163 | } 164 | 165 | .footer-heading { 166 | font-size: 18px; 167 | font-weight: 300; 168 | letter-spacing: -.5px; 169 | margin-bottom: 15px; 170 | } 171 | 172 | .site-footer .column { float: left; margin-bottom: 15px; } 173 | 174 | .footer-col-1 { 175 | width: 270px; /*fallback*/ 176 | width: -webkit-calc(35% - 10px); 177 | width: -moz-calc(35% - 10px); 178 | width: -o-calc(35% - 10px); 179 | width: calc(35% - 10px); 180 | margin-right: 10px 181 | } 182 | .footer-col-2 { 183 | width: 175px; /*fallback*/ 184 | width: -webkit-calc(23.125% - 10px); 185 | width: -moz-calc(23.125% - 10px); 186 | width: -o-calc(23.125% - 10px); 187 | width: calc(23.125% - 10px); 188 | margin-right: 10px 189 | } 190 | .footer-col-3 { 191 | width: 335px; /*fallback*/ 192 | width: -webkit-calc(41.875%); 193 | width: -moz-calc(41.875%); 194 | width: -o-calc(41.875%); 195 | width: calc(41.875%); 196 | } 197 | 198 | .site-footer ul { list-style: none; } 199 | 200 | .site-footer li, 201 | .site-footer p { 202 | font-size: 15px; 203 | letter-spacing: -.3px; 204 | color: #828282; 205 | } 206 | 207 | .github-icon-svg, 208 | .twitter-icon-svg { 209 | display: inline-block; 210 | width: 16px; 211 | height: 16px; 212 | position: relative; 213 | top: 3px; 214 | } 215 | 216 | 217 | /* Page Content styles */ 218 | /* ----------------------------------------------------------*/ 219 | 220 | .page-content { 221 | padding: 30px 0; 222 | background-color: #fff; 223 | } 224 | 225 | 226 | /* Home styles */ 227 | /* ----------------------------------------------------------*/ 228 | 229 | .home h1 { margin-bottom: 25px; } 230 | 231 | .posts { list-style-type: none; } 232 | 233 | .posts li { margin-bottom: 30px; } 234 | 235 | .posts .post-link { 236 | font-size: 24px; 237 | letter-spacing: -1px; 238 | line-height: 1; 239 | } 240 | 241 | .posts .post-date { 242 | display: block; 243 | font-size: 15px; 244 | color: #818181; 245 | } 246 | 247 | 248 | /* Post styles */ 249 | /* ----------------------------------------------------------*/ 250 | 251 | .post-header { margin: 10px 0 30px; } 252 | 253 | .post-header h1 { 254 | font-size: 30px; 255 | letter-spacing: -1.75px; 256 | line-height: 1; 257 | font-weight: 300; 258 | } 259 | 260 | .post-header .meta { 261 | font-size: 15px; 262 | color: #818181; 263 | margin-top: 5px; 264 | } 265 | 266 | .post-content { margin: 0 0 30px; } 267 | 268 | .post-content > * { margin: 20px 0; } 269 | 270 | 271 | .post-content h1, 272 | .post-content h2, 273 | .post-content h3, 274 | .post-content h4, 275 | .post-content h5, 276 | .post-content h6 { 277 | line-height: 1; 278 | font-weight: 300; 279 | margin: 40px 0 20px; 280 | } 281 | 282 | .post-content h2 { 283 | font-size: 32px; 284 | letter-spacing: -1.25px; 285 | } 286 | 287 | .post-content h3 { 288 | font-size: 26px; 289 | letter-spacing: -1px; 290 | } 291 | 292 | .post-content h4 { 293 | font-size: 20px; 294 | letter-spacing: -1px; 295 | } 296 | 297 | .post-content blockquote { 298 | border-left: 4px solid #e8e8e8; 299 | padding-left: 20px; 300 | font-size: 18px; 301 | opacity: .6; 302 | letter-spacing: -1px; 303 | font-style: italic; 304 | margin: 30px 0; 305 | } 306 | 307 | .post-content ul, 308 | .post-content ol { padding-left: 20px; } 309 | 310 | .post pre, 311 | .post code { 312 | border: 1px solid #d5d5e9; 313 | background-color: #eef; 314 | padding: 8px 12px; 315 | -webkit-border-radius: 3px; 316 | -moz-border-radius: 3px; 317 | border-radius: 3px; 318 | font-size: 15px; 319 | overflow:scroll; 320 | } 321 | 322 | .post code { padding: 1px 5px; } 323 | 324 | .post ul, 325 | .post ol { margin-left: 1.35em; } 326 | 327 | .post pre code { 328 | border: 0; 329 | padding-right: 0; 330 | padding-left: 0; 331 | } 332 | 333 | /* terminal */ 334 | .post pre.terminal { 335 | border: 1px solid #000; 336 | background-color: #333; 337 | color: #FFF; 338 | -webkit-border-radius: 3px; 339 | -moz-border-radius: 3px; 340 | border-radius: 3px; 341 | } 342 | 343 | .post pre.terminal code { background-color: #333; } 344 | 345 | /* Syntax highlighting styles */ 346 | /* ----------------------------------------------------------*/ 347 | 348 | .highlight { background: #ffffff; } 349 | .highlight .c { color: #999988; font-style: italic } /* Comment */ 350 | .highlight .err { color: #a61717; background-color: #e3d2d2 } /* Error */ 351 | .highlight .k { font-weight: bold } /* Keyword */ 352 | .highlight .o { font-weight: bold } /* Operator */ 353 | .highlight .cm { color: #999988; font-style: italic } /* Comment.Multiline */ 354 | .highlight .cp { color: #999999; font-weight: bold } /* Comment.Preproc */ 355 | .highlight .c1 { color: #999988; font-style: italic } /* Comment.Single */ 356 | .highlight .cs { color: #999999; font-weight: bold; font-style: italic } /* Comment.Special */ 357 | .highlight .gd { color: #000000; background-color: #ffdddd } /* Generic.Deleted */ 358 | .highlight .gd .x { color: #000000; background-color: #ffaaaa } /* Generic.Deleted.Specific */ 359 | .highlight .ge { font-style: italic } /* Generic.Emph */ 360 | .highlight .gr { color: #aa0000 } /* Generic.Error */ 361 | .highlight .gh { color: #999999 } /* Generic.Heading */ 362 | .highlight .gi { color: #000000; background-color: #ddffdd } /* Generic.Inserted */ 363 | .highlight .gi .x { color: #000000; background-color: #aaffaa } /* Generic.Inserted.Specific */ 364 | .highlight .go { color: #888888 } /* Generic.Output */ 365 | .highlight .gp { color: #555555 } /* Generic.Prompt */ 366 | .highlight .gs { font-weight: bold } /* Generic.Strong */ 367 | .highlight .gu { color: #aaaaaa } /* Generic.Subheading */ 368 | .highlight .gt { color: #aa0000 } /* Generic.Traceback */ 369 | .highlight .kc { font-weight: bold } /* Keyword.Constant */ 370 | .highlight .kd { font-weight: bold } /* Keyword.Declaration */ 371 | .highlight .kp { font-weight: bold } /* Keyword.Pseudo */ 372 | .highlight .kr { font-weight: bold } /* Keyword.Reserved */ 373 | .highlight .kt { color: #445588; font-weight: bold } /* Keyword.Type */ 374 | .highlight .m { color: #009999 } /* Literal.Number */ 375 | .highlight .s { color: #d14 } /* Literal.String */ 376 | .highlight .na { color: #008080 } /* Name.Attribute */ 377 | .highlight .nb { color: #0086B3 } /* Name.Builtin */ 378 | .highlight .nc { color: #445588; font-weight: bold } /* Name.Class */ 379 | .highlight .no { color: #008080 } /* Name.Constant */ 380 | .highlight .ni { color: #800080 } /* Name.Entity */ 381 | .highlight .ne { color: #990000; font-weight: bold } /* Name.Exception */ 382 | .highlight .nf { color: #990000; font-weight: bold } /* Name.Function */ 383 | .highlight .nn { color: #555555 } /* Name.Namespace */ 384 | .highlight .nt { color: #000080 } /* Name.Tag */ 385 | .highlight .nv { color: #008080 } /* Name.Variable */ 386 | .highlight .ow { font-weight: bold } /* Operator.Word */ 387 | .highlight .w { color: #bbbbbb } /* Text.Whitespace */ 388 | .highlight .mf { color: #009999 } /* Literal.Number.Float */ 389 | .highlight .mh { color: #009999 } /* Literal.Number.Hex */ 390 | .highlight .mi { color: #009999 } /* Literal.Number.Integer */ 391 | .highlight .mo { color: #009999 } /* Literal.Number.Oct */ 392 | .highlight .sb { color: #d14 } /* Literal.String.Backtick */ 393 | .highlight .sc { color: #d14 } /* Literal.String.Char */ 394 | .highlight .sd { color: #d14 } /* Literal.String.Doc */ 395 | .highlight .s2 { color: #d14 } /* Literal.String.Double */ 396 | .highlight .se { color: #d14 } /* Literal.String.Escape */ 397 | .highlight .sh { color: #d14 } /* Literal.String.Heredoc */ 398 | .highlight .si { color: #d14 } /* Literal.String.Interpol */ 399 | .highlight .sx { color: #d14 } /* Literal.String.Other */ 400 | .highlight .sr { color: #009926 } /* Literal.String.Regex */ 401 | .highlight .s1 { color: #d14 } /* Literal.String.Single */ 402 | .highlight .ss { color: #990073 } /* Literal.String.Symbol */ 403 | .highlight .bp { color: #999999 } /* Name.Builtin.Pseudo */ 404 | .highlight .vc { color: #008080 } /* Name.Variable.Class */ 405 | .highlight .vg { color: #008080 } /* Name.Variable.Global */ 406 | .highlight .vi { color: #008080 } /* Name.Variable.Instance */ 407 | .highlight .il { color: #009999 } /* Literal.Number.Integer.Long */ 408 | 409 | /* media queries */ 410 | /* ----------------------------------------------------------*/ 411 | 412 | 413 | @media screen and (max-width: 750px) { 414 | 415 | .footer-col-1 { width: 50%; } 416 | 417 | .footer-col-2 { 418 | width: 45%; /*fallback*/ 419 | width: -webkit-calc(50% - 10px); 420 | width: -moz-calc(50% - 10px); 421 | width: -o-calc(50% - 10px); 422 | width: calc(50% - 10px); 423 | margin-right: 0; 424 | } 425 | 426 | .site-footer .column.footer-col-3 { 427 | width: auto; 428 | float: none; 429 | clear: both; 430 | } 431 | 432 | } 433 | 434 | @media screen and (max-width: 600px) { 435 | 436 | .wrap { padding: 0 12px; } 437 | 438 | .site-nav { 439 | position: fixed; 440 | z-index: 10; 441 | top: 14px; right: 8px; 442 | background-color: white; 443 | -webkit-border-radius: 5px; 444 | -moz-border-radius: 5px; 445 | border-radius: 5px; 446 | border: 1px solid #e8e8e8; 447 | } 448 | 449 | .site-nav .menu-icon { 450 | display: block; 451 | font-size: 24px; 452 | color: #505050; 453 | float: right; 454 | width: 36px; 455 | text-align: center; 456 | line-height: 36px; 457 | } 458 | 459 | .site-nav .menu-icon svg { width: 18px; height: 16px; } 460 | 461 | .site-nav .trigger { 462 | clear: both; 463 | margin-bottom: 5px; 464 | display: none; 465 | } 466 | 467 | .site-nav:hover .trigger { display: block; } 468 | 469 | .site-nav .page-link { 470 | display: block; 471 | text-align: right; 472 | line-height: 1.25; 473 | padding: 5px 10px; 474 | margin: 0; 475 | } 476 | 477 | .post-header h1 { font-size: 36px; } 478 | .post-content h2 { font-size: 28px; } 479 | .post-content h3 { font-size: 22px; } 480 | .post-content h4 { font-size: 18px; } 481 | .post-content blockquote { padding-left: 10px; } 482 | .post-content ul, 483 | .post-content ol { padding-left: 10px; } 484 | 485 | .site-footer .column { 486 | float: none; 487 | clear: both; 488 | width: auto; 489 | margin: 0 0 15px; } 490 | 491 | } 492 | -------------------------------------------------------------------------------- /google_cloud_tutorial.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | title: Google Cloud Tutorial 4 | permalink: /gce-tutorial/ 5 | --- 6 | # Google Cloud Tutorial # 7 | 8 | ## BEFORE WE BEGIN ## 9 | ### BIG REMINDER: Make sure you stop your instances! ### 10 | (We know you won't read until the very bottom once your assignment is running, so we are printing this at the top too since it is ***super important***) 11 | 12 | Don't forget to ***stop your instance*** when you are done (by clicking on the stop button at the top of the page showing your instances), otherwise you will ***run out of credits*** and that will be very sad. :( 13 | 14 | If you follow our instructions below correctly, you should be able to restart your instance and the downloaded software will still be available. 15 | 16 |
17 | 18 |
19 | 20 | 21 | ## Create and Configure Your Account ## 22 | For the class project and assignments, we offer an option to use Google Compute Engine for developing and testing your 23 | implementations. This tutorial lists the necessary steps of working on the assignments using Google Cloud. **We expect this tutorial to take about an hour. Don't get intimidated by the steps, we tried to make the tutorial detailed so that you are less likely to get stuck on a particular step. Please tag all questions related to Google Cloud with google_cloud on Piazza.** 24 | 25 | This tutorial goes through how to set up your own Google Compute Engine (GCE) instance to work on the assignments. Each student will have $100 in credit throughout the quarter. When you sign up for the first time, you also receive $300 credits from Google by default. Please try to use the resources judiciously. But if $100 ends up not being enough, we will try to adjust this number as the quarter goes on. **Note: for assignment 1, we are only supporting python version 2.7 (the default installation from the script) and 3.5.3**. 26 | 27 | First, if you don't have a Google Cloud account already, create one by going to the [Google Cloud homepage](https://cloud.google.com/?utm_source=google&utm_medium=cpc&utm_campaign=2015-q2-cloud-na-gcp-skws-freetrial-en&gclid=CP2e4PPpiNMCFU9bfgodGHsA1A "Title") and clicking on **Compute**. When you get to the next page, click on the blue **TRY IT FREE** button. If you are not logged into gmail, you will see a page that looks like the one below. Sign into your gmail account or create a new one if you do not already have an account. 28 | 29 |
30 | 31 |
32 | 33 | If you already have a gmail account, it will direct you to a signup page which looks like the following. 34 |
35 | 36 |
37 | 38 | Click the appropriate **yes** or **no** button for the first option, and check **yes** for the latter two options after you have read the required agreements. Press the blue **Agree and continue** button to continue to the next page to enter the requested information (your name, billing address and credit card information). Once you have entered the required information, press the blue **Start my free trial** button. You will be greeted by a page like this: 39 | 40 |
41 | 42 |
43 | 44 | To change the name of your project, click on **Manage project settings** on the **Project info** button and save your changes. 45 | 46 |
47 | 48 |
49 | 50 | ## Launch a Virtual Instance ## 51 | To launch a virtual instance, go to the **Compute Engine** menu on the left column of your dashboard and click on **VM instances**. Then click on the blue **CREATE** button on the next page. This will take you to a page that looks like the screenshot below. **(NOTE: Please carefully read the instructions in addition to looking at the screenshots. The instructions tell you exactly what values to fill in).** 52 | 53 |
54 | 55 |
56 | 57 | Make sure that the Zone is set to be **us-west1-b** (especially for assignments where you need to use GPU instances). Under **Machine type** pick the **8 vCPUs** option. Click on the **customize** button under **Machine type** and make sure that the number of cores is set to 8 and the number of GPUs is set to **None** (we will not be using GPUs in assignment 1 and this tutorial will be updated with instructions for GPU usage). Click on the **Change** button under **Boot disk**, choose **OS images**, check **Ubuntu 16.04 LTS** and click on the blue **select** button. Check **Allow HTTP traffic** and **Allow HTTPS traffic**. Click on **disk** and then **Disks** and uncheck **Delete boot disk when instance is deleted** (Note that the "Disks" option may be hiding under an expandable URL at the bottom of that webform). Click on the blue **Create** button at the bottom of the page. You should have now successfully created a Google Compute Instance, it might take a few minutes to start running. Your screen should look something like the one below. When you want to stop running the instance, click on the blue stop button above. 58 | 59 |
60 | 61 |
62 | 63 | Take note of your \, in this case, my instance name is instance-2. 64 | 65 | ## Connect to Your Virtual Instance and Download the Assignment ## 66 | Now that you have created your virtual GCE, you want to be able to connect to it from your computer. The rest of this tutorial goes over how to do that using the command line. First, download the Google Cloud SDK that is appropriate for your platform from [here](https://cloud.google.com/sdk/docs/ "Title") and follow their installation instructions. **NOTE: this tutorial assumes that you have performed step #4 on the website which they list as optional**. When prompted, make sure you select us-west1-b as the time zone. The easiest way to connect is using the gcloud compute command below. The tool takes care of authentication for you. On OS X, run: 67 | 68 | ``` 69 | .//bin/gcloud compute ssh --zone=us-west1-b 70 | ``` 71 | 72 | See [this page](https://cloud.google.com/compute/docs/instances/connecting-to-instance "Title") for more detailed instructions. You are now ready to work on the assignments on Google Cloud. 73 | 74 | Run the following command to download the current assignment onto your GCE: 75 | 76 | ``` 77 | wget http://cs231n.stanford.edu/assignments/2017/spring1617_assignment1.zip 78 | ``` 79 | 80 | Then run: 81 | 82 | ``` 83 | sudo apt-get install unzip 84 | ``` 85 | 86 | and 87 | 88 | ``` 89 | unzip spring1617_assignment1.zip 90 | ``` 91 | 92 | to get the contents. You should now see a folder titled assignment**X**. To install the necessary dependencies for assignment 1 (**NOTE:** you only need to do this for assignment 1), cd into the assignment directory and run the provided shell script: **(Note: you will need to hit the [*enter*] key at all the "[Y/n]" prompts)** 93 | 94 | ``` 95 | cd assignment1 96 | ./setup_googlecloud.sh 97 | ``` 98 | 99 | You will be prompted to enter Y/N at various times during the download. Press enter for every prompt. You should now have all the software you need for assignment**X**. If you had no errors, you can proceed to work with your virtualenv as normal. 100 | 101 | I.e. run 102 | 103 | ``` 104 | source .env/bin/activate 105 | ``` 106 | 107 | in your assignment directory to load the venv, and run 108 | 109 | ``` 110 | deactivate 111 | ``` 112 | to exit the venv. See assignment handout for details. 113 | 114 | **NOTE**: The instructions above will run everything needed using Python 2.7. If you would like to use Python 3.5 instead, edit setup_googlecloud.sh to replce the line 115 | 116 | ``` 117 | virtualenv .env 118 | ``` 119 | 120 | with 121 | 122 | ``` 123 | virtualenv -p python3 .env 124 | ``` 125 | 126 | before running 127 | 128 | ``` 129 | ./setup_googlecloud.sh 130 | ``` 131 | 132 | ## Using Jupyter Notebook with Google Compute Engine ## 133 | Many of the assignments will involve using Jupyter Notebook. Below, we discuss how to run Jupyter Notebook from your GCE instance and use it on your local browser. 134 | 135 | ### Getting a Static IP Address ### 136 | Change the Extenal IP address of your GCE instance to be static (see screenshot below). 137 |
138 | 139 |
140 | 141 | To Do this, click on the 3 line icon next to the **Google Cloud Platform** button on the top left corner of your screen, go to **Networking** and **External IP addresses** (see screenshot below). 142 | 143 |
144 | 145 |
146 | 147 | To have a static IP address, change **Type** from **Ephemeral** to **Static**. Enter your preffered name for your static IP, mine is assignment-1 (see screenshot below). And click on Reserve. Remember to release the static IP address when you are done because according to [this page](https://jeffdelaney.me/blog/running-jupyter-notebook-google-cloud-platform/ "Title") Google charges a small fee for unused static IPs. **Type** should now be set to **Static**. 148 | 149 |
150 | 151 |
152 | 153 | Take note of your Static IP address (circled on the screenshot below). I used 104.196.224.11 for this tutorial. 154 | 155 |
156 | 157 |
158 | 159 | ### Adding a Firewall rule ### 160 | One last thing you have to do is adding a new firewall rule allowing TCP acess to a particular \. I usually use 7000 or 8000 for \. Click on the 3 line icon at the top of the page next to **Google Cloud Platform**. On the menu that pops up on the left column, go to **Networking** and **Firewall rules** (see the screenshot below). 161 | 162 |
163 | 164 |
165 | 166 | Click on the blue **CREATE FIREWALL RULE** button. Enter whatever name you want: I used assignment1-rules. Enter 0.0.0.0/0 for **Source IP ranges** and tcp:\ for **Allowed protocols and ports** where \ is the number you used above. Click on the blue **Create** button. See the screen shot below. 167 | 168 |
169 | 170 |
171 | 172 | **NOTE:** Some people are seeing a different screen where instead of **Allowed protocols and ports** there is a field titled **Specified protocols and ports**. You should enter tcp:\ for this field if this is the page you see. Also, if you see a field titled **Targets** select **All instances in the network**. 173 | 174 | ### Configuring Jupyter Notebook ### 175 | The following instructions are excerpts from [this page](https://haroldsoh.com/2016/04/28/set-up-anaconda-ipython-tensorflow-julia-on-a-google-compute-engine-vm/ "Title") that has more detailed instructions. 176 | 177 | On your GCE instance check where the Jupyter configuration file is located: 178 | 179 | ``` 180 | ls ~/.jupyter/jupyter_notebook_config.py 181 | ``` 182 | Mine was in /home/timnitgebru/.jupyter/jupyter_notebook_config.py 183 | 184 | If it doesn’t exist, create one: 185 | 186 | ``` 187 | # Remember to activate your virtualenv ('source .env/bin/activate') so you can actually run jupyter :) 188 | jupyter notebook --generate-config 189 | ``` 190 | 191 | Using your favorite editor (vim, emacs etc...) add the following lines to the config file, (e.g.: /home/timnitgebru/.jupyter/jupyter_notebook_config.py): 192 | 193 | ``` 194 | c = get_config() 195 | 196 | c.NotebookApp.ip = '*' 197 | 198 | c.NotebookApp.open_browser = False 199 | 200 | c.NotebookApp.port = 201 | ``` 202 | 203 | Where \ is the same number you used in the prior section. Save your changes and close the file. 204 | 205 | ### Launching and connecting to Jupyter Notebook ### 206 | The instructions below assume that you have SSH'd into your GCE instance using the prior instructions, have already downloaded and unzipped the current assignment folder into assignment**X** (where X is the assignment number), and have successfully configured Jupyter Notebook. 207 | 208 | 209 | If you are not already in the assignment directory, cd into it by running the following command: 210 | 211 | ``` 212 | cd assignment1 213 | ``` 214 | If you haven't already done so, activate your virtualenv by running: 215 | 216 | ``` 217 | source .env/bin/activate 218 | ``` 219 | 220 | Launch Jupyter notebook using: 221 | 222 | ``` 223 | jupyter-notebook --no-browser --port= 224 | ``` 225 | 226 | Where \ is what you wrote in the prior section. 227 | 228 | On your local browser, if you go to http://\:\, you should see something like the screen below. My value for \ was 104.196.224.11 as mentioned above. You should now be able to start working on your assignments. 229 | 230 |
231 | 232 |
233 | 234 | ## Submission: Transferring Files From Your Instance To Your Computer ## 235 | Once you are done with your assignments, run the submission script in your assignment folder. For assignment1, this will create a zip file called `assignment1.zip` containing the files you need to upload to Canvas. If you're not in the assignment1 directory already, CD into it by running 236 | 237 | ``` 238 | cd assignment1 239 | ``` 240 | 241 | install **zip** by running 242 | ``` 243 | sudo apt-get install zip 244 | ``` 245 | 246 | and then run 247 | 248 | ``` 249 | bash collectSubmission.sh 250 | ``` 251 | 252 | to create the zip file that you need to upload to canvas. Then copy the file to your local computer using the gcloud compute copy-file command as shown below. **NOTE: run this command on your local computer**: 253 | 254 | ``` 255 | gcloud compute copy-files [INSTANCE_NAME]:[REMOTE_FILE_PATH] [LOCAL_FILE_PATH] 256 | ``` 257 | 258 | For example, to copy my files to my desktop I ran: 259 | 260 | ``` 261 | gcloud compute copy-files instance-2:~/assignment1/assignment1.zip ~/Desktop 262 | ``` 263 | Another (perhaps easier) option proposed by a student is to directly download the zip file from Jupyter. After running the submission script and creating assignment1.zip, you can download that file directly from Jupyter. To do this, go to Jupyter Notebook and click on the zip file (in this case assignment1.zip). The file will be downloaded to your local computer. 264 | 265 | Finally, remember to upload the zip file containing your submission to [***Canvas***](https://canvas.stanford.edu/courses/66461). (You can unzip the file locally if you want to double check your ipython notebooks and other code files are correctly inside). 266 | 267 | You can refer to [this page](https://cloud.google.com/compute/docs/instances/transfer-files "Title") for more details on transferring files to/from Google Cloud. 268 | 269 | # BIG REMINDER: Make sure you stop your instances! # 270 | Don't forget to stop your instance when you are done (by clicking on the stop button at the top of the page showing your instances). You can restart your instance and the downloaded software will still be available. 271 | -------------------------------------------------------------------------------- /neural-networks-case-study.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | permalink: /neural-networks-case-study/ 4 | --- 5 | 6 | Table of Contents: 7 | 8 | - [Generating some data](#data) 9 | - [Training a Softmax Linear Classifier](#linear) 10 | - [Initialize the parameters](#init) 11 | - [Compute the class scores](#scores) 12 | - [Compute the loss](#loss) 13 | - [Computing the analytic gradient with backpropagation](#grad) 14 | - [Performing a parameter update](#update) 15 | - [Putting it all together: Training a Softmax Classifier](#together) 16 | - [Training a Neural Network](#net) 17 | - [Summary](#summary) 18 | 19 | In this section we'll walk through a complete implementation of a toy Neural Network in 2 dimensions. We'll first implement a simple linear classifier and then extend the code to a 2-layer Neural Network. As we'll see, this extension is surprisingly simple and very few changes are necessary. 20 | 21 | 22 | 23 | ## Generating some data 24 | 25 | Lets generate a classification dataset that is not easily linearly separable. Our favorite example is the spiral dataset, which can be generated as follows: 26 | 27 | ```python 28 | N = 100 # number of points per class 29 | D = 2 # dimensionality 30 | K = 3 # number of classes 31 | X = np.zeros((N*K,D)) # data matrix (each row = single example) 32 | y = np.zeros(N*K, dtype='uint8') # class labels 33 | for j in xrange(K): 34 | ix = range(N*j,N*(j+1)) 35 | r = np.linspace(0.0,1,N) # radius 36 | t = np.linspace(j*4,(j+1)*4,N) + np.random.randn(N)*0.2 # theta 37 | X[ix] = np.c_[r*np.sin(t), r*np.cos(t)] 38 | y[ix] = j 39 | # lets visualize the data: 40 | plt.scatter(X[:, 0], X[:, 1], c=y, s=40, cmap=plt.cm.Spectral) 41 | plt.show() 42 | ``` 43 | 44 |
45 | 46 |
47 | The toy spiral data consists of three classes (blue, red, yellow) that are not linearly separable. 48 |
49 |
50 | 51 | Normally we would want to preprocess the dataset so that each feature has zero mean and unit standard deviation, but in this case the features are already in a nice range from -1 to 1, so we skip this step. 52 | 53 | 54 | 55 | ## Training a Softmax Linear Classifier 56 | 57 | 58 | 59 | ### Initialize the parameters 60 | 61 | Lets first train a Softmax classifier on this classification dataset. As we saw in the previous sections, the Softmax classifier has a linear score function and uses the cross-entropy loss. The parameters of the linear classifier consist of a weight matrix `W` and a bias vector `b` for each class. Lets first initialize these parameters to be random numbers: 62 | 63 | ```python 64 | # initialize parameters randomly 65 | W = 0.01 * np.random.randn(D,K) 66 | b = np.zeros((1,K)) 67 | ``` 68 | 69 | Recall that we `D = 2` is the dimensionality and `K = 3` is the number of classes. 70 | 71 | 72 | 73 | ### Compute the class scores 74 | 75 | Since this is a linear classifier, we can compute all class scores very simply in parallel with a single matrix multiplication: 76 | 77 | ```python 78 | # compute class scores for a linear classifier 79 | scores = np.dot(X, W) + b 80 | ``` 81 | 82 | In this example we have 300 2-D points, so after this multiplication the array `scores` will have size [300 x 3], where each row gives the class scores corresponding to the 3 classes (blue, red, yellow). 83 | 84 | 85 | 86 | ### Compute the loss 87 | 88 | The second key ingredient we need is a loss function, which is a differentiable objective that quantifies our unhappiness with the computed class scores. Intuitively, we want the correct class to have a higher score than the other classes. When this is the case, the loss should be low and otherwise the loss should be high. There are many ways to quantify this intuition, but in this example lets use the cross-entropy loss that is associated with the Softmax classifier. Recall that if \\(f\\) is the array of class scores for a single example (e.g. array of 3 numbers here), then the Softmax classifier computes the loss for that example as: 89 | 90 | $$ 91 | L_i = -\log\left(\frac{e^{f_{y_i}}}{ \sum_j e^{f_j} }\right) 92 | $$ 93 | 94 | We can see that the Softmax classifier interprets every element of \\(f\\) as holding the (unnormalized) log probabilities of the three classes. We exponentiate these to get (unnormalized) probabilities, and then normalize them to get probabilites. Therefore, the expression inside the log is the normalized probability of the correct class. Note how this expression works: this quantity is always between 0 and 1. When the probability of the correct class is very small (near 0), the loss will go towards (positive) infinity. Conversely, when the correct class probability goes towards 1, the loss will go towards zero because \\(log(1) = 0\\). Hence, the expression for \\(L_i\\) is low when the correct class probability is high, and it's very high when it is low. 95 | 96 | Recall also that the full Softmax classifier loss is then defined as the average cross-entropy loss over the training examples and the regularization: 97 | 98 | $$ 99 | L = \underbrace{ \frac{1}{N} \sum_i L_i }_\text{data loss} + \underbrace{ \frac{1}{2} \lambda \sum_k\sum_l W_{k,l}^2 }_\text{regularization loss} \\\\ 100 | $$ 101 | 102 | Given the array of `scores` we've computed above, we can compute the loss. First, the way to obtain the probabilities is straight forward: 103 | 104 | ```python 105 | num_examples = X.shape[0] 106 | # get unnormalized probabilities 107 | exp_scores = np.exp(scores) 108 | # normalize them for each example 109 | probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True) 110 | ``` 111 | 112 | We now have an array `probs` of size [300 x 3], where each row now contains the class probabilities. In particular, since we've normalized them every row now sums to one. We can now query for the log probabilities assigned to the correct classes in each example: 113 | 114 | ```python 115 | corect_logprobs = -np.log(probs[range(num_examples),y]) 116 | ``` 117 | 118 | The array `correct_logprobs` is a 1D array of just the probabilities assigned to the correct classes for each example. The full loss is then the average of these log probabilities and the regularization loss: 119 | 120 | ```python 121 | # compute the loss: average cross-entropy loss and regularization 122 | data_loss = np.sum(corect_logprobs)/num_examples 123 | reg_loss = 0.5*reg*np.sum(W*W) 124 | loss = data_loss + reg_loss 125 | ``` 126 | 127 | In this code, the regularization strength \\(\lambda\\) is stored inside the `reg`. The convenience factor of `0.5` multiplying the regularization will become clear in a second. Evaluating this in the beginning (with random parameters) might give us `loss = 1.1`, which is `np.log(1.0/3)`, since with small initial random weights all probabilities assigned to all classes are about one third. We now want to make the loss as low as possible, with `loss = 0` as the absolute lower bound. But the lower the loss is, the higher are the probabilities assigned to the correct classes for all examples. 128 | 129 | 130 | 131 | ### Computing the Analytic Gradient with Backpropagation 132 | 133 | We have a way of evaluating the loss, and now we have to minimize it. We'll do so with gradient descent. That is, we start with random parameters (as shown above), and evaluate the gradient of the loss function with respect to the parameters, so that we know how we should change the parameters to decrease the loss. Lets introduce the intermediate variable \\(p\\), which is a vector of the (normalized) probabilities. The loss for one example is: 134 | 135 | $$ 136 | p_k = \frac{e^{f_k}}{ \sum_j e^{f_j} } \hspace{1in} L_i =-\log\left(p_{y_i}\right) 137 | $$ 138 | 139 | We now wish to understand how the computed scores inside \\(f\\) should change to decrease the loss \\(L_i\\) that this example contributes to the full objective. In other words, we want to derive the gradient \\( \partial L_i / \partial f_k \\). The loss \\(L_i\\) is computed from \\(p\\), which in turn depends on \\(f\\). It's a fun exercise to the reader to use the chain rule to derive the gradient, but it turns out to be extremely simple and interpretible in the end, after a lot of things cancel out: 140 | 141 | $$ 142 | \frac{\partial L_i }{ \partial f_k } = p_k - \mathbb{1}(y_i = k) 143 | $$ 144 | 145 | Notice how elegant and simple this expression is. Suppose the probabilities we computed were `p = [0.2, 0.3, 0.5]`, and that the correct class was the middle one (with probability 0.3). According to this derivation the gradient on the scores would be `df = [0.2, -0.7, 0.5]`. Recalling what the interpretation of the gradient, we see that this result is highly intuitive: increasing the first or last element of the score vector `f` (the scores of the incorrect classes) leads to an *increased* loss (due to the positive signs +0.2 and +0.5) - and increasing the loss is bad, as expected. However, increasing the score of the correct class has *negative* influence on the loss. The gradient of -0.7 is telling us that increasing the correct class score would lead to a decrease of the loss \\(L_i\\), which makes sense. 146 | 147 | All of this boils down to the following code. Recall that `probs` stores the probabilities of all classes (as rows) for each example. To get the gradient on the scores, which we call `dscores`, we proceed as follows: 148 | 149 | ```python 150 | dscores = probs 151 | dscores[range(num_examples),y] -= 1 152 | dscores /= num_examples 153 | ``` 154 | 155 | Lastly, we had that `scores = np.dot(X, W) + b`, so armed with the gradient on `scores` (stored in `dscores`), we can now backpropagate into `W` and `b`: 156 | 157 | ```python 158 | dW = np.dot(X.T, dscores) 159 | db = np.sum(dscores, axis=0, keepdims=True) 160 | dW += reg*W # don't forget the regularization gradient 161 | ``` 162 | 163 | Where we see that we have backpropped through the matrix multiply operation, and also added the contribution from the regularization. Note that the regularization gradient has the very simple form `reg*W` since we used the constant `0.5` for its loss contribution (i.e. \\(\frac{d}{dw} ( \frac{1}{2} \lambda w^2) = \lambda w\\). This is a common convenience trick that simplifies the gradient expression. 164 | 165 | 166 | 167 | ### Performing a parameter update 168 | 169 | Now that we've evaluated the gradient we know how every parameter influences the loss function. We will now perform a parameter update in the *negative* gradient direction to *decrease* the loss: 170 | 171 | ```python 172 | # perform a parameter update 173 | W += -step_size * dW 174 | b += -step_size * db 175 | ``` 176 | 177 | 178 | 179 | ### Putting it all together: Training a Softmax Classifier 180 | 181 | Putting all of this together, here is the full code for training a Softmax classifier with Gradient descent: 182 | 183 | ```python 184 | #Train a Linear Classifier 185 | 186 | # initialize parameters randomly 187 | W = 0.01 * np.random.randn(D,K) 188 | b = np.zeros((1,K)) 189 | 190 | # some hyperparameters 191 | step_size = 1e-0 192 | reg = 1e-3 # regularization strength 193 | 194 | # gradient descent loop 195 | num_examples = X.shape[0] 196 | for i in xrange(200): 197 | 198 | # evaluate class scores, [N x K] 199 | scores = np.dot(X, W) + b 200 | 201 | # compute the class probabilities 202 | exp_scores = np.exp(scores) 203 | probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True) # [N x K] 204 | 205 | # compute the loss: average cross-entropy loss and regularization 206 | corect_logprobs = -np.log(probs[range(num_examples),y]) 207 | data_loss = np.sum(corect_logprobs)/num_examples 208 | reg_loss = 0.5*reg*np.sum(W*W) 209 | loss = data_loss + reg_loss 210 | if i % 10 == 0: 211 | print "iteration %d: loss %f" % (i, loss) 212 | 213 | # compute the gradient on scores 214 | dscores = probs 215 | dscores[range(num_examples),y] -= 1 216 | dscores /= num_examples 217 | 218 | # backpropate the gradient to the parameters (W,b) 219 | dW = np.dot(X.T, dscores) 220 | db = np.sum(dscores, axis=0, keepdims=True) 221 | 222 | dW += reg*W # regularization gradient 223 | 224 | # perform a parameter update 225 | W += -step_size * dW 226 | b += -step_size * db 227 | ``` 228 | 229 | Running this prints the output: 230 | 231 | ``` 232 | iteration 0: loss 1.096956 233 | iteration 10: loss 0.917265 234 | iteration 20: loss 0.851503 235 | iteration 30: loss 0.822336 236 | iteration 40: loss 0.807586 237 | iteration 50: loss 0.799448 238 | iteration 60: loss 0.794681 239 | iteration 70: loss 0.791764 240 | iteration 80: loss 0.789920 241 | iteration 90: loss 0.788726 242 | iteration 100: loss 0.787938 243 | iteration 110: loss 0.787409 244 | iteration 120: loss 0.787049 245 | iteration 130: loss 0.786803 246 | iteration 140: loss 0.786633 247 | iteration 150: loss 0.786514 248 | iteration 160: loss 0.786431 249 | iteration 170: loss 0.786373 250 | iteration 180: loss 0.786331 251 | iteration 190: loss 0.786302 252 | ``` 253 | 254 | We see that we've converged to something after about 190 iterations. We can evaluate the training set accuracy: 255 | 256 | ```python 257 | # evaluate training set accuracy 258 | scores = np.dot(X, W) + b 259 | predicted_class = np.argmax(scores, axis=1) 260 | print 'training accuracy: %.2f' % (np.mean(predicted_class == y)) 261 | ``` 262 | 263 | This prints **49%**. Not very good at all, but also not surprising given that the dataset is constructed so it is not linearly separable. We can also plot the learned decision boundaries: 264 | 265 |
266 | 267 |
268 | Linear classifier fails to learn the toy spiral dataset. 269 |
270 |
271 | 272 | 273 | 274 | ## Training a Neural Network 275 | 276 | Clearly, a linear classifier is inadequate for this dataset and we would like to use a Neural Network. One additional hidden layer will suffice for this toy data. We will now need two sets of weights and biases (for the first and second layers): 277 | 278 | ```python 279 | # initialize parameters randomly 280 | h = 100 # size of hidden layer 281 | W = 0.01 * np.random.randn(D,h) 282 | b = np.zeros((1,h)) 283 | W2 = 0.01 * np.random.randn(h,K) 284 | b2 = np.zeros((1,K)) 285 | ``` 286 | 287 | The forward pass to compute scores now changes form: 288 | 289 | ```python 290 | # evaluate class scores with a 2-layer Neural Network 291 | hidden_layer = np.maximum(0, np.dot(X, W) + b) # note, ReLU activation 292 | scores = np.dot(hidden_layer, W2) + b2 293 | ``` 294 | 295 | Notice that the only change from before is one extra line of code, where we first compute the hidden layer representation and then the scores based on this hidden layer. Crucially, we've also added a non-linearity, which in this case is simple ReLU that thresholds the activations on the hidden layer at zero. 296 | 297 | Everything else remains the same. We compute the loss based on the scores exactly as before, and get the gradient for the scores `dscores` exactly as before. However, the way we backpropagate that gradient into the model parameters now changes form, of course. First lets backpropagate the second layer of the Neural Network. This looks identical to the code we had for the Softmax classifier, except we're replacing `X` (the raw data), with the variable `hidden_layer`): 298 | 299 | ```python 300 | # backpropate the gradient to the parameters 301 | # first backprop into parameters W2 and b2 302 | dW2 = np.dot(hidden_layer.T, dscores) 303 | db2 = np.sum(dscores, axis=0, keepdims=True) 304 | ``` 305 | 306 | However, unlike before we are not yet done, because `hidden_layer` is itself a function of other parameters and the data! We need to continue backpropagation through this variable. Its gradient can be computed as: 307 | 308 | ```python 309 | dhidden = np.dot(dscores, W2.T) 310 | ``` 311 | 312 | Now we have the gradient on the outputs of the hidden layer. Next, we have to backpropagate the ReLU non-linearity. This turns out to be easy because ReLU during the backward pass is effectively a switch. Since \\(r = max(0, x)\\), we have that \\(\frac{dr}{dx} = 1(x > 0) \\). Combined with the chain rule, we see that the ReLU unit lets the gradient pass through unchanged if its input was greater than 0, but *kills it* if its input was less than zero during the forward pass. Hence, we can backpropagate the ReLU in place simply with: 313 | 314 | ```python 315 | # backprop the ReLU non-linearity 316 | dhidden[hidden_layer <= 0] = 0 317 | ``` 318 | 319 | And now we finally continue to the first layer weights and biases: 320 | 321 | ```python 322 | # finally into W,b 323 | dW = np.dot(X.T, dhidden) 324 | db = np.sum(dhidden, axis=0, keepdims=True) 325 | ``` 326 | 327 | We're done! We have the gradients `dW,db,dW2,db2` and can perform the parameter update. Everything else remains unchanged. The full code looks very similar: 328 | 329 | ```python 330 | # initialize parameters randomly 331 | h = 100 # size of hidden layer 332 | W = 0.01 * np.random.randn(D,h) 333 | b = np.zeros((1,h)) 334 | W2 = 0.01 * np.random.randn(h,K) 335 | b2 = np.zeros((1,K)) 336 | 337 | # some hyperparameters 338 | step_size = 1e-0 339 | reg = 1e-3 # regularization strength 340 | 341 | # gradient descent loop 342 | num_examples = X.shape[0] 343 | for i in xrange(10000): 344 | 345 | # evaluate class scores, [N x K] 346 | hidden_layer = np.maximum(0, np.dot(X, W) + b) # note, ReLU activation 347 | scores = np.dot(hidden_layer, W2) + b2 348 | 349 | # compute the class probabilities 350 | exp_scores = np.exp(scores) 351 | probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True) # [N x K] 352 | 353 | # compute the loss: average cross-entropy loss and regularization 354 | corect_logprobs = -np.log(probs[range(num_examples),y]) 355 | data_loss = np.sum(corect_logprobs)/num_examples 356 | reg_loss = 0.5*reg*np.sum(W*W) + 0.5*reg*np.sum(W2*W2) 357 | loss = data_loss + reg_loss 358 | if i % 1000 == 0: 359 | print "iteration %d: loss %f" % (i, loss) 360 | 361 | # compute the gradient on scores 362 | dscores = probs 363 | dscores[range(num_examples),y] -= 1 364 | dscores /= num_examples 365 | 366 | # backpropate the gradient to the parameters 367 | # first backprop into parameters W2 and b2 368 | dW2 = np.dot(hidden_layer.T, dscores) 369 | db2 = np.sum(dscores, axis=0, keepdims=True) 370 | # next backprop into hidden layer 371 | dhidden = np.dot(dscores, W2.T) 372 | # backprop the ReLU non-linearity 373 | dhidden[hidden_layer <= 0] = 0 374 | # finally into W,b 375 | dW = np.dot(X.T, dhidden) 376 | db = np.sum(dhidden, axis=0, keepdims=True) 377 | 378 | # add regularization gradient contribution 379 | dW2 += reg * W2 380 | dW += reg * W 381 | 382 | # perform a parameter update 383 | W += -step_size * dW 384 | b += -step_size * db 385 | W2 += -step_size * dW2 386 | b2 += -step_size * db2 387 | ``` 388 | 389 | This prints: 390 | 391 | ``` 392 | iteration 0: loss 1.098744 393 | iteration 1000: loss 0.294946 394 | iteration 2000: loss 0.259301 395 | iteration 3000: loss 0.248310 396 | iteration 4000: loss 0.246170 397 | iteration 5000: loss 0.245649 398 | iteration 6000: loss 0.245491 399 | iteration 7000: loss 0.245400 400 | iteration 8000: loss 0.245335 401 | iteration 9000: loss 0.245292 402 | ``` 403 | 404 | The training accuracy is now: 405 | 406 | ```python 407 | # evaluate training set accuracy 408 | hidden_layer = np.maximum(0, np.dot(X, W) + b) 409 | scores = np.dot(hidden_layer, W2) + b2 410 | predicted_class = np.argmax(scores, axis=1) 411 | print 'training accuracy: %.2f' % (np.mean(predicted_class == y)) 412 | ``` 413 | 414 | Which prints **98%**!. We can also visualize the decision boundaries: 415 | 416 |
417 | 418 |
419 | Neural Network classifier crushes the spiral dataset. 420 |
421 |
422 | 423 | ## Summary 424 | 425 | We've worked with a toy 2D dataset and trained both a linear network and a 2-layer Neural Network. We saw that the change from a linear classifier to a Neural Network involves very few changes in the code. The score function changes its form (1 line of code difference), and the backpropagation changes its form (we have to perform one more round of backprop through the hidden layer to the first layer of the network). 426 | 427 | - You may want to look at this IPython Notebook code [rendered as HTML](http://cs.stanford.edu/people/karpathy/cs231nfiles/minimal_net.html). 428 | - Or download the [ipynb file](http://cs.stanford.edu/people/karpathy/cs231nfiles/minimal_net.ipynb) 429 | 430 | --------------------------------------------------------------------------------