├── code ├── hmc │ ├── __init__.py │ ├── test_hmc.py │ └── hmc.py ├── utils.py ├── test.py ├── logistic_cg.py ├── rnnrbm.py ├── cA.py ├── convolutional_mlp.py ├── mlp.py └── dA.py ├── .hgignore ├── doc ├── Makefile ├── images │ ├── bm.png │ ├── DBN3.png │ ├── mlp.png │ ├── rbm.png │ ├── mnist_0.png │ ├── mnist_1.png │ ├── mnist_2.png │ ├── mnist_3.png │ ├── mnist_4.png │ ├── mnist_5.png │ ├── mylenet.png │ ├── rnnrbm.png │ ├── sample1.png │ ├── sample2.png │ ├── samples.png │ ├── 3wolfmoon.jpg │ ├── conv_1D_nn.png │ ├── markov_chain.png │ ├── sparse_1D_nn.png │ ├── cnn_explained.png │ ├── 3wolfmoon_output.png │ ├── filters_at_epoch_14.png │ ├── filters_corruption_0.png │ └── filters_corruption_30.png ├── contents.txt ├── .templates │ └── layout.html ├── LICENSE.txt ├── scripts │ └── docgen.py ├── references.txt ├── intro.txt ├── deep.txt ├── utilities.txt ├── conf.py ├── rnnrbm.txt ├── SdA.txt ├── logreg.txt ├── DBN.txt └── mlp.txt ├── data ├── training_colorpatches_16x16_demo.mat └── download.sh ├── .gitignore ├── issues_open ├── 3_RBM_scan_GPU.txt ├── 5_results.txt ├── 1_SdA_performance.txt ├── 4_RBM_scan.txt └── 6_benchmarking_pybrain.txt ├── issues_closed └── 2_RBM_cost_fn.txt ├── README.rst ├── misc └── do_nightly_build └── .travis.yml /code/hmc/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /.hgignore: -------------------------------------------------------------------------------- 1 | syntax: glob 2 | *.pyc 3 | *.png 4 | *~ 5 | -------------------------------------------------------------------------------- /doc/Makefile: -------------------------------------------------------------------------------- 1 | all: 2 | python scripts/docgen.py 3 | -------------------------------------------------------------------------------- /doc/images/bm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/bm.png -------------------------------------------------------------------------------- /doc/images/DBN3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/DBN3.png -------------------------------------------------------------------------------- /doc/images/mlp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/mlp.png -------------------------------------------------------------------------------- /doc/images/rbm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/rbm.png -------------------------------------------------------------------------------- /doc/images/mnist_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/mnist_0.png -------------------------------------------------------------------------------- /doc/images/mnist_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/mnist_1.png -------------------------------------------------------------------------------- /doc/images/mnist_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/mnist_2.png -------------------------------------------------------------------------------- /doc/images/mnist_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/mnist_3.png -------------------------------------------------------------------------------- /doc/images/mnist_4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/mnist_4.png -------------------------------------------------------------------------------- /doc/images/mnist_5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/mnist_5.png -------------------------------------------------------------------------------- /doc/images/mylenet.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/mylenet.png -------------------------------------------------------------------------------- /doc/images/rnnrbm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/rnnrbm.png -------------------------------------------------------------------------------- /doc/images/sample1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/sample1.png -------------------------------------------------------------------------------- /doc/images/sample2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/sample2.png -------------------------------------------------------------------------------- /doc/images/samples.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/samples.png -------------------------------------------------------------------------------- /doc/images/3wolfmoon.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/3wolfmoon.jpg -------------------------------------------------------------------------------- /doc/images/conv_1D_nn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/conv_1D_nn.png -------------------------------------------------------------------------------- /doc/images/markov_chain.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/markov_chain.png -------------------------------------------------------------------------------- /doc/images/sparse_1D_nn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/sparse_1D_nn.png -------------------------------------------------------------------------------- /doc/images/cnn_explained.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/cnn_explained.png -------------------------------------------------------------------------------- /doc/images/3wolfmoon_output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/3wolfmoon_output.png -------------------------------------------------------------------------------- /doc/images/filters_at_epoch_14.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/filters_at_epoch_14.png -------------------------------------------------------------------------------- /doc/images/filters_corruption_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/filters_corruption_0.png -------------------------------------------------------------------------------- /doc/images/filters_corruption_30.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/doc/images/filters_corruption_30.png -------------------------------------------------------------------------------- /data/training_colorpatches_16x16_demo.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/findmyway/DeepLearningTutorials/master/data/training_colorpatches_16x16_demo.mat -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | code/*.pyc 2 | code/tmp* 3 | code/midi 4 | data/mnist.pkl.gz 5 | data/mnist_py3k.pkl.gz 6 | data/Nottingham.zip 7 | data/Nottingham 8 | data/midi.zip 9 | html 10 | *.pyc 11 | *~ 12 | *.swp 13 | -------------------------------------------------------------------------------- /issues_open/3_RBM_scan_GPU.txt: -------------------------------------------------------------------------------- 1 | Reported by : Razvan 2 | 3 | Scan is not GPU ready.. making RBM tutorial slow on GPU (not tested yet). 4 | Quick fix is a optimization that removes scan if you're doing CD-1. 5 | -------------------------------------------------------------------------------- /issues_open/5_results.txt: -------------------------------------------------------------------------------- 1 | Reported by : Razvan 2 | 3 | We should produce results + time for CPU float32 / CPU float64 / GPU . We should also 4 | specify the batchsize (or number of updates) pointing out that you can't always just 5 | compare the number of epochs. 6 | -------------------------------------------------------------------------------- /issues_closed/2_RBM_cost_fn.txt: -------------------------------------------------------------------------------- 1 | Reported by : Razvan 2 | 3 | Cost function (delta of free energy) has a reversed sign (i.e. free_energy(positive) - free_energy(negative) ). I'm not sure 4 | where the minus pops in .. but is confusing when going from theory to code. 5 | 6 | 7 | FIXED 8 | -------------------------------------------------------------------------------- /issues_open/1_SdA_performance.txt: -------------------------------------------------------------------------------- 1 | Reported by : Razvan 2 | 3 | Best performance for SdA float64 CPU : 1.23% 4 | float32 CPU : 1.30% 5 | target : 1.10% 6 | 7 | Possible reasons: 8 | - bug !? 9 | - random seed / weights initialization / finetuning early stopping parameters 10 | -------------------------------------------------------------------------------- /doc/contents.txt: -------------------------------------------------------------------------------- 1 | 2 | .. _contents: 3 | 4 | ======== 5 | Contents 6 | ======== 7 | 8 | .. toctree:: 9 | :maxdepth: 2 10 | 11 | LICENSE 12 | intro 13 | gettingstarted 14 | logreg 15 | mlp 16 | lenet 17 | dA 18 | SdA 19 | rbm 20 | DBN 21 | hmc 22 | rnnrbm 23 | utilities 24 | references 25 | -------------------------------------------------------------------------------- /issues_open/4_RBM_scan.txt: -------------------------------------------------------------------------------- 1 | Reported by : Razvan 2 | 3 | The bug can be reproduced if you do : 4 | z = scan(..) 5 | c = f(z[-1]) 6 | gp = T.grad(c, p, consider_constant = [ z[-1] ] ) 7 | 8 | In this case grad will not consider z[-1] constant. Workaround: 9 | 10 | z = scan(..) 11 | z_1 = z[-1] 12 | c = f(z_1) 13 | gp = T.grad(c,p, consider_constant = [z_1]) 14 | 15 | Note : I need to make sure this actually happens .. it might have been an 16 | artifact of something else when I first got this. 17 | -------------------------------------------------------------------------------- /data/download.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | which wget >/dev/null 2>&1 4 | WGET=$? 5 | which curl >/dev/null 2>&1 6 | CURL=$? 7 | if [ "$WGET" -eq 0 ]; then 8 | DL_CMD="wget -c" 9 | elif [ "$CURL" -eq 0 ]; then 10 | DL_CMD="curl -C - -O" 11 | else 12 | echo "You need wget or curl installed to download" 13 | exit 1 14 | fi 15 | 16 | $DL_CMD http://www.iro.umontreal.ca/~lisa/deep/data/mnist/mnist.pkl.gz 17 | $DL_CMD http://www.iro.umontreal.ca/~lisa/deep/data/mnist/mnist_py3k.pkl.gz 18 | $DL_CMD http://www.iro.umontreal.ca/~lisa/deep/data/Nottingham.zip && unzip -u Nottingham.zip 19 | $DL_CMD http://www.iro.umontreal.ca/~lisa/deep/midi.zip && unzip -u midi.zip -d ../code && echo "extracted Modified Python MIDI package (GPL)" 20 | -------------------------------------------------------------------------------- /doc/.templates/layout.html: -------------------------------------------------------------------------------- 1 | {% extends "!layout.html" %} 2 | 3 | {%- block extrahead %} 4 | {{ super() }} 5 | 10 | {% endblock %} 11 | 12 | {% block footer %} 13 | {{ super() }} 14 | 23 | {% endblock %} 24 | 25 | -------------------------------------------------------------------------------- /doc/LICENSE.txt: -------------------------------------------------------------------------------- 1 | .. _license: 2 | 3 | LICENSE 4 | ======= 5 | 6 | Copyright (c) 2008--2013, Theano Development Team 7 | All rights reserved. 8 | 9 | Redistribution and use in source and binary forms, with or without 10 | modification, are permitted provided that the following conditions are met: 11 | 12 | * Redistributions of source code must retain the above copyright 13 | notice, this list of conditions and the following disclaimer. 14 | * Redistributions in binary form must reproduce the above copyright 15 | notice, this list of conditions and the following disclaimer in the 16 | documentation and/or other materials provided with the distribution. 17 | * Neither the name of Theano nor the names of its contributors may be 18 | used to endorse or promote products derived from this software without 19 | specific prior written permission. 20 | 21 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ''AS IS'' AND ANY 22 | EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 23 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 24 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS BE LIABLE FOR ANY 25 | DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 26 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 27 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 28 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 29 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 30 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 31 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | Deep Learning Tutorials 2 | ======================= 3 | 4 | Deep Learning is a new area of Machine Learning research, which has been 5 | introduced with the objective of moving Machine Learning closer to one of its 6 | original goals: Artificial Intelligence. Deep Learning is about learning 7 | multiple levels of representation and abstraction that help to make sense of 8 | data such as images, sound, and text. The tutorials presented here will 9 | introduce you to some of the most important deep learning algorithms and will 10 | also show you how to run them using Theano. Theano is a python library that 11 | makes writing deep learning models easy, and gives the option of training them 12 | on a GPU. 13 | 14 | The easiest way to follow the tutorials is to `browse them online 15 | `_. 16 | 17 | `Main development `_ 18 | of this project. 19 | 20 | .. image:: https://secure.travis-ci.org/lisa-lab/DeepLearningTutorials.png 21 | :target: http://travis-ci.org/lisa-lab/DeepLearningTutorials 22 | 23 | Project Layout 24 | -------------- 25 | 26 | Subdirectories: 27 | 28 | - code - Python files corresponding to each tutorial 29 | - data - data and scripts to download data that is used by the tutorials 30 | - doc - restructured text used by Sphinx to build the tutorial website 31 | - html - built automatically by doc/Makefile, contains tutorial website 32 | - issues_closed - issue tracking 33 | - issues_open - issue tracking 34 | - misc - administrative scripts 35 | 36 | 37 | Build instructions 38 | ------------------ 39 | 40 | To build the html version of the tutorials, install sphinx and run doc/Makefile 41 | -------------------------------------------------------------------------------- /misc/do_nightly_build: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #we set the compiledir to the /Tmp dir to make the test faster by bypassing the nfs network. 3 | date 4 | ROOT_CWD=/Tmp/nightly_build 5 | COMPILEDIR=/Tmp/lisa_theano_compile_dir_deeplearning 6 | NOSETESTS=${ROOT_CWD}/Theano/bin/theano-nose 7 | 8 | FLAGS=warn.ignore_bug_before=0.5,compiledir=${COMPILEDIR} 9 | export PYTHONPATH=${ROOT_CWD}/Theano:${ROOT_CWD}/Pylearn:$PYTHONPATH 10 | 11 | cd ${ROOT_CWD}/DeepLearningTutorials/data 12 | ./download.sh 13 | 14 | cd ${ROOT_CWD}/Theano 15 | echo "git version for Theano:" `git rev-parse HEAD` 16 | cd ${ROOT_CWD}/DeepLearningTutorials/code 17 | echo "git version:" `git rev-parse HEAD` 18 | 19 | #echo "executing nosetests with mode=FAST_COMPILE" 20 | #THEANO_FLAGS=${FLAGS},mode=FAST_COMPILE ${NOSETESTS} 21 | echo "executing nosetests speed with mode=FAST_RUN" 22 | THEANO_FLAGS=${FLAGS},mode=FAST_RUN ${NOSETESTS} test.py:speed 23 | #echo "executing nosetests speed with mode=FAST_RUN and OMP_NUM_THREADS=2" 24 | #OMP_NUM_THREADS=2 THEANO_FLAGS=${FLAGS},mode=FAST_RUN ${NOSETESTS} test.py:speed 25 | echo "executing nosetests with mode=FAST_RUN,floatX=float32" 26 | THEANO_FLAGS=${FLAGS},mode=FAST_RUN,floatX=float32 ${NOSETESTS} 27 | 28 | #we change the seed and record it everyday to test different combination. We record it to be able to reproduce bug caused by different seed. We don't want multiple test in DEBUG_MODE each day as this take too long. 29 | #seed=$RANDOM 30 | #echo "executing nosetests with mode=DEBUG_MODE with seed of the day $seed" 31 | #THEANO_DEBUGMODE_CHECK_STRIDES=0 THEANO_DEBUGMODE_PATIENCE=3 THEANO_COMPILEDIR=/Tmp/lisa_theano_compile_dir_deeplearning THEANO_UNITTEST_SEED=$seed THEANO_DEFAULT_MODE=DEBUG_MODE ${NOSETESTS} 32 | 33 | -------------------------------------------------------------------------------- /doc/scripts/docgen.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | import sys 3 | import os 4 | import shutil 5 | 6 | import getopt 7 | from collections import defaultdict 8 | 9 | if __name__ == '__main__': 10 | 11 | throot = "/".join(sys.path[0].split("/")[:-2]) 12 | 13 | options = defaultdict(bool) 14 | output_arg = getopt.getopt(sys.argv[1:], 'o:', ['rst', 'help', 'nopdf'])[0] 15 | options.update(dict([x, y or True] for x, y in output_arg)) 16 | if options['--help']: 17 | print('Usage: %s [OPTIONS]' % sys.argv[0]) 18 | print(' -o : output the html files in the specified dir') 19 | print(' --rst: only compile the doc (requires sphinx)') 20 | print(' --nopdf: do not produce a PDF file from the doc, only HTML') 21 | print(' --help: this help') 22 | sys.exit(0) 23 | 24 | options['--all'] = not bool(options['--rst']) 25 | 26 | def mkdir(path): 27 | try: 28 | os.mkdir(path) 29 | except OSError: 30 | pass 31 | 32 | outdir = options['-o'] or (throot + '/html') 33 | mkdir(outdir) 34 | os.chdir(outdir) 35 | mkdir("doc") 36 | 37 | # Make sure the appropriate 'deeplearning' directory is in the PYTHONPATH 38 | pythonpath = os.environ.get('PYTHONPATH', '') 39 | pythonpath = throot + ':' + pythonpath 40 | os.environ['PYTHONPATH'] = pythonpath 41 | 42 | if options['--all'] or options['--rst']: 43 | import sphinx 44 | sys.path[0:0] = [os.path.join(throot, 'doc')] 45 | sphinx.main(['', '-E', os.path.join(throot, 'doc'), '.']) 46 | 47 | if not options['--nopdf']: 48 | # Generate latex file in a temp directory 49 | import tempfile 50 | workdir = tempfile.mkdtemp() 51 | sphinx.main(['', '-E', '-b', 'latex', 52 | os.path.join(throot, 'doc'), workdir]) 53 | # Compile to PDF 54 | os.chdir(workdir) 55 | os.system('make') 56 | try: 57 | shutil.copy(os.path.join(workdir, 'deeplearning.pdf'), outdir) 58 | os.chdir(outdir) 59 | shutil.rmtree(workdir) 60 | except OSError as e: 61 | print('OSError:', e) 62 | except IOError as e: 63 | print('IOError:', e) 64 | -------------------------------------------------------------------------------- /code/hmc/test_hmc.py: -------------------------------------------------------------------------------- 1 | import numpy 2 | from scipy import linalg 3 | import theano 4 | 5 | from hmc import HMC_sampler 6 | 7 | 8 | def sampler_on_nd_gaussian(sampler_cls, burnin, n_samples, dim=10): 9 | batchsize = 3 10 | 11 | rng = numpy.random.RandomState(123) 12 | 13 | # Define a covariance and mu for a gaussian 14 | mu = numpy.array(rng.rand(dim) * 10, dtype=theano.config.floatX) 15 | cov = numpy.array(rng.rand(dim, dim), dtype=theano.config.floatX) 16 | cov = (cov + cov.T) / 2. 17 | cov[numpy.arange(dim), numpy.arange(dim)] = 1.0 18 | cov_inv = linalg.inv(cov) 19 | 20 | # Define energy function for a multi-variate Gaussian 21 | def gaussian_energy(x): 22 | return 0.5 * (theano.tensor.dot((x - mu), cov_inv) * 23 | (x - mu)).sum(axis=1) 24 | 25 | # Declared shared random variable for positions 26 | position = rng.randn(batchsize, dim).astype(theano.config.floatX) 27 | position = theano.shared(position) 28 | 29 | # Create HMC sampler 30 | sampler = sampler_cls(position, gaussian_energy, 31 | initial_stepsize=1e-3, stepsize_max=0.5) 32 | 33 | # Start with a burn-in process 34 | garbage = [sampler.draw() for r in xrange(burnin)] # burn-in Draw 35 | # `n_samples`: result is a 3D tensor of dim [n_samples, batchsize, 36 | # dim] 37 | _samples = numpy.asarray([sampler.draw() for r in xrange(n_samples)]) 38 | # Flatten to [n_samples * batchsize, dim] 39 | samples = _samples.T.reshape(dim, -1).T 40 | 41 | print '****** TARGET VALUES ******' 42 | print 'target mean:', mu 43 | print 'target cov:\n', cov 44 | 45 | print '****** EMPIRICAL MEAN/COV USING HMC ******' 46 | print 'empirical mean: ', samples.mean(axis=0) 47 | print 'empirical_cov:\n', numpy.cov(samples.T) 48 | 49 | print '****** HMC INTERNALS ******' 50 | print 'final stepsize', sampler.stepsize.get_value() 51 | print 'final acceptance_rate', sampler.avg_acceptance_rate.get_value() 52 | 53 | return sampler 54 | 55 | 56 | def test_hmc(): 57 | sampler = sampler_on_nd_gaussian(HMC_sampler.new_from_shared_positions, 58 | burnin=1000, n_samples=1000, dim=5) 59 | assert abs(sampler.avg_acceptance_rate.get_value() - 60 | sampler.target_acceptance_rate) < .1 61 | assert sampler.stepsize.get_value() >= sampler.stepsize_min 62 | assert sampler.stepsize.get_value() <= sampler.stepsize_max 63 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | # After changing this file, check it on: 2 | # http://lint.travis-ci.org/ 3 | 4 | #We can't get scipy installed with the python language 5 | #So we will use the system python from the c language. 6 | language: c 7 | #language: python 8 | #python: 9 | # - "2.5" 10 | # - "2.7" 11 | # - "3.2" 12 | # command to install dependencies 13 | before_install: 14 | #zlib1g-dev is needed to allow PIL to uncompress the dataset. 15 | - sudo apt-get install -qq libatlas3gf-base libatlas-dev zlib1g-dev zip unzip zlibc libzip-dev libjpeg8 libjpeg62-dev libfreetype6 libfreetype6-dev python-numpy python-scipy python-pip python-nose python-yaml pyflakes python-imaging 16 | 17 | install: 18 | # - "pip install -q numpy --use-mirrors" 19 | # Use Pillow instead of PIL as it is better packaged 20 | # - "pip install -q Pillow --use-mirrors" 21 | #If we don't install numpy before SciPy 0.10.1, the SciPy installations fails. 22 | # - "pip install -q scipy --use-mirrors" 23 | - "sudo pip install --no-deps git+git://github.com/Theano/Theano.git" 24 | 25 | env: 26 | - PART="test.py:test_logistic_sgd test.py:test_logistic_cg test.py:test_mlp test.py:test_convolutional_mlp test.py:test_dA" 27 | - PART="test.py:test_SdA" 28 | - PART="test.py:test_dbn" 29 | - PART="test.py:test_rbm test.py:test_rnnrbm" 30 | - PART="-e test.py" 31 | 32 | #i7-2600K CPU @ 3.40GHz 33 | #166.572s #8 test.test_rbm OK 34 | #155.114s #7 test.test_dbn OK 35 | #152.365s #9 test.test_rnnrbm OK 36 | #127.286s #6 test.test_SdA OK 37 | #39.252s #5 test.test_dA OK 38 | #27.56s #4 test.test_convolutional_mlp OK 39 | #15.454s #3 test.test_mlp OK 40 | #12.732s #1 test.test_logistic_sgd OK 41 | #12.638s #2 test.test_logistic_cg OK 42 | 43 | #i7-920 44 | #296.475s #7 code.test.test_dbn OK 45 | #257.272s #6 code.test.test_SdA OK 46 | #234.776s #9 code.test.test_rnnrbm OK 47 | #233.896s #8 code.test.test_rbm OK 48 | #65.737s #5 code.test.test_dA OK 49 | #37.658s #4 code.test.test_convolutional_mlp OK 50 | #24.172s #3 code.test.test_mlp OK 51 | #20.401s #1 code.test.test_logistic_sgd OK 52 | #17.546s #2 code.test.test_logistic_cg OK 53 | 54 | # On Core2 duo E8500 with MRG 55 | #308.004s #7 code.test.test_dbn OK 56 | #277.268s #6 code.test.test_SdA OK 57 | #126.102s #8 code.test.test_rbm OK 58 | #123.652s #9 code.test.test_rnnrbm OK 59 | #77.101s #5 code.test.test_dA OK 60 | #39.75s #4 code.test.test_convolutional_mlp OK 61 | #30.406s #3 code.test.test_mlp OK 62 | #21.132s #2 code.test.test_logistic_cg OK 63 | #17.945s #1 code.test.test_logistic_sgd OK 64 | 65 | # Unknown computer with older version of Theano 66 | #569.882s #9 code.test.test_rbm OK 67 | #298.992s #8 code.test.test_dbn OK 68 | #268.901s #7 code.test.test_SdA OK 69 | #67.292s #6 code.test.test_dA OK 70 | #27.485s #4 code.test.test_mlp OK 71 | #26.204s #5 code.test.test_convolutional_mlp OK 72 | #14.676s #3 code.test.test_logistic_cg OK 73 | #10.66s #2 code.test.test_logistic_sgd OK 74 | #5.795s #1 code.hmc.test_hmc.test_hmc OK 75 | 76 | script: 77 | - cd data 78 | - ./download.sh 79 | - ls 80 | - cd ../code 81 | - pwd 82 | - ls 83 | - export THEANO_FLAGS=warn.ignore_bug_before=all,on_opt_error=raise,on_shape_error=raise 84 | - python --version 85 | - nosetests $PART 86 | 87 | -------------------------------------------------------------------------------- /doc/references.txt: -------------------------------------------------------------------------------- 1 | .. _references: 2 | 3 | ========== 4 | References 5 | ========== 6 | 7 | .. [Bengio07] Y. Bengio, P. Lamblin, D. Popovici and H. Larochelle, `Greedy Layer-Wise Training of Deep Networks `_, in Advances in Neural Information Processing Systems 19 (NIPS'06), pages 153-160, MIT Press 2007. 8 | 9 | .. [Bengio09] Y. Bengio, `Learning deep architectures for AI `_, Foundations and Trends in Machine Learning 1(2) pages 1-127. 10 | 11 | .. [BengioDelalleau09] Y. Bengio, O. Delalleau, Justifying and Generalizing Contrastive Divergence (2009), Neural Computation, 21(6): 1601-1621. 12 | 13 | .. [BoulangerLewandowski12] N Boulanger-Lewandowski, Y. Bengio and P. Vincent, `Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription `_, in Proceedings of the 29th International Conference on Machine Learning (ICML), 2012. 14 | 15 | .. [Fukushima] Fukushima, K. (1980). Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36, 193–202. 16 | 17 | .. [Hinton06] G.E. Hinton and R.R. Salakhutdinov, `Reducing the Dimensionality of Data with Neural Networks `_, Science, 28 July 2006, Vol. 313. no. 5786, pp. 504 - 507. 18 | 19 | .. [Hinton07] G.E. Hinton, S. Osindero, and Y. Teh, "A fast learning algorithm for deep belief nets", Neural Computation, vol 18, 2006 20 | 21 | .. [Hubel68] Hubel, D. and Wiesel, T. (1968). Receptive fields and functional architecture of monkey striate cortex. Journal of Physiology (London), 195, 215–243. 22 | 23 | .. [LeCun98] LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998d). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324. 24 | 25 | .. [Lee08] H. Lee, C. Ekanadham, and A.Y. Ng., `Sparse deep belief net model for visual area V2 `_, in Advances in Neural Information Processing Systems (NIPS) 20, 2008. 26 | 27 | .. [Lee09] H. Lee, R. Grosse, R. Ranganath, and A.Y. Ng, "Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations.", ICML 2009 28 | 29 | .. [Ranzato10] M. Ranzato, A. Krizhevsky, G. Hinton, "Factored 3-Way Restricted Boltzmann Machines for Modeling Natural Images". Proc. of the 13-th International Conference on Artificial Intelligence and Statistics (AISTATS 2010), Italy, 2010 30 | 31 | .. [Ranzato07] M.A. Ranzato, C. Poultney, S. Chopra and Y. LeCun, in J. Platt et al., `Efficient Learning of Sparse Representations with an Energy-Based Model `_, Advances in Neural Information Processing Systems (NIPS 2006), MIT Press, 2007. 32 | 33 | .. [Serre07] Serre, T., Wolf, L., Bileschi, S., and Riesenhuber, M. (2007). Robust object recog- nition with cortex-like mechanisms. IEEE Trans. Pattern Anal. Mach. Intell., 29(3), 411–426. Member-Poggio, Tomaso. 34 | 35 | .. [Vincent08] P. Vincent, H. Larochelle Y. Bengio and P.A. Manzagol, `Extracting and Composing Robust Features with Denoising Autoencoders `_, Proceedings of the Twenty-fifth International Conference on Machine Learning (ICML'08), pages 1096 - 1103, ACM, 2008. 36 | 37 | .. [Tieleman08] T. Tieleman, Training restricted boltzmann machines using approximations to the likelihood gradient, ICML 2008. 38 | 39 | .. [Xavier10] Y. Bengio, X. Glorot, Understanding the difficulty of training deep feedforward neuralnetworks, AISTATS 2010 40 | -------------------------------------------------------------------------------- /issues_open/6_benchmarking_pybrain.txt: -------------------------------------------------------------------------------- 1 | Reported by : Razvan 2 | 3 | Observations : 4 | 5 | 1. First thing, working with their dataset model is a pain ! Either I had 6 | not figure it out, or it allows you to add only one datapoint at a time 7 | in the dataset. This seems to me highly unoptimal ... 8 | 9 | 2. You do not get batches for sgd ! The only thing you can do is compare with 10 | batch size of 1. 11 | 12 | 3. Their early stopping is different from ours. Differences : 13 | - You can not set how often you do a pass on the validation set 14 | (i.e. ``patience`` in our case). You always do one epoch of training 15 | and then you go through the validation set. 16 | - You do not have an improvement thereshold, any improvement in 17 | validation score leads to storing the new best parameters, and 18 | increasing the time you will still look for better parameters 19 | - The increase is not by multiplication but summation. So if at 20 | epoch x you do better on the validation step, you will go on for 21 | x+y epochs to look for something better ( we do x*y ) 22 | 23 | 4. The errors return by pyBrain are divided by the number of 24 | classes. So if you do classification, you take the number of 25 | errors and divide it by the number of test examples times the 26 | number of classes. For MNIST this yields 10 times smaller 27 | errors. Is this something standard .. should we do it ? It 28 | definetelly makes error look smaller. 29 | 30 | 5. There is no straight forward way of adding L1/L2 regularization (from 31 | what I've seen), unless you go into their code and change it. That is not 32 | ard to do .. but for now I do not want to meangle with the library 33 | 34 | 6. The code for RBM is not ready (they say that it is work in progress). It seems to me that the 35 | code is wrong .. They have 3 loops, which to me would mean that the inner most is for CD-k ( 36 | second is for one epoch / third for training). But they update the weights after each Gibbs 37 | step in CD-k .. which results in a strage form of CD-1 that sees same example several time before 38 | moving to the next one. I could (?) potentially fix the code but it is outside the scope of 39 | benchmarking. 40 | 41 | 7. There are question marks of how easy it would be to implement a SdA ( autoassociators might be 42 | easy to do though). 43 | 44 | 45 | RESULTS : 46 | logistic_sgd on maggie46 47 | 48 | Total error: 0.015611011103 49 | Total error: 0.00966772673335 50 | Total error: 0.00860664508883 51 | Time spend per epoch: 43.32 52 | Final error is : 10.44 53 | Time spend per epoch: 43.32 54 | Final error is : 10.44 55 | 56 | Arac : 57 | 58 | Total error: 0.0366924968888 59 | Total error: 0.0366576944937 60 | Total error: 0.0367442383338 61 | Time spend per epoch: 24.71 62 | Final error is : 69.28 63 | Time spend per epoch: 24.71 64 | Final error is : 69.28 65 | 66 | 67 | ** Our thing with batchsize =1 ** 68 | 69 | test error of best model 8.45 70 | time : 12.99 71 | 12.01 72 | 73 | 74 | 75 | 76 | Results : 77 | mlp on maggie46 78 | 79 | 80 | pybrain :: 81 | 82 | Total error: 0.0124744609817 83 | Total error: 0.00722484141084 84 | Total error: 0.00599591269763 85 | Time spend per epoch : 1226.69 86 | Final error is : 8.68 87 | Time spend per epoch: 1226.69 88 | Final error is : 8.68 89 | 90 | 20.4448 min 91 | 92 | arac:: 93 | 94 | Total error: 0.0318599056504 95 | Total error: 0.0316029246672 96 | Total error: 0.0315542295953 97 | Time spend per epoch: 860.336666667 (s) 98 | Final error is : 58.59 99 | 100 | our thing:: 101 | 102 | test error of best model 3.88 103 | time: 381.92 104 | 105 | -------------------------------------------------------------------------------- /doc/intro.txt: -------------------------------------------------------------------------------- 1 | ======================= 2 | Deep Learning Tutorials 3 | ======================= 4 | 5 | Deep Learning is a new area of Machine Learning research, which 6 | has been introduced with the objective of moving Machine Learning 7 | closer to one of its original goals: Artificial Intelligence. 8 | See these course notes for a `brief introduction to Machine Learning for AI `_ 9 | and an `introduction to Deep Learning algorithms `_. 10 | 11 | Deep Learning is about learning multiple levels of representation 12 | and abstraction that help to 13 | make sense of data such as images, sound, and text. 14 | For more about deep learning algorithms, see for example: 15 | 16 | - The monograph or review paper `Learning Deep Architectures for AI `_ (Foundations & Trends in Machine Learning, 2009). 17 | - The ICML 2009 Workshop on Learning Feature Hierarchies `webpage `_ has a `list of references `_. 18 | - The LISA `public wiki `_ has a `reading list `_ and a `bibliography `_. 19 | - Geoff Hinton has `readings `_ from last year's `NIPS tutorial `_. 20 | 21 | The tutorials presented here will introduce you to some of the most important deep learning 22 | algorithms and will also show you how to run them using Theano_. Theano is a python library that makes writing deep learning models easy, and gives the option of 23 | training them on a GPU. 24 | 25 | The algorithm tutorials have some prerequisites. You should know some python, 26 | and be familiar with numpy. Since this tutorial is about using Theano, you 27 | should read over the `Theano basic tutorial`_ first. Once you've done that, 28 | read through our :ref:`gettingstarted` chapter -- it introduces the notation, and [downloadable] datasets used in the algorithm tutorials, and the way we do optimization by stochastic gradient descent. 29 | 30 | The purely supervised learning algorithms are meant to be read in order: 31 | 32 | #. :ref:`Logistic Regression ` - using Theano for something simple 33 | #. :ref:`Multilayer perceptron ` - introduction to layers 34 | #. :ref:`Deep Convolutional Network ` - a simplified version of LeNet5 35 | 36 | The unsupervised and semi-supervised learning algorithms can be read in any 37 | order (the auto-encoders can be read independently of the RBM/DBN thread): 38 | 39 | * :ref:`Auto Encoders, Denoising Autoencoders ` - description of autoencoders 40 | * :ref:`Stacked Denoising Auto-Encoders ` - easy steps into unsupervised pre-training for deep nets 41 | * :ref:`Restricted Boltzmann Machines ` - single layer generative RBM model 42 | * :ref:`Deep Belief Networks ` - unsupervised generative pre-training of stacked RBMs followed by supervised fine-tuning 43 | 44 | Building towards including the mcRBM model, we have a new tutorial on sampling 45 | from energy models: 46 | 47 | * :ref:`HMC Sampling ` - hybrid (aka Hamiltonian) Monte-Carlo sampling with scan() 48 | 49 | Building towards including the Contractive auto-encoders tutorial, we have the code for now: 50 | * `Contractive auto-encoders`_ code - There is some basic doc in the code. 51 | 52 | Energy-based recurrent neural network (RNN-RBM): 53 | * :ref:`Modeling and generating sequences of polyphonic music ` 54 | 55 | .. _Theano: http://deeplearning.net/software/theano 56 | 57 | .. _Theano basic tutorial: http://deeplearning.net/software/theano/tutorial 58 | 59 | .. _Contractive auto-encoders: https://github.com/lisa-lab/DeepLearningTutorials/blob/master/code/cA.py 60 | -------------------------------------------------------------------------------- /doc/deep.txt: -------------------------------------------------------------------------------- 1 | .. _deep: 2 | 3 | Deep Learning 4 | ============= 5 | 6 | The breakthrough to effective training strategies for deep architectures came in 7 | 2006 with the algorithms for training deep belief networks 8 | (DBN) [Hinton07]_ and stacked auto-encoders [Ranzato07]_ , [Bengio07]_ . 9 | All these methods are based on a similar approach: **greedy layer-wise unsupervised 10 | pre-training** followed by **supervised fine-tuning**. 11 | 12 | The pretraining strategy consists in using unsupervised learning to guide the 13 | training of intermediate levels of representation. Each layer is pre-trained 14 | with an unsupervised learning algorithm, which attempts to learn a nonlinear 15 | transformation of its input, in order to captures its main variations. Higher 16 | levels of abstractions are created by feeding the output of one layer, to the 17 | input of the subsequent layer. 18 | 19 | The resulting an architecture can then be seen in two lights: 20 | 21 | * the pre-trained deep network can be used to initialize the weights of all, but 22 | the last layer of a deep neural network. The weights are then further adapted 23 | to a supervised task (such as classification) through traditional gradient 24 | descent (see :ref:`Multilayer perceptron `). This is referred to as the 25 | fine-tuning step. 26 | 27 | * the pre-trained deep network can also serve solely as a feature extractor. The 28 | output of the last layer is fed to a classifier, such as logistic regression, 29 | which is trained independently. Better results can be obtained by 30 | concatenating the output of the last layer, with the hidden representations of 31 | all intermediate layers [Lee09]_. 32 | 33 | For the purposes of this tutorial, we will focus on the first interpretation, 34 | as that is what was first proposed in [Hinton06]_. 35 | 36 | Deep Coding 37 | +++++++++++ 38 | 39 | Since Deep Belief Networks (DBN) and Stacked Denoising-AutoEncoders (SDA) share 40 | much of the same architecture and have very similar training algorithms (in 41 | terms of pretraining and fine-tuning stages), it makes sense to implement them 42 | in a similar fashion, as part of a "Deep Learning" framework. 43 | 44 | We thus define a generic interface, which both of these architectures will 45 | share. 46 | 47 | .. code-block:: python 48 | 49 | class DeepLayerwiseModel(object): 50 | 51 | def layerwise_pretrain(self, layer_fns, pretrain_amounts): 52 | """ 53 | """ 54 | 55 | def finetune(self, datasets, lr, batch_size): 56 | """ 57 | 58 | class DBN(DeepLayerwiseModel): 59 | """ 60 | """ 61 | 62 | class StackedDAA(DeepLayerwiseModel): 63 | """ 64 | """ 65 | 66 | .. code-block:: python 67 | 68 | def deep_main(learning_rate=0.1, 69 | pretraining_epochs=20, 70 | pretrain_lr=0.1, 71 | training_epochs=1000, 72 | batch_size=20, 73 | mnist_file='mnist.pkl.gz'): 74 | 75 | n_train_examples, train_valid_test = load_mnist(mnist_file) 76 | 77 | # instantiate model 78 | deep_model = ... 79 | 80 | #### 81 | #### Phase 1: Pre-training 82 | #### 83 | 84 | # create an array of functions, which will be used for the greedy 85 | # layer-wise unsupervised training procedure 86 | 87 | pretrain_functions = deep_model.pretrain_functions( 88 | batch_size=batch_size, 89 | train_set_x=train_set_x, 90 | learning_rate=pretrain_lr, 91 | ... 92 | ) 93 | 94 | # loop over all the layers in our network 95 | for layer_idx, pretrain_fn in enumerate(pretrain_functions): 96 | 97 | # iterate over a certain number of epochs) 98 | for i in xrange(pretraining_epochs * n_train_examples / batch_size): 99 | 100 | # follow one step in the gradient of the unsupervised cost 101 | # function, at the given layer 102 | layer_fn(i) 103 | 104 | 105 | .. code-block:: python 106 | 107 | #### 108 | #### Phase 2: Fine Tuning 109 | #### 110 | 111 | # create theano functions for fine-tuning, as well as 112 | # validation and testing our model. 113 | 114 | train_fn, valid_scores, test_scores =\ 115 | deep_model.finetune_functions( 116 | train_valid_test[0][0], # training dataset 117 | learning_rate=finetune_lr, # the learning rate 118 | batch_size=batch_size) # number of examples to use at once 119 | 120 | 121 | # use these functions as part of the generic early-stopping procedure 122 | for i in xrange(patience_max): 123 | 124 | if i >= patience: 125 | break 126 | 127 | cost_i = train_fn(i) 128 | 129 | ... 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | -------------------------------------------------------------------------------- /code/utils.py: -------------------------------------------------------------------------------- 1 | """ This file contains different utility functions that are not connected 2 | in anyway to the networks presented in the tutorials, but rather help in 3 | processing the outputs into a more understandable way. 4 | 5 | For example ``tile_raster_images`` helps in generating a easy to grasp 6 | image from a set of samples or weights. 7 | """ 8 | 9 | 10 | import numpy 11 | 12 | 13 | def scale_to_unit_interval(ndar, eps=1e-8): 14 | """ Scales all values in the ndarray ndar to be between 0 and 1 """ 15 | ndar = ndar.copy() 16 | ndar -= ndar.min() 17 | ndar *= 1.0 / (ndar.max() + eps) 18 | return ndar 19 | 20 | 21 | def tile_raster_images(X, img_shape, tile_shape, tile_spacing=(0, 0), 22 | scale_rows_to_unit_interval=True, 23 | output_pixel_vals=True): 24 | """ 25 | Transform an array with one flattened image per row, into an array in 26 | which images are reshaped and layed out like tiles on a floor. 27 | 28 | This function is useful for visualizing datasets whose rows are images, 29 | and also columns of matrices for transforming those rows 30 | (such as the first layer of a neural net). 31 | 32 | :type X: a 2-D ndarray or a tuple of 4 channels, elements of which can 33 | be 2-D ndarrays or None; 34 | :param X: a 2-D array in which every row is a flattened image. 35 | 36 | :type img_shape: tuple; (height, width) 37 | :param img_shape: the original shape of each image 38 | 39 | :type tile_shape: tuple; (rows, cols) 40 | :param tile_shape: the number of images to tile (rows, cols) 41 | 42 | :param output_pixel_vals: if output should be pixel values (i.e. int8 43 | values) or floats 44 | 45 | :param scale_rows_to_unit_interval: if the values need to be scaled before 46 | being plotted to [0,1] or not 47 | 48 | 49 | :returns: array suitable for viewing as an image. 50 | (See:`Image.fromarray`.) 51 | :rtype: a 2-d array with same dtype as X. 52 | 53 | """ 54 | 55 | assert len(img_shape) == 2 56 | assert len(tile_shape) == 2 57 | assert len(tile_spacing) == 2 58 | 59 | # The expression below can be re-written in a more C style as 60 | # follows : 61 | # 62 | # out_shape = [0,0] 63 | # out_shape[0] = (img_shape[0]+tile_spacing[0])*tile_shape[0] - 64 | # tile_spacing[0] 65 | # out_shape[1] = (img_shape[1]+tile_spacing[1])*tile_shape[1] - 66 | # tile_spacing[1] 67 | out_shape = [ 68 | (ishp + tsp) * tshp - tsp 69 | for ishp, tshp, tsp in zip(img_shape, tile_shape, tile_spacing) 70 | ] 71 | 72 | if isinstance(X, tuple): 73 | assert len(X) == 4 74 | # Create an output numpy ndarray to store the image 75 | if output_pixel_vals: 76 | out_array = numpy.zeros((out_shape[0], out_shape[1], 4), 77 | dtype='uint8') 78 | else: 79 | out_array = numpy.zeros((out_shape[0], out_shape[1], 4), 80 | dtype=X.dtype) 81 | 82 | #colors default to 0, alpha defaults to 1 (opaque) 83 | if output_pixel_vals: 84 | channel_defaults = [0, 0, 0, 255] 85 | else: 86 | channel_defaults = [0., 0., 0., 1.] 87 | 88 | for i in xrange(4): 89 | if X[i] is None: 90 | # if channel is None, fill it with zeros of the correct 91 | # dtype 92 | dt = out_array.dtype 93 | if output_pixel_vals: 94 | dt = 'uint8' 95 | out_array[:, :, i] = numpy.zeros( 96 | out_shape, 97 | dtype=dt 98 | ) + channel_defaults[i] 99 | else: 100 | # use a recurrent call to compute the channel and store it 101 | # in the output 102 | out_array[:, :, i] = tile_raster_images( 103 | X[i], img_shape, tile_shape, tile_spacing, 104 | scale_rows_to_unit_interval, output_pixel_vals) 105 | return out_array 106 | 107 | else: 108 | # if we are dealing with only one channel 109 | H, W = img_shape 110 | Hs, Ws = tile_spacing 111 | 112 | # generate a matrix to store the output 113 | dt = X.dtype 114 | if output_pixel_vals: 115 | dt = 'uint8' 116 | out_array = numpy.zeros(out_shape, dtype=dt) 117 | 118 | for tile_row in xrange(tile_shape[0]): 119 | for tile_col in xrange(tile_shape[1]): 120 | if tile_row * tile_shape[1] + tile_col < X.shape[0]: 121 | this_x = X[tile_row * tile_shape[1] + tile_col] 122 | if scale_rows_to_unit_interval: 123 | # if we should scale values to be between 0 and 1 124 | # do this by calling the `scale_to_unit_interval` 125 | # function 126 | this_img = scale_to_unit_interval( 127 | this_x.reshape(img_shape)) 128 | else: 129 | this_img = this_x.reshape(img_shape) 130 | # add the slice to the corresponding position in the 131 | # output array 132 | c = 1 133 | if output_pixel_vals: 134 | c = 255 135 | out_array[ 136 | tile_row * (H + Hs): tile_row * (H + Hs) + H, 137 | tile_col * (W + Ws): tile_col * (W + Ws) + W 138 | ] = this_img * c 139 | return out_array 140 | -------------------------------------------------------------------------------- /doc/utilities.txt: -------------------------------------------------------------------------------- 1 | ============= 2 | Miscellaneous 3 | ============= 4 | 5 | .. _how-to-plot: 6 | 7 | Plotting Samples and Filters 8 | ++++++++++++++++++++++++++++ 9 | 10 | .. note:: 11 | The code for this section is available for download `here`_. 12 | 13 | .. _here: http://deeplearning.net/tutorial/code/utils.py 14 | 15 | 16 | To plot a sample, what we need to do is to take the visible units, which 17 | are a flattened image (there is no 2D structure to the visible units, 18 | just a 1D string of unit activations) and reshape it into a 2D image. The order in 19 | which the points from the 1D array go into the 2D image is given by the 20 | order in which the inital MNIST images where converted into a 1D array. 21 | Lucky for us this is just a call of the ``numpy.reshape`` function. 22 | 23 | Plotting the weights is a bit more tricky. We have ``n_hidden`` hidden 24 | units, each of them corresponding to a column of the weight matrix. A 25 | column has the same shape as the visible, where the weight corresponding 26 | to the connection with visible unit `j` is at position `j`. Therefore, 27 | if we reshape every such column, using ``numpy.reshape``, we get a 28 | filter image that tells us how this hidden unit is influenced by 29 | the input image. 30 | 31 | We need a utility function that takes a minibatch, or the weight matrix, 32 | and converts each row ( for the weight matrix we do a transpose ) into a 33 | 2D image and then tile these images together. Once we converted the 34 | minibatch or the weights in this image of tiles, we can use PIL to plot 35 | and save. `PIL `_ is a standard 36 | python libarary to deal with images. 37 | 38 | Tiling minibatches together is done for us by the 39 | ``tile_raster_image`` function which we provide here. 40 | 41 | .. code-block:: python 42 | 43 | 44 | def scale_to_unit_interval(ndar, eps=1e-8): 45 | """ Scales all values in the ndarray ndar to be between 0 and 1 """ 46 | ndar = ndar.copy() 47 | ndar -= ndar.min() 48 | ndar *= 1.0 / (ndar.max() + eps) 49 | return ndar 50 | 51 | 52 | def tile_raster_images(X, img_shape, tile_shape, tile_spacing=(0, 0), 53 | scale_rows_to_unit_interval=True, 54 | output_pixel_vals=True): 55 | """ 56 | Transform an array with one flattened image per row, into an array in 57 | which images are reshaped and layed out like tiles on a floor. 58 | 59 | This function is useful for visualizing datasets whose rows are images, 60 | and also columns of matrices for transforming those rows 61 | (such as the first layer of a neural net). 62 | 63 | :type X: a 2-D ndarray or a tuple of 4 channels, elements of which can 64 | be 2-D ndarrays or None; 65 | :param X: a 2-D array in which every row is a flattened image. 66 | 67 | :type img_shape: tuple; (height, width) 68 | :param img_shape: the original shape of each image 69 | 70 | :type tile_shape: tuple; (rows, cols) 71 | :param tile_shape: the number of images to tile (rows, cols) 72 | 73 | :param output_pixel_vals: if output should be pixel values (i.e. int8 74 | values) or floats 75 | 76 | :param scale_rows_to_unit_interval: if the values need to be scaled before 77 | being plotted to [0,1] or not 78 | 79 | 80 | :returns: array suitable for viewing as an image. 81 | (See:`Image.fromarray`.) 82 | :rtype: a 2-d array with same dtype as X. 83 | 84 | """ 85 | 86 | assert len(img_shape) == 2 87 | assert len(tile_shape) == 2 88 | assert len(tile_spacing) == 2 89 | 90 | # The expression below can be re-written in a more C style as 91 | # follows : 92 | # 93 | # out_shape = [0,0] 94 | # out_shape[0] = (img_shape[0] + tile_spacing[0]) * tile_shape[0] - 95 | # tile_spacing[0] 96 | # out_shape[1] = (img_shape[1] + tile_spacing[1]) * tile_shape[1] - 97 | # tile_spacing[1] 98 | out_shape = [(ishp + tsp) * tshp - tsp for ishp, tshp, tsp 99 | in zip(img_shape, tile_shape, tile_spacing)] 100 | 101 | if isinstance(X, tuple): 102 | assert len(X) == 4 103 | # Create an output numpy ndarray to store the image 104 | if output_pixel_vals: 105 | out_array = numpy.zeros((out_shape[0], out_shape[1], 4), dtype='uint8') 106 | else: 107 | out_array = numpy.zeros((out_shape[0], out_shape[1], 4), dtype=X.dtype) 108 | 109 | #colors default to 0, alpha defaults to 1 (opaque) 110 | if output_pixel_vals: 111 | channel_defaults = [0, 0, 0, 255] 112 | else: 113 | channel_defaults = [0., 0., 0., 1.] 114 | 115 | for i in xrange(4): 116 | if X[i] is None: 117 | # if channel is None, fill it with zeros of the correct 118 | # dtype 119 | out_array[:, :, i] = numpy.zeros(out_shape, 120 | dtype='uint8' if output_pixel_vals else out_array.dtype 121 | ) + channel_defaults[i] 122 | else: 123 | # use a recurrent call to compute the channel and store it 124 | # in the output 125 | out_array[:, :, i] = tile_raster_images(X[i], img_shape, tile_shape, tile_spacing, scale_rows_to_unit_interval, output_pixel_vals) 126 | return out_array 127 | 128 | else: 129 | # if we are dealing with only one channel 130 | H, W = img_shape 131 | Hs, Ws = tile_spacing 132 | 133 | # generate a matrix to store the output 134 | out_array = numpy.zeros(out_shape, dtype='uint8' if output_pixel_vals else X.dtype) 135 | 136 | 137 | for tile_row in xrange(tile_shape[0]): 138 | for tile_col in xrange(tile_shape[1]): 139 | if tile_row * tile_shape[1] + tile_col < X.shape[0]: 140 | if scale_rows_to_unit_interval: 141 | # if we should scale values to be between 0 and 1 142 | # do this by calling the `scale_to_unit_interval` 143 | # function 144 | this_img = scale_to_unit_interval(X[tile_row * tile_shape[1] + tile_col].reshape(img_shape)) 145 | else: 146 | this_img = X[tile_row * tile_shape[1] + tile_col].reshape(img_shape) 147 | # add the slice to the corresponding position in the 148 | # output array 149 | out_array[ 150 | tile_row * (H+Hs): tile_row * (H + Hs) + H, 151 | tile_col * (W+Ws): tile_col * (W + Ws) + W 152 | ] \ 153 | = this_img * (255 if output_pixel_vals else 1) 154 | return out_array 155 | -------------------------------------------------------------------------------- /doc/conf.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # theano documentation build configuration file, created by 4 | # sphinx-quickstart on Tue Oct 7 16:34:06 2008. 5 | # 6 | # This file is execfile()d with the current directory set to its containing dir. 7 | # 8 | # The contents of this file are pickled, so don't put values in the namespace 9 | # that aren't pickleable (module imports are okay, they're removed automatically). 10 | # 11 | # All configuration values have a default value; values that are commented out 12 | # serve to show the default value. 13 | import sys, os 14 | 15 | # If your extensions are in another directory, add it here. If the directory 16 | # is relative to the documentation root, use os.path.abspath to make it 17 | # absolute, like shown here. 18 | #sys.path.append(os.path.abspath('some/directory')) 19 | 20 | # General configuration 21 | # --------------------- 22 | 23 | # Add any Sphinx extension module names here, as strings. They can be extensions 24 | # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. 25 | extensions = ['sphinx.ext.autodoc', 'sphinx.ext.todo'] 26 | 27 | try: 28 | from sphinx.ext import pngmath 29 | extensions.append('sphinx.ext.pngmath') 30 | except ImportError: 31 | print >>sys.stderr, 'Warning: could not import sphinx.ext.pngmath' 32 | pass 33 | 34 | # Add any paths that contain templates here, relative to this directory. 35 | templates_path = ['.templates'] 36 | 37 | # The suffix of source filenames. 38 | source_suffix = '.txt' 39 | 40 | # The master toctree document. 41 | master_doc = 'contents' 42 | 43 | # General substitutions. 44 | project = 'DeepLearning' 45 | copyright = '2008--2010, LISA lab' 46 | 47 | # The default replacements for |version| and |release|, also used in various 48 | # other places throughout the built documents. 49 | # 50 | # The short X.Y version. 51 | version = '0.1' 52 | # The full version, including alpha/beta/rc tags. 53 | release = '0.1' 54 | 55 | # There are two options for replacing |today|: either, you set today to some 56 | # non-false value, then it is used: 57 | #today = '' 58 | # Else, today_fmt is used as the format for a strftime call. 59 | today_fmt = '%B %d, %Y' 60 | 61 | # List of documents that shouldn't be included in the build. 62 | #unused_docs = [] 63 | 64 | # List of directories, relative to source directories, that shouldn't be searched 65 | # for source files. 66 | exclude_dirs = ['scripts'] 67 | 68 | # The reST default role (used for this markup: `text`) to use for all documents. 69 | #default_role = None 70 | 71 | # If true, '()' will be appended to :func: etc. cross-reference text. 72 | #add_function_parentheses = True 73 | 74 | # If true, the current module name will be prepended to all description 75 | # unit titles (such as .. function::). 76 | #add_module_names = True 77 | 78 | # If true, sectionauthor and moduleauthor directives will be shown in the 79 | # output. They are ignored by default. 80 | #show_authors = False 81 | 82 | # The name of the Pygments (syntax highlighting) style to use. 83 | pygments_style = 'sphinx' 84 | 85 | 86 | # Options for HTML output 87 | # ----------------------- 88 | 89 | # The style sheet to use for HTML and HTML Help pages. A file of that name 90 | # must exist either in Sphinx' static/ path, or in one of the custom paths 91 | # given in html_static_path. 92 | #html_style = 'default.css' 93 | html_theme = 'sphinxdoc' 94 | 95 | # The name for this set of Sphinx documents. If None, it defaults to 96 | # " v documentation". 97 | #html_title = None 98 | 99 | # A shorter title for the navigation bar. Default is the same as html_title. 100 | #html_short_title = None 101 | 102 | # The name of an image file (within the static path) to place at the top of 103 | # the sidebar. 104 | #html_logo = None 105 | 106 | # The name of an image file (within the static path) to use as favicon of the 107 | # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 108 | # pixels large. 109 | #html_favicon = None 110 | 111 | # Add any paths that contain custom static files (such as style sheets) here, 112 | # relative to this directory. They are copied after the builtin static files, 113 | # so a file named "default.css" will overwrite the builtin "default.css". 114 | #html_static_path = ['.static', 'images'] 115 | html_static_path = ['images'] 116 | 117 | # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, 118 | # using the given strftime format. 119 | html_last_updated_fmt = '%b %d, %Y' 120 | 121 | # If true, SmartyPants will be used to convert quotes and dashes to 122 | # typographically correct entities. 123 | html_use_smartypants = True 124 | 125 | # Custom sidebar templates, maps document names to template names. 126 | #html_sidebars = {} 127 | 128 | # Additional templates that should be rendered to pages, maps page names to 129 | # template names. 130 | #html_additional_pages = {} 131 | 132 | # If false, no module index is generated. 133 | html_use_modindex = True 134 | 135 | # If false, no index is generated. 136 | html_use_index = True 137 | 138 | # If true, the index is split into individual pages for each letter. 139 | #html_split_index = False 140 | 141 | # If true, the reST sources are included in the HTML build as _sources/. 142 | #html_copy_source = True 143 | 144 | # If true, an OpenSearch description file will be output, and all pages will 145 | # contain a tag referring to it. The value of this option must be the 146 | # base URL from which the finished HTML is served. 147 | #html_use_opensearch = '' 148 | 149 | # If nonempty, this is the file name suffix for HTML files (e.g. ".xhtml"). 150 | #html_file_suffix = '' 151 | 152 | # Output file base name for HTML help builder. 153 | htmlhelp_basename = 'deeplearningdoc' 154 | 155 | 156 | # Options for LaTeX output 157 | # ------------------------ 158 | 159 | # The paper size ('letter' or 'a4'). 160 | #latex_paper_size = 'letter' 161 | 162 | # The font size ('10pt', '11pt' or '12pt'). 163 | latex_font_size = '11pt' 164 | 165 | # Grouping the document tree into LaTeX files. List of tuples 166 | # (source start file, target name, title, author, document class [howto/manual]). 167 | latex_documents = [ 168 | ('contents', 'deeplearning.tex', 'Deep Learning Tutorial', 169 | 'LISA lab, University of Montreal', 'manual'), 170 | ] 171 | 172 | # The name of an image file (relative to this directory) to place at the top of 173 | # the title page. 174 | latex_logo = None 175 | 176 | # For "manual" documents, if this is true, then toplevel headings are parts, 177 | # not chapters. 178 | #latex_use_parts = False 179 | 180 | # Additional stuff for the LaTeX preamble. 181 | #latex_preamble = '' 182 | 183 | # Documents to append as an appendix to all manuals. 184 | #latex_appendices = [] 185 | 186 | # If false, no module index is generated. 187 | #latex_use_modindex = True 188 | 189 | default_role = 'math' 190 | pngmath_divpng_args = ['-gamma 1.5','-D 110'] 191 | pngmath_latex_preamble = '\\usepackage{amsmath}\n'+\ 192 | '\\usepackage{amsfonts}\n'+\ 193 | '\\usepackage{amssymb}\n'+\ 194 | '\\def\\E{\\mathbf{E}}\n'+\ 195 | '\\def\\F{\\mathbf{F}}\n'+\ 196 | '\\def\\x{\\mathbf{x}}\n'+\ 197 | '\\def\\h{\\mathbf{h}}\n'+\ 198 | '\\def\\v{\\mathbf{v}}\n'+\ 199 | '\\def\\nv{\\mathbf{v^{{\bf -}}}}\n'+\ 200 | '\\def\\nh{\\mathbf{h^{{\bf -}}}}\n'+\ 201 | '\\def\\s{\\mathbf{s}}\n'+\ 202 | '\\def\\b{\\mathbf{b}}\n'+\ 203 | '\\def\\c{\\mathbf{c}}\n'+\ 204 | '\\def\\W{\\mathbf{W}}\n'+\ 205 | '\\def\\C{\\mathbf{C}}\n'+\ 206 | '\\def\\P{\\mathbf{P}}\n'+\ 207 | '\\def\\T{{\\bf \\mathcal T}}\n'+\ 208 | '\\def\\B{{\\bf \\mathcal B}}\n' 209 | -------------------------------------------------------------------------------- /doc/rnnrbm.txt: -------------------------------------------------------------------------------- 1 | .. _rnnrbm: 2 | 3 | Modeling and generating sequences of polyphonic music with the RNN-RBM 4 | ======================================================================== 5 | 6 | .. note:: 7 | This tutorial demonstrates a basic implementation of the RNN-RBM as described in [BoulangerLewandowski12]_ 8 | (`pdf `_). 9 | We assume the reader is familiar with 10 | `recurrent neural networks using the scan op `_ 11 | and `restricted Boltzmann machines (RBM) `_. 12 | 13 | .. note:: 14 | The code for this section is available for download here: `rnnrbm.py `_. 15 | 16 | You will need the modified `Python MIDI package (GPL license) `_ in your ``$PYTHONPATH`` or in the working directory in order to convert MIDI files to and from piano-rolls. 17 | The script also assumes that the content of the `Nottingham Database of folk tunes `_ has been extracted in the ``../data`` directory. 18 | Alternative MIDI datasets are available `here `_. 19 | 20 | Note that both dependencies above can be setup automatically by running the ``download.sh`` script in the ``../data`` directory. 21 | 22 | .. caution:: 23 | Need Theano 0.6 or more recent. 24 | 25 | 26 | The RNN-RBM 27 | +++++++++++++++++++++++++ 28 | 29 | The RNN-RBM is an energy-based model for density estimation of temporal sequences, where the feature vector :math:`v^{(t)}` at time step :math:`t` may be high-dimensional. 30 | It allows to describe multimodal conditional distributions of :math:`v^{(t)}|\mathcal A^{(t)}`, where :math:`\mathcal A^{(t)}\equiv \{v_\tau|\tau