├── 2017 ├── README.md ├── assignments │ ├── chatbot │ │ ├── README.md │ │ ├── chatbot.py │ │ ├── config.py │ │ ├── data.py │ │ ├── model.py │ │ └── output_convo.txt │ ├── exercises │ │ ├── e01.py │ │ └── e01_sol.py │ ├── style_transfer │ │ ├── content │ │ │ ├── deadpool.jpg │ │ │ └── resized_deadpool.jpg │ │ ├── readme.md │ │ ├── style_transfer.py │ │ ├── styles │ │ │ ├── guernica.jpg │ │ │ ├── harlequin.jpg │ │ │ ├── pattern.jpg │ │ │ ├── resized_guernica.jpg │ │ │ └── starry_night.jpg │ │ ├── utils.py │ │ └── vgg_model.py │ └── style_transfer_starter │ │ ├── content │ │ ├── deadpool.jpg │ │ └── resized_deadpool.jpg │ │ ├── readme.md │ │ ├── style_transfer.py │ │ ├── styles │ │ ├── guernica.jpg │ │ ├── harlequin.jpg │ │ ├── pattern.jpg │ │ ├── resized_guernica.jpg │ │ └── starry_night.jpg │ │ ├── utils.py │ │ └── vgg_model.py ├── data │ ├── arvix_abstracts.txt │ ├── fire_theft.xls │ ├── friday.jpg │ ├── friday.tfrecord │ ├── heart.csv │ └── heart.txt ├── examples │ ├── 02_feed_dict.py │ ├── 02_lazy_loading.py │ ├── 02_simple_tf.py │ ├── 02_variables.py │ ├── 03_linear_regression_sol.py │ ├── 03_linear_regression_starter.py │ ├── 03_logistic_regression_mnist_sol.py │ ├── 03_logistic_regression_mnist_starter.py │ ├── 04_word2vec_no_frills.py │ ├── 04_word2vec_starter.py │ ├── 04_word2vec_visualize.py │ ├── 05_csv_reader.py │ ├── 05_randomization.py │ ├── 07_basic_filters.py │ ├── 07_convnet_mnist.py │ ├── 07_convnet_mnist_starter.py │ ├── 09_queue_example.py │ ├── 09_tfrecord_example.py │ ├── 11_char_rnn_gist.py │ ├── autoencoder │ │ ├── autoencoder.py │ │ ├── layer_utils.py │ │ ├── layers.py │ │ ├── train.py │ │ ├── utils.py │ │ └── vis │ │ │ ├── test_1.png │ │ │ ├── test_10.png │ │ │ ├── test_2.png │ │ │ ├── test_3.png │ │ │ ├── test_4.png │ │ │ ├── test_5.png │ │ │ ├── test_6.png │ │ │ ├── test_7.png │ │ │ ├── test_8.png │ │ │ └── test_9.png │ ├── cgru │ │ ├── README.md │ │ ├── custom_getter.py │ │ ├── data_reader.py │ │ ├── my_layers.py │ │ └── neural_gpu_v3.py │ ├── data │ │ ├── arvix_abstracts.txt │ │ ├── fire_theft.xls │ │ ├── friday.jpg │ │ ├── heart.csv │ │ └── heart.txt │ ├── deepdream │ │ ├── deepdream_exercise.py │ │ ├── deepdream_solution.py │ │ └── output.jpg │ ├── graphs │ │ ├── gist │ │ │ ├── events.out.tfevents.1499787135.MacBook-Pro │ │ │ ├── events.out.tfevents.1499787150.MacBook-Pro │ │ │ └── events.out.tfevents.1499787321.MacBook-Pro │ │ ├── l2 │ │ │ ├── events.out.tfevents.1499786503.MacBook-Pro │ │ │ └── events.out.tfevents.1499786515.MacBook-Pro │ │ └── linear_reg │ │ │ └── events.out.tfevents.1499786822.MacBook-Pro │ ├── kernels.py │ ├── process_data.py │ └── utils.py └── setup │ ├── requirements.txt │ └── setup_instruction.md ├── .gitignore ├── LICENSE ├── README.md ├── assignments ├── 01 │ ├── q1.py │ └── q1_sol.py ├── 02_style_transfer │ ├── load_vgg.py │ ├── load_vgg_sol.py │ ├── style_transfer.py │ ├── style_transfer_sol.py │ └── utils.py ├── chatbot │ ├── README.md │ ├── chatbot.py │ ├── config.py │ ├── data.py │ ├── model.py │ └── output_convo.txt ├── trump_bot │ └── trump_tweets.txt └── word_transform │ ├── common.en.vocab │ ├── eval.vocab │ └── train.vocab ├── examples ├── 02_lazy_loading.py ├── 02_placeholder.py ├── 02_simple_tf.py ├── 02_variables.py ├── 03_linreg_dataset.py ├── 03_linreg_placeholder.py ├── 03_linreg_starter.py ├── 03_logreg.py ├── 03_logreg_placeholder.py ├── 03_logreg_starter.py ├── 04_linreg_eager.py ├── 04_linreg_eager_starter.py ├── 04_word2vec.py ├── 04_word2vec_eager.py ├── 04_word2vec_eager_starter.py ├── 04_word2vec_visualize.py ├── 05_randomization.py ├── 05_variable_sharing.py ├── 07_convnet_layers.py ├── 07_convnet_mnist.py ├── 07_convnet_mnist_starter.py ├── 07_run_kernels.py ├── 11_char_rnn.py ├── data │ ├── abstracts.txt │ ├── birth_life_2010.txt │ └── trump_tweets.txt ├── kernels.py ├── utils.py └── word2vec_utils.py └── setup ├── requirements.txt └── setup_instruction.md /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | *.pdf 3 | 4 | *.SUNet 5 | *.pyc 6 | .env/* 7 | examples/data 8 | examples/graphs/* 9 | examples/checkpoints/* 10 | examples/visualization/* 11 | -------------------------------------------------------------------------------- /2017/README.md: -------------------------------------------------------------------------------- 1 | # tf-stanford-tutorials 2 | This repository contains code examples for the 2017 course CS 20SI: TensorFlow for Deep Learning Research.
3 | Detailed syllabus and lecture notes can be found [here](http://cs20si.stanford.edu/2017).
4 | For this year's course, please see [here](http://cs20.stanford.edu) 5 | 6 | # Note (as of July 11, 2017) 7 | I've updated the code to TensorFlow 1.2 and Python3, except the code for chatbot. I will update the code for chatbot soon. 8 | 9 | 10 | 11 | ## Models include:
12 | ### In the folder "examples":
13 | Linear Regression with Chicago's Fire-Theft dataset
14 | Logistic Regression with MNIST
15 | Word2vec skip-gram model with NCE loss
16 | Convnets with MNIST
17 | Autoencoder (by Nishith Khandwala)
18 | Deepdream (by Jon Shlens)
19 | Character-level language modeling
20 |
21 | ### In the folder "assignments":
22 | Style Transfer
23 | Chatbot using sequence to sequence with attention
24 |
25 | ## Misc
26 | Examples on how to use data readers, TFRecord
27 | Embedding visualization with TensorBoard
28 | Usage of summary ops
29 | Exercises to be familiar with other special TensorFlow ops
30 | Demonstration of the danger of lazy loading
31 | Convolutional GRU (CRGU) (by Lukasz Kaiser) -------------------------------------------------------------------------------- /2017/assignments/chatbot/README.md: -------------------------------------------------------------------------------- 1 | <<<<<<< 2 | A neural chatbot using sequence to sequence model with 3 | attentional decoder. This is a fully functional chatbot. 4 | 5 | This is based on Google Translate Tensorflow model 6 | https://github.com/tensorflow/models/blob/master/tutorials/rnn/translate/ 7 | 8 | Sequence to sequence model by Cho et al.(2014) 9 | 10 | Created by Chip Huyen as the starter code for assignment 3, 11 | class CS 20SI: "TensorFlow for Deep Learning Research" 12 | cs20si.stanford.edu 13 | 14 | The detailed assignment handout and information on training time can be found at http://web.stanford.edu/class/cs20si/assignments/a3.pdf 15 | 16 |

Sample conversation

17 | HUMAN ++++ hi
18 | BOT ++++ hi . what ' s your name ?
19 | HUMAN ++++ i'm chip
20 | BOT ++++ please .
21 | HUMAN ++++ please what?
22 | BOT ++++ i don ' t know what to say .
23 | HUMAN ++++ where are you from?
24 | BOT ++++ california .
25 | HUMAN ++++ do you like california?
26 | BOT ++++ i ' m okay .
27 | HUMAN ++++ i like california a lot
28 | BOT ++++ of what ?
29 | HUMAN ++++ i live here
30 | BOT ++++ and what is it ?
31 | HUMAN ++++ california
32 | BOT ++++ i don ' t know what to say .
33 | HUMAN ++++ i'm a student here
34 | BOT ++++ and get it out of here .
35 | HUMAN ++++ what do you like to do in your free time?
36 | BOT ++++ and what about the gun ?
37 | HUMAN ++++ do you have a gun?
38 | BOT ++++ yes .
39 | HUMAN ++++ you're dangerous!
40 | BOT ++++ i ' m not .
41 | HUMAN ++++ do you go hunting?
42 | BOT ++++ i can do .
43 | HUMAN ++++ what do you hunt?
44 | BOT ++++ virgin .
45 | HUMAN ++++ that's very bad
46 | BOT ++++ and you in love ?
47 | HUMAN ++++ no i'm not
48 | BOT ++++ please - -
49 | HUMAN ++++ okay, i am
50 | BOT ++++ and you are good .
51 | HUMAN ++++ yeah i am good too.
52 | BOT ++++ good good to you .
53 | HUMAN ++++ are you in love?
54 | BOT ++++ i ' m ready .
55 | HUMAN ++++ ready for what?
56 | BOT ++++ for what ?
57 | HUMAN ++++ for what?
58 | BOT ++++ i don ' t know what to say .
59 | HUMAN ++++ tell me
60 | BOT ++++ i can do it .
61 | HUMAN ++++ okay, tell me what you're ready for
62 | BOT ++++ i ' m doing you .
63 | HUMAN ++++ no you're not
64 | BOT ++++ please !
65 | HUMAN ++++ this is a strange conversation
66 | BOT ++++ i don ' t know what to say .
67 | HUMAN ++++ are you a guy?
68 | BOT ++++ of what ?
69 | HUMAN ++++ are you a boy?
70 | BOT ++++ i ' m fine .
71 | HUMAN ++++ are you a man?
72 | BOT ++++ i ' m a little .
73 | HUMAN ++++ what's your name?
74 | BOT ++++ chris -
75 | HUMAN ++++ chris who?
76 | BOT ++++ cornelius .
77 | HUMAN ++++ so i guess you're a guy.
78 | BOT ++++ i am .
79 | HUMAN ++++ okay
80 | BOT ++++ okay ?
81 | HUMAN ++++ do you have a girlfriend?
82 | BOT ++++ i ' m sure .
83 | HUMAN ++++ what's her name?
84 | BOT ++++ let ' s talk about something else .
85 | 86 | See output_convo.txt for more sample conversations. 87 | 88 |

Usage

89 | 90 | Step 1: create a data folder in your project directory, download 91 | the Cornell Movie-Dialogs Corpus from 92 | https://www.cs.cornell.edu/~cristian/Cornell_Movie-Dialogs_Corpus.html 93 | Unzip it 94 | 95 | Step 2: python data.py 96 |
This will do all the pre-processing for the Cornell dataset. 97 | 98 | Step 3: 99 | python chatbot.py --mode [train/chat]
100 | If mode is train, then you train the chatbot. By default, the model will 101 | restore the previously trained weights (if there is any) and continue 102 | training up on that. 103 | 104 | If you want to start training from scratch, please delete all the checkpoints 105 | in the checkpoints folder. 106 | 107 | If the mode is chat, you'll go into the interaction mode with the bot. 108 | 109 | By default, all the conversations you have with the chatbot will be written 110 | into the file output_convo.txt in the processed folder. If you run this chatbot, 111 | I kindly ask you to send me the output_convo.txt so that I can improve 112 | the chatbot. My email is huyenn@stanford.edu 113 | 114 | If you find the tutorial helpful, please head over to Anonymous Chatlog Donation 115 | to see how you can help us create the first realistic dialogue dataset. 116 | 117 | Thank you very much! 118 | >>>>>>> origin/master 119 | -------------------------------------------------------------------------------- /2017/assignments/chatbot/config.py: -------------------------------------------------------------------------------- 1 | """ A neural chatbot using sequence to sequence model with 2 | attentional decoder. 3 | 4 | This is based on Google Translate Tensorflow model 5 | https://github.com/tensorflow/models/blob/master/tutorials/rnn/translate/ 6 | 7 | Sequence to sequence model by Cho et al.(2014) 8 | 9 | Created by Chip Huyen as the starter code for assignment 3, 10 | class CS 20SI: "TensorFlow for Deep Learning Research" 11 | cs20si.stanford.edu 12 | 13 | This file contains the hyperparameters for the model. 14 | 15 | See readme.md for instruction on how to run the starter code. 16 | """ 17 | 18 | # parameters for processing the dataset 19 | DATA_PATH = '/Users/Chip/data/cornell movie-dialogs corpus' 20 | CONVO_FILE = 'movie_conversations.txt' 21 | LINE_FILE = 'movie_lines.txt' 22 | OUTPUT_FILE = 'output_convo.txt' 23 | PROCESSED_PATH = 'processed' 24 | CPT_PATH = 'checkpoints' 25 | 26 | THRESHOLD = 2 27 | 28 | PAD_ID = 0 29 | UNK_ID = 1 30 | START_ID = 2 31 | EOS_ID = 3 32 | 33 | TESTSET_SIZE = 25000 34 | 35 | # model parameters 36 | """ Train encoder length distribution: 37 | [175, 92, 11883, 8387, 10656, 13613, 13480, 12850, 11802, 10165, 38 | 8973, 7731, 7005, 6073, 5521, 5020, 4530, 4421, 3746, 3474, 3192, 39 | 2724, 2587, 2413, 2252, 2015, 1816, 1728, 1555, 1392, 1327, 1248, 40 | 1128, 1084, 1010, 884, 843, 755, 705, 660, 649, 594, 558, 517, 475, 41 | 426, 444, 388, 349, 337] 42 | These buckets size seem to work the best 43 | """ 44 | # [19530, 17449, 17585, 23444, 22884, 16435, 17085, 18291, 18931] 45 | # BUCKETS = [(6, 8), (8, 10), (10, 12), (13, 15), (16, 19), (19, 22), (23, 26), (29, 32), (39, 44)] 46 | 47 | # [37049, 33519, 30223, 33513, 37371] 48 | # BUCKETS = [(8, 10), (12, 14), (16, 19), (23, 26), (39, 43)] 49 | 50 | # BUCKETS = [(8, 10), (12, 14), (16, 19)] 51 | BUCKETS = [(16, 19)] 52 | 53 | NUM_LAYERS = 3 54 | HIDDEN_SIZE = 256 55 | BATCH_SIZE = 64 56 | 57 | LR = 0.5 58 | MAX_GRAD_NORM = 5.0 59 | 60 | NUM_SAMPLES = 512 61 | -------------------------------------------------------------------------------- /2017/assignments/exercises/e01.py: -------------------------------------------------------------------------------- 1 | """ 2 | Simple exercises to get used to TensorFlow API 3 | You should thoroughly test your code 4 | """ 5 | import os 6 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 7 | 8 | import tensorflow as tf 9 | 10 | sess = tf.InteractiveSession() 11 | ############################################################################### 12 | # 1a: Create two random 0-d tensors x and y of any distribution. 13 | # Create a TensorFlow object that returns x + y if x > y, and x - y otherwise. 14 | # Hint: look up tf.cond() 15 | # I do the first problem for you 16 | ############################################################################### 17 | 18 | x = tf.random_uniform([]) # Empty array as shape creates a scalar. 19 | y = tf.random_uniform([]) 20 | out = tf.cond(tf.greater(x, y), lambda: tf.add(x, y), lambda: tf.subtract(x, y)) 21 | print(sess.run(out)) 22 | 23 | ############################################################################### 24 | # 1b: Create two 0-d tensors x and y randomly selected from the range [-1, 1). 25 | # Return x + y if x < y, x - y if x > y, 0 otherwise. 26 | # Hint: Look up tf.case(). 27 | ############################################################################### 28 | 29 | # YOUR CODE 30 | 31 | ############################################################################### 32 | # 1c: Create the tensor x of the value [[0, -2, -1], [0, 1, 2]] 33 | # and y as a tensor of zeros with the same shape as x. 34 | # Return a boolean tensor that yields Trues if x equals y element-wise. 35 | # Hint: Look up tf.equal(). 36 | ############################################################################### 37 | 38 | # YOUR CODE 39 | 40 | ############################################################################### 41 | # 1d: Create the tensor x of value 42 | # [29.05088806, 27.61298943, 31.19073486, 29.35532951, 43 | # 30.97266006, 26.67541885, 38.08450317, 20.74983215, 44 | # 34.94445419, 34.45999146, 29.06485367, 36.01657104, 45 | # 27.88236427, 20.56035233, 30.20379066, 29.51215172, 46 | # 33.71149445, 28.59134293, 36.05556488, 28.66994858]. 47 | # Get the indices of elements in x whose values are greater than 30. 48 | # Hint: Use tf.where(). 49 | # Then extract elements whose values are greater than 30. 50 | # Hint: Use tf.gather(). 51 | ############################################################################### 52 | 53 | # YOUR CODE 54 | 55 | ############################################################################### 56 | # 1e: Create a diagnoal 2-d tensor of size 6 x 6 with the diagonal values of 1, 57 | # 2, ..., 6 58 | # Hint: Use tf.range() and tf.diag(). 59 | ############################################################################### 60 | 61 | # YOUR CODE 62 | 63 | ############################################################################### 64 | # 1f: Create a random 2-d tensor of size 10 x 10 from any distribution. 65 | # Calculate its determinant. 66 | # Hint: Look at tf.matrix_determinant(). 67 | ############################################################################### 68 | 69 | # YOUR CODE 70 | 71 | ############################################################################### 72 | # 1g: Create tensor x with value [5, 2, 3, 5, 10, 6, 2, 3, 4, 2, 1, 1, 0, 9]. 73 | # Return the unique elements in x 74 | # Hint: use tf.unique(). Keep in mind that tf.unique() returns a tuple. 75 | ############################################################################### 76 | 77 | # YOUR CODE 78 | 79 | ############################################################################### 80 | # 1h: Create two tensors x and y of shape 300 from any normal distribution, 81 | # as long as they are from the same distribution. 82 | # Use tf.cond() to return: 83 | # - The mean squared error of (x - y) if the average of all elements in (x - y) 84 | # is negative, or 85 | # - The sum of absolute value of all elements in the tensor (x - y) otherwise. 86 | # Hint: see the Huber loss function in the lecture slides 3. 87 | ############################################################################### 88 | 89 | # YOUR CODE -------------------------------------------------------------------------------- /2017/assignments/exercises/e01_sol.py: -------------------------------------------------------------------------------- 1 | """ 2 | Solution to simple TensorFlow exercises 3 | For the problems 4 | """ 5 | import tensorflow as tf 6 | 7 | ############################################################################### 8 | # 1a: Create two random 0-d tensors x and y of any distribution. 9 | # Create a TensorFlow object that returns x + y if x > y, and x - y otherwise. 10 | # Hint: look up tf.cond() 11 | # I do the first problem for you 12 | ############################################################################### 13 | 14 | x = tf.random_uniform([]) # Empty array as shape creates a scalar. 15 | y = tf.random_uniform([]) 16 | out = tf.cond(tf.greater(x, y), lambda: tf.add(x, y), lambda: tf.subtract(x, y)) 17 | 18 | ############################################################################### 19 | # 1b: Create two 0-d tensors x and y randomly selected from the range [-1, 1). 20 | # Return x + y if x < y, x - y if x > y, 0 otherwise. 21 | # Hint: Look up tf.case(). 22 | ############################################################################### 23 | 24 | x = tf.random_uniform([], -1, 1, dtype=tf.float32) 25 | y = tf.random_uniform([], -1, 1, dtype=tf.float32) 26 | out = tf.case({tf.less(x, y): lambda: tf.add(x, y), 27 | tf.greater(x, y): lambda: tf.subtract(x, y)}, 28 | default=lambda: tf.constant(0.0), exclusive=True) 29 | print(x) 30 | sess = tf.InteractiveSession() 31 | print(sess.run(x)) 32 | 33 | ############################################################################### 34 | # 1c: Create the tensor x of the value [[0, -2, -1], [0, 1, 2]] 35 | # and y as a tensor of zeros with the same shape as x. 36 | # Return a boolean tensor that yields Trues if x equals y element-wise. 37 | # Hint: Look up tf.equal(). 38 | ############################################################################### 39 | 40 | x = tf.constant([[0, -2, -1], [0, 1, 2]]) 41 | y = tf.zeros_like(x) 42 | out = tf.equal(x, y) 43 | 44 | ############################################################################### 45 | # 1d: Create the tensor x of value 46 | # [29.05088806, 27.61298943, 31.19073486, 29.35532951, 47 | # 30.97266006, 26.67541885, 38.08450317, 20.74983215, 48 | # 34.94445419, 34.45999146, 29.06485367, 36.01657104, 49 | # 27.88236427, 20.56035233, 30.20379066, 29.51215172, 50 | # 33.71149445, 28.59134293, 36.05556488, 28.66994858]. 51 | # Get the indices of elements in x whose values are greater than 30. 52 | # Hint: Use tf.where(). 53 | # Then extract elements whose values are greater than 30. 54 | # Hint: Use tf.gather(). 55 | ############################################################################### 56 | 57 | x = tf.constant([29.05088806, 27.61298943, 31.19073486, 29.35532951, 58 | 30.97266006, 26.67541885, 38.08450317, 20.74983215, 59 | 34.94445419, 34.45999146, 29.06485367, 36.01657104, 60 | 27.88236427, 20.56035233, 30.20379066, 29.51215172, 61 | 33.71149445, 28.59134293, 36.05556488, 28.66994858]) 62 | indices = tf.where(x > 30) 63 | out = tf.gather(x, indices) 64 | 65 | ############################################################################### 66 | # 1e: Create a diagnoal 2-d tensor of size 6 x 6 with the diagonal values of 1, 67 | # 2, ..., 6 68 | # Hint: Use tf.range() and tf.diag(). 69 | ############################################################################### 70 | 71 | values = tf.range(1, 7) 72 | out = tf.diag(values) 73 | 74 | ############################################################################### 75 | # 1f: Create a random 2-d tensor of size 10 x 10 from any distribution. 76 | # Calculate its determinant. 77 | # Hint: Look at tf.matrix_determinant(). 78 | ############################################################################### 79 | 80 | m = tf.random_normal([10, 10], mean=10, stddev=1) 81 | out = tf.matrix_determinant(m) 82 | 83 | ############################################################################### 84 | # 1g: Create tensor x with value [5, 2, 3, 5, 10, 6, 2, 3, 4, 2, 1, 1, 0, 9]. 85 | # Return the unique elements in x 86 | # Hint: use tf.unique(). Keep in mind that tf.unique() returns a tuple. 87 | ############################################################################### 88 | 89 | x = tf.constant([5, 2, 3, 5, 10, 6, 2, 3, 4, 2, 1, 1, 0, 9]) 90 | unique_values, indices = tf.unique(x) 91 | 92 | ############################################################################### 93 | # 1h: Create two tensors x and y of shape 300 from any normal distribution, 94 | # as long as they are from the same distribution. 95 | # Use tf.cond() to return: 96 | # - The mean squared error of (x - y) if the average of all elements in (x - y) 97 | # is negative, or 98 | # - The sum of absolute value of all elements in the tensor (x - y) otherwise. 99 | # Hint: see the Huber loss function in the lecture slides 3. 100 | ############################################################################### 101 | 102 | x = tf.random_normal([300], mean=5, stddev=1) 103 | y = tf.random_normal([300], mean=5, stddev=1) 104 | average = tf.reduce_mean(x - y) 105 | def f1(): return tf.reduce_mean(tf.square(x - y)) 106 | def f2(): return tf.reduce_sum(tf.abs(x - y)) 107 | out = tf.cond(average < 0, f1, f2) -------------------------------------------------------------------------------- /2017/assignments/style_transfer/content/deadpool.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/assignments/style_transfer/content/deadpool.jpg -------------------------------------------------------------------------------- /2017/assignments/style_transfer/content/resized_deadpool.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/assignments/style_transfer/content/resized_deadpool.jpg -------------------------------------------------------------------------------- /2017/assignments/style_transfer/readme.md: -------------------------------------------------------------------------------- 1 | For detailed instruction, you should read the assignment handout on the course website: http://web.stanford.edu/class/cs20si/assignments/a2.pdf 2 | -------------------------------------------------------------------------------- /2017/assignments/style_transfer/styles/guernica.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/assignments/style_transfer/styles/guernica.jpg -------------------------------------------------------------------------------- /2017/assignments/style_transfer/styles/harlequin.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/assignments/style_transfer/styles/harlequin.jpg -------------------------------------------------------------------------------- /2017/assignments/style_transfer/styles/pattern.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/assignments/style_transfer/styles/pattern.jpg -------------------------------------------------------------------------------- /2017/assignments/style_transfer/styles/resized_guernica.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/assignments/style_transfer/styles/resized_guernica.jpg -------------------------------------------------------------------------------- /2017/assignments/style_transfer/styles/starry_night.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/assignments/style_transfer/styles/starry_night.jpg -------------------------------------------------------------------------------- /2017/assignments/style_transfer/utils.py: -------------------------------------------------------------------------------- 1 | """ Utils needed for the implementation of the paper "A Neural Algorithm of Artistic Style" 2 | by Gatys et al. in TensorFlow. 3 | 4 | Author: Chip Huyen (huyenn@stanford.edu) 5 | Prepared for the class CS 20SI: "TensorFlow for Deep Learning Research" 6 | For more details, please read the assignment handout: 7 | http://web.stanford.edu/class/cs20si/assignments/a2.pdf 8 | """ 9 | from __future__ import print_function 10 | 11 | import os 12 | 13 | from PIL import Image, ImageOps 14 | import numpy as np 15 | import scipy.misc 16 | from six.moves import urllib 17 | 18 | def download(download_link, file_name, expected_bytes): 19 | """ Download the pretrained VGG-19 model if it's not already downloaded """ 20 | if os.path.exists(file_name): 21 | print("VGG-19 pre-trained model ready") 22 | return 23 | print("Downloading the VGG pre-trained model. This might take a while ...") 24 | file_name, _ = urllib.request.urlretrieve(download_link, file_name) 25 | file_stat = os.stat(file_name) 26 | if file_stat.st_size == expected_bytes: 27 | print('Successfully downloaded VGG-19 pre-trained model', file_name) 28 | else: 29 | raise Exception('File ' + file_name + 30 | ' might be corrupted. You should try downloading it with a browser.') 31 | 32 | def get_resized_image(img_path, height, width, save=True): 33 | image = Image.open(img_path) 34 | # it's because PIL is column major so you have to change place of width and height 35 | # this is stupid, i know 36 | image = ImageOps.fit(image, (width, height), Image.ANTIALIAS) 37 | if save: 38 | image_dirs = img_path.split('/') 39 | image_dirs[-1] = 'resized_' + image_dirs[-1] 40 | out_path = '/'.join(image_dirs) 41 | if not os.path.exists(out_path): 42 | image.save(out_path) 43 | image = np.asarray(image, np.float32) 44 | return np.expand_dims(image, 0) 45 | 46 | def generate_noise_image(content_image, height, width, noise_ratio=0.6): 47 | noise_image = np.random.uniform(-20, 20, 48 | (1, height, width, 3)).astype(np.float32) 49 | return noise_image * noise_ratio + content_image * (1 - noise_ratio) 50 | 51 | def save_image(path, image): 52 | # Output should add back the mean pixels we subtracted at the beginning 53 | image = image[0] # the image 54 | image = np.clip(image, 0, 255).astype('uint8') 55 | scipy.misc.imsave(path, image) 56 | 57 | def make_dir(path): 58 | """ Create a directory if there isn't one already. """ 59 | try: 60 | os.mkdir(path) 61 | except OSError: 62 | pass -------------------------------------------------------------------------------- /2017/assignments/style_transfer/vgg_model.py: -------------------------------------------------------------------------------- 1 | """ Load VGGNet weights needed for the implementation of the paper 2 | "A Neural Algorithm of Artistic Style" by Gatys et al. in TensorFlow. 3 | 4 | Author: Chip Huyen (huyenn@stanford.edu) 5 | Prepared for the class CS 20SI: "TensorFlow for Deep Learning Research" 6 | For more details, please read the assignment handout: 7 | http://web.stanford.edu/class/cs20si/assignments/a2.pdf 8 | """ 9 | 10 | import numpy as np 11 | import tensorflow as tf 12 | import scipy.io 13 | 14 | def _weights(vgg_layers, layer, expected_layer_name): 15 | """ Return the weights and biases already trained by VGG 16 | """ 17 | W = vgg_layers[0][layer][0][0][2][0][0] 18 | b = vgg_layers[0][layer][0][0][2][0][1] 19 | layer_name = vgg_layers[0][layer][0][0][0][0] 20 | assert layer_name == expected_layer_name 21 | return W, b.reshape(b.size) 22 | 23 | def _conv2d_relu(vgg_layers, prev_layer, layer, layer_name): 24 | """ Return the Conv2D layer with RELU using the weights, biases from the VGG 25 | model at 'layer'. 26 | Inputs: 27 | vgg_layers: holding all the layers of VGGNet 28 | prev_layer: the output tensor from the previous layer 29 | layer: the index to current layer in vgg_layers 30 | layer_name: the string that is the name of the current layer. 31 | It's used to specify variable_scope. 32 | 33 | Output: 34 | relu applied on the convolution. 35 | 36 | Note that you first need to obtain W and b from vgg-layers using the function 37 | _weights() defined above. 38 | W and b returned from _weights() are numpy arrays, so you have 39 | to convert them to TF tensors using tf.constant. 40 | Note that you'll have to do apply relu on the convolution. 41 | Hint for choosing strides size: 42 | for small images, you probably don't want to skip any pixel 43 | """ 44 | with tf.variable_scope(layer_name) as scope: 45 | W, b = _weights(vgg_layers, layer, layer_name) 46 | W = tf.constant(W, name='weights') 47 | b = tf.constant(b, name='bias') 48 | conv2d = tf.nn.conv2d(prev_layer, filter=W, strides=[1, 1, 1, 1], padding='SAME') 49 | return tf.nn.relu(conv2d + b) 50 | 51 | def _avgpool(prev_layer): 52 | """ Return the average pooling layer. The paper suggests that average pooling 53 | actually works better than max pooling. 54 | Input: 55 | prev_layer: the output tensor from the previous layer 56 | 57 | Output: 58 | the output of the tf.nn.avg_pool() function. 59 | Hint for choosing strides and kszie: choose what you feel appropriate 60 | """ 61 | return tf.nn.avg_pool(prev_layer, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], 62 | padding='SAME', name='avg_pool_') 63 | 64 | def load_vgg(path, input_image): 65 | """ Load VGG into a TensorFlow model. 66 | Use a dictionary to hold the model instead of using a Python class 67 | """ 68 | vgg = scipy.io.loadmat(path) 69 | vgg_layers = vgg['layers'] 70 | 71 | graph = {} 72 | graph['conv1_1'] = _conv2d_relu(vgg_layers, input_image, 0, 'conv1_1') 73 | graph['conv1_2'] = _conv2d_relu(vgg_layers, graph['conv1_1'], 2, 'conv1_2') 74 | graph['avgpool1'] = _avgpool(graph['conv1_2']) 75 | graph['conv2_1'] = _conv2d_relu(vgg_layers, graph['avgpool1'], 5, 'conv2_1') 76 | graph['conv2_2'] = _conv2d_relu(vgg_layers, graph['conv2_1'], 7, 'conv2_2') 77 | graph['avgpool2'] = _avgpool(graph['conv2_2']) 78 | graph['conv3_1'] = _conv2d_relu(vgg_layers, graph['avgpool2'], 10, 'conv3_1') 79 | graph['conv3_2'] = _conv2d_relu(vgg_layers, graph['conv3_1'], 12, 'conv3_2') 80 | graph['conv3_3'] = _conv2d_relu(vgg_layers, graph['conv3_2'], 14, 'conv3_3') 81 | graph['conv3_4'] = _conv2d_relu(vgg_layers, graph['conv3_3'], 16, 'conv3_4') 82 | graph['avgpool3'] = _avgpool(graph['conv3_4']) 83 | graph['conv4_1'] = _conv2d_relu(vgg_layers, graph['avgpool3'], 19, 'conv4_1') 84 | graph['conv4_2'] = _conv2d_relu(vgg_layers, graph['conv4_1'], 21, 'conv4_2') 85 | graph['conv4_3'] = _conv2d_relu(vgg_layers, graph['conv4_2'], 23, 'conv4_3') 86 | graph['conv4_4'] = _conv2d_relu(vgg_layers, graph['conv4_3'], 25, 'conv4_4') 87 | graph['avgpool4'] = _avgpool(graph['conv4_4']) 88 | graph['conv5_1'] = _conv2d_relu(vgg_layers, graph['avgpool4'], 28, 'conv5_1') 89 | graph['conv5_2'] = _conv2d_relu(vgg_layers, graph['conv5_1'], 30, 'conv5_2') 90 | graph['conv5_3'] = _conv2d_relu(vgg_layers, graph['conv5_2'], 32, 'conv5_3') 91 | graph['conv5_4'] = _conv2d_relu(vgg_layers, graph['conv5_3'], 34, 'conv5_4') 92 | graph['avgpool5'] = _avgpool(graph['conv5_4']) 93 | 94 | return graph -------------------------------------------------------------------------------- /2017/assignments/style_transfer_starter/content/deadpool.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/assignments/style_transfer_starter/content/deadpool.jpg -------------------------------------------------------------------------------- /2017/assignments/style_transfer_starter/content/resized_deadpool.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/assignments/style_transfer_starter/content/resized_deadpool.jpg -------------------------------------------------------------------------------- /2017/assignments/style_transfer_starter/readme.md: -------------------------------------------------------------------------------- 1 | For detailed instruction, you should read the assignment handout on the course website: http://web.stanford.edu/class/cs20si/assignments/a2.pdf 2 | -------------------------------------------------------------------------------- /2017/assignments/style_transfer_starter/styles/guernica.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/assignments/style_transfer_starter/styles/guernica.jpg -------------------------------------------------------------------------------- /2017/assignments/style_transfer_starter/styles/harlequin.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/assignments/style_transfer_starter/styles/harlequin.jpg -------------------------------------------------------------------------------- /2017/assignments/style_transfer_starter/styles/pattern.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/assignments/style_transfer_starter/styles/pattern.jpg -------------------------------------------------------------------------------- /2017/assignments/style_transfer_starter/styles/resized_guernica.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/assignments/style_transfer_starter/styles/resized_guernica.jpg -------------------------------------------------------------------------------- /2017/assignments/style_transfer_starter/styles/starry_night.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/assignments/style_transfer_starter/styles/starry_night.jpg -------------------------------------------------------------------------------- /2017/assignments/style_transfer_starter/utils.py: -------------------------------------------------------------------------------- 1 | """ Utils needed for the implementation of the paper "A Neural Algorithm of Artistic Style" 2 | by Gatys et al. in TensorFlow. 3 | 4 | Author: Chip Huyen (huyenn@stanford.edu) 5 | Prepared for the class CS 20SI: "TensorFlow for Deep Learning Research" 6 | For more details, please read the assignment handout: 7 | http://web.stanford.edu/class/cs20si/assignments/a2.pdf 8 | """ 9 | from __future__ import print_function 10 | 11 | import os 12 | 13 | from PIL import Image, ImageOps 14 | import numpy as np 15 | import scipy.misc 16 | from six.moves import urllib 17 | 18 | def download(download_link, file_name, expected_bytes): 19 | """ Download the pretrained VGG-19 model if it's not already downloaded """ 20 | if os.path.exists(file_name): 21 | print("VGG-19 pre-trained model ready") 22 | return 23 | print("Downloading the VGG pre-trained model. This might take a while ...") 24 | file_name, _ = urllib.request.urlretrieve(download_link, file_name) 25 | file_stat = os.stat(file_name) 26 | if file_stat.st_size == expected_bytes: 27 | print('Successfully downloaded VGG-19 pre-trained model', file_name) 28 | else: 29 | raise Exception('File ' + file_name + 30 | ' might be corrupted. You should try downloading it with a browser.') 31 | 32 | def get_resized_image(img_path, height, width, save=True): 33 | image = Image.open(img_path) 34 | # it's because PIL is column major so you have to change place of width and height 35 | # this is stupid, i know 36 | image = ImageOps.fit(image, (width, height), Image.ANTIALIAS) 37 | if save: 38 | image_dirs = img_path.split('/') 39 | image_dirs[-1] = 'resized_' + image_dirs[-1] 40 | out_path = '/'.join(image_dirs) 41 | if not os.path.exists(out_path): 42 | image.save(out_path) 43 | image = np.asarray(image, np.float32) 44 | return np.expand_dims(image, 0) 45 | 46 | def generate_noise_image(content_image, height, width, noise_ratio=0.6): 47 | noise_image = np.random.uniform(-20, 20, 48 | (1, height, width, 3)).astype(np.float32) 49 | return noise_image * noise_ratio + content_image * (1 - noise_ratio) 50 | 51 | def save_image(path, image): 52 | # Output should add back the mean pixels we subtracted at the beginning 53 | image = image[0] # the image 54 | image = np.clip(image, 0, 255).astype('uint8') 55 | scipy.misc.imsave(path, image) 56 | 57 | def make_dir(path): 58 | """ Create a directory if there isn't one already. """ 59 | try: 60 | os.mkdir(path) 61 | except OSError: 62 | pass -------------------------------------------------------------------------------- /2017/assignments/style_transfer_starter/vgg_model.py: -------------------------------------------------------------------------------- 1 | """ Load VGGNet weights needed for the implementation of the paper 2 | "A Neural Algorithm of Artistic Style" by Gatys et al. in TensorFlow. 3 | 4 | Author: Chip Huyen (huyenn@stanford.edu) 5 | Prepared for the class CS 20SI: "TensorFlow for Deep Learning Research" 6 | For more details, please read the assignment handout: 7 | http://web.stanford.edu/class/cs20si/assignments/a2.pdf 8 | """ 9 | 10 | import numpy as np 11 | import tensorflow as tf 12 | import scipy.io 13 | 14 | def _weights(vgg_layers, layer, expected_layer_name): 15 | """ Return the weights and biases already trained by VGG 16 | """ 17 | W = vgg_layers[0][layer][0][0][2][0][0] 18 | b = vgg_layers[0][layer][0][0][2][0][1] 19 | layer_name = vgg_layers[0][layer][0][0][0][0] 20 | assert layer_name == expected_layer_name 21 | return W, b.reshape(b.size) 22 | 23 | def _conv2d_relu(vgg_layers, prev_layer, layer, layer_name): 24 | """ Return the Conv2D layer with RELU using the weights, biases from the VGG 25 | model at 'layer'. 26 | Inputs: 27 | vgg_layers: holding all the layers of VGGNet 28 | prev_layer: the output tensor from the previous layer 29 | layer: the index to current layer in vgg_layers 30 | layer_name: the string that is the name of the current layer. 31 | It's used to specify variable_scope. 32 | 33 | Output: 34 | relu applied on the convolution. 35 | 36 | Note that you first need to obtain W and b from vgg-layers using the function 37 | _weights() defined above. 38 | W and b returned from _weights() are numpy arrays, so you have 39 | to convert them to TF tensors using tf.constant. 40 | Note that you'll have to do apply relu on the convolution. 41 | Hint for choosing strides size: 42 | for small images, you probably don't want to skip any pixel 43 | """ 44 | pass 45 | 46 | def _avgpool(prev_layer): 47 | """ Return the average pooling layer. The paper suggests that average pooling 48 | actually works better than max pooling. 49 | Input: 50 | prev_layer: the output tensor from the previous layer 51 | 52 | Output: 53 | the output of the tf.nn.avg_pool() function. 54 | Hint for choosing strides and kszie: choose what you feel appropriate 55 | """ 56 | pass 57 | 58 | def load_vgg(path, input_image): 59 | """ Load VGG into a TensorFlow model. 60 | Use a dictionary to hold the model instead of using a Python class 61 | """ 62 | vgg = scipy.io.loadmat(path) 63 | vgg_layers = vgg['layers'] 64 | 65 | graph = {} 66 | graph['conv1_1'] = _conv2d_relu(vgg_layers, input_image, 0, 'conv1_1') 67 | graph['conv1_2'] = _conv2d_relu(vgg_layers, graph['conv1_1'], 2, 'conv1_2') 68 | graph['avgpool1'] = _avgpool(graph['conv1_2']) 69 | graph['conv2_1'] = _conv2d_relu(vgg_layers, graph['avgpool1'], 5, 'conv2_1') 70 | graph['conv2_2'] = _conv2d_relu(vgg_layers, graph['conv2_1'], 7, 'conv2_2') 71 | graph['avgpool2'] = _avgpool(graph['conv2_2']) 72 | graph['conv3_1'] = _conv2d_relu(vgg_layers, graph['avgpool2'], 10, 'conv3_1') 73 | graph['conv3_2'] = _conv2d_relu(vgg_layers, graph['conv3_1'], 12, 'conv3_2') 74 | graph['conv3_3'] = _conv2d_relu(vgg_layers, graph['conv3_2'], 14, 'conv3_3') 75 | graph['conv3_4'] = _conv2d_relu(vgg_layers, graph['conv3_3'], 16, 'conv3_4') 76 | graph['avgpool3'] = _avgpool(graph['conv3_4']) 77 | graph['conv4_1'] = _conv2d_relu(vgg_layers, graph['avgpool3'], 19, 'conv4_1') 78 | graph['conv4_2'] = _conv2d_relu(vgg_layers, graph['conv4_1'], 21, 'conv4_2') 79 | graph['conv4_3'] = _conv2d_relu(vgg_layers, graph['conv4_2'], 23, 'conv4_3') 80 | graph['conv4_4'] = _conv2d_relu(vgg_layers, graph['conv4_3'], 25, 'conv4_4') 81 | graph['avgpool4'] = _avgpool(graph['conv4_4']) 82 | graph['conv5_1'] = _conv2d_relu(vgg_layers, graph['avgpool4'], 28, 'conv5_1') 83 | graph['conv5_2'] = _conv2d_relu(vgg_layers, graph['conv5_1'], 30, 'conv5_2') 84 | graph['conv5_3'] = _conv2d_relu(vgg_layers, graph['conv5_2'], 32, 'conv5_3') 85 | graph['conv5_4'] = _conv2d_relu(vgg_layers, graph['conv5_3'], 34, 'conv5_4') 86 | graph['avgpool5'] = _avgpool(graph['conv5_4']) 87 | 88 | return graph -------------------------------------------------------------------------------- /2017/data/fire_theft.xls: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/data/fire_theft.xls -------------------------------------------------------------------------------- /2017/data/friday.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/data/friday.jpg -------------------------------------------------------------------------------- /2017/data/friday.tfrecord: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/data/friday.tfrecord -------------------------------------------------------------------------------- /2017/examples/02_feed_dict.py: -------------------------------------------------------------------------------- 1 | """ Example to demonstrate the use of feed_dict 2 | Author: Chip Huyen 3 | Prepared for the class CS 20SI: "TensorFlow for Deep Learning Research" 4 | cs20si.stanford.edu 5 | """ 6 | import os 7 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 8 | 9 | import tensorflow as tf 10 | 11 | # Example 1: feed_dict with placeholder 12 | # create a placeholder of type float 32-bit, value is a vector of 3 elements 13 | a = tf.placeholder(tf.float32, shape=[3]) 14 | 15 | # create a constant of type float 32-bit, value is a vector of 3 elements 16 | b = tf.constant([5, 5, 5], tf.float32) 17 | 18 | # use the placeholder as you would a constant 19 | c = a + b # short for tf.add(a, b) 20 | 21 | with tf.Session() as sess: 22 | # print(sess.run(c)) # InvalidArgumentError because a doesn’t have any value 23 | 24 | # feed [1, 2, 3] to placeholder a via the dict {a: [1, 2, 3]} 25 | # fetch value of c 26 | print(sess.run(c, {a: [1, 2, 3]})) # >> [6. 7. 8.] 27 | 28 | 29 | # Example 2: feed_dict with variables 30 | a = tf.add(2, 5) 31 | b = tf.multiply(a, 3) 32 | 33 | with tf.Session() as sess: 34 | # define a dictionary that says to replace the value of 'a' with 15 35 | replace_dict = {a: 15} 36 | 37 | # Run the session, passing in 'replace_dict' as the value to 'feed_dict' 38 | print(sess.run(b, feed_dict=replace_dict)) # >> 45 -------------------------------------------------------------------------------- /2017/examples/02_lazy_loading.py: -------------------------------------------------------------------------------- 1 | """ Example to demonstrate how the graph definition gets 2 | bloated because of lazy loading 3 | Author: Chip Huyen 4 | Prepared for the class CS 20SI: "TensorFlow for Deep Learning Research" 5 | cs20si.stanford.edu 6 | """ 7 | import os 8 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 9 | 10 | import tensorflow as tf 11 | 12 | ######################################## 13 | ## NORMAL LOADING ## 14 | ## print out a graph with 1 Add node ## 15 | ######################################## 16 | 17 | x = tf.Variable(10, name='x') 18 | y = tf.Variable(20, name='y') 19 | z = tf.add(x, y) 20 | 21 | with tf.Session() as sess: 22 | sess.run(tf.global_variables_initializer()) 23 | writer = tf.summary.FileWriter('./graphs/l2', sess.graph) 24 | for _ in range(10): 25 | sess.run(z) 26 | print(tf.get_default_graph().as_graph_def()) 27 | writer.close() 28 | 29 | ######################################## 30 | ## LAZY LOADING ## 31 | ## print out a graph with 10 Add nodes## 32 | ######################################## 33 | 34 | x = tf.Variable(10, name='x') 35 | y = tf.Variable(20, name='y') 36 | 37 | with tf.Session() as sess: 38 | sess.run(tf.global_variables_initializer()) 39 | writer = tf.summary.FileWriter('./graphs/l2', sess.graph) 40 | for _ in range(10): 41 | sess.run(tf.add(x, y)) 42 | print(tf.get_default_graph().as_graph_def()) 43 | writer.close() -------------------------------------------------------------------------------- /2017/examples/02_simple_tf.py: -------------------------------------------------------------------------------- 1 | """ Some simple TensorFlow's ops 2 | Author: Chip Huyen 3 | Prepared for the class CS 20SI: "TensorFlow for Deep Learning Research" 4 | cs20si.stanford.edu 5 | """ 6 | import os 7 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 8 | 9 | import numpy as np 10 | import tensorflow as tf 11 | 12 | 13 | a = tf.constant(2) 14 | b = tf.constant(3) 15 | x = tf.add(a, b) 16 | with tf.Session() as sess: 17 | writer = tf.summary.FileWriter('./graphs', sess.graph) 18 | print(sess.run(x)) 19 | writer.close() # close the writer when you’re done using it 20 | 21 | 22 | a = tf.constant([2, 2], name='a') 23 | b = tf.constant([[0, 1], [2, 3]], name='b') 24 | x = tf.multiply(a, b, name='dot_product') 25 | with tf.Session() as sess: 26 | print(sess.run(x)) 27 | # >> [[0 2] 28 | # [4 6]] 29 | 30 | tf.zeros(shape, dtype=tf.float32, name=None) 31 | #creates a tensor of shape and all elements will be zeros (when ran in session) 32 | 33 | x = tf.zeros([2, 3], tf.int32) 34 | y = tf.zeros_like(x, optimize=True) 35 | print(y) 36 | print(tf.get_default_graph().as_graph_def()) 37 | with tf.Session() as sess: 38 | y = sess.run(y) 39 | 40 | 41 | with tf.Session() as sess: 42 | print(sess.run(tf.linspace(10.0, 13.0, 4))) 43 | print(sess.run(tf.range(5))) 44 | for i in np.arange(5): 45 | print(i) 46 | 47 | samples = tf.multinomial(tf.constant([[1., 3., 1]]), 5) 48 | 49 | with tf.Session() as sess: 50 | for _ in range(10): 51 | print(sess.run(samples)) 52 | 53 | t_0 = 19 54 | x = tf.zeros_like(t_0) # ==> 0 55 | y = tf.ones_like(t_0) # ==> 1 56 | 57 | with tf.Session() as sess: 58 | print(sess.run([x, y])) 59 | 60 | t_1 = ['apple', 'peach', 'banana'] 61 | x = tf.zeros_like(t_1) # ==> ['' '' ''] 62 | y = tf.ones_like(t_1) # ==> TypeError: Expected string, got 1 of type 'int' instead. 63 | 64 | t_2 = [[True, False, False], 65 | [False, False, True], 66 | [False, True, False]] 67 | x = tf.zeros_like(t_2) # ==> 2x2 tensor, all elements are False 68 | y = tf.ones_like(t_2) # ==> 2x2 tensor, all elements are True 69 | with tf.Session() as sess: 70 | print(sess.run([x, y])) 71 | 72 | with tf.variable_scope('meh') as scope: 73 | a = tf.get_variable('a', [10]) 74 | b = tf.get_variable('b', [100]) 75 | 76 | writer = tf.summary.FileWriter('test', tf.get_default_graph()) 77 | 78 | 79 | x = tf.Variable(2.0) 80 | y = 2.0 * (x ** 3) 81 | z = 3.0 + y ** 2 82 | grad_z = tf.gradients(z, [x, y]) 83 | with tf.Session() as sess: 84 | sess.run(x.initializer) 85 | print(sess.run(grad_z)) 86 | -------------------------------------------------------------------------------- /2017/examples/02_variables.py: -------------------------------------------------------------------------------- 1 | """ 2 | Example to demonstrate the ops of tf.Variables() 3 | """ 4 | import os 5 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 6 | 7 | import tensorflow as tf 8 | 9 | # Example 1: how to run assign op 10 | W = tf.Variable(10) 11 | assign_op = W.assign(100) 12 | 13 | with tf.Session() as sess: 14 | sess.run(W.initializer) 15 | print(W.eval()) # >> 10 16 | print(sess.run(assign_op)) # >> 100 17 | 18 | # Example 2: tricky example 19 | # create a variable whose original value is 2 20 | my_var = tf.Variable(2, name="my_var") 21 | 22 | # assign 2 * my_var to my_var and run the op my_var_times_two 23 | my_var_times_two = my_var.assign(2 * my_var) 24 | 25 | with tf.Session() as sess: 26 | sess.run(tf.global_variables_initializer()) 27 | print(sess.run(my_var_times_two)) # >> 4 28 | print(sess.run(my_var_times_two)) # >> 8 29 | print(sess.run(my_var_times_two)) # >> 16 30 | 31 | # Example 3: each session maintains its own copy of variables 32 | W = tf.Variable(10) 33 | sess1 = tf.Session() 34 | sess2 = tf.Session() 35 | 36 | # You have to initialize W at each session 37 | sess1.run(W.initializer) 38 | sess2.run(W.initializer) 39 | 40 | print(sess1.run(W.assign_add(10))) # >> 20 41 | print(sess2.run(W.assign_sub(2))) # >> 8 42 | 43 | print(sess1.run(W.assign_add(100))) # >> 120 44 | print(sess2.run(W.assign_sub(50))) # >> -42 45 | 46 | sess1.close() 47 | sess2.close() -------------------------------------------------------------------------------- /2017/examples/03_linear_regression_sol.py: -------------------------------------------------------------------------------- 1 | """ Simple linear regression example in TensorFlow 2 | This program tries to predict the number of thefts from 3 | the number of fire in the city of Chicago 4 | Author: Chip Huyen 5 | Prepared for the class CS 20SI: "TensorFlow for Deep Learning Research" 6 | cs20si.stanford.edu 7 | """ 8 | import os 9 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 10 | 11 | import numpy as np 12 | import matplotlib.pyplot as plt 13 | import tensorflow as tf 14 | import xlrd 15 | 16 | import utils 17 | 18 | DATA_FILE = 'data/fire_theft.xls' 19 | 20 | # Step 1: read in data from the .xls file 21 | book = xlrd.open_workbook(DATA_FILE, encoding_override="utf-8") 22 | sheet = book.sheet_by_index(0) 23 | data = np.asarray([sheet.row_values(i) for i in range(1, sheet.nrows)]) 24 | n_samples = sheet.nrows - 1 25 | 26 | # Step 2: create placeholders for input X (number of fire) and label Y (number of theft) 27 | X = tf.placeholder(tf.float32, name='X') 28 | Y = tf.placeholder(tf.float32, name='Y') 29 | 30 | # Step 3: create weight and bias, initialized to 0 31 | w = tf.Variable(0.0, name='weights') 32 | b = tf.Variable(0.0, name='bias') 33 | 34 | # Step 4: build model to predict Y 35 | Y_predicted = X * w + b 36 | 37 | # Step 5: use the square error as the loss function 38 | loss = tf.square(Y - Y_predicted, name='loss') 39 | # loss = utils.huber_loss(Y, Y_predicted) 40 | 41 | # Step 6: using gradient descent with learning rate of 0.01 to minimize loss 42 | optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001).minimize(loss) 43 | 44 | with tf.Session() as sess: 45 | # Step 7: initialize the necessary variables, in this case, w and b 46 | sess.run(tf.global_variables_initializer()) 47 | 48 | writer = tf.summary.FileWriter('./graphs/linear_reg', sess.graph) 49 | 50 | # Step 8: train the model 51 | for i in range(50): # train the model 100 epochs 52 | total_loss = 0 53 | for x, y in data: 54 | # Session runs train_op and fetch values of loss 55 | _, l = sess.run([optimizer, loss], feed_dict={X: x, Y:y}) 56 | total_loss += l 57 | print('Epoch {0}: {1}'.format(i, total_loss/n_samples)) 58 | 59 | # close the writer when you're done using it 60 | writer.close() 61 | 62 | # Step 9: output the values of w and b 63 | w, b = sess.run([w, b]) 64 | 65 | # plot the results 66 | X, Y = data.T[0], data.T[1] 67 | plt.plot(X, Y, 'bo', label='Real data') 68 | plt.plot(X, X * w + b, 'r', label='Predicted data') 69 | plt.legend() 70 | plt.show() -------------------------------------------------------------------------------- /2017/examples/03_linear_regression_starter.py: -------------------------------------------------------------------------------- 1 | """ Simple linear regression example in TensorFlow 2 | This program tries to predict the number of thefts from 3 | the number of fire in the city of Chicago 4 | Author: Chip Huyen 5 | Prepared for the class CS 20SI: "TensorFlow for Deep Learning Research" 6 | cs20si.stanford.edu 7 | """ 8 | import os 9 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 10 | 11 | import numpy as np 12 | import matplotlib.pyplot as plt 13 | import tensorflow as tf 14 | import xlrd 15 | 16 | import utils 17 | 18 | DATA_FILE = 'data/fire_theft.xls' 19 | 20 | # Phase 1: Assemble the graph 21 | # Step 1: read in data from the .xls file 22 | book = xlrd.open_workbook(DATA_FILE, encoding_override='utf-8') 23 | sheet = book.sheet_by_index(0) 24 | data = np.asarray([sheet.row_values(i) for i in range(1, sheet.nrows)]) 25 | n_samples = sheet.nrows - 1 26 | 27 | # Step 2: create placeholders for input X (number of fire) and label Y (number of theft) 28 | # Both have the type float32 29 | 30 | 31 | # Step 3: create weight and bias, initialized to 0 32 | # name your variables w and b 33 | 34 | 35 | # Step 4: predict Y (number of theft) from the number of fire 36 | # name your variable Y_predicted 37 | 38 | 39 | # Step 5: use the square error as the loss function 40 | # name your variable loss 41 | 42 | 43 | # Step 6: using gradient descent with learning rate of 0.01 to minimize loss 44 | 45 | # Phase 2: Train our model 46 | with tf.Session() as sess: 47 | # Step 7: initialize the necessary variables, in this case, w and b 48 | # TO - DO 49 | 50 | 51 | # Step 8: train the model 52 | for i in range(50): # run 100 epochs 53 | total_loss = 0 54 | for x, y in data: 55 | # Session runs optimizer to minimize loss and fetch the value of loss. Name the received value as l 56 | # TO DO: write sess.run() 57 | 58 | total_loss += l 59 | print("Epoch {0}: {1}".format(i, total_loss/n_samples)) 60 | 61 | # plot the results 62 | # X, Y = data.T[0], data.T[1] 63 | # plt.plot(X, Y, 'bo', label='Real data') 64 | # plt.plot(X, X * w + b, 'r', label='Predicted data') 65 | # plt.legend() 66 | # plt.show() -------------------------------------------------------------------------------- /2017/examples/03_logistic_regression_mnist_sol.py: -------------------------------------------------------------------------------- 1 | """ Simple logistic regression model to solve OCR task 2 | with MNIST in TensorFlow 3 | MNIST dataset: yann.lecun.com/exdb/mnist/ 4 | Author: Chip Huyen 5 | Prepared for the class CS 20SI: "TensorFlow for Deep Learning Research" 6 | cs20si.stanford.edu 7 | """ 8 | import os 9 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 10 | 11 | import numpy as np 12 | import tensorflow as tf 13 | from tensorflow.examples.tutorials.mnist import input_data 14 | import time 15 | 16 | # Define paramaters for the model 17 | learning_rate = 0.01 18 | batch_size = 128 19 | n_epochs = 30 20 | 21 | # Step 1: Read in data 22 | # using TF Learn's built in function to load MNIST data to the folder data/mnist 23 | mnist = input_data.read_data_sets('/data/mnist', one_hot=True) 24 | 25 | # Step 2: create placeholders for features and labels 26 | # each image in the MNIST data is of shape 28*28 = 784 27 | # therefore, each image is represented with a 1x784 tensor 28 | # there are 10 classes for each image, corresponding to digits 0 - 9. 29 | # each lable is one hot vector. 30 | X = tf.placeholder(tf.float32, [batch_size, 784], name='X_placeholder') 31 | Y = tf.placeholder(tf.int32, [batch_size, 10], name='Y_placeholder') 32 | 33 | # Step 3: create weights and bias 34 | # w is initialized to random variables with mean of 0, stddev of 0.01 35 | # b is initialized to 0 36 | # shape of w depends on the dimension of X and Y so that Y = tf.matmul(X, w) 37 | # shape of b depends on Y 38 | w = tf.Variable(tf.random_normal(shape=[784, 10], stddev=0.01), name='weights') 39 | b = tf.Variable(tf.zeros([1, 10]), name="bias") 40 | 41 | # Step 4: build model 42 | # the model that returns the logits. 43 | # this logits will be later passed through softmax layer 44 | logits = tf.matmul(X, w) + b 45 | 46 | # Step 5: define loss function 47 | # use cross entropy of softmax of logits as the loss function 48 | entropy = tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=Y, name='loss') 49 | loss = tf.reduce_mean(entropy) # computes the mean over all the examples in the batch 50 | 51 | # Step 6: define training op 52 | # using gradient descent with learning rate of 0.01 to minimize loss 53 | optimizer = tf.train.AdamOptimizer(learning_rate).minimize(loss) 54 | 55 | with tf.Session() as sess: 56 | # to visualize using TensorBoard 57 | writer = tf.summary.FileWriter('./graphs/logistic_reg', sess.graph) 58 | 59 | start_time = time.time() 60 | sess.run(tf.global_variables_initializer()) 61 | n_batches = int(mnist.train.num_examples/batch_size) 62 | for i in range(n_epochs): # train the model n_epochs times 63 | total_loss = 0 64 | 65 | for _ in range(n_batches): 66 | X_batch, Y_batch = mnist.train.next_batch(batch_size) 67 | _, loss_batch = sess.run([optimizer, loss], feed_dict={X: X_batch, Y:Y_batch}) 68 | total_loss += loss_batch 69 | print('Average loss epoch {0}: {1}'.format(i, total_loss/n_batches)) 70 | 71 | print('Total time: {0} seconds'.format(time.time() - start_time)) 72 | 73 | print('Optimization Finished!') # should be around 0.35 after 25 epochs 74 | 75 | # test the model 76 | 77 | preds = tf.nn.softmax(logits) 78 | correct_preds = tf.equal(tf.argmax(preds, 1), tf.argmax(Y, 1)) 79 | accuracy = tf.reduce_sum(tf.cast(correct_preds, tf.float32)) # need numpy.count_nonzero(boolarr) :( 80 | 81 | n_batches = int(mnist.test.num_examples/batch_size) 82 | total_correct_preds = 0 83 | 84 | for i in range(n_batches): 85 | X_batch, Y_batch = mnist.test.next_batch(batch_size) 86 | accuracy_batch = sess.run([accuracy], feed_dict={X: X_batch, Y:Y_batch}) 87 | total_correct_preds += accuracy_batch 88 | 89 | print('Accuracy {0}'.format(total_correct_preds/mnist.test.num_examples)) 90 | 91 | writer.close() 92 | -------------------------------------------------------------------------------- /2017/examples/03_logistic_regression_mnist_starter.py: -------------------------------------------------------------------------------- 1 | """ Starter code for logistic regression model to solve OCR task 2 | with MNIST in TensorFlow 3 | MNIST dataset: yann.lecun.com/exdb/mnist/ 4 | Author: Chip Huyen 5 | Prepared for the class CS 20SI: "TensorFlow for Deep Learning Research" 6 | cs20si.stanford.edu 7 | """ 8 | import os 9 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 10 | 11 | import numpy as np 12 | import tensorflow as tf 13 | from tensorflow.examples.tutorials.mnist import input_data 14 | import time 15 | 16 | # Define paramaters for the model 17 | learning_rate = 0.01 18 | batch_size = 128 19 | n_epochs = 10 20 | 21 | # Step 1: Read in data 22 | # using TF Learn's built in function to load MNIST data to the folder data/mnist 23 | mnist = input_data.read_data_sets('/data/mnist', one_hot=True) 24 | 25 | # Step 2: create placeholders for features and labels 26 | # each image in the MNIST data is of shape 28*28 = 784 27 | # therefore, each image is represented with a 1x784 tensor 28 | # there are 10 classes for each image, corresponding to digits 0 - 9. 29 | # Features are of the type float, and labels are of the type int 30 | 31 | 32 | # Step 3: create weights and bias 33 | # weights and biases are initialized to 0 34 | # shape of w depends on the dimension of X and Y so that Y = X * w + b 35 | # shape of b depends on Y 36 | 37 | 38 | # Step 4: build model 39 | # the model that returns the logits. 40 | # this logits will be later passed through softmax layer 41 | # to get the probability distribution of possible label of the image 42 | # DO NOT DO SOFTMAX HERE 43 | 44 | 45 | # Step 5: define loss function 46 | # use cross entropy loss of the real labels with the softmax of logits 47 | # use the method: 48 | # tf.nn.softmax_cross_entropy_with_logits(logits, Y) 49 | # then use tf.reduce_mean to get the mean loss of the batch 50 | 51 | 52 | # Step 6: define training op 53 | # using gradient descent to minimize loss 54 | 55 | 56 | with tf.Session() as sess: 57 | start_time = time.time() 58 | sess.run(tf.global_variables_initializer()) 59 | n_batches = int(mnist.train.num_examples/batch_size) 60 | for i in range(n_epochs): # train the model n_epochs times 61 | total_loss = 0 62 | 63 | for _ in range(n_batches): 64 | X_batch, Y_batch = mnist.train.next_batch(batch_size) 65 | # TO-DO: run optimizer + fetch loss_batch 66 | # 67 | # 68 | total_loss += loss_batch 69 | print('Average loss epoch {0}: {1}'.format(i, total_loss/n_batches)) 70 | 71 | print('Total time: {0} seconds'.format(time.time() - start_time)) 72 | 73 | print('Optimization Finished!') # should be around 0.35 after 25 epochs 74 | 75 | # test the model 76 | preds = tf.nn.softmax(logits) 77 | correct_preds = tf.equal(tf.argmax(preds, 1), tf.argmax(Y, 1)) 78 | accuracy = tf.reduce_sum(tf.cast(correct_preds, tf.float32)) # need numpy.count_nonzero(boolarr) :( 79 | 80 | n_batches = int(mnist.test.num_examples/batch_size) 81 | total_correct_preds = 0 82 | 83 | for i in range(n_batches): 84 | X_batch, Y_batch = mnist.test.next_batch(batch_size) 85 | accuracy_batch = sess.run([accuracy], feed_dict={X: X_batch, Y:Y_batch}) 86 | total_correct_preds += accuracy_batch 87 | 88 | print('Accuracy {0}'.format(total_correct_preds/mnist.test.num_examples)) 89 | -------------------------------------------------------------------------------- /2017/examples/04_word2vec_no_frills.py: -------------------------------------------------------------------------------- 1 | """ The no frills implementation of word2vec skip-gram model using NCE loss. 2 | Author: Chip Huyen 3 | Prepared for the class CS 20SI: "TensorFlow for Deep Learning Research" 4 | cs20si.stanford.edu 5 | """ 6 | 7 | from __future__ import absolute_import 8 | from __future__ import division 9 | from __future__ import print_function 10 | 11 | import os 12 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 13 | 14 | import numpy as np 15 | import tensorflow as tf 16 | from tensorflow.contrib.tensorboard.plugins import projector 17 | 18 | from process_data import process_data 19 | 20 | VOCAB_SIZE = 50000 21 | BATCH_SIZE = 128 22 | EMBED_SIZE = 128 # dimension of the word embedding vectors 23 | SKIP_WINDOW = 1 # the context window 24 | NUM_SAMPLED = 64 # Number of negative examples to sample. 25 | LEARNING_RATE = 1.0 26 | NUM_TRAIN_STEPS = 10000 27 | SKIP_STEP = 2000 # how many steps to skip before reporting the loss 28 | 29 | def word2vec(batch_gen): 30 | """ Build the graph for word2vec model and train it """ 31 | # Step 1: define the placeholders for input and output 32 | with tf.name_scope('data'): 33 | center_words = tf.placeholder(tf.int32, shape=[BATCH_SIZE], name='center_words') 34 | target_words = tf.placeholder(tf.int32, shape=[BATCH_SIZE, 1], name='target_words') 35 | 36 | # Assemble this part of the graph on the CPU. You can change it to GPU if you have GPU 37 | # Step 2: define weights. In word2vec, it's actually the weights that we care about 38 | 39 | with tf.name_scope('embedding_matrix'): 40 | embed_matrix = tf.Variable(tf.random_uniform([VOCAB_SIZE, EMBED_SIZE], -1.0, 1.0), 41 | name='embed_matrix') 42 | 43 | # Step 3: define the inference 44 | with tf.name_scope('loss'): 45 | embed = tf.nn.embedding_lookup(embed_matrix, center_words, name='embed') 46 | 47 | # Step 4: construct variables for NCE loss 48 | nce_weight = tf.Variable(tf.truncated_normal([VOCAB_SIZE, EMBED_SIZE], 49 | stddev=1.0 / (EMBED_SIZE ** 0.5)), 50 | name='nce_weight') 51 | nce_bias = tf.Variable(tf.zeros([VOCAB_SIZE]), name='nce_bias') 52 | 53 | # define loss function to be NCE loss function 54 | loss = tf.reduce_mean(tf.nn.nce_loss(weights=nce_weight, 55 | biases=nce_bias, 56 | labels=target_words, 57 | inputs=embed, 58 | num_sampled=NUM_SAMPLED, 59 | num_classes=VOCAB_SIZE), name='loss') 60 | 61 | # Step 5: define optimizer 62 | optimizer = tf.train.GradientDescentOptimizer(LEARNING_RATE).minimize(loss) 63 | 64 | with tf.Session() as sess: 65 | sess.run(tf.global_variables_initializer()) 66 | 67 | total_loss = 0.0 # we use this to calculate late average loss in the last SKIP_STEP steps 68 | writer = tf.summary.FileWriter('./graphs/no_frills/', sess.graph) 69 | for index in range(NUM_TRAIN_STEPS): 70 | centers, targets = next(batch_gen) 71 | loss_batch, _ = sess.run([loss, optimizer], 72 | feed_dict={center_words: centers, target_words: targets}) 73 | total_loss += loss_batch 74 | if (index + 1) % SKIP_STEP == 0: 75 | print('Average loss at step {}: {:5.1f}'.format(index, total_loss / SKIP_STEP)) 76 | total_loss = 0.0 77 | writer.close() 78 | 79 | def main(): 80 | batch_gen = process_data(VOCAB_SIZE, BATCH_SIZE, SKIP_WINDOW) 81 | word2vec(batch_gen) 82 | 83 | if __name__ == '__main__': 84 | main() -------------------------------------------------------------------------------- /2017/examples/04_word2vec_starter.py: -------------------------------------------------------------------------------- 1 | """ The mo frills implementation of word2vec skip-gram model using NCE loss. 2 | Author: Chip Huyen 3 | Prepared for the class CS 20SI: "TensorFlow for Deep Learning Research" 4 | cs20si.stanford.edu 5 | """ 6 | 7 | from __future__ import absolute_import 8 | from __future__ import division 9 | from __future__ import print_function 10 | 11 | import os 12 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 13 | 14 | import numpy as np 15 | import tensorflow as tf 16 | from tensorflow.contrib.tensorboard.plugins import projector 17 | 18 | from process_data import process_data 19 | 20 | VOCAB_SIZE = 50000 21 | BATCH_SIZE = 128 22 | EMBED_SIZE = 128 # dimension of the word embedding vectors 23 | SKIP_WINDOW = 1 # the context window 24 | NUM_SAMPLED = 64 # Number of negative examples to sample. 25 | LEARNING_RATE = 1.0 26 | NUM_TRAIN_STEPS = 20000 27 | SKIP_STEP = 2000 # how many steps to skip before reporting the loss 28 | 29 | def word2vec(batch_gen): 30 | """ Build the graph for word2vec model and train it """ 31 | # Step 1: define the placeholders for input and output 32 | # center_words have to be int to work on embedding lookup 33 | 34 | # TO DO 35 | 36 | 37 | # Step 2: define weights. In word2vec, it's actually the weights that we care about 38 | # vocab size x embed size 39 | # initialized to random uniform -1 to 1 40 | 41 | # TOO DO 42 | 43 | 44 | # Step 3: define the inference 45 | # get the embed of input words using tf.nn.embedding_lookup 46 | # embed = tf.nn.embedding_lookup(embed_matrix, center_words, name='embed') 47 | 48 | # TO DO 49 | 50 | 51 | # Step 4: construct variables for NCE loss 52 | # tf.nn.nce_loss(weights, biases, labels, inputs, num_sampled, num_classes, ...) 53 | # nce_weight (vocab size x embed size), intialized to truncated_normal stddev=1.0 / (EMBED_SIZE ** 0.5) 54 | # bias: vocab size, initialized to 0 55 | 56 | # TO DO 57 | 58 | 59 | # define loss function to be NCE loss function 60 | # tf.nn.nce_loss(weights, biases, labels, inputs, num_sampled, num_classes, ...) 61 | # need to get the mean accross the batch 62 | # note: you should use embedding of center words for inputs, not center words themselves 63 | 64 | # TO DO 65 | 66 | 67 | # Step 5: define optimizer 68 | 69 | # TO DO 70 | 71 | 72 | 73 | with tf.Session() as sess: 74 | # TO DO: initialize variables 75 | 76 | 77 | total_loss = 0.0 # we use this to calculate the average loss in the last SKIP_STEP steps 78 | writer = tf.summary.FileWriter('./graphs/no_frills/', sess.graph) 79 | for index in range(NUM_TRAIN_STEPS): 80 | centers, targets = next(batch_gen) 81 | # TO DO: create feed_dict, run optimizer, fetch loss_batch 82 | 83 | total_loss += loss_batch 84 | if (index + 1) % SKIP_STEP == 0: 85 | print('Average loss at step {}: {:5.1f}'.format(index, total_loss / SKIP_STEP)) 86 | total_loss = 0.0 87 | writer.close() 88 | 89 | def main(): 90 | batch_gen = process_data(VOCAB_SIZE, BATCH_SIZE, SKIP_WINDOW) 91 | word2vec(batch_gen) 92 | 93 | if __name__ == '__main__': 94 | main() 95 | -------------------------------------------------------------------------------- /2017/examples/05_csv_reader.py: -------------------------------------------------------------------------------- 1 | """ Some people tried to use TextLineReader for the assignment 1 2 | but seem to have problems getting it work, so here is a short 3 | script demonstrating the use of CSV reader on the heart dataset. 4 | Note that the heart dataset is originally in txt so I first 5 | converted it to csv to take advantage of the already laid out columns. 6 | 7 | You can download heart.csv in the data folder. 8 | Author: Chip Huyen 9 | Prepared for the class CS 20SI: "TensorFlow for Deep Learning Research" 10 | cs20si.stanford.edu 11 | """ 12 | import os 13 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 14 | 15 | import sys 16 | sys.path.append('..') 17 | 18 | import tensorflow as tf 19 | 20 | DATA_PATH = 'data/heart.csv' 21 | BATCH_SIZE = 2 22 | N_FEATURES = 9 23 | 24 | def batch_generator(filenames): 25 | """ filenames is the list of files you want to read from. 26 | In this case, it contains only heart.csv 27 | """ 28 | filename_queue = tf.train.string_input_producer(filenames) 29 | reader = tf.TextLineReader(skip_header_lines=1) # skip the first line in the file 30 | _, value = reader.read(filename_queue) 31 | 32 | # record_defaults are the default values in case some of our columns are empty 33 | # This is also to tell tensorflow the format of our data (the type of the decode result) 34 | # for this dataset, out of 9 feature columns, 35 | # 8 of them are floats (some are integers, but to make our features homogenous, 36 | # we consider them floats), and 1 is string (at position 5) 37 | # the last column corresponds to the lable is an integer 38 | 39 | record_defaults = [[1.0] for _ in range(N_FEATURES)] 40 | record_defaults[4] = [''] 41 | record_defaults.append([1]) 42 | 43 | # read in the 10 rows of data 44 | content = tf.decode_csv(value, record_defaults=record_defaults) 45 | 46 | # convert the 5th column (present/absent) to the binary value 0 and 1 47 | content[4] = tf.cond(tf.equal(content[4], tf.constant('Present')), lambda: tf.constant(1.0), lambda: tf.constant(0.0)) 48 | 49 | # pack all 9 features into a tensor 50 | features = tf.stack(content[:N_FEATURES]) 51 | 52 | # assign the last column to label 53 | label = content[-1] 54 | 55 | # minimum number elements in the queue after a dequeue, used to ensure 56 | # that the samples are sufficiently mixed 57 | # I think 10 times the BATCH_SIZE is sufficient 58 | min_after_dequeue = 10 * BATCH_SIZE 59 | 60 | # the maximum number of elements in the queue 61 | capacity = 20 * BATCH_SIZE 62 | 63 | # shuffle the data to generate BATCH_SIZE sample pairs 64 | data_batch, label_batch = tf.train.shuffle_batch([features, label], batch_size=BATCH_SIZE, 65 | capacity=capacity, min_after_dequeue=min_after_dequeue) 66 | 67 | return data_batch, label_batch 68 | 69 | def generate_batches(data_batch, label_batch): 70 | with tf.Session() as sess: 71 | coord = tf.train.Coordinator() 72 | threads = tf.train.start_queue_runners(coord=coord) 73 | for _ in range(10): # generate 10 batches 74 | features, labels = sess.run([data_batch, label_batch]) 75 | print(features) 76 | coord.request_stop() 77 | coord.join(threads) 78 | 79 | def main(): 80 | data_batch, label_batch = batch_generator([DATA_PATH]) 81 | generate_batches(data_batch, label_batch) 82 | 83 | if __name__ == '__main__': 84 | main() 85 | -------------------------------------------------------------------------------- /2017/examples/05_randomization.py: -------------------------------------------------------------------------------- 1 | """ Examples to demonstrate ops level randomization 2 | Author: Chip Huyen 3 | Prepared for the class CS 20SI: "TensorFlow for Deep Learning Research" 4 | cs20si.stanford.edu 5 | """ 6 | import os 7 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 8 | 9 | import tensorflow as tf 10 | 11 | # Example 1: session is the thing that keeps track of random state 12 | c = tf.random_uniform([], -10, 10, seed=2) 13 | 14 | with tf.Session() as sess: 15 | print(sess.run(c)) # >> 3.57493 16 | print(sess.run(c)) # >> -5.97319 17 | 18 | # Example 2: each new session will start the random state all over again. 19 | c = tf.random_uniform([], -10, 10, seed=2) 20 | 21 | with tf.Session() as sess: 22 | print(sess.run(c)) # >> 3.57493 23 | 24 | with tf.Session() as sess: 25 | print(sess.run(c)) # >> 3.57493 26 | 27 | # Example 3: with operation level random seed, each op keeps its own seed. 28 | c = tf.random_uniform([], -10, 10, seed=2) 29 | d = tf.random_uniform([], -10, 10, seed=2) 30 | 31 | with tf.Session() as sess: 32 | print(sess.run(c)) # >> 3.57493 33 | print(sess.run(d)) # >> 3.57493 34 | 35 | # Example 4: graph level random seed 36 | tf.set_random_seed(2) 37 | c = tf.random_uniform([], -10, 10) 38 | d = tf.random_uniform([], -10, 10) 39 | 40 | with tf.Session() as sess: 41 | print(sess.run(c)) # >> 9.12393 42 | print(sess.run(d)) # >> -4.53404 43 | -------------------------------------------------------------------------------- /2017/examples/07_basic_filters.py: -------------------------------------------------------------------------------- 1 | """ 2 | Simple examples of convolution to do some basic filters 3 | Also demonstrates the use of TensorFlow data readers. 4 | 5 | We will use some popular filters for our image. 6 | It seems to be working with grayscale images, but not with rgb images. 7 | It's probably because I didn't choose the right kernels for rgb images. 8 | 9 | kernels for rgb images have dimensions 3 x 3 x 3 x 3 10 | kernels for grayscale images have dimensions 3 x 3 x 1 x 1 11 | 12 | Note: 13 | When you call tf.train.string_input_producer, 14 | a tf.train.QueueRunner is added to the graph, which must be run using 15 | e.g. tf.train.start_queue_runners() else your session will run into deadlock 16 | and your program will crash. 17 | 18 | And to run QueueRunner, you need a coordinator to close to your queue for you. 19 | Without coordinator, your threads will keep on running outside session and you will have the error: 20 | ERROR:tensorflow:Exception in QueueRunner: Attempted to use a closed Session. 21 | 22 | Author: Chip Huyen 23 | Prepared for the class CS 20SI: "TensorFlow for Deep Learning Research" 24 | cs20si.stanford.edu 25 | 26 | """ 27 | import os 28 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 29 | 30 | import sys 31 | sys.path.append('..') 32 | 33 | from matplotlib import gridspec as gridspec 34 | from matplotlib import pyplot as plt 35 | import tensorflow as tf 36 | 37 | import kernels 38 | 39 | FILENAME = 'data/friday.jpg' 40 | 41 | def read_one_image(filename): 42 | """ This is just to demonstrate how to open an image in TensorFlow, 43 | but it's actually a lot easier to use Pillow 44 | """ 45 | filename_queue = tf.train.string_input_producer([filename]) 46 | image_reader = tf.WholeFileReader() 47 | _, image_file = image_reader.read(filename_queue) 48 | image = tf.image.decode_jpeg(image_file, channels=3) 49 | image = tf.cast(image, tf.float32) / 256.0 # cast to float to make conv2d work 50 | return image 51 | 52 | def convolve(image, kernels, rgb=True, strides=[1, 3, 3, 1], padding='SAME'): 53 | images = [image[0]] 54 | for i, kernel in enumerate(kernels): 55 | filtered_image = tf.nn.conv2d(image, kernel, strides=strides, padding=padding)[0] 56 | if i == 2: 57 | filtered_image = tf.minimum(tf.nn.relu(filtered_image), 255) 58 | images.append(filtered_image) 59 | return images 60 | 61 | def get_real_images(images): 62 | with tf.Session() as sess: 63 | coord = tf.train.Coordinator() 64 | threads = tf.train.start_queue_runners(coord=coord) 65 | images = sess.run(images) 66 | coord.request_stop() 67 | coord.join(threads) 68 | return images 69 | 70 | def show_images(images, rgb=True): 71 | gs = gridspec.GridSpec(1, len(images)) 72 | for i, image in enumerate(images): 73 | plt.subplot(gs[0, i]) 74 | if rgb: 75 | plt.imshow(image) 76 | else: 77 | image = image.reshape(image.shape[0], image.shape[1]) 78 | plt.imshow(image, cmap='gray') 79 | plt.axis('off') 80 | plt.show() 81 | 82 | def main(): 83 | rgb = False 84 | if rgb: 85 | kernels_list = [kernels.BLUR_FILTER_RGB, kernels.SHARPEN_FILTER_RGB, kernels.EDGE_FILTER_RGB, 86 | kernels.TOP_SOBEL_RGB, kernels.EMBOSS_FILTER_RGB] 87 | else: 88 | kernels_list = [kernels.BLUR_FILTER, kernels.SHARPEN_FILTER, kernels.EDGE_FILTER, 89 | kernels.TOP_SOBEL, kernels.EMBOSS_FILTER] 90 | 91 | image = read_one_image(FILENAME) 92 | if not rgb: 93 | image = tf.image.rgb_to_grayscale(image) 94 | image = tf.expand_dims(image, 0) # to make it into a batch of 1 element 95 | images = convolve(image, kernels_list, rgb) 96 | images = get_real_images(images) 97 | show_images(images, rgb) 98 | 99 | if __name__ == '__main__': 100 | main() -------------------------------------------------------------------------------- /2017/examples/09_queue_example.py: -------------------------------------------------------------------------------- 1 | """ Example to demonstrate how to use queues 2 | Author: Chip Huyen 3 | Prepared for the class CS 20SI: "TensorFlow for Deep Learning Research" 4 | cs20si.stanford.edu 5 | """ 6 | import os 7 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 8 | 9 | import numpy as np 10 | import tensorflow as tf 11 | 12 | N_SAMPLES = 1000 13 | NUM_THREADS = 4 14 | # Generating some simple data 15 | # create 1000 random samples, each is a 1D array from the normal distribution (10, 1) 16 | data = 10 * np.random.randn(N_SAMPLES, 4) + 1 17 | # create 1000 random labels of 0 and 1 18 | target = np.random.randint(0, 2, size=N_SAMPLES) 19 | 20 | queue = tf.FIFOQueue(capacity=50, dtypes=[tf.float32, tf.int32], shapes=[[4], []]) 21 | 22 | enqueue_op = queue.enqueue_many([data, target]) 23 | data_sample, label_sample = queue.dequeue() 24 | 25 | # create ops that do something with data_sample and label_sample 26 | 27 | # create NUM_THREADS to do enqueue 28 | qr = tf.train.QueueRunner(queue, [enqueue_op] * NUM_THREADS) 29 | with tf.Session() as sess: 30 | # create a coordinator, launch the queue runner threads. 31 | coord = tf.train.Coordinator() 32 | enqueue_threads = qr.create_threads(sess, coord=coord, start=True) 33 | try: 34 | for step in range(100): # do to 100 iterations 35 | if coord.should_stop(): 36 | break 37 | data_batch, label_batch = sess.run([data_sample, label_sample]) 38 | print(data_batch) 39 | print(label_batch) 40 | except Exception as e: 41 | coord.request_stop(e) 42 | finally: 43 | coord.request_stop() 44 | coord.join(enqueue_threads) -------------------------------------------------------------------------------- /2017/examples/09_tfrecord_example.py: -------------------------------------------------------------------------------- 1 | """ Examples to demonstrate how to write an image file to a TFRecord, 2 | and how to read a TFRecord file using TFRecordReader. 3 | Author: Chip Huyen 4 | Prepared for the class CS 20SI: "TensorFlow for Deep Learning Research" 5 | cs20si.stanford.edu 6 | """ 7 | import os 8 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 9 | 10 | import sys 11 | sys.path.append('..') 12 | 13 | from PIL import Image 14 | import numpy as np 15 | import matplotlib.pyplot as plt 16 | import tensorflow as tf 17 | 18 | # image supposed to have shape: 480 x 640 x 3 = 921600 19 | IMAGE_PATH = 'data/' 20 | 21 | def _int64_feature(value): 22 | return tf.train.Feature(int64_list=tf.train.Int64List(value=[value])) 23 | 24 | def _bytes_feature(value): 25 | return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value])) 26 | 27 | def get_image_binary(filename): 28 | """ You can read in the image using tensorflow too, but it's a drag 29 | since you have to create graphs. It's much easier using Pillow and NumPy 30 | """ 31 | image = Image.open(filename) 32 | image = np.asarray(image, np.uint8) 33 | shape = np.array(image.shape, np.int32) 34 | return shape.tobytes(), image.tobytes() # convert image to raw data bytes in the array. 35 | 36 | def write_to_tfrecord(label, shape, binary_image, tfrecord_file): 37 | """ This example is to write a sample to TFRecord file. If you want to write 38 | more samples, just use a loop. 39 | """ 40 | writer = tf.python_io.TFRecordWriter(tfrecord_file) 41 | # write label, shape, and image content to the TFRecord file 42 | example = tf.train.Example(features=tf.train.Features(feature={ 43 | 'label': _int64_feature(label), 44 | 'shape': _bytes_feature(shape), 45 | 'image': _bytes_feature(binary_image) 46 | })) 47 | writer.write(example.SerializeToString()) 48 | writer.close() 49 | 50 | def write_tfrecord(label, image_file, tfrecord_file): 51 | shape, binary_image = get_image_binary(image_file) 52 | write_to_tfrecord(label, shape, binary_image, tfrecord_file) 53 | 54 | def read_from_tfrecord(filenames): 55 | tfrecord_file_queue = tf.train.string_input_producer(filenames, name='queue') 56 | reader = tf.TFRecordReader() 57 | _, tfrecord_serialized = reader.read(tfrecord_file_queue) 58 | 59 | # label and image are stored as bytes but could be stored as 60 | # int64 or float64 values in a serialized tf.Example protobuf. 61 | tfrecord_features = tf.parse_single_example(tfrecord_serialized, 62 | features={ 63 | 'label': tf.FixedLenFeature([], tf.int64), 64 | 'shape': tf.FixedLenFeature([], tf.string), 65 | 'image': tf.FixedLenFeature([], tf.string), 66 | }, name='features') 67 | # image was saved as uint8, so we have to decode as uint8. 68 | image = tf.decode_raw(tfrecord_features['image'], tf.uint8) 69 | shape = tf.decode_raw(tfrecord_features['shape'], tf.int32) 70 | # the image tensor is flattened out, so we have to reconstruct the shape 71 | image = tf.reshape(image, shape) 72 | label = tfrecord_features['label'] 73 | return label, shape, image 74 | 75 | def read_tfrecord(tfrecord_file): 76 | label, shape, image = read_from_tfrecord([tfrecord_file]) 77 | 78 | with tf.Session() as sess: 79 | coord = tf.train.Coordinator() 80 | threads = tf.train.start_queue_runners(coord=coord) 81 | label, image, shape = sess.run([label, image, shape]) 82 | coord.request_stop() 83 | coord.join(threads) 84 | print(label) 85 | print(shape) 86 | plt.imshow(image) 87 | plt.show() 88 | 89 | def main(): 90 | # assume the image has the label Chihuahua, which corresponds to class number 1 91 | label = 1 92 | image_file = IMAGE_PATH + 'friday.jpg' 93 | tfrecord_file = IMAGE_PATH + 'friday.tfrecord' 94 | write_tfrecord(label, image_file, tfrecord_file) 95 | read_tfrecord(tfrecord_file) 96 | 97 | if __name__ == '__main__': 98 | main() 99 | 100 | -------------------------------------------------------------------------------- /2017/examples/11_char_rnn_gist.py: -------------------------------------------------------------------------------- 1 | """ A clean, no_frills character-level generative language model. 2 | Created by Danijar Hafner (danijar.com), edited by Chip Huyen 3 | for the class CS 20SI: "TensorFlow for Deep Learning Research" 4 | 5 | Based on Andrej Karpathy's blog: 6 | http://karpathy.github.io/2015/05/21/rnn-effectiveness/ 7 | """ 8 | import os 9 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 10 | import sys 11 | sys.path.append('..') 12 | 13 | import time 14 | 15 | import tensorflow as tf 16 | 17 | import utils 18 | 19 | DATA_PATH = 'data/arvix_abstracts.txt' 20 | HIDDEN_SIZE = 200 21 | BATCH_SIZE = 64 22 | NUM_STEPS = 50 23 | SKIP_STEP = 40 24 | TEMPRATURE = 0.7 25 | LR = 0.003 26 | LEN_GENERATED = 300 27 | 28 | def vocab_encode(text, vocab): 29 | return [vocab.index(x) + 1 for x in text if x in vocab] 30 | 31 | def vocab_decode(array, vocab): 32 | return ''.join([vocab[x - 1] for x in array]) 33 | 34 | def read_data(filename, vocab, window=NUM_STEPS, overlap=NUM_STEPS//2): 35 | for text in open(filename): 36 | text = vocab_encode(text, vocab) 37 | for start in range(0, len(text) - window, overlap): 38 | chunk = text[start: start + window] 39 | chunk += [0] * (window - len(chunk)) 40 | yield chunk 41 | 42 | def read_batch(stream, batch_size=BATCH_SIZE): 43 | batch = [] 44 | for element in stream: 45 | batch.append(element) 46 | if len(batch) == batch_size: 47 | yield batch 48 | batch = [] 49 | yield batch 50 | 51 | def create_rnn(seq, hidden_size=HIDDEN_SIZE): 52 | cell = tf.contrib.rnn.GRUCell(hidden_size) 53 | in_state = tf.placeholder_with_default( 54 | cell.zero_state(tf.shape(seq)[0], tf.float32), [None, hidden_size]) 55 | # this line to calculate the real length of seq 56 | # all seq are padded to be of the same length which is NUM_STEPS 57 | length = tf.reduce_sum(tf.reduce_max(tf.sign(seq), 2), 1) 58 | output, out_state = tf.nn.dynamic_rnn(cell, seq, length, in_state) 59 | return output, in_state, out_state 60 | 61 | def create_model(seq, temp, vocab, hidden=HIDDEN_SIZE): 62 | seq = tf.one_hot(seq, len(vocab)) 63 | output, in_state, out_state = create_rnn(seq, hidden) 64 | # fully_connected is syntactic sugar for tf.matmul(w, output) + b 65 | # it will create w and b for us 66 | logits = tf.contrib.layers.fully_connected(output, len(vocab), None) 67 | loss = tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits(logits=logits[:, :-1], labels=seq[:, 1:])) 68 | # sample the next character from Maxwell-Boltzmann Distribution with temperature temp 69 | # it works equally well without tf.exp 70 | sample = tf.multinomial(tf.exp(logits[:, -1] / temp), 1)[:, 0] 71 | return loss, sample, in_state, out_state 72 | 73 | def training(vocab, seq, loss, optimizer, global_step, temp, sample, in_state, out_state): 74 | saver = tf.train.Saver() 75 | start = time.time() 76 | with tf.Session() as sess: 77 | writer = tf.summary.FileWriter('graphs/gist', sess.graph) 78 | sess.run(tf.global_variables_initializer()) 79 | 80 | ckpt = tf.train.get_checkpoint_state(os.path.dirname('checkpoints/arvix/checkpoint')) 81 | if ckpt and ckpt.model_checkpoint_path: 82 | saver.restore(sess, ckpt.model_checkpoint_path) 83 | 84 | iteration = global_step.eval() 85 | for batch in read_batch(read_data(DATA_PATH, vocab)): 86 | batch_loss, _ = sess.run([loss, optimizer], {seq: batch}) 87 | if (iteration + 1) % SKIP_STEP == 0: 88 | print('Iter {}. \n Loss {}. Time {}'.format(iteration, batch_loss, time.time() - start)) 89 | online_inference(sess, vocab, seq, sample, temp, in_state, out_state) 90 | start = time.time() 91 | saver.save(sess, 'checkpoints/arvix/char-rnn', iteration) 92 | iteration += 1 93 | 94 | def online_inference(sess, vocab, seq, sample, temp, in_state, out_state, seed='T'): 95 | """ Generate sequence one character at a time, based on the previous character 96 | """ 97 | sentence = seed 98 | state = None 99 | for _ in range(LEN_GENERATED): 100 | batch = [vocab_encode(sentence[-1], vocab)] 101 | feed = {seq: batch, temp: TEMPRATURE} 102 | # for the first decoder step, the state is None 103 | if state is not None: 104 | feed.update({in_state: state}) 105 | index, state = sess.run([sample, out_state], feed) 106 | sentence += vocab_decode(index, vocab) 107 | print(sentence) 108 | 109 | def main(): 110 | vocab = ( 111 | " $%'()+,-./0123456789:;=?ABCDEFGHIJKLMNOPQRSTUVWXYZ" 112 | "\\^_abcdefghijklmnopqrstuvwxyz{|}") 113 | seq = tf.placeholder(tf.int32, [None, None]) 114 | temp = tf.placeholder(tf.float32) 115 | loss, sample, in_state, out_state = create_model(seq, temp, vocab) 116 | global_step = tf.Variable(0, dtype=tf.int32, trainable=False, name='global_step') 117 | optimizer = tf.train.AdamOptimizer(LR).minimize(loss, global_step=global_step) 118 | utils.make_dir('checkpoints') 119 | utils.make_dir('checkpoints/arvix') 120 | training(vocab, seq, loss, optimizer, global_step, temp, sample, in_state, out_state) 121 | 122 | if __name__ == '__main__': 123 | main() -------------------------------------------------------------------------------- /2017/examples/autoencoder/autoencoder.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | 3 | from layers import * 4 | 5 | def encoder(input): 6 | # Create a conv network with 3 conv layers and 1 FC layer 7 | # Conv 1: filter: [3, 3, 1], stride: [2, 2], relu 8 | 9 | # Conv 2: filter: [3, 3, 8], stride: [2, 2], relu 10 | 11 | # Conv 3: filter: [3, 3, 8], stride: [2, 2], relu 12 | 13 | # FC: output_dim: 100, no non-linearity 14 | raise NotImplementedError 15 | 16 | def decoder(input): 17 | # Create a deconv network with 1 FC layer and 3 deconv layers 18 | # FC: output dim: 128, relu 19 | 20 | # Reshape to [batch_size, 4, 4, 8] 21 | 22 | # Deconv 1: filter: [3, 3, 8], stride: [2, 2], relu 23 | 24 | # Deconv 2: filter: [8, 8, 1], stride: [2, 2], padding: valid, relu 25 | 26 | # Deconv 3: filter: [7, 7, 1], stride: [1, 1], padding: valid, sigmoid 27 | raise NotImplementedError 28 | 29 | def autoencoder(input_shape): 30 | # Define place holder with input shape 31 | 32 | # Define variable scope for autoencoder 33 | with tf.variable_scope('autoencoder') as scope: 34 | # Pass input to encoder to obtain encoding 35 | 36 | # Pass encoding into decoder to obtain reconstructed image 37 | 38 | # Return input image (placeholder) and reconstructed image 39 | pass 40 | -------------------------------------------------------------------------------- /2017/examples/autoencoder/layer_utils.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | 3 | def get_deconv2d_output_dims(input_dims, filter_dims, stride_dims, padding): 4 | # Returns the height and width of the output of a deconvolution layer. 5 | batch_size, input_h, input_w, num_channels_in = input_dims 6 | filter_h, filter_w, num_channels_out = filter_dims 7 | stride_h, stride_w = stride_dims 8 | 9 | # Compute the height in the output, based on the padding. 10 | if padding == 'SAME': 11 | out_h = input_h * stride_h 12 | elif padding == 'VALID': 13 | out_h = (input_h - 1) * stride_h + filter_h 14 | 15 | # Compute the width in the output, based on the padding. 16 | if padding == 'SAME': 17 | out_w = input_w * stride_w 18 | elif padding == 'VALID': 19 | out_w = (input_w - 1) * stride_w + filter_w 20 | 21 | return [batch_size, out_h, out_w, num_channels_out] 22 | -------------------------------------------------------------------------------- /2017/examples/autoencoder/layers.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | 3 | from layer_utils import get_deconv2d_output_dims 4 | 5 | def conv(input, name, filter_dims, stride_dims, padding='SAME', 6 | non_linear_fn=tf.nn.relu): 7 | input_dims = input.get_shape().as_list() 8 | assert(len(input_dims) == 4) # batch_size, height, width, num_channels_in 9 | assert(len(filter_dims) == 3) # height, width and num_channels out 10 | assert(len(stride_dims) == 2) # stride height and width 11 | 12 | num_channels_in = input_dims[-1] 13 | filter_h, filter_w, num_channels_out = filter_dims 14 | stride_h, stride_w = stride_dims 15 | 16 | # Define a variable scope for the conv layer 17 | with tf.variable_scope(name) as scope: 18 | # Create filter weight variable 19 | 20 | # Create bias variable 21 | 22 | # Define the convolution flow graph 23 | 24 | # Add bias to conv output 25 | 26 | # Apply non-linearity (if asked) and return output 27 | pass 28 | 29 | def deconv(input, name, filter_dims, stride_dims, padding='SAME', 30 | non_linear_fn=tf.nn.relu): 31 | input_dims = input.get_shape().as_list() 32 | assert(len(input_dims) == 4) # batch_size, height, width, num_channels_in 33 | assert(len(filter_dims) == 3) # height, width and num_channels out 34 | assert(len(stride_dims) == 2) # stride height and width 35 | 36 | num_channels_in = input_dims[-1] 37 | filter_h, filter_w, num_channels_out = filter_dims 38 | stride_h, stride_w = stride_dims 39 | # Let's step into this function 40 | output_dims = get_deconv2d_output_dims(input_dims, 41 | filter_dims, 42 | stride_dims, 43 | padding) 44 | 45 | # Define a variable scope for the deconv layer 46 | with tf.variable_scope(name) as scope: 47 | # Create filter weight variable 48 | # Note that num_channels_out and in positions are flipped for deconv. 49 | 50 | # Create bias variable 51 | 52 | # Define the deconv flow graph 53 | 54 | # Add bias to deconv output 55 | 56 | # Apply non-linearity (if asked) and return output 57 | pass 58 | 59 | def max_pool(input, name, filter_dims, stride_dims, padding='SAME'): 60 | assert(len(filter_dims) == 2) # filter height and width 61 | assert(len(stride_dims) == 2) # stride height and width 62 | 63 | filter_h, filter_w = filter_dims 64 | stride_h, stride_w = stride_dims 65 | 66 | # Define the max pool flow graph and return output 67 | pass 68 | 69 | def fc(input, name, out_dim, non_linear_fn=tf.nn.relu): 70 | assert(type(out_dim) == int) 71 | 72 | # Define a variable scope for the FC layer 73 | with tf.variable_scope(name) as scope: 74 | input_dims = input.get_shape().as_list() 75 | # the input to the fc layer should be flattened 76 | if len(input_dims) == 4: 77 | # for eg. the output of a conv layer 78 | batch_size, input_h, input_w, num_channels = input_dims 79 | # ignore the batch dimension 80 | in_dim = input_h * input_w * num_channels 81 | flat_input = tf.reshape(input, [batch_size, in_dim]) 82 | else: 83 | in_dim = input_dims[-1] 84 | flat_input = input 85 | 86 | # Create weight variable 87 | 88 | # Create bias variable 89 | 90 | # Define FC flow graph 91 | 92 | # Apply non-linearity (if asked) and return output 93 | pass 94 | -------------------------------------------------------------------------------- /2017/examples/autoencoder/train.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | 3 | from utils import * 4 | from autoencoder import * 5 | 6 | batch_size = 100 7 | batch_shape = (batch_size, 28, 28, 1) 8 | num_visualize = 10 9 | 10 | lr = 0.01 11 | num_epochs = 50 12 | 13 | def calculate_loss(original, reconstructed): 14 | return tf.div(tf.reduce_sum(tf.square(tf.sub(reconstructed, 15 | original))), 16 | tf.constant(float(batch_size))) 17 | 18 | def train(dataset): 19 | input_image, reconstructed_image = autoencoder(batch_shape) 20 | loss = calculate_loss(input_image, reconstructed_image) 21 | optimizer = tf.train.GradientDescentOptimizer(lr).minimize(loss) 22 | 23 | init = tf.global_variables_initializer() 24 | with tf.Session() as session: 25 | session.run(init) 26 | 27 | dataset_size = len(dataset.train.images) 28 | print "Dataset size:", dataset_size 29 | num_iters = (num_epochs * dataset_size)/batch_size 30 | print "Num iters:", num_iters 31 | for step in xrange(num_iters): 32 | input_batch = get_next_batch(dataset.train, batch_size) 33 | loss_val, _ = session.run([loss, optimizer], 34 | feed_dict={input_image: input_batch}) 35 | if step % 1000 == 0: 36 | print "Loss at step", step, ":", loss_val 37 | 38 | test_batch = get_next_batch(dataset.test, batch_size) 39 | reconstruction = session.run(reconstructed_image, 40 | feed_dict={input_image: test_batch}) 41 | visualize(test_batch, reconstruction, num_visualize) 42 | 43 | if __name__ == '__main__': 44 | dataset = load_dataset() 45 | train(dataset) 46 | 47 | -------------------------------------------------------------------------------- /2017/examples/autoencoder/utils.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import tensorflow 4 | import numpy as np 5 | 6 | import matplotlib 7 | matplotlib.use('TKAgg') 8 | from matplotlib import pyplot as plt 9 | 10 | from tensorflow.examples.tutorials.mnist import input_data 11 | 12 | mnist_image_shape = [28, 28, 1] 13 | 14 | def load_dataset(): 15 | return input_data.read_data_sets('MNIST_data') 16 | 17 | def get_next_batch(dataset, batch_size): 18 | # dataset should be mnist.(train/val/test) 19 | batch, _ = dataset.next_batch(batch_size) 20 | batch_shape = [batch_size] + mnist_image_shape 21 | return np.reshape(batch, batch_shape) 22 | 23 | def visualize(_original, _reconstructions, num_visualize): 24 | vis_folder = './vis/' 25 | if not os.path.exists(vis_folder): 26 | os.makedirs(vis_folder) 27 | 28 | original = _original[:num_visualize] 29 | reconstructions = _reconstructions[:num_visualize] 30 | 31 | count = 1 32 | for (orig, rec) in zip(original, reconstructions): 33 | orig = np.reshape(orig, (mnist_image_shape[0], 34 | mnist_image_shape[1])) 35 | rec = np.reshape(rec, (mnist_image_shape[0], 36 | mnist_image_shape[1])) 37 | f, ax = plt.subplots(1,2) 38 | ax[0].imshow(orig, cmap='gray') 39 | ax[1].imshow(rec, cmap='gray') 40 | plt.savefig(vis_folder + "test_%d.png" % count) 41 | count += 1 42 | -------------------------------------------------------------------------------- /2017/examples/autoencoder/vis/test_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/examples/autoencoder/vis/test_1.png -------------------------------------------------------------------------------- /2017/examples/autoencoder/vis/test_10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/examples/autoencoder/vis/test_10.png -------------------------------------------------------------------------------- /2017/examples/autoencoder/vis/test_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/examples/autoencoder/vis/test_2.png -------------------------------------------------------------------------------- /2017/examples/autoencoder/vis/test_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/examples/autoencoder/vis/test_3.png -------------------------------------------------------------------------------- /2017/examples/autoencoder/vis/test_4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/examples/autoencoder/vis/test_4.png -------------------------------------------------------------------------------- /2017/examples/autoencoder/vis/test_5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/examples/autoencoder/vis/test_5.png -------------------------------------------------------------------------------- /2017/examples/autoencoder/vis/test_6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/examples/autoencoder/vis/test_6.png -------------------------------------------------------------------------------- /2017/examples/autoencoder/vis/test_7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/examples/autoencoder/vis/test_7.png -------------------------------------------------------------------------------- /2017/examples/autoencoder/vis/test_8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/examples/autoencoder/vis/test_8.png -------------------------------------------------------------------------------- /2017/examples/autoencoder/vis/test_9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/examples/autoencoder/vis/test_9.png -------------------------------------------------------------------------------- /2017/examples/cgru/README.md: -------------------------------------------------------------------------------- 1 | This is the files used to explain convolutional GRU (CGRU) by Lukasz Kaiser at Google Brain. The accompanied slides can be found at http://web.stanford.edu/class/cs20si/lectures/slides_12.pdf 2 | -------------------------------------------------------------------------------- /2017/examples/cgru/custom_getter.py: -------------------------------------------------------------------------------- 1 | # From [github]/tensorflow/python/kernel_tests/variable_scope_test.py 2 | def testGetterThatCreatesTwoVariablesAndSumsThem(self): 3 | 4 | def custom_getter(getter, name, *args, **kwargs): 5 | g_0 = getter("%s/0" % name, *args, **kwargs) 6 | g_1 = getter("%s/1" % name, *args, **kwargs) 7 | with tf.name_scope("custom_getter"): 8 | return g_0 + g_1 # or g_0 * const / ||g_0|| or anything you want 9 | 10 | with variable_scope.variable_scope("scope", custom_getter=custom_getter): 11 | v = variable_scope.get_variable("v", [1, 2, 3]) 12 | # Or a full model if you wish. OO layers are ok. 13 | 14 | self.assertEqual([1, 2, 3], v.get_shape()) 15 | true_vars = variables_lib.trainable_variables() 16 | self.assertEqual(2, len(true_vars)) 17 | self.assertEqual("scope/v/0:0", true_vars[0].name) 18 | self.assertEqual("scope/v/1:0", true_vars[1].name) 19 | self.assertEqual("custom_getter/add:0", v.name) 20 | with self.test_session() as sess: 21 | variables_lib.global_variables_initializer().run() 22 | np_vars, np_v = sess.run([true_vars, v]) 23 | self.assertAllClose(np_v, sum(np_vars)) 24 | -------------------------------------------------------------------------------- /2017/examples/cgru/my_layers.py: -------------------------------------------------------------------------------- 1 | def saturating_sigmoid(x): 2 | """Saturating sigmoid: 1.2 * sigmoid(x) - 0.1 cut to [0, 1].""" 3 | with tf.name_scope("saturating_sigmoid", [x]): 4 | y = tf.sigmoid(x) 5 | return tf.minimum(1.0, tf.maximum(0.0, 1.2 * y - 0.1)) 6 | 7 | 8 | def embedding(x, vocab_size, dense_size, name=None, reuse=None): 9 | """Embed x of type int64 into dense vectors, reducing to max 4 dimensions.""" 10 | with tf.variable_scope(name, default_name="embedding", 11 | values=[x], reuse=reuse): 12 | embedding_var = tf.get_variable("kernel", [vocab_size, dense_size]) 13 | return tf.gather(embedding_var, x) 14 | 15 | 16 | def conv_gru(x, kernel_size, filters, padding="same", dilation_rate=1, 17 | name=None, reuse=None): 18 | """Convolutional GRU in 1 dimension.""" 19 | # Let's make a shorthand for conv call first. 20 | def do_conv(args, name, bias_start, padding): 21 | return tf.layers.conv1d(args, filters, kernel_size, 22 | padding=padding, dilation_rate=dilation_rate, 23 | bias_initializer=tf.constant_initializer(bias_start), name=name) 24 | # Here comes the GRU gate. 25 | with tf.variable_scope(name, default_name="conv_gru", 26 | values=[x], reuse=reuse): 27 | reset = saturating_sigmoid(do_conv(x, "reset", 1.0, padding)) 28 | gate = saturating_sigmoid(do_conv(x, "gate", 1.0, padding)) 29 | candidate = tf.tanh(do_conv(reset * x, "candidate", 0.0, padding)) 30 | return gate * x + (1 - gate) * candidate 31 | -------------------------------------------------------------------------------- /2017/examples/cgru/neural_gpu_v3.py: -------------------------------------------------------------------------------- 1 | def neural_gpu(features, hparams, name=None): 2 | """The core Neural GPU.""" 3 | with tf.variable_scope(name, "neural_gpu"): 4 | inputs = features["inputs"] 5 | emb_inputs = common_layers.embedding( 6 | inputs, hparams.vocab_size, hparams.hidden_size) 7 | 8 | def step(state, inp): 9 | x = tf.nn.dropout(state, 1.0 - hparams.dropout) 10 | for layer in xrange(hparams.num_hidden_layers): 11 | x = common_layers.conv_gru( 12 | x, hparams.kernel_size, hparams.hidden_size, name="cgru_%d" % layer) 13 | return tf.where(inp == 0, state, x) # No-op where inp is just padding=0. 14 | 15 | final = tf.foldl(step, tf.transpose(inputs, [1, 0]), 16 | initializer=emb_inputs, 17 | parallel_iterations=1, swap_memory=True) 18 | return common_layers.conv(final, hparams.vocab_size, 3, padding="same") 19 | 20 | 21 | def mixed_curriculum(inputs, hparams): 22 | """Mixed curriculum: skip short sequences, but only with some probability.""" 23 | with tf.name_scope("mixed_curriculum"): 24 | inputs_length = tf.to_float(tf.shape(inputs)[1]) 25 | used_length = tf.cond(tf.less(tf.random_uniform([]), 26 | hparams.curriculum_mixing_probability), 27 | lambda: tf.constant(0.0), 28 | lambda: inputs_length) 29 | step = tf.to_float(tf.contrib.framework.get_global_step()) 30 | relative_step = step / hparams.curriculum_lengths_per_step 31 | return used_length - hparams.curriculum_min_length > relative_step 32 | 33 | 34 | def neural_gpu_curriculum(features, hparams, mode): 35 | """The Neural GPU model with curriculum.""" 36 | with tf.name_scope("neural_gpu_with_curriculum"): 37 | inputs = features["inputs"] 38 | is_training = mode == tf.contrib.learn.ModeKeys.TRAIN 39 | should_skip = tf.logical_and(is_training, mixed_curriculum(inputs, hparams)) 40 | final_shape = tf.concat([tf.shape(inputs), 41 | tf.constant([hparams.vocab_size])], axis=0) 42 | outputs = tf.cond(should_skip, 43 | lambda: tf.zeros(final_shape), 44 | lambda: neural_gpu(features, hparams)) 45 | return outputs, should_skip 46 | 47 | 48 | def basic_params1(): 49 | """A set of basic hyperparameters.""" 50 | return tf.HParams(batch_size=32, 51 | num_hidden_layers=4, 52 | kernel_size=3, 53 | hidden_size=64, 54 | vocab_size=256, 55 | dropout=0.2, 56 | clip_grad_norm=2.0, 57 | initializer="orthogonal", 58 | initializer_gain=1.5, 59 | label_smoothing=0.1, 60 | optimizer="Adam", 61 | optimizer_adam_epsilon=1e-4, 62 | optimizer_momentum_momentum=0.9, 63 | max_train_length=512, 64 | learning_rate_decay_scheme="none", 65 | learning_rate_warmup_steps=100, 66 | learning_rate=0.1) 67 | 68 | 69 | def curriculum_params1(): 70 | """Set of hyperparameters with curriculum settings.""" 71 | hparams = common_hparams.basic_params1() 72 | hparams.add_hparam("curriculum_mixing_probability", 0.1) 73 | hparams.add_hparam("curriculum_lengths_per_step", 1000.0) 74 | hparams.add_hparam("curriculum_min_length", 10) 75 | return hparams 76 | -------------------------------------------------------------------------------- /2017/examples/data/fire_theft.xls: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/examples/data/fire_theft.xls -------------------------------------------------------------------------------- /2017/examples/data/friday.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/examples/data/friday.jpg -------------------------------------------------------------------------------- /2017/examples/deepdream/deepdream_exercise.py: -------------------------------------------------------------------------------- 1 | """DeepDream. 2 | """ 3 | from __future__ import absolute_import 4 | from __future__ import division 5 | from __future__ import print_function 6 | 7 | import os.path 8 | import zipfile 9 | 10 | import numpy as np 11 | import PIL.Image 12 | 13 | import tensorflow as tf 14 | 15 | FLAGS = tf.app.flags.FLAGS 16 | 17 | 18 | tf.app.flags.DEFINE_string('data_dir', 19 | '/tmp/inception/', 20 | 'Directory for storing Inception network.') 21 | 22 | tf.app.flags.DEFINE_string('jpeg_file', 23 | 'output.jpg', 24 | 'Where to save the resulting JPEG.') 25 | 26 | 27 | def get_layer(layer): 28 | """Helper for getting layer output Tensor in model Graph. 29 | 30 | Args: 31 | layer: string, layer name 32 | 33 | Returns: 34 | Tensor for that layer. 35 | """ 36 | graph = tf.get_default_graph() 37 | return graph.get_tensor_by_name('import/%s:0' % layer) 38 | 39 | 40 | def maybe_download(data_dir): 41 | """Maybe download pretrained Inception network. 42 | 43 | Args: 44 | data_dir: string, path to data 45 | """ 46 | url = ('https://storage.googleapis.com/download.tensorflow.org/models/' 47 | 'inception5h.zip') 48 | basename = 'inception5h.zip' 49 | local_file = tf.contrib.learn.python.learn.datasets.base.maybe_download( 50 | basename, data_dir, url) 51 | 52 | # Uncompress the pretrained Inception network. 53 | print('Extracting', local_file) 54 | zip_ref = zipfile.ZipFile(local_file, 'r') 55 | zip_ref.extractall(FLAGS.data_dir) 56 | zip_ref.close() 57 | 58 | 59 | def normalize_image(image): 60 | """Stretch the range and prepare the image for saving as a JPEG. 61 | 62 | Args: 63 | image: numpy array 64 | 65 | Returns: 66 | numpy array of image in uint8 67 | """ 68 | # Clip to [0, 1] and then convert to uint8. 69 | image = np.clip(image, 0, 1) 70 | image = np.uint8(image * 255) 71 | return image 72 | 73 | 74 | def save_jpeg(jpeg_file, image): 75 | pil_image = PIL.Image.fromarray(image) 76 | pil_image.save(jpeg_file) 77 | print('Saved to file: ', jpeg_file) 78 | 79 | 80 | def main(unused_argv): 81 | # Maybe download and uncompress pretrained Inception network. 82 | maybe_download(FLAGS.data_dir) 83 | 84 | model_fn = os.path.join(FLAGS.data_dir, 'tensorflow_inception_graph.pb') 85 | 86 | # Load the pretrained Inception model as a GraphDef. 87 | with tf.gfile.FastGFile(model_fn, 'rb') as f: 88 | graph_def = tf.GraphDef() 89 | graph_def.ParseFromString(f.read()) 90 | 91 | with tf.Graph().as_default(): 92 | # Input for the network. 93 | input_image = tf.placeholder(np.float32, name='input') 94 | pixel_mean = 117.0 95 | input_preprocessed = tf.expand_dims(input_image - pixel_mean, 0) 96 | tf.import_graph_def(graph_def, {'input': input_preprocessed}) 97 | 98 | # Grab a list of the names of Tensor's that are the output of convolutions. 99 | graph = tf.get_default_graph() 100 | layers = [op.name for op in graph.get_operations() 101 | if op.type == 'Conv2D' and 'import/' in op.name] 102 | feature_nums = [int(graph.get_tensor_by_name(name+':0').get_shape()[-1]) 103 | for name in layers] 104 | # print('Layers available: %s' % ','.join(layers)) 105 | print('Number of layers', len(layers)) 106 | print('Number of features:', sum(feature_nums)) 107 | 108 | # Pick an internal layer and node to visualize. 109 | # Note that we use outputs before applying the ReLU nonlinearity to 110 | # have non-zero gradients for features with negative initial activations. 111 | layer = 'mixed4d_3x3_bottleneck_pre_relu' 112 | channel = 139 113 | layer_channel = get_layer(layer)[:, :, :, channel] 114 | print('layer %s, channel %d: %s' % (layer, channel, layer_channel)) 115 | 116 | # Define the optimization as the average across all spatial locations. 117 | score = tf.reduce_mean(layer_channel) 118 | 119 | # Automatic differentiation with TensorFlow. Magic! 120 | input_gradient = tf.gradients(score, input_image)[0] 121 | 122 | # Employ random noise as a image. 123 | noise_image = np.random.uniform(size=(224, 224, 3)) + 100.0 124 | image = noise_image.copy() 125 | 126 | ################################################################ 127 | # EXERCISE: Implemement the Deep Dream algorithm here! 128 | ################################################################ 129 | 130 | # Save the image. 131 | stddev = 0.1 132 | image = (image - image.mean()) / max(image.std(), 1e-4) * stddev + 0.5 133 | image = normalize_image(image) 134 | save_jpeg(FLAGS.jpeg_file, image) 135 | 136 | 137 | if __name__ == '__main__': 138 | tf.app.run() -------------------------------------------------------------------------------- /2017/examples/deepdream/output.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/examples/deepdream/output.jpg -------------------------------------------------------------------------------- /2017/examples/graphs/gist/events.out.tfevents.1499787135.MacBook-Pro: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/examples/graphs/gist/events.out.tfevents.1499787135.MacBook-Pro -------------------------------------------------------------------------------- /2017/examples/graphs/gist/events.out.tfevents.1499787150.MacBook-Pro: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/examples/graphs/gist/events.out.tfevents.1499787150.MacBook-Pro -------------------------------------------------------------------------------- /2017/examples/graphs/gist/events.out.tfevents.1499787321.MacBook-Pro: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/examples/graphs/gist/events.out.tfevents.1499787321.MacBook-Pro -------------------------------------------------------------------------------- /2017/examples/graphs/l2/events.out.tfevents.1499786503.MacBook-Pro: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/examples/graphs/l2/events.out.tfevents.1499786503.MacBook-Pro -------------------------------------------------------------------------------- /2017/examples/graphs/l2/events.out.tfevents.1499786515.MacBook-Pro: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/examples/graphs/l2/events.out.tfevents.1499786515.MacBook-Pro -------------------------------------------------------------------------------- /2017/examples/graphs/linear_reg/events.out.tfevents.1499786822.MacBook-Pro: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chiphuyen/stanford-tensorflow-tutorials/51e53daaa2a32cfe7a1966f060b28dbbd081791c/2017/examples/graphs/linear_reg/events.out.tfevents.1499786822.MacBook-Pro -------------------------------------------------------------------------------- /2017/examples/kernels.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import tensorflow as tf 3 | 4 | a = np.zeros([3, 3, 3, 3]) 5 | a[1, 1, :, :] = 0.25 6 | a[0, 1, :, :] = 0.125 7 | a[1, 0, :, :] = 0.125 8 | a[2, 1, :, :] = 0.125 9 | a[1, 2, :, :] = 0.125 10 | a[0, 0, :, :] = 0.0625 11 | a[0, 2, :, :] = 0.0625 12 | a[2, 0, :, :] = 0.0625 13 | a[2, 2, :, :] = 0.0625 14 | 15 | BLUR_FILTER_RGB = tf.constant(a, dtype=tf.float32) 16 | 17 | a = np.zeros([3, 3, 1, 1]) 18 | # a[1, 1, :, :] = 0.25 19 | # a[0, 1, :, :] = 0.125 20 | # a[1, 0, :, :] = 0.125 21 | # a[2, 1, :, :] = 0.125 22 | # a[1, 2, :, :] = 0.125 23 | # a[0, 0, :, :] = 0.0625 24 | # a[0, 2, :, :] = 0.0625 25 | # a[2, 0, :, :] = 0.0625 26 | # a[2, 2, :, :] = 0.0625 27 | a[1, 1, :, :] = 1.0 28 | a[0, 1, :, :] = 1.0 29 | a[1, 0, :, :] = 1.0 30 | a[2, 1, :, :] = 1.0 31 | a[1, 2, :, :] = 1.0 32 | a[0, 0, :, :] = 1.0 33 | a[0, 2, :, :] = 1.0 34 | a[2, 0, :, :] = 1.0 35 | a[2, 2, :, :] = 1.0 36 | BLUR_FILTER = tf.constant(a, dtype=tf.float32) 37 | 38 | a = np.zeros([3, 3, 3, 3]) 39 | a[1, 1, :, :] = 5 40 | a[0, 1, :, :] = -1 41 | a[1, 0, :, :] = -1 42 | a[2, 1, :, :] = -1 43 | a[1, 2, :, :] = -1 44 | 45 | SHARPEN_FILTER_RGB = tf.constant(a, dtype=tf.float32) 46 | 47 | a = np.zeros([3, 3, 1, 1]) 48 | a[1, 1, :, :] = 5 49 | a[0, 1, :, :] = -1 50 | a[1, 0, :, :] = -1 51 | a[2, 1, :, :] = -1 52 | a[1, 2, :, :] = -1 53 | 54 | SHARPEN_FILTER = tf.constant(a, dtype=tf.float32) 55 | 56 | # a = np.zeros([3, 3, 3, 3]) 57 | # a[:, :, :, :] = -1 58 | # a[1, 1, :, :] = 8 59 | 60 | # EDGE_FILTER_RGB = tf.constant(a, dtype=tf.float32) 61 | 62 | EDGE_FILTER_RGB = tf.constant([ 63 | [[[ -1., 0., 0.], [ 0., -1., 0.], [ 0., 0., -1.]], 64 | [[ -1., 0., 0.], [ 0., -1., 0.], [ 0., 0., -1.]], 65 | [[ -1., 0., 0.], [ 0., -1., 0.], [ 0., 0., -1.]]], 66 | [[[ -1., 0., 0.], [ 0., -1., 0.], [ 0., 0., -1.]], 67 | [[ 8., 0., 0.], [ 0., 8., 0.], [ 0., 0., 8.]], 68 | [[ -1., 0., 0.], [ 0., -1., 0.], [ 0., 0., -1.]]], 69 | [[[ -1., 0., 0.], [ 0., -1., 0.], [ 0., 0., -1.]], 70 | [[ -1., 0., 0.], [ 0., -1., 0.], [ 0., 0., -1.]], 71 | [[ -1., 0., 0.], [ 0., -1., 0.], [ 0., 0., -1.]]] 72 | ]) 73 | 74 | a = np.zeros([3, 3, 1, 1]) 75 | # a[:, :, :, :] = -1 76 | # a[1, 1, :, :] = 8 77 | a[0, 1, :, :] = -1 78 | a[1, 0, :, :] = -1 79 | a[1, 2, :, :] = -1 80 | a[2, 1, :, :] = -1 81 | a[1, 1, :, :] = 4 82 | 83 | EDGE_FILTER = tf.constant(a, dtype=tf.float32) 84 | 85 | a = np.zeros([3, 3, 3, 3]) 86 | a[0, :, :, :] = 1 87 | a[0, 1, :, :] = 2 # originally 2 88 | a[2, :, :, :] = -1 89 | a[2, 1, :, :] = -2 90 | 91 | TOP_SOBEL_RGB = tf.constant(a, dtype=tf.float32) 92 | 93 | a = np.zeros([3, 3, 1, 1]) 94 | a[0, :, :, :] = 1 95 | a[0, 1, :, :] = 2 # originally 2 96 | a[2, :, :, :] = -1 97 | a[2, 1, :, :] = -2 98 | 99 | TOP_SOBEL = tf.constant(a, dtype=tf.float32) 100 | 101 | a = np.zeros([3, 3, 3, 3]) 102 | a[0, 0, :, :] = -2 103 | a[0, 1, :, :] = -1 104 | a[1, 0, :, :] = -1 105 | a[1, 1, :, :] = 1 106 | a[1, 2, :, :] = 1 107 | a[2, 1, :, :] = 1 108 | a[2, 2, :, :] = 2 109 | 110 | EMBOSS_FILTER_RGB = tf.constant(a, dtype=tf.float32) 111 | 112 | a = np.zeros([3, 3, 1, 1]) 113 | a[0, 0, :, :] = -2 114 | a[0, 1, :, :] = -1 115 | a[1, 0, :, :] = -1 116 | a[1, 1, :, :] = 1 117 | a[1, 2, :, :] = 1 118 | a[2, 1, :, :] = 1 119 | a[2, 2, :, :] = 2 120 | EMBOSS_FILTER = tf.constant(a, dtype=tf.float32) -------------------------------------------------------------------------------- /2017/examples/process_data.py: -------------------------------------------------------------------------------- 1 | from __future__ import absolute_import 2 | from __future__ import division 3 | from __future__ import print_function 4 | 5 | from collections import Counter 6 | import random 7 | import os 8 | import sys 9 | sys.path.append('..') 10 | import zipfile 11 | 12 | import numpy as np 13 | from six.moves import urllib 14 | import tensorflow as tf 15 | 16 | import utils 17 | 18 | # Parameters for downloading data 19 | DOWNLOAD_URL = 'http://mattmahoney.net/dc/' 20 | EXPECTED_BYTES = 31344016 21 | DATA_FOLDER = 'data/' 22 | FILE_NAME = 'text8.zip' 23 | 24 | def download(file_name, expected_bytes): 25 | """ Download the dataset text8 if it's not already downloaded """ 26 | file_path = DATA_FOLDER + file_name 27 | if os.path.exists(file_path): 28 | print("Dataset ready") 29 | return file_path 30 | file_name, _ = urllib.request.urlretrieve(DOWNLOAD_URL + file_name, file_path) 31 | file_stat = os.stat(file_path) 32 | if file_stat.st_size == expected_bytes: 33 | print('Successfully downloaded the file', file_name) 34 | else: 35 | raise Exception('File ' + file_name + 36 | ' might be corrupted. You should try downloading it with a browser.') 37 | return file_path 38 | 39 | def read_data(file_path): 40 | """ Read data into a list of tokens 41 | There should be 17,005,207 tokens 42 | """ 43 | with zipfile.ZipFile(file_path) as f: 44 | words = tf.compat.as_str(f.read(f.namelist()[0])).split() 45 | # tf.compat.as_str() converts the input into the string 46 | return words 47 | 48 | def build_vocab(words, vocab_size): 49 | """ Build vocabulary of VOCAB_SIZE most frequent words """ 50 | dictionary = dict() 51 | count = [('UNK', -1)] 52 | count.extend(Counter(words).most_common(vocab_size - 1)) 53 | index = 0 54 | utils.make_dir('processed') 55 | with open('processed/vocab_1000.tsv', "w") as f: 56 | for word, _ in count: 57 | dictionary[word] = index 58 | if index < 1000: 59 | f.write(word + "\n") 60 | index += 1 61 | index_dictionary = dict(zip(dictionary.values(), dictionary.keys())) 62 | return dictionary, index_dictionary 63 | 64 | def convert_words_to_index(words, dictionary): 65 | """ Replace each word in the dataset with its index in the dictionary """ 66 | return [dictionary[word] if word in dictionary else 0 for word in words] 67 | 68 | def generate_sample(index_words, context_window_size): 69 | """ Form training pairs according to the skip-gram model. """ 70 | for index, center in enumerate(index_words): 71 | context = random.randint(1, context_window_size) 72 | # get a random target before the center word 73 | for target in index_words[max(0, index - context): index]: 74 | yield center, target 75 | # get a random target after the center wrod 76 | for target in index_words[index + 1: index + context + 1]: 77 | yield center, target 78 | 79 | def get_batch(iterator, batch_size): 80 | """ Group a numerical stream into batches and yield them as Numpy arrays. """ 81 | while True: 82 | center_batch = np.zeros(batch_size, dtype=np.int32) 83 | target_batch = np.zeros([batch_size, 1]) 84 | for index in range(batch_size): 85 | center_batch[index], target_batch[index] = next(iterator) 86 | yield center_batch, target_batch 87 | 88 | def process_data(vocab_size, batch_size, skip_window): 89 | file_path = download(FILE_NAME, EXPECTED_BYTES) 90 | words = read_data(file_path) 91 | dictionary, _ = build_vocab(words, vocab_size) 92 | index_words = convert_words_to_index(words, dictionary) 93 | del words # to save memory 94 | single_gen = generate_sample(index_words, skip_window) 95 | return get_batch(single_gen, batch_size) 96 | 97 | def get_index_vocab(vocab_size): 98 | file_path = download(FILE_NAME, EXPECTED_BYTES) 99 | words = read_data(file_path) 100 | return build_vocab(words, vocab_size) 101 | -------------------------------------------------------------------------------- /2017/examples/utils.py: -------------------------------------------------------------------------------- 1 | import os 2 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 3 | import tensorflow as tf 4 | 5 | def huber_loss(labels, predictions, delta=1.0): 6 | residual = tf.abs(predictions - labels) 7 | def f1(): return 0.5 * tf.square(residual) 8 | def f2(): return delta * residual - 0.5 * tf.square(delta) 9 | return tf.cond(residual < delta, f1, f2) 10 | 11 | def make_dir(path): 12 | """ Create a directory if there isn't one already. """ 13 | try: 14 | os.mkdir(path) 15 | except OSError: 16 | pass -------------------------------------------------------------------------------- /2017/setup/requirements.txt: -------------------------------------------------------------------------------- 1 | tensorflow==1.2.1 2 | scipy==0.19.1 3 | scikit-learn==0.18.2 4 | matplotlib==2.0.2 5 | xlrd==1.0.0 6 | ipdb==0.10.1 7 | Pillow==4.2.1 8 | lxml==3.8.0 9 | -------------------------------------------------------------------------------- /2017/setup/setup_instruction.md: -------------------------------------------------------------------------------- 1 | Tensorflow supports both Python 2.7 and Python 3.3+. Note that for Windows, TensorFlow supports only 64-bit Python 3.5. 2 | For this course, I will use Python 2.7. But you’re welcome to use either Python 2 or Python 3 for the assignments. The starter code, though, will be in Python 2.7 3 | 4 | Google has a pretty detailed instruction on how to download and setup Tensorflow. You can follow it here: https://www.tensorflow.org/get_started/os_setup 5 | 6 | Unless your computer has GPU, you should install Tensorflow without GPU support. My recommendation is always set up Tensorflow using virtualenv. For the list of dependencies, please consult the file requirements.txt. This list will be updated as the course progresses. 7 | 8 | Below is a simpler instruction on how to install tensorflow for people using Mac OS. If you have any problem installing Tensorflow, feel free to post it on Piazza: piazza.com/stanford/winter2017/cs20si 9 | 10 | ## Install TensorFlow
11 | ### For Mac OS 12 | 13 | If you get “permission denied” error in any command, use “sudo” in front of that command. 14 | 15 | You will need pip (or pip3 if you use Python 3), and virtualenv. 16 | 17 | Step 1: set up pip and virtual environment 18 | ```bash 19 | $ sudo easy_install pip 20 | $ sudo easy_install --upgrade six 21 | $ pip install virtualenv 22 | ``` 23 | 24 | Step 2: set up a project directory. You will do all work for this class in this directory 25 | ```bash 26 | $ mkdir [my project] 27 | ``` 28 | 29 | Step 3: set up virtual environment for the project directory. 30 | ```bash 31 | $ cd [my project] 32 | $ virtualenv venv --distribute 33 | ``` 34 | These commands create a venv subdirectory in your project where everything is installed. 35 | 36 | Step 4: to activate the virtual environment 37 | ```bash 38 | $ source venv/bin/activate 39 | ``` 40 | 41 | If you type: 42 | ```bash 43 | $ pip freeze 44 | ``` 45 | 46 | You will see that nothing is shown, which means no package is installed in your virtual environment. So you have to install all packages that you need. For the list of packages you need for this class, refer to requirements.txt 47 | Step 5: Install Tensorflow and other dependencies 48 | ```bash 49 | $ pip install tensorflow 50 | $ pip freeze > requirements.txt 51 | ``` 52 | 53 | Step n: 54 | To exit the virtual environment, use: 55 | ```bash 56 | $ deactivate 57 | ``` 58 | 59 | If you want your virtual environment to inherit globally installed packages, (not recommended), use: 60 | ```bash 61 | $ virtualenv venv --distribute --system-site-packages 62 | ``` 63 | ### For Ubuntu 64 | 65 | 66 | ### For Windows 67 | 68 | 69 | ### On the cloud 70 | If you don't want to install TensorFlow, you can use TensorFlow over the web. 71 | 72 | #### SageMath 73 | You can use Tensorflow over the web at https://cloud.sagemath.com/ 74 | Simply click on the link, create an account (or log in with your GitHub), and create a TensorFlow project. 75 | 76 | #### Jupyter 77 | You can also use Jupyter notebook to write TensorFlow programs. 78 | 79 | # Possible set up problems 80 | ## Matplotlib 81 | If you have problem with using Matplotlib in virtual environment, here is a simple fix.
82 | If you installed matplotlib using pip, there is a directory in you root called ~/.matplotlib. 83 | Go there and create a file ~/.matplotlib/matplotlibrc there and add the following code: ```backend: TkAgg``` 84 | 85 | Or you can simply add this after importing matplotlib: ```matplotlib.use("TkAgg")``` 86 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2017 Huyen Nguyen (Chip Huyen) 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE) 2 | [![Join the https://gitter.im/stanford-tensorflow-tutorials](https://badges.gitter.im/tflearn/tflearn.svg)](https://gitter.im/stanford-tensorflow-tutorials) 3 | 4 | # stanford-tensorflow-tutorials 5 | This repository contains code examples for the course CS 20: TensorFlow for Deep Learning Research.
6 | It will be updated as the class progresses.
7 | Detailed syllabus and lecture notes can be found [here](http://cs20.stanford.edu).
8 | For this course, I use python3.6 and TensorFlow 1.4.1. 9 | 10 | For the code and notes of the previous year's course, please see the folder 2017 and the website https://web.stanford.edu/class/cs20si/2017 11 | 12 | For setup instruction and the list of dependencies, please see the setup folder of this repository. -------------------------------------------------------------------------------- /assignments/01/q1.py: -------------------------------------------------------------------------------- 1 | """ 2 | Simple exercises to get used to TensorFlow API 3 | You should thoroughly test your code. 4 | TensorFlow's official documentation should be your best friend here 5 | CS20: "TensorFlow for Deep Learning Research" 6 | cs20.stanford.edu 7 | Created by Chip Huyen (chiphuyen@cs.stanford.edu) 8 | """ 9 | import os 10 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 11 | 12 | import tensorflow as tf 13 | 14 | sess = tf.InteractiveSession() 15 | ############################################################################### 16 | # 1a: Create two random 0-d tensors x and y of any distribution. 17 | # Create a TensorFlow object that returns x + y if x > y, and x - y otherwise. 18 | # Hint: look up tf.cond() 19 | # I do the first problem for you 20 | ############################################################################### 21 | 22 | x = tf.random_uniform([]) # Empty array as shape creates a scalar. 23 | y = tf.random_uniform([]) 24 | out = tf.cond(tf.greater(x, y), lambda: x + y, lambda: x - y) 25 | print(sess.run(out)) 26 | 27 | ############################################################################### 28 | # 1b: Create two 0-d tensors x and y randomly selected from the range [-1, 1). 29 | # Return x + y if x < y, x - y if x > y, 0 otherwise. 30 | # Hint: Look up tf.case(). 31 | ############################################################################### 32 | 33 | # YOUR CODE 34 | 35 | ############################################################################### 36 | # 1c: Create the tensor x of the value [[0, -2, -1], [0, 1, 2]] 37 | # and y as a tensor of zeros with the same shape as x. 38 | # Return a boolean tensor that yields Trues if x equals y element-wise. 39 | # Hint: Look up tf.equal(). 40 | ############################################################################### 41 | 42 | # YOUR CODE 43 | 44 | ############################################################################### 45 | # 1d: Create the tensor x of value 46 | # [29.05088806, 27.61298943, 31.19073486, 29.35532951, 47 | # 30.97266006, 26.67541885, 38.08450317, 20.74983215, 48 | # 34.94445419, 34.45999146, 29.06485367, 36.01657104, 49 | # 27.88236427, 20.56035233, 30.20379066, 29.51215172, 50 | # 33.71149445, 28.59134293, 36.05556488, 28.66994858]. 51 | # Get the indices of elements in x whose values are greater than 30. 52 | # Hint: Use tf.where(). 53 | # Then extract elements whose values are greater than 30. 54 | # Hint: Use tf.gather(). 55 | ############################################################################### 56 | 57 | # YOUR CODE 58 | 59 | ############################################################################### 60 | # 1e: Create a diagnoal 2-d tensor of size 6 x 6 with the diagonal values of 1, 61 | # 2, ..., 6 62 | # Hint: Use tf.range() and tf.diag(). 63 | ############################################################################### 64 | 65 | # YOUR CODE 66 | 67 | ############################################################################### 68 | # 1f: Create a random 2-d tensor of size 10 x 10 from any distribution. 69 | # Calculate its determinant. 70 | # Hint: Look at tf.matrix_determinant(). 71 | ############################################################################### 72 | 73 | # YOUR CODE 74 | 75 | ############################################################################### 76 | # 1g: Create tensor x with value [5, 2, 3, 5, 10, 6, 2, 3, 4, 2, 1, 1, 0, 9]. 77 | # Return the unique elements in x 78 | # Hint: use tf.unique(). Keep in mind that tf.unique() returns a tuple. 79 | ############################################################################### 80 | 81 | # YOUR CODE 82 | 83 | ############################################################################### 84 | # 1h: Create two tensors x and y of shape 300 from any normal distribution, 85 | # as long as they are from the same distribution. 86 | # Use tf.cond() to return: 87 | # - The mean squared error of (x - y) if the average of all elements in (x - y) 88 | # is negative, or 89 | # - The sum of absolute value of all elements in the tensor (x - y) otherwise. 90 | # Hint: see the Huber loss function in the lecture slides 3. 91 | ############################################################################### 92 | 93 | # YOUR CODE -------------------------------------------------------------------------------- /assignments/01/q1_sol.py: -------------------------------------------------------------------------------- 1 | """ 2 | Solution to simple exercises to get used to TensorFlow API 3 | You should thoroughly test your code. 4 | TensorFlow's official documentation should be your best friend here 5 | CS20: "TensorFlow for Deep Learning Research" 6 | cs20.stanford.edu 7 | Created by Chip Huyen (chiphuyen@cs.stanford.edu) 8 | """ 9 | import os 10 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 11 | 12 | import tensorflow as tf 13 | 14 | sess = tf.InteractiveSession() 15 | ############################################################################### 16 | # 1a: Create two random 0-d tensors x and y of any distribution. 17 | # Create a TensorFlow object that returns x + y if x > y, and x - y otherwise. 18 | # Hint: look up tf.cond() 19 | # I do the first problem for you 20 | ############################################################################### 21 | 22 | x = tf.random_uniform([]) # Empty array as shape creates a scalar. 23 | y = tf.random_uniform([]) 24 | out = tf.cond(tf.greater(x, y), lambda: tf.add(x, y), lambda: tf.subtract(x, y)) 25 | 26 | ############################################################################### 27 | # 1b: Create two 0-d tensors x and y randomly selected from the range [-1, 1). 28 | # Return x + y if x < y, x - y if x > y, 0 otherwise. 29 | # Hint: Look up tf.case(). 30 | ############################################################################### 31 | 32 | x = tf.random_uniform([], -1, 1, dtype=tf.float32) 33 | y = tf.random_uniform([], -1, 1, dtype=tf.float32) 34 | out = tf.case({tf.less(x, y): lambda: tf.add(x, y), 35 | tf.greater(x, y): lambda: tf.subtract(x, y)}, 36 | default=lambda: tf.constant(0.0), exclusive=True) 37 | 38 | 39 | ############################################################################### 40 | # 1c: Create the tensor x of the value [[0, -2, -1], [0, 1, 2]] 41 | # and y as a tensor of zeros with the same shape as x. 42 | # Return a boolean tensor that yields Trues if x equals y element-wise. 43 | # Hint: Look up tf.equal(). 44 | ############################################################################### 45 | 46 | x = tf.constant([[0, -2, -1], [0, 1, 2]]) 47 | y = tf.zeros_like(x) 48 | out = tf.equal(x, y) 49 | 50 | ############################################################################### 51 | # 1d: Create the tensor x of value 52 | # [29.05088806, 27.61298943, 31.19073486, 29.35532951, 53 | # 30.97266006, 26.67541885, 38.08450317, 20.74983215, 54 | # 34.94445419, 34.45999146, 29.06485367, 36.01657104, 55 | # 27.88236427, 20.56035233, 30.20379066, 29.51215172, 56 | # 33.71149445, 28.59134293, 36.05556488, 28.66994858]. 57 | # Get the indices of elements in x whose values are greater than 30. 58 | # Hint: Use tf.where(). 59 | # Then extract elements whose values are greater than 30. 60 | # Hint: Use tf.gather(). 61 | ############################################################################### 62 | 63 | x = tf.constant([29.05088806, 27.61298943, 31.19073486, 29.35532951, 64 | 30.97266006, 26.67541885, 38.08450317, 20.74983215, 65 | 34.94445419, 34.45999146, 29.06485367, 36.01657104, 66 | 27.88236427, 20.56035233, 30.20379066, 29.51215172, 67 | 33.71149445, 28.59134293, 36.05556488, 28.66994858]) 68 | indices = tf.where(x > 30) 69 | out = tf.gather(x, indices) 70 | 71 | ############################################################################### 72 | # 1e: Create a diagnoal 2-d tensor of size 6 x 6 with the diagonal values of 1, 73 | # 2, ..., 6 74 | # Hint: Use tf.range() and tf.diag(). 75 | ############################################################################### 76 | 77 | values = tf.range(1, 7) 78 | out = tf.diag(values) 79 | 80 | ############################################################################### 81 | # 1f: Create a random 2-d tensor of size 10 x 10 from any distribution. 82 | # Calculate its determinant. 83 | # Hint: Look at tf.matrix_determinant(). 84 | ############################################################################### 85 | 86 | m = tf.random_normal([10, 10], mean=10, stddev=1) 87 | out = tf.matrix_determinant(m) 88 | 89 | ############################################################################### 90 | # 1g: Create tensor x with value [5, 2, 3, 5, 10, 6, 2, 3, 4, 2, 1, 1, 0, 9]. 91 | # Return the unique elements in x 92 | # Hint: use tf.unique(). Keep in mind that tf.unique() returns a tuple. 93 | ############################################################################### 94 | 95 | x = tf.constant([5, 2, 3, 5, 10, 6, 2, 3, 4, 2, 1, 1, 0, 9]) 96 | unique_values, indices = tf.unique(x) 97 | 98 | ############################################################################### 99 | # 1h: Create two tensors x and y of shape 300 from any normal distribution, 100 | # as long as they are from the same distribution. 101 | # Use tf.cond() to return: 102 | # - The mean squared error of (x - y) if the average of all elements in (x - y) 103 | # is negative, or 104 | # - The sum of absolute value of all elements in the tensor (x - y) otherwise. 105 | # Hint: see the Huber loss function in the lecture slides 3. 106 | ############################################################################### 107 | 108 | x = tf.random_normal([300], mean=5, stddev=1) 109 | y = tf.random_normal([300], mean=5, stddev=1) 110 | average = tf.reduce_mean(x - y) 111 | def f1(): return tf.reduce_mean(tf.square(x - y)) 112 | def f2(): return tf.reduce_sum(tf.abs(x - y)) 113 | out = tf.cond(average < 0, f1, f2) -------------------------------------------------------------------------------- /assignments/02_style_transfer/load_vgg.py: -------------------------------------------------------------------------------- 1 | """ Load VGGNet weights needed for the implementation in TensorFlow 2 | of the paper A Neural Algorithm of Artistic Style (Gatys et al., 2016) 3 | 4 | Created by Chip Huyen (chiphuyen@cs.stanford.edu) 5 | CS20: "TensorFlow for Deep Learning Research" 6 | cs20.stanford.edu 7 | 8 | For more details, please read the assignment handout: 9 | https://docs.google.com/document/d/1FpueD-3mScnD0SJQDtwmOb1FrSwo1NGowkXzMwPoLH4/edit?usp=sharing 10 | 11 | """ 12 | import numpy as np 13 | import scipy.io 14 | import tensorflow as tf 15 | 16 | import utils 17 | 18 | # VGG-19 parameters file 19 | VGG_DOWNLOAD_LINK = 'http://www.vlfeat.org/matconvnet/models/imagenet-vgg-verydeep-19.mat' 20 | VGG_FILENAME = 'imagenet-vgg-verydeep-19.mat' 21 | EXPECTED_BYTES = 534904783 22 | 23 | class VGG(object): 24 | def __init__(self, input_img): 25 | utils.download(VGG_DOWNLOAD_LINK, VGG_FILENAME, EXPECTED_BYTES) 26 | self.vgg_layers = scipy.io.loadmat(VGG_FILENAME)['layers'] 27 | self.input_img = input_img 28 | self.mean_pixels = np.array([123.68, 116.779, 103.939]).reshape((1,1,1,3)) 29 | 30 | def _weights(self, layer_idx, expected_layer_name): 31 | """ Return the weights and biases at layer_idx already trained by VGG 32 | """ 33 | W = self.vgg_layers[0][layer_idx][0][0][2][0][0] 34 | b = self.vgg_layers[0][layer_idx][0][0][2][0][1] 35 | layer_name = self.vgg_layers[0][layer_idx][0][0][0][0] 36 | assert layer_name == expected_layer_name 37 | return W, b.reshape(b.size) 38 | 39 | def conv2d_relu(self, prev_layer, layer_idx, layer_name): 40 | """ Create a convolution layer with RELU using the weights and 41 | biases extracted from the VGG model at 'layer_idx'. You should use 42 | the function _weights() defined above to extract weights and biases. 43 | 44 | _weights() returns numpy arrays, so you have to convert them to TF tensors. 45 | 46 | Don't forget to apply relu to the output from the convolution. 47 | Inputs: 48 | prev_layer: the output tensor from the previous layer 49 | layer_idx: the index to current layer in vgg_layers 50 | layer_name: the string that is the name of the current layer. 51 | It's used to specify variable_scope. 52 | Hint for choosing strides size: 53 | for small images, you probably don't want to skip any pixel 54 | """ 55 | ############################### 56 | ## TO DO 57 | out = None 58 | ############################### 59 | setattr(self, layer_name, out) 60 | 61 | def avgpool(self, prev_layer, layer_name): 62 | """ Create the average pooling layer. The paper suggests that 63 | average pooling works better than max pooling. 64 | 65 | Input: 66 | prev_layer: the output tensor from the previous layer 67 | layer_name: the string that you want to name the layer. 68 | It's used to specify variable_scope. 69 | 70 | Hint for choosing strides and kszie: choose what you feel appropriate 71 | """ 72 | ############################### 73 | ## TO DO 74 | out = None 75 | ############################### 76 | setattr(self, layer_name, out) 77 | 78 | def load(self): 79 | self.conv2d_relu(self.input_img, 0, 'conv1_1') 80 | self.conv2d_relu(self.conv1_1, 2, 'conv1_2') 81 | self.avgpool(self.conv1_2, 'avgpool1') 82 | self.conv2d_relu(self.avgpool1, 5, 'conv2_1') 83 | self.conv2d_relu(self.conv2_1, 7, 'conv2_2') 84 | self.avgpool(self.conv2_2, 'avgpool2') 85 | self.conv2d_relu(self.avgpool2, 10, 'conv3_1') 86 | self.conv2d_relu(self.conv3_1, 12, 'conv3_2') 87 | self.conv2d_relu(self.conv3_2, 14, 'conv3_3') 88 | self.conv2d_relu(self.conv3_3, 16, 'conv3_4') 89 | self.avgpool(self.conv3_4, 'avgpool3') 90 | self.conv2d_relu(self.avgpool3, 19, 'conv4_1') 91 | self.conv2d_relu(self.conv4_1, 21, 'conv4_2') 92 | self.conv2d_relu(self.conv4_2, 23, 'conv4_3') 93 | self.conv2d_relu(self.conv4_3, 25, 'conv4_4') 94 | self.avgpool(self.conv4_4, 'avgpool4') 95 | self.conv2d_relu(self.avgpool4, 28, 'conv5_1') 96 | self.conv2d_relu(self.conv5_1, 30, 'conv5_2') 97 | self.conv2d_relu(self.conv5_2, 32, 'conv5_3') 98 | self.conv2d_relu(self.conv5_3, 34, 'conv5_4') 99 | self.avgpool(self.conv5_4, 'avgpool5') -------------------------------------------------------------------------------- /assignments/02_style_transfer/load_vgg_sol.py: -------------------------------------------------------------------------------- 1 | """ Load VGGNet weights needed for the implementation in TensorFlow 2 | of the paper A Neural Algorithm of Artistic Style (Gatys et al., 2016) 3 | 4 | Created by Chip Huyen (chiphuyen@cs.stanford.edu) 5 | CS20: "TensorFlow for Deep Learning Research" 6 | cs20.stanford.edu 7 | 8 | For more details, please read the assignment handout: 9 | 10 | """ 11 | import numpy as np 12 | import scipy.io 13 | import tensorflow as tf 14 | 15 | import utils 16 | 17 | # VGG-19 parameters file 18 | VGG_DOWNLOAD_LINK = 'http://www.vlfeat.org/matconvnet/models/imagenet-vgg-verydeep-19.mat' 19 | VGG_FILENAME = 'imagenet-vgg-verydeep-19.mat' 20 | EXPECTED_BYTES = 534904783 21 | 22 | class VGG(object): 23 | def __init__(self, input_img): 24 | utils.download(VGG_DOWNLOAD_LINK, VGG_FILENAME, EXPECTED_BYTES) 25 | self.vgg_layers = scipy.io.loadmat(VGG_FILENAME)['layers'] 26 | self.input_img = input_img 27 | self.mean_pixels = np.array([123.68, 116.779, 103.939]).reshape((1,1,1,3)) 28 | 29 | def _weights(self, layer_idx, expected_layer_name): 30 | """ Return the weights and biases at layer_idx already trained by VGG 31 | """ 32 | W = self.vgg_layers[0][layer_idx][0][0][2][0][0] 33 | b = self.vgg_layers[0][layer_idx][0][0][2][0][1] 34 | layer_name = self.vgg_layers[0][layer_idx][0][0][0][0] 35 | assert layer_name == expected_layer_name 36 | return W, b.reshape(b.size) 37 | 38 | def conv2d_relu(self, prev_layer, layer_idx, layer_name): 39 | """ Return the Conv2D layer with RELU using the weights, 40 | biases from the VGG model at 'layer_idx'. 41 | Don't forget to apply relu to the output from the convolution. 42 | Inputs: 43 | prev_layer: the output tensor from the previous layer 44 | layer_idx: the index to current layer in vgg_layers 45 | layer_name: the string that is the name of the current layer. 46 | It's used to specify variable_scope. 47 | 48 | 49 | Note that you first need to obtain W and b from from the corresponding VGG's layer 50 | using the function _weights() defined above. 51 | W and b returned from _weights() are numpy arrays, so you have 52 | to convert them to TF tensors. One way to do it is with tf.constant. 53 | 54 | Hint for choosing strides size: 55 | for small images, you probably don't want to skip any pixel 56 | """ 57 | ############################### 58 | ## TO DO 59 | with tf.variable_scope(layer_name) as scope: 60 | W, b = self._weights(layer_idx, layer_name) 61 | W = tf.constant(W, name='weights') 62 | b = tf.constant(b, name='bias') 63 | conv2d = tf.nn.conv2d(prev_layer, 64 | filter=W, 65 | strides=[1, 1, 1, 1], 66 | padding='SAME') 67 | out = tf.nn.relu(conv2d + b) 68 | ############################### 69 | setattr(self, layer_name, out) 70 | 71 | def avgpool(self, prev_layer, layer_name): 72 | """ Return the average pooling layer. The paper suggests that 73 | average pooling works better than max pooling. 74 | Input: 75 | prev_layer: the output tensor from the previous layer 76 | layer_name: the string that you want to name the layer. 77 | It's used to specify variable_scope. 78 | 79 | Hint for choosing strides and kszie: choose what you feel appropriate 80 | """ 81 | ############################### 82 | ## TO DO 83 | with tf.variable_scope(layer_name): 84 | out = tf.nn.avg_pool(prev_layer, 85 | ksize=[1, 2, 2, 1], 86 | strides=[1, 2, 2, 1], 87 | padding='SAME') 88 | ############################### 89 | setattr(self, layer_name, out) 90 | 91 | def load(self): 92 | self.conv2d_relu(self.input_img, 0, 'conv1_1') 93 | self.conv2d_relu(self.conv1_1, 2, 'conv1_2') 94 | self.avgpool(self.conv1_2, 'avgpool1') 95 | self.conv2d_relu(self.avgpool1, 5, 'conv2_1') 96 | self.conv2d_relu(self.conv2_1, 7, 'conv2_2') 97 | self.avgpool(self.conv2_2, 'avgpool2') 98 | self.conv2d_relu(self.avgpool2, 10, 'conv3_1') 99 | self.conv2d_relu(self.conv3_1, 12, 'conv3_2') 100 | self.conv2d_relu(self.conv3_2, 14, 'conv3_3') 101 | self.conv2d_relu(self.conv3_3, 16, 'conv3_4') 102 | self.avgpool(self.conv3_4, 'avgpool3') 103 | self.conv2d_relu(self.avgpool3, 19, 'conv4_1') 104 | self.conv2d_relu(self.conv4_1, 21, 'conv4_2') 105 | self.conv2d_relu(self.conv4_2, 23, 'conv4_3') 106 | self.conv2d_relu(self.conv4_3, 25, 'conv4_4') 107 | self.avgpool(self.conv4_4, 'avgpool4') 108 | self.conv2d_relu(self.avgpool4, 28, 'conv5_1') 109 | self.conv2d_relu(self.conv5_1, 30, 'conv5_2') 110 | self.conv2d_relu(self.conv5_2, 32, 'conv5_3') 111 | self.conv2d_relu(self.conv5_3, 34, 'conv5_4') 112 | self.avgpool(self.conv5_4, 'avgpool5') -------------------------------------------------------------------------------- /assignments/02_style_transfer/utils.py: -------------------------------------------------------------------------------- 1 | """ Utils needed for the implementation in TensorFlow 2 | of the paper A Neural Algorithm of Artistic Style (Gatys et al., 2016) 3 | 4 | Created by Chip Huyen (chiphuyen@cs.stanford.edu) 5 | CS20: "TensorFlow for Deep Learning Research" 6 | cs20.stanford.edu 7 | 8 | For more details, please read the assignment handout: 9 | https://docs.google.com/document/d/1FpueD-3mScnD0SJQDtwmOb1FrSwo1NGowkXzMwPoLH4/edit?usp=sharing 10 | 11 | """ 12 | 13 | import os 14 | 15 | from PIL import Image, ImageOps 16 | import numpy as np 17 | import scipy.misc 18 | from six.moves import urllib 19 | 20 | def download(download_link, file_name, expected_bytes): 21 | """ Download the pretrained VGG-19 model if it's not already downloaded """ 22 | if os.path.exists(file_name): 23 | print("VGG-19 pre-trained model is ready") 24 | return 25 | print("Downloading the VGG pre-trained model. This might take a while ...") 26 | file_name, _ = urllib.request.urlretrieve(download_link, file_name) 27 | file_stat = os.stat(file_name) 28 | if file_stat.st_size == expected_bytes: 29 | print('Successfully downloaded VGG-19 pre-trained model', file_name) 30 | else: 31 | raise Exception('File ' + file_name + 32 | ' might be corrupted. You should try downloading it with a browser.') 33 | 34 | def get_resized_image(img_path, width, height, save=True): 35 | image = Image.open(img_path) 36 | # PIL is column major so you have to swap the places of width and height 37 | image = ImageOps.fit(image, (width, height), Image.ANTIALIAS) 38 | if save: 39 | image_dirs = img_path.split('/') 40 | image_dirs[-1] = 'resized_' + image_dirs[-1] 41 | out_path = '/'.join(image_dirs) 42 | if not os.path.exists(out_path): 43 | image.save(out_path) 44 | image = np.asarray(image, np.float32) 45 | return np.expand_dims(image, 0) 46 | 47 | def generate_noise_image(content_image, width, height, noise_ratio=0.6): 48 | noise_image = np.random.uniform(-20, 20, (1, height, width, 3)).astype(np.float32) 49 | return noise_image * noise_ratio + content_image * (1 - noise_ratio) 50 | 51 | def save_image(path, image): 52 | image = image[0] 53 | image = np.clip(image, 0, 255).astype('uint8') 54 | scipy.misc.imsave(path, image) 55 | 56 | def safe_mkdir(path): 57 | """ Create a directory if there isn't one already. """ 58 | try: 59 | os.mkdir(path) 60 | except OSError: 61 | pass -------------------------------------------------------------------------------- /assignments/chatbot/README.md: -------------------------------------------------------------------------------- 1 | A neural chatbot using sequence to sequence model with 2 | attentional decoder. This is a fully functional chatbot. 3 | 4 | This is based on Google Translate Tensorflow model 5 | https://github.com/tensorflow/models/blob/master/tutorials/rnn/translate/ 6 | 7 | Sequence to sequence model by Cho et al.(2014) 8 | 9 | Created by Chip Huyen (chiphuyen@cs.stanford.edu) 10 | CS20: "TensorFlow for Deep Learning Research" 11 | cs20.stanford.edu 12 | 13 | The detailed assignment handout can be found at: 14 | https://docs.google.com/document/d/1GJfn2B6EI8JueDiBwzTAdD34d6pC99BSt6vldOmUCPQ/edit?usp=sharing 15 | 16 | 17 |

Sample conversation

18 | HUMAN ++++ hi
19 | BOT ++++ hi . what ' s your name ?
20 | HUMAN ++++ i'm chip
21 | BOT ++++ please .
22 | HUMAN ++++ please what?
23 | BOT ++++ i don ' t know what to say .
24 | HUMAN ++++ where are you from?
25 | BOT ++++ california .
26 | HUMAN ++++ do you like california?
27 | BOT ++++ i ' m okay .
28 | HUMAN ++++ i like california a lot
29 | BOT ++++ of what ?
30 | HUMAN ++++ i live here
31 | BOT ++++ and what is it ?
32 | HUMAN ++++ california
33 | BOT ++++ i don ' t know what to say .
34 | HUMAN ++++ i'm a student here
35 | BOT ++++ and get it out of here .
36 | HUMAN ++++ what do you like to do in your free time?
37 | BOT ++++ and what about the gun ?
38 | HUMAN ++++ do you have a gun?
39 | BOT ++++ yes .
40 | HUMAN ++++ you're dangerous!
41 | BOT ++++ i ' m not .
42 | HUMAN ++++ do you go hunting?
43 | BOT ++++ i can do .
44 | HUMAN ++++ what do you hunt?
45 | BOT ++++ virgin .
46 | HUMAN ++++ that's very bad
47 | BOT ++++ and you in love ?
48 | HUMAN ++++ no i'm not
49 | BOT ++++ please - -
50 | HUMAN ++++ okay, i am
51 | BOT ++++ and you are good .
52 | HUMAN ++++ yeah i am good too.
53 | BOT ++++ good good to you .
54 | HUMAN ++++ are you in love?
55 | BOT ++++ i ' m ready .
56 | HUMAN ++++ ready for what?
57 | BOT ++++ for what ?
58 | HUMAN ++++ for what?
59 | BOT ++++ i don ' t know what to say .
60 | HUMAN ++++ tell me
61 | BOT ++++ i can do it .
62 | HUMAN ++++ okay, tell me what you're ready for
63 | BOT ++++ i ' m doing you .
64 | HUMAN ++++ no you're not
65 | BOT ++++ please !
66 | HUMAN ++++ this is a strange conversation
67 | BOT ++++ i don ' t know what to say .
68 | HUMAN ++++ are you a guy?
69 | BOT ++++ of what ?
70 | HUMAN ++++ are you a boy?
71 | BOT ++++ i ' m fine .
72 | HUMAN ++++ are you a man?
73 | BOT ++++ i ' m a little .
74 | HUMAN ++++ what's your name?
75 | BOT ++++ chris -
76 | HUMAN ++++ chris who?
77 | BOT ++++ cornelius .
78 | HUMAN ++++ so i guess you're a guy.
79 | BOT ++++ i am .
80 | HUMAN ++++ okay
81 | BOT ++++ okay ?
82 | HUMAN ++++ do you have a girlfriend?
83 | BOT ++++ i ' m sure .
84 | HUMAN ++++ what's her name?
85 | BOT ++++ let ' s talk about something else .
86 | 87 | See output_convo.txt for more sample conversations. 88 | 89 |

Usage

90 | 91 | Step 1: create a data folder in your project directory, download 92 | the Cornell Movie-Dialogs Corpus from 93 | https://www.cs.cornell.edu/~cristian/Cornell_Movie-Dialogs_Corpus.html 94 | Unzip it 95 | 96 | Step 2: update config.py file
97 | Change DATA_PATH to where you store your data 98 | 99 | Step 3: python3 data.py
100 | This will do all the pre-processing for the Cornell dataset. 101 | 102 | Step 4: 103 | python3 chatbot.py --mode [train/chat]
104 | If mode is train, then you train the chatbot. By default, the model will 105 | restore the previously trained weights (if there is any) and continue 106 | training up on that. 107 | 108 | If you want to start training from scratch, please delete all the checkpoints 109 | in the checkpoints folder. 110 | 111 | If the mode is chat, you'll go into the interaction mode with the bot. 112 | 113 | By default, all the conversations you have with the chatbot will be written 114 | into the file output_convo.txt in the processed folder. If you run this chatbot, 115 | I kindly ask you to send me the output_convo.txt so that I can improve 116 | the chatbot. 117 | 118 | 119 | Thank you very much! 120 | -------------------------------------------------------------------------------- /assignments/chatbot/config.py: -------------------------------------------------------------------------------- 1 | """ A neural chatbot using sequence to sequence model with 2 | attentional decoder. 3 | 4 | This is based on Google Translate Tensorflow model 5 | https://github.com/tensorflow/models/blob/master/tutorials/rnn/translate/ 6 | 7 | Sequence to sequence model by Cho et al.(2014) 8 | 9 | Created by Chip Huyen (chiphuyen@cs.stanford.edu) 10 | CS20: "TensorFlow for Deep Learning Research" 11 | cs20.stanford.edu 12 | 13 | This file contains the hyperparameters for the model. 14 | 15 | See README.md for instruction on how to run the starter code. 16 | """ 17 | 18 | # parameters for processing the dataset 19 | DATA_PATH = 'data/cornell movie-dialogs corpus' 20 | CONVO_FILE = 'movie_conversations.txt' 21 | LINE_FILE = 'movie_lines.txt' 22 | OUTPUT_FILE = 'output_convo.txt' 23 | PROCESSED_PATH = 'processed' 24 | CPT_PATH = 'checkpoints' 25 | 26 | THRESHOLD = 2 27 | 28 | PAD_ID = 0 29 | UNK_ID = 1 30 | START_ID = 2 31 | EOS_ID = 3 32 | 33 | TESTSET_SIZE = 25000 34 | 35 | BUCKETS = [(19, 19), (28, 28), (33, 33), (40, 43), (50, 53), (60, 63)] 36 | 37 | 38 | CONTRACTIONS = [("i ' m ", "i 'm "), ("' d ", "'d "), ("' s ", "'s "), 39 | ("don ' t ", "do n't "), ("didn ' t ", "did n't "), ("doesn ' t ", "does n't "), 40 | ("can ' t ", "ca n't "), ("shouldn ' t ", "should n't "), ("wouldn ' t ", "would n't "), 41 | ("' ve ", "'ve "), ("' re ", "'re "), ("in ' ", "in' ")] 42 | 43 | NUM_LAYERS = 3 44 | HIDDEN_SIZE = 256 45 | BATCH_SIZE = 64 46 | 47 | LR = 0.5 48 | MAX_GRAD_NORM = 5.0 49 | 50 | NUM_SAMPLES = 512 51 | -------------------------------------------------------------------------------- /assignments/word_transform/eval.vocab: -------------------------------------------------------------------------------- 1 | tiene,has 2 | habían,had 3 | entendido,understood 4 | clase,class 5 | harry,harry 6 | pluma,pen 7 | guerra,war 8 | tan,so 9 | dios,god 10 | le,you 11 | estés,are 12 | marea,tide 13 | mr,mr 14 | el,he 15 | jerry,jerry 16 | puedo,can 17 | coño,cone 18 | marca,brand 19 | debió,must 20 | diferente,different 21 | tras,after 22 | rival,rival 23 | películas,films 24 | ésta,this 25 | piel,skin 26 | intención,intention 27 | show,show 28 | ir,go 29 | os,you 30 | aumentar,increase 31 | país,country 32 | marcador,marker 33 | perfecta,perfect 34 | ben,ben 35 | presión,pressure 36 | pasada,pass 37 | deje,leave 38 | dia,day 39 | dólares,dollars 40 | porque,why 41 | maldita,damn 42 | locura,madness 43 | fotos,photos 44 | hinchar,swell 45 | regresar,return 46 | alto,high 47 | chico,boy 48 | soberanía,sovereignty 49 | aquella,that 50 | hables,speak 51 | poder,power 52 | tomado,taken 53 | verde,green 54 | nube,cloud 55 | playa,beach 56 | mercado,market 57 | nadie,nobody 58 | contrario,contrary 59 | olvidar,forget 60 | jodido,fucking 61 | altavoz,speaker 62 | pobre,poor 63 | oigan,hear 64 | viuda,widow 65 | vivo,alive 66 | verle,see 67 | creí,believed 68 | malas,bad 69 | hubiera,would 70 | perra,dog 71 | muestra,sample 72 | bienvenidos,welcome 73 | calcetines,socks 74 | dónde,where 75 | teléfono,phone 76 | huele,smells 77 | clientes,customers 78 | sería,would 79 | biblioteca,library 80 | paciente,patient 81 | ruido,noise 82 | pasa,happens 83 | diplomático,diplomatic 84 | llamaba,called 85 | prosperar,prosper 86 | nosotros,us 87 | vas,go 88 | emergencia,emergency 89 | sucia,dirty 90 | desastre,disaster 91 | david,david 92 | pensar,think 93 | real,real 94 | humano,human 95 | vuelvas,return 96 | estaría,be 97 | comprar,buy 98 | red,net 99 | sea,be 100 | ray,ray 101 | presa,dam 102 | ganado,won 103 | sexo,sex 104 | oficina,office 105 | recibir,receive 106 | maravilloso,wonderful 107 | dura,hard 108 | estupendo,great 109 | depende,depends 110 | bastardo,bastard 111 | media,half 112 | pedazo,piece 113 | unas,nail 114 | ojalá,hopefully 115 | banda,band 116 | metros,meters 117 | siente,feels 118 | posibilidad,possibility 119 | inevitable,inevitable 120 | batalla,battle 121 | señorita,miss 122 | peor,worst 123 | naval,naval 124 | buenas,good 125 | completamente,completely 126 | sientes,feel 127 | paso,passed 128 | callejón,alley 129 | observación,observation 130 | perfecto,perfect 131 | flor,flower 132 | imposible,impossible 133 | hagan,make 134 | conversión,conversion 135 | trasero,rear 136 | diez,ten 137 | línea,line 138 | c,c 139 | buena,good 140 | adelante,ahead 141 | ee,ee 142 | otras,other 143 | voz,voice 144 | mofeta,skunk 145 | política,politics 146 | ah,ah 147 | nombres,name 148 | maestro,teacher 149 | ablandar,soften 150 | dará,give 151 | encantado,charmed 152 | cállate,quiet 153 | ocho,eight 154 | fuimos,went 155 | fiesta,party 156 | quedo,remain 157 | sentí,felt 158 | cansado,tired 159 | oro,gold 160 | abierta,open 161 | cámara,camera 162 | magnético,magnetic 163 | ratón,mouse 164 | seguro,insurance 165 | como,as 166 | imagino,imagine 167 | guantes,gloves 168 | espacio,space 169 | otros,others 170 | bailando,dancing 171 | herido,injured 172 | oportunidad,opportunity 173 | bobby,bobby 174 | robert,robert 175 | uso,use 176 | encontrado,found 177 | manos,hands 178 | ver,see 179 | afuera,outside 180 | habéis,have 181 | quienes,who 182 | iluminación,lighting 183 | fácil,easy 184 | menor,less 185 | dirección,address 186 | negocios,business 187 | privado,private 188 | lengua,language 189 | informática,computing 190 | mary,mary 191 | tratando,trying 192 | ejército,army 193 | perros,dogs 194 | cosecha,harvest 195 | siempre,always 196 | vienes,viennese 197 | cabra,goat 198 | gana,desire 199 | empieza,starts 200 | deben,should 201 | vengo,come 202 | tuvo,had 203 | dolor,pain 204 | tuve,had 205 | efecto,effect 206 | quedado,left 207 | llegue,arrived 208 | caluroso,hot 209 | organizado,organized 210 | quede,stay 211 | estarás,be 212 | eso,that 213 | hijos,children 214 | tuvimos,had 215 | vergüenza,shame 216 | alegra,happy 217 | gobierno,government 218 | caro,expensive 219 | oscuridad,darkness 220 | investigación,investigation 221 | mike,mike 222 | dinero,money 223 | hacia,toward 224 | dulce,sweet 225 | siéntate,sit 226 | parecer,seem 227 | vistazo,glance 228 | historias,stories 229 | vender,sell 230 | roja,red 231 | gallo,rooster 232 | vayan,go 233 | chicos,boys 234 | contrato,contract 235 | -------------------------------------------------------------------------------- /examples/02_lazy_loading.py: -------------------------------------------------------------------------------- 1 | """ Example of lazy vs normal loading 2 | Created by Chip Huyen (chiphuyen@cs.stanford.edu) 3 | CS20: "TensorFlow for Deep Learning Research" 4 | cs20.stanford.edu 5 | Lecture 02 6 | """ 7 | import os 8 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 9 | 10 | import tensorflow as tf 11 | 12 | ######################################## 13 | ## NORMAL LOADING ## 14 | ## print out a graph with 1 Add node ## 15 | ######################################## 16 | 17 | x = tf.Variable(10, name='x') 18 | y = tf.Variable(20, name='y') 19 | z = tf.add(x, y) 20 | 21 | with tf.Session() as sess: 22 | sess.run(tf.global_variables_initializer()) 23 | writer = tf.summary.FileWriter('graphs/normal_loading', sess.graph) 24 | for _ in range(10): 25 | sess.run(z) 26 | print(tf.get_default_graph().as_graph_def()) 27 | writer.close() 28 | 29 | ######################################## 30 | ## LAZY LOADING ## 31 | ## print out a graph with 10 Add nodes## 32 | ######################################## 33 | 34 | x = tf.Variable(10, name='x') 35 | y = tf.Variable(20, name='y') 36 | 37 | with tf.Session() as sess: 38 | sess.run(tf.global_variables_initializer()) 39 | writer = tf.summary.FileWriter('graphs/lazy_loading', sess.graph) 40 | for _ in range(10): 41 | sess.run(tf.add(x, y)) 42 | print(tf.get_default_graph().as_graph_def()) 43 | writer.close() -------------------------------------------------------------------------------- /examples/02_placeholder.py: -------------------------------------------------------------------------------- 1 | """ Placeholder and feed_dict example 2 | Created by Chip Huyen (chiphuyen@cs.stanford.edu) 3 | CS20: "TensorFlow for Deep Learning Research" 4 | cs20.stanford.edu 5 | Lecture 02 6 | """ 7 | import os 8 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 9 | 10 | import tensorflow as tf 11 | 12 | # Example 1: feed_dict with placeholder 13 | 14 | # a is a placeholderfor a vector of 3 elements, type tf.float32 15 | a = tf.placeholder(tf.float32, shape=[3]) 16 | b = tf.constant([5, 5, 5], tf.float32) 17 | 18 | # use the placeholder as you would a constant 19 | c = a + b # short for tf.add(a, b) 20 | 21 | writer = tf.summary.FileWriter('graphs/placeholders', tf.get_default_graph()) 22 | with tf.Session() as sess: 23 | # compute the value of c given the value of a is [1, 2, 3] 24 | print(sess.run(c, {a: [1, 2, 3]})) # [6. 7. 8.] 25 | writer.close() 26 | 27 | 28 | # Example 2: feed_dict with variables 29 | a = tf.add(2, 5) 30 | b = tf.multiply(a, 3) 31 | 32 | with tf.Session() as sess: 33 | print(sess.run(b)) # >> 21 34 | # compute the value of b given the value of a is 15 35 | print(sess.run(b, feed_dict={a: 15})) # >> 45 -------------------------------------------------------------------------------- /examples/02_simple_tf.py: -------------------------------------------------------------------------------- 1 | """ Simple TensorFlow's ops 2 | Created by Chip Huyen (chiphuyen@cs.stanford.edu) 3 | CS20: "TensorFlow for Deep Learning Research" 4 | cs20.stanford.edu 5 | """ 6 | import os 7 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 8 | 9 | import numpy as np 10 | import tensorflow as tf 11 | 12 | # Example 1: Simple ways to create log file writer 13 | a = tf.constant(2, name='a') 14 | b = tf.constant(3, name='b') 15 | x = tf.add(a, b, name='add') 16 | writer = tf.summary.FileWriter('./graphs/simple', tf.get_default_graph()) 17 | with tf.Session() as sess: 18 | # writer = tf.summary.FileWriter('./graphs', sess.graph) 19 | print(sess.run(x)) 20 | writer.close() # close the writer when you’re done using it 21 | 22 | # Example 2: The wonderful wizard of div 23 | a = tf.constant([2, 2], name='a') 24 | b = tf.constant([[0, 1], [2, 3]], name='b') 25 | 26 | with tf.Session() as sess: 27 | print(sess.run(tf.div(b, a))) 28 | print(sess.run(tf.divide(b, a))) 29 | print(sess.run(tf.truediv(b, a))) 30 | print(sess.run(tf.floordiv(b, a))) 31 | # print(sess.run(tf.realdiv(b, a))) 32 | print(sess.run(tf.truncatediv(b, a))) 33 | print(sess.run(tf.floor_div(b, a))) 34 | 35 | # Example 3: multiplying tensors 36 | a = tf.constant([10, 20], name='a') 37 | b = tf.constant([2, 3], name='b') 38 | 39 | with tf.Session() as sess: 40 | print(sess.run(tf.multiply(a, b))) 41 | print(sess.run(tf.tensordot(a, b, 1))) 42 | 43 | # Example 4: Python native type 44 | t_0 = 19 45 | x = tf.zeros_like(t_0) # ==> 0 46 | y = tf.ones_like(t_0) # ==> 1 47 | 48 | t_1 = ['apple', 'peach', 'banana'] 49 | x = tf.zeros_like(t_1) # ==> ['' '' ''] 50 | # y = tf.ones_like(t_1) # ==> TypeError: Expected string, got 1 of type 'int' instead. 51 | 52 | t_2 = [[True, False, False], 53 | [False, False, True], 54 | [False, True, False]] 55 | x = tf.zeros_like(t_2) # ==> 3x3 tensor, all elements are False 56 | y = tf.ones_like(t_2) # ==> 3x3 tensor, all elements are True 57 | 58 | print(tf.int32.as_numpy_dtype()) 59 | 60 | # Example 5: printing your graph's definition 61 | my_const = tf.constant([1.0, 2.0], name='my_const') 62 | print(tf.get_default_graph().as_graph_def()) -------------------------------------------------------------------------------- /examples/02_variables.py: -------------------------------------------------------------------------------- 1 | """ Variable exmaples 2 | Created by Chip Huyen (chiphuyen@cs.stanford.edu) 3 | CS20: "TensorFlow for Deep Learning Research" 4 | cs20.stanford.edu 5 | Lecture 02 6 | """ 7 | import os 8 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 9 | 10 | import numpy as np 11 | import tensorflow as tf 12 | 13 | # Example 1: creating variables 14 | s = tf.Variable(2, name='scalar') 15 | m = tf.Variable([[0, 1], [2, 3]], name='matrix') 16 | W = tf.Variable(tf.zeros([784,10]), name='big_matrix') 17 | V = tf.Variable(tf.truncated_normal([784, 10]), name='normal_matrix') 18 | 19 | s = tf.get_variable('scalar', initializer=tf.constant(2)) 20 | m = tf.get_variable('matrix', initializer=tf.constant([[0, 1], [2, 3]])) 21 | W = tf.get_variable('big_matrix', shape=(784, 10), initializer=tf.zeros_initializer()) 22 | V = tf.get_variable('normal_matrix', shape=(784, 10), initializer=tf.truncated_normal_initializer()) 23 | 24 | with tf.Session() as sess: 25 | sess.run(tf.global_variables_initializer()) 26 | print(V.eval()) 27 | 28 | # Example 2: assigning values to variables 29 | W = tf.Variable(10) 30 | W.assign(100) 31 | with tf.Session() as sess: 32 | sess.run(W.initializer) 33 | print(sess.run(W)) # >> 10 34 | 35 | W = tf.Variable(10) 36 | assign_op = W.assign(100) 37 | with tf.Session() as sess: 38 | sess.run(assign_op) 39 | print(W.eval()) # >> 100 40 | 41 | # create a variable whose original value is 2 42 | a = tf.get_variable('scalar', initializer=tf.constant(2)) 43 | a_times_two = a.assign(a * 2) 44 | with tf.Session() as sess: 45 | sess.run(tf.global_variables_initializer()) 46 | sess.run(a_times_two) # >> 4 47 | sess.run(a_times_two) # >> 8 48 | sess.run(a_times_two) # >> 16 49 | 50 | W = tf.Variable(10) 51 | with tf.Session() as sess: 52 | sess.run(W.initializer) 53 | print(sess.run(W.assign_add(10))) # >> 20 54 | print(sess.run(W.assign_sub(2))) # >> 18 55 | 56 | # Example 3: Each session has its own copy of variable 57 | W = tf.Variable(10) 58 | sess1 = tf.Session() 59 | sess2 = tf.Session() 60 | sess1.run(W.initializer) 61 | sess2.run(W.initializer) 62 | print(sess1.run(W.assign_add(10))) # >> 20 63 | print(sess2.run(W.assign_sub(2))) # >> 8 64 | print(sess1.run(W.assign_add(100))) # >> 120 65 | print(sess2.run(W.assign_sub(50))) # >> -42 66 | sess1.close() 67 | sess2.close() 68 | 69 | # Example 4: create a variable with the initial value depending on another variable 70 | W = tf.Variable(tf.truncated_normal([700, 10])) 71 | U = tf.Variable(W * 2) -------------------------------------------------------------------------------- /examples/03_linreg_dataset.py: -------------------------------------------------------------------------------- 1 | """ Solution for simple linear regression example using tf.data 2 | Created by Chip Huyen (chiphuyen@cs.stanford.edu) 3 | CS20: "TensorFlow for Deep Learning Research" 4 | cs20.stanford.edu 5 | Lecture 03 6 | """ 7 | import os 8 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 9 | import time 10 | 11 | import numpy as np 12 | import matplotlib.pyplot as plt 13 | import tensorflow as tf 14 | 15 | import utils 16 | 17 | DATA_FILE = 'data/birth_life_2010.txt' 18 | 19 | # Step 1: read in the data 20 | data, n_samples = utils.read_birth_life_data(DATA_FILE) 21 | 22 | # Step 2: create Dataset and iterator 23 | dataset = tf.data.Dataset.from_tensor_slices((data[:,0], data[:,1])) 24 | 25 | iterator = dataset.make_initializable_iterator() 26 | X, Y = iterator.get_next() 27 | 28 | # Step 3: create weight and bias, initialized to 0 29 | w = tf.get_variable('weights', initializer=tf.constant(0.0)) 30 | b = tf.get_variable('bias', initializer=tf.constant(0.0)) 31 | 32 | # Step 4: build model to predict Y 33 | Y_predicted = X * w + b 34 | 35 | # Step 5: use the square error as the loss function 36 | loss = tf.square(Y - Y_predicted, name='loss') 37 | # loss = utils.huber_loss(Y, Y_predicted) 38 | 39 | # Step 6: using gradient descent with learning rate of 0.001 to minimize loss 40 | optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001).minimize(loss) 41 | 42 | start = time.time() 43 | with tf.Session() as sess: 44 | # Step 7: initialize the necessary variables, in this case, w and b 45 | sess.run(tf.global_variables_initializer()) 46 | writer = tf.summary.FileWriter('./graphs/linear_reg', sess.graph) 47 | 48 | # Step 8: train the model for 100 epochs 49 | for i in range(100): 50 | sess.run(iterator.initializer) # initialize the iterator 51 | total_loss = 0 52 | try: 53 | while True: 54 | _, l = sess.run([optimizer, loss]) 55 | total_loss += l 56 | except tf.errors.OutOfRangeError: 57 | pass 58 | 59 | print('Epoch {0}: {1}'.format(i, total_loss/n_samples)) 60 | 61 | # close the writer when you're done using it 62 | writer.close() 63 | 64 | # Step 9: output the values of w and b 65 | w_out, b_out = sess.run([w, b]) 66 | print('w: %f, b: %f' %(w_out, b_out)) 67 | print('Took: %f seconds' %(time.time() - start)) 68 | 69 | # plot the results 70 | plt.plot(data[:,0], data[:,1], 'bo', label='Real data') 71 | plt.plot(data[:,0], data[:,0] * w_out + b_out, 'r', label='Predicted data with squared error') 72 | # plt.plot(data[:,0], data[:,0] * (-5.883589) + 85.124306, 'g', label='Predicted data with Huber loss') 73 | plt.legend() 74 | plt.show() -------------------------------------------------------------------------------- /examples/03_linreg_placeholder.py: -------------------------------------------------------------------------------- 1 | """ Solution for simple linear regression example using placeholders 2 | Created by Chip Huyen (chiphuyen@cs.stanford.edu) 3 | CS20: "TensorFlow for Deep Learning Research" 4 | cs20.stanford.edu 5 | Lecture 03 6 | """ 7 | import os 8 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 9 | import time 10 | 11 | import numpy as np 12 | import matplotlib.pyplot as plt 13 | import tensorflow as tf 14 | 15 | import utils 16 | 17 | DATA_FILE = 'data/birth_life_2010.txt' 18 | 19 | # Step 1: read in data from the .txt file 20 | data, n_samples = utils.read_birth_life_data(DATA_FILE) 21 | 22 | # Step 2: create placeholders for X (birth rate) and Y (life expectancy) 23 | X = tf.placeholder(tf.float32, name='X') 24 | Y = tf.placeholder(tf.float32, name='Y') 25 | 26 | # Step 3: create weight and bias, initialized to 0 27 | w = tf.get_variable('weights', initializer=tf.constant(0.0)) 28 | b = tf.get_variable('bias', initializer=tf.constant(0.0)) 29 | 30 | # Step 4: build model to predict Y 31 | Y_predicted = w * X + b 32 | 33 | # Step 5: use the squared error as the loss function 34 | # you can use either mean squared error or Huber loss 35 | loss = tf.square(Y - Y_predicted, name='loss') 36 | # loss = utils.huber_loss(Y, Y_predicted) 37 | 38 | # Step 6: using gradient descent with learning rate of 0.001 to minimize loss 39 | optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001).minimize(loss) 40 | 41 | 42 | start = time.time() 43 | writer = tf.summary.FileWriter('./graphs/linear_reg', tf.get_default_graph()) 44 | with tf.Session() as sess: 45 | # Step 7: initialize the necessary variables, in this case, w and b 46 | sess.run(tf.global_variables_initializer()) 47 | 48 | # Step 8: train the model for 100 epochs 49 | for i in range(100): 50 | total_loss = 0 51 | for x, y in data: 52 | # Session execute optimizer and fetch values of loss 53 | _, l = sess.run([optimizer, loss], feed_dict={X: x, Y:y}) 54 | total_loss += l 55 | print('Epoch {0}: {1}'.format(i, total_loss/n_samples)) 56 | 57 | # close the writer when you're done using it 58 | writer.close() 59 | 60 | # Step 9: output the values of w and b 61 | w_out, b_out = sess.run([w, b]) 62 | 63 | print('Took: %f seconds' %(time.time() - start)) 64 | 65 | # plot the results 66 | plt.plot(data[:,0], data[:,1], 'bo', label='Real data') 67 | plt.plot(data[:,0], data[:,0] * w_out + b_out, 'r', label='Predicted data') 68 | plt.legend() 69 | plt.show() -------------------------------------------------------------------------------- /examples/03_linreg_starter.py: -------------------------------------------------------------------------------- 1 | """ Starter code for simple linear regression example using placeholders 2 | Created by Chip Huyen (huyenn@cs.stanford.edu) 3 | CS20: "TensorFlow for Deep Learning Research" 4 | cs20.stanford.edu 5 | Lecture 03 6 | """ 7 | import os 8 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 9 | import time 10 | 11 | import numpy as np 12 | import matplotlib.pyplot as plt 13 | import tensorflow as tf 14 | 15 | import utils 16 | 17 | DATA_FILE = 'data/birth_life_2010.txt' 18 | 19 | # Step 1: read in data from the .txt file 20 | data, n_samples = utils.read_birth_life_data(DATA_FILE) 21 | 22 | # Step 2: create placeholders for X (birth rate) and Y (life expectancy) 23 | # Remember both X and Y are scalars with type float 24 | X, Y = None, None 25 | ############################# 26 | ########## TO DO ############ 27 | ############################# 28 | 29 | # Step 3: create weight and bias, initialized to 0.0 30 | # Make sure to use tf.get_variable 31 | w, b = None, None 32 | ############################# 33 | ########## TO DO ############ 34 | ############################# 35 | 36 | # Step 4: build model to predict Y 37 | # e.g. how would you derive at Y_predicted given X, w, and b 38 | Y_predicted = None 39 | ############################# 40 | ########## TO DO ############ 41 | ############################# 42 | 43 | # Step 5: use the square error as the loss function 44 | loss = None 45 | ############################# 46 | ########## TO DO ############ 47 | ############################# 48 | 49 | # Step 6: using gradient descent with learning rate of 0.001 to minimize loss 50 | optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001).minimize(loss) 51 | 52 | start = time.time() 53 | 54 | # Create a filewriter to write the model's graph to TensorBoard 55 | ############################# 56 | ########## TO DO ############ 57 | ############################# 58 | 59 | with tf.Session() as sess: 60 | # Step 7: initialize the necessary variables, in this case, w and b 61 | ############################# 62 | ########## TO DO ############ 63 | ############################# 64 | 65 | # Step 8: train the model for 100 epochs 66 | for i in range(100): 67 | total_loss = 0 68 | for x, y in data: 69 | # Execute train_op and get the value of loss. 70 | # Don't forget to feed in data for placeholders 71 | _, loss = ########## TO DO ############ 72 | total_loss += loss 73 | 74 | print('Epoch {0}: {1}'.format(i, total_loss/n_samples)) 75 | 76 | # close the writer when you're done using it 77 | ############################# 78 | ########## TO DO ############ 79 | ############################# 80 | writer.close() 81 | 82 | # Step 9: output the values of w and b 83 | w_out, b_out = None, None 84 | ############################# 85 | ########## TO DO ############ 86 | ############################# 87 | 88 | print('Took: %f seconds' %(time.time() - start)) 89 | 90 | # uncomment the following lines to see the plot 91 | # plt.plot(data[:,0], data[:,1], 'bo', label='Real data') 92 | # plt.plot(data[:,0], data[:,0] * w_out + b_out, 'r', label='Predicted data') 93 | # plt.legend() 94 | # plt.show() -------------------------------------------------------------------------------- /examples/03_logreg.py: -------------------------------------------------------------------------------- 1 | """ Solution for simple logistic regression model for MNIST 2 | with tf.data module 3 | MNIST dataset: yann.lecun.com/exdb/mnist/ 4 | Created by Chip Huyen (chiphuyen@cs.stanford.edu) 5 | CS20: "TensorFlow for Deep Learning Research" 6 | cs20.stanford.edu 7 | Lecture 03 8 | """ 9 | import os 10 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 11 | 12 | import numpy as np 13 | import tensorflow as tf 14 | import time 15 | 16 | import utils 17 | 18 | # Define paramaters for the model 19 | learning_rate = 0.01 20 | batch_size = 128 21 | n_epochs = 30 22 | n_train = 60000 23 | n_test = 10000 24 | 25 | # Step 1: Read in data 26 | mnist_folder = 'data/mnist' 27 | utils.download_mnist(mnist_folder) 28 | train, val, test = utils.read_mnist(mnist_folder, flatten=True) 29 | 30 | # Step 2: Create datasets and iterator 31 | train_data = tf.data.Dataset.from_tensor_slices(train) 32 | train_data = train_data.shuffle(10000) # if you want to shuffle your data 33 | train_data = train_data.batch(batch_size) 34 | 35 | test_data = tf.data.Dataset.from_tensor_slices(test) 36 | test_data = test_data.batch(batch_size) 37 | 38 | iterator = tf.data.Iterator.from_structure(train_data.output_types, 39 | train_data.output_shapes) 40 | img, label = iterator.get_next() 41 | 42 | train_init = iterator.make_initializer(train_data) # initializer for train_data 43 | test_init = iterator.make_initializer(test_data) # initializer for train_data 44 | 45 | # Step 3: create weights and bias 46 | # w is initialized to random variables with mean of 0, stddev of 0.01 47 | # b is initialized to 0 48 | # shape of w depends on the dimension of X and Y so that Y = tf.matmul(X, w) 49 | # shape of b depends on Y 50 | w = tf.get_variable(name='weights', shape=(784, 10), initializer=tf.random_normal_initializer(0, 0.01)) 51 | b = tf.get_variable(name='bias', shape=(1, 10), initializer=tf.zeros_initializer()) 52 | 53 | # Step 4: build model 54 | # the model that returns the logits. 55 | # this logits will be later passed through softmax layer 56 | logits = tf.matmul(img, w) + b 57 | 58 | # Step 5: define loss function 59 | # use cross entropy of softmax of logits as the loss function 60 | entropy = tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=label, name='entropy') 61 | loss = tf.reduce_mean(entropy, name='loss') # computes the mean over all the examples in the batch 62 | 63 | # Step 6: define training op 64 | # using gradient descent with learning rate of 0.01 to minimize loss 65 | optimizer = tf.train.AdamOptimizer(learning_rate).minimize(loss) 66 | 67 | # Step 7: calculate accuracy with test set 68 | preds = tf.nn.softmax(logits) 69 | correct_preds = tf.equal(tf.argmax(preds, 1), tf.argmax(label, 1)) 70 | accuracy = tf.reduce_sum(tf.cast(correct_preds, tf.float32)) 71 | 72 | writer = tf.summary.FileWriter('./graphs/logreg', tf.get_default_graph()) 73 | with tf.Session() as sess: 74 | 75 | start_time = time.time() 76 | sess.run(tf.global_variables_initializer()) 77 | 78 | # train the model n_epochs times 79 | for i in range(n_epochs): 80 | sess.run(train_init) # drawing samples from train_data 81 | total_loss = 0 82 | n_batches = 0 83 | try: 84 | while True: 85 | _, l = sess.run([optimizer, loss]) 86 | total_loss += l 87 | n_batches += 1 88 | except tf.errors.OutOfRangeError: 89 | pass 90 | print('Average loss epoch {0}: {1}'.format(i, total_loss/n_batches)) 91 | print('Total time: {0} seconds'.format(time.time() - start_time)) 92 | 93 | # test the model 94 | sess.run(test_init) # drawing samples from test_data 95 | total_correct_preds = 0 96 | try: 97 | while True: 98 | accuracy_batch = sess.run(accuracy) 99 | total_correct_preds += accuracy_batch 100 | except tf.errors.OutOfRangeError: 101 | pass 102 | 103 | print('Accuracy {0}'.format(total_correct_preds/n_test)) 104 | writer.close() 105 | -------------------------------------------------------------------------------- /examples/03_logreg_placeholder.py: -------------------------------------------------------------------------------- 1 | """ Solution for simple logistic regression model for MNIST 2 | with placeholder 3 | MNIST dataset: yann.lecun.com/exdb/mnist/ 4 | Created by Chip Huyen (huyenn@cs.stanford.edu) 5 | CS20: "TensorFlow for Deep Learning Research" 6 | cs20.stanford.edu 7 | Lecture 03 8 | """ 9 | import os 10 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 11 | 12 | import numpy as np 13 | import tensorflow as tf 14 | from tensorflow.examples.tutorials.mnist import input_data 15 | import time 16 | 17 | import utils 18 | 19 | # Define paramaters for the model 20 | learning_rate = 0.01 21 | batch_size = 128 22 | n_epochs = 30 23 | 24 | # Step 1: Read in data 25 | # using TF Learn's built in function to load MNIST data to the folder data/mnist 26 | mnist = input_data.read_data_sets('data/mnist', one_hot=True) 27 | X_batch, Y_batch = mnist.train.next_batch(batch_size) 28 | 29 | # Step 2: create placeholders for features and labels 30 | # each image in the MNIST data is of shape 28*28 = 784 31 | # therefore, each image is represented with a 1x784 tensor 32 | # there are 10 classes for each image, corresponding to digits 0 - 9. 33 | # each lable is one hot vector. 34 | X = tf.placeholder(tf.float32, [batch_size, 784], name='image') 35 | Y = tf.placeholder(tf.int32, [batch_size, 10], name='label') 36 | 37 | # Step 3: create weights and bias 38 | # w is initialized to random variables with mean of 0, stddev of 0.01 39 | # b is initialized to 0 40 | # shape of w depends on the dimension of X and Y so that Y = tf.matmul(X, w) 41 | # shape of b depends on Y 42 | w = tf.get_variable(name='weights', shape=(784, 10), initializer=tf.random_normal_initializer()) 43 | b = tf.get_variable(name='bias', shape=(1, 10), initializer=tf.zeros_initializer()) 44 | 45 | # Step 4: build model 46 | # the model that returns the logits. 47 | # this logits will be later passed through softmax layer 48 | logits = tf.matmul(X, w) + b 49 | 50 | # Step 5: define loss function 51 | # use cross entropy of softmax of logits as the loss function 52 | entropy = tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=Y, name='loss') 53 | loss = tf.reduce_mean(entropy) # computes the mean over all the examples in the batch 54 | # loss = tf.reduce_mean(-tf.reduce_sum(tf.nn.softmax(logits) * tf.log(Y), reduction_indices=[1])) 55 | 56 | # Step 6: define training op 57 | # using gradient descent with learning rate of 0.01 to minimize loss 58 | optimizer = tf.train.AdamOptimizer(learning_rate).minimize(loss) 59 | 60 | # Step 7: calculate accuracy with test set 61 | preds = tf.nn.softmax(logits) 62 | correct_preds = tf.equal(tf.argmax(preds, 1), tf.argmax(Y, 1)) 63 | accuracy = tf.reduce_sum(tf.cast(correct_preds, tf.float32)) 64 | 65 | writer = tf.summary.FileWriter('./graphs/logreg_placeholder', tf.get_default_graph()) 66 | with tf.Session() as sess: 67 | start_time = time.time() 68 | sess.run(tf.global_variables_initializer()) 69 | n_batches = int(mnist.train.num_examples/batch_size) 70 | 71 | # train the model n_epochs times 72 | for i in range(n_epochs): 73 | total_loss = 0 74 | 75 | for j in range(n_batches): 76 | X_batch, Y_batch = mnist.train.next_batch(batch_size) 77 | _, loss_batch = sess.run([optimizer, loss], {X: X_batch, Y:Y_batch}) 78 | total_loss += loss_batch 79 | print('Average loss epoch {0}: {1}'.format(i, total_loss/n_batches)) 80 | print('Total time: {0} seconds'.format(time.time() - start_time)) 81 | 82 | # test the model 83 | n_batches = int(mnist.test.num_examples/batch_size) 84 | total_correct_preds = 0 85 | 86 | for i in range(n_batches): 87 | X_batch, Y_batch = mnist.test.next_batch(batch_size) 88 | accuracy_batch = sess.run(accuracy, {X: X_batch, Y:Y_batch}) 89 | total_correct_preds += accuracy_batch 90 | 91 | print('Accuracy {0}'.format(total_correct_preds/mnist.test.num_examples)) 92 | 93 | writer.close() 94 | -------------------------------------------------------------------------------- /examples/03_logreg_starter.py: -------------------------------------------------------------------------------- 1 | """ Starter code for simple logistic regression model for MNIST 2 | with tf.data module 3 | MNIST dataset: yann.lecun.com/exdb/mnist/ 4 | Created by Chip Huyen (chiphuyen@cs.stanford.edu) 5 | CS20: "TensorFlow for Deep Learning Research" 6 | cs20.stanford.edu 7 | Lecture 03 8 | """ 9 | import os 10 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 11 | 12 | import numpy as np 13 | import tensorflow as tf 14 | import time 15 | 16 | import utils 17 | 18 | # Define paramaters for the model 19 | learning_rate = 0.01 20 | batch_size = 128 21 | n_epochs = 30 22 | n_train = 60000 23 | n_test = 10000 24 | 25 | # Step 1: Read in data 26 | mnist_folder = 'data/mnist' 27 | utils.download_mnist(mnist_folder) 28 | train, val, test = utils.read_mnist(mnist_folder, flatten=True) 29 | 30 | # Step 2: Create datasets and iterator 31 | # create training Dataset and batch it 32 | train_data = tf.data.Dataset.from_tensor_slices(train) 33 | train_data = train_data.shuffle(10000) # if you want to shuffle your data 34 | train_data = train_data.batch(batch_size) 35 | 36 | # create testing Dataset and batch it 37 | test_data = None 38 | ############################# 39 | ########## TO DO ############ 40 | ############################# 41 | 42 | 43 | # create one iterator and initialize it with different datasets 44 | iterator = tf.data.Iterator.from_structure(train_data.output_types, 45 | train_data.output_shapes) 46 | img, label = iterator.get_next() 47 | 48 | train_init = iterator.make_initializer(train_data) # initializer for train_data 49 | test_init = iterator.make_initializer(test_data) # initializer for train_data 50 | 51 | # Step 3: create weights and bias 52 | # w is initialized to random variables with mean of 0, stddev of 0.01 53 | # b is initialized to 0 54 | # shape of w depends on the dimension of X and Y so that Y = tf.matmul(X, w) 55 | # shape of b depends on Y 56 | w, b = None, None 57 | ############################# 58 | ########## TO DO ############ 59 | ############################# 60 | 61 | 62 | # Step 4: build model 63 | # the model that returns the logits. 64 | # this logits will be later passed through softmax layer 65 | logits = None 66 | ############################# 67 | ########## TO DO ############ 68 | ############################# 69 | 70 | 71 | # Step 5: define loss function 72 | # use cross entropy of softmax of logits as the loss function 73 | loss = None 74 | ############################# 75 | ########## TO DO ############ 76 | ############################# 77 | 78 | 79 | # Step 6: define optimizer 80 | # using Adamn Optimizer with pre-defined learning rate to minimize loss 81 | optimizer = None 82 | ############################# 83 | ########## TO DO ############ 84 | ############################# 85 | 86 | 87 | # Step 7: calculate accuracy with test set 88 | preds = tf.nn.softmax(logits) 89 | correct_preds = tf.equal(tf.argmax(preds, 1), tf.argmax(label, 1)) 90 | accuracy = tf.reduce_sum(tf.cast(correct_preds, tf.float32)) 91 | 92 | writer = tf.summary.FileWriter('./graphs/logreg', tf.get_default_graph()) 93 | with tf.Session() as sess: 94 | 95 | start_time = time.time() 96 | sess.run(tf.global_variables_initializer()) 97 | 98 | # train the model n_epochs times 99 | for i in range(n_epochs): 100 | sess.run(train_init) # drawing samples from train_data 101 | total_loss = 0 102 | n_batches = 0 103 | try: 104 | while True: 105 | _, l = sess.run([optimizer, loss]) 106 | total_loss += l 107 | n_batches += 1 108 | except tf.errors.OutOfRangeError: 109 | pass 110 | print('Average loss epoch {0}: {1}'.format(i, total_loss/n_batches)) 111 | print('Total time: {0} seconds'.format(time.time() - start_time)) 112 | 113 | # test the model 114 | sess.run(test_init) # drawing samples from test_data 115 | total_correct_preds = 0 116 | try: 117 | while True: 118 | accuracy_batch = sess.run(accuracy) 119 | total_correct_preds += accuracy_batch 120 | except tf.errors.OutOfRangeError: 121 | pass 122 | 123 | print('Accuracy {0}'.format(total_correct_preds/n_test)) 124 | writer.close() -------------------------------------------------------------------------------- /examples/04_linreg_eager.py: -------------------------------------------------------------------------------- 1 | """ Starter code for a simple regression example using eager execution. 2 | Created by Akshay Agrawal (akshayka@cs.stanford.edu) 3 | CS20: "TensorFlow for Deep Learning Research" 4 | cs20.stanford.edu 5 | Lecture 04 6 | """ 7 | import time 8 | 9 | import tensorflow as tf 10 | import tensorflow.contrib.eager as tfe 11 | import matplotlib.pyplot as plt 12 | 13 | import utils 14 | 15 | DATA_FILE = 'data/birth_life_2010.txt' 16 | 17 | # In order to use eager execution, `tfe.enable_eager_execution()` must be 18 | # called at the very beginning of a TensorFlow program. 19 | tfe.enable_eager_execution() 20 | 21 | # Read the data into a dataset. 22 | data, n_samples = utils.read_birth_life_data(DATA_FILE) 23 | dataset = tf.data.Dataset.from_tensor_slices((data[:,0], data[:,1])) 24 | 25 | # Create variables. 26 | w = tfe.Variable(0.0) 27 | b = tfe.Variable(0.0) 28 | 29 | # Define the linear predictor. 30 | def prediction(x): 31 | return x * w + b 32 | 33 | # Define loss functions of the form: L(y, y_predicted) 34 | def squared_loss(y, y_predicted): 35 | return (y - y_predicted) ** 2 36 | 37 | def huber_loss(y, y_predicted, m=1.0): 38 | """Huber loss.""" 39 | t = y - y_predicted 40 | # Note that enabling eager execution lets you use Python control flow and 41 | # specificy dynamic TensorFlow computations. Contrast this implementation 42 | # to the graph-construction one found in `utils`, which uses `tf.cond`. 43 | return t ** 2 if tf.abs(t) <= m else m * (2 * tf.abs(t) - m) 44 | 45 | def train(loss_fn): 46 | """Train a regression model evaluated using `loss_fn`.""" 47 | print('Training; loss function: ' + loss_fn.__name__) 48 | optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01) 49 | 50 | # Define the function through which to differentiate. 51 | def loss_for_example(x, y): 52 | return loss_fn(y, prediction(x)) 53 | 54 | # `grad_fn(x_i, y_i)` returns (1) the value of `loss_for_example` 55 | # evaluated at `x_i`, `y_i` and (2) the gradients of any variables used in 56 | # calculating it. 57 | grad_fn = tfe.implicit_value_and_gradients(loss_for_example) 58 | 59 | start = time.time() 60 | for epoch in range(100): 61 | total_loss = 0.0 62 | for x_i, y_i in tfe.Iterator(dataset): 63 | loss, gradients = grad_fn(x_i, y_i) 64 | # Take an optimization step and update variables. 65 | optimizer.apply_gradients(gradients) 66 | total_loss += loss 67 | if epoch % 10 == 0: 68 | print('Epoch {0}: {1}'.format(epoch, total_loss / n_samples)) 69 | print('Took: %f seconds' % (time.time() - start)) 70 | print('Eager execution exhibits significant overhead per operation. ' 71 | 'As you increase your batch size, the impact of the overhead will ' 72 | 'become less noticeable. Eager execution is under active development: ' 73 | 'expect performance to increase substantially in the near future!') 74 | 75 | train(huber_loss) 76 | plt.plot(data[:,0], data[:,1], 'bo') 77 | # The `.numpy()` method of a tensor retrieves the NumPy array backing it. 78 | # In future versions of eager, you won't need to call `.numpy()` and will 79 | # instead be able to, in most cases, pass Tensors wherever NumPy arrays are 80 | # expected. 81 | plt.plot(data[:,0], data[:,0] * w.numpy() + b.numpy(), 'r', 82 | label="huber regression") 83 | plt.legend() 84 | plt.show() 85 | -------------------------------------------------------------------------------- /examples/04_linreg_eager_starter.py: -------------------------------------------------------------------------------- 1 | """ Starter code for a simple regression example using eager execution. 2 | Created by Akshay Agrawal (akshayka@cs.stanford.edu) 3 | CS20: "TensorFlow for Deep Learning Research" 4 | cs20.stanford.edu 5 | Lecture 04 6 | """ 7 | import time 8 | 9 | import tensorflow as tf 10 | import tensorflow.contrib.eager as tfe 11 | import matplotlib.pyplot as plt 12 | 13 | import utils 14 | 15 | DATA_FILE = 'data/birth_life_2010.txt' 16 | 17 | # In order to use eager execution, `tfe.enable_eager_execution()` must be 18 | # called at the very beginning of a TensorFlow program. 19 | ############################# 20 | ########## TO DO ############ 21 | ############################# 22 | 23 | # Read the data into a dataset. 24 | data, n_samples = utils.read_birth_life_data(DATA_FILE) 25 | dataset = tf.data.Dataset.from_tensor_slices((data[:,0], data[:,1])) 26 | 27 | # Create weight and bias variables, initialized to 0.0. 28 | ############################# 29 | ########## TO DO ############ 30 | ############################# 31 | w = None 32 | b = None 33 | 34 | # Define the linear predictor. 35 | def prediction(x): 36 | ############################# 37 | ########## TO DO ############ 38 | ############################# 39 | pass 40 | 41 | # Define loss functions of the form: L(y, y_predicted) 42 | def squared_loss(y, y_predicted): 43 | ############################# 44 | ########## TO DO ############ 45 | ############################# 46 | pass 47 | 48 | def huber_loss(y, y_predicted): 49 | """Huber loss with `m` set to `1.0`.""" 50 | ############################# 51 | ########## TO DO ############ 52 | ############################# 53 | pass 54 | 55 | def train(loss_fn): 56 | """Train a regression model evaluated using `loss_fn`.""" 57 | print('Training; loss function: ' + loss_fn.__name__) 58 | optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01) 59 | 60 | # Define the function through which to differentiate. 61 | ############################# 62 | ########## TO DO ############ 63 | ############################# 64 | def loss_for_example(x, y): 65 | pass 66 | 67 | # Obtain a gradients function using `tfe.implicit_value_and_gradients`. 68 | ############################# 69 | ########## TO DO ############ 70 | ############################# 71 | grad_fn = None 72 | 73 | start = time.time() 74 | for epoch in range(100): 75 | total_loss = 0.0 76 | for x_i, y_i in tfe.Iterator(dataset): 77 | # Compute the loss and gradient, and take an optimization step. 78 | ############################# 79 | ########## TO DO ############ 80 | ############################# 81 | optimizer.apply_gradients(gradients) 82 | total_loss += loss 83 | if epoch % 10 == 0: 84 | print('Epoch {0}: {1}'.format(epoch, total_loss / n_samples)) 85 | print('Took: %f seconds' % (time.time() - start)) 86 | print('Eager execution exhibits significant overhead per operation. ' 87 | 'As you increase your batch size, the impact of the overhead will ' 88 | 'become less noticeable. Eager execution is under active development: ' 89 | 'expect performance to increase substantially in the near future!') 90 | 91 | train(huber_loss) 92 | plt.plot(data[:,0], data[:,1], 'bo') 93 | # The `.numpy()` method of a tensor retrieves the NumPy array backing it. 94 | # In future versions of eager, you won't need to call `.numpy()` and will 95 | # instead be able to, in most cases, pass Tensors wherever NumPy arrays are 96 | # expected. 97 | plt.plot(data[:,0], data[:,0] * w.numpy() + b.numpy(), 'r', 98 | label="huber regression") 99 | plt.legend() 100 | plt.show() 101 | -------------------------------------------------------------------------------- /examples/04_word2vec.py: -------------------------------------------------------------------------------- 1 | """ starter code for word2vec skip-gram model with NCE loss 2 | CS 20: "TensorFlow for Deep Learning Research" 3 | cs20.stanford.edu 4 | Chip Huyen (chiphuyen@cs.stanford.edu) 5 | Lecture 04 6 | """ 7 | 8 | import os 9 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 10 | 11 | import numpy as np 12 | from tensorflow.contrib.tensorboard.plugins import projector 13 | import tensorflow as tf 14 | 15 | import utils 16 | import word2vec_utils 17 | 18 | # Model hyperparameters 19 | VOCAB_SIZE = 50000 20 | BATCH_SIZE = 128 21 | EMBED_SIZE = 128 # dimension of the word embedding vectors 22 | SKIP_WINDOW = 1 # the context window 23 | NUM_SAMPLED = 64 # number of negative examples to sample 24 | LEARNING_RATE = 1.0 25 | NUM_TRAIN_STEPS = 100000 26 | VISUAL_FLD = 'visualization' 27 | SKIP_STEP = 5000 28 | 29 | # Parameters for downloading data 30 | DOWNLOAD_URL = 'http://mattmahoney.net/dc/text8.zip' 31 | EXPECTED_BYTES = 31344016 32 | NUM_VISUALIZE = 3000 # number of tokens to visualize 33 | 34 | 35 | def word2vec(dataset): 36 | """ Build the graph for word2vec model and train it """ 37 | # Step 1: get input, output from the dataset 38 | with tf.name_scope('data'): 39 | iterator = dataset.make_initializable_iterator() 40 | center_words, target_words = iterator.get_next() 41 | 42 | """ Step 2 + 3: define weights and embedding lookup. 43 | In word2vec, it's actually the weights that we care about 44 | """ 45 | with tf.name_scope('embed'): 46 | embed_matrix = tf.get_variable('embed_matrix', 47 | shape=[VOCAB_SIZE, EMBED_SIZE], 48 | initializer=tf.random_uniform_initializer()) 49 | embed = tf.nn.embedding_lookup(embed_matrix, center_words, name='embedding') 50 | 51 | # Step 4: construct variables for NCE loss and define loss function 52 | with tf.name_scope('loss'): 53 | nce_weight = tf.get_variable('nce_weight', shape=[VOCAB_SIZE, EMBED_SIZE], 54 | initializer=tf.truncated_normal_initializer(stddev=1.0 / (EMBED_SIZE ** 0.5))) 55 | nce_bias = tf.get_variable('nce_bias', initializer=tf.zeros([VOCAB_SIZE])) 56 | 57 | # define loss function to be NCE loss function 58 | loss = tf.reduce_mean(tf.nn.nce_loss(weights=nce_weight, 59 | biases=nce_bias, 60 | labels=target_words, 61 | inputs=embed, 62 | num_sampled=NUM_SAMPLED, 63 | num_classes=VOCAB_SIZE), name='loss') 64 | 65 | # Step 5: define optimizer 66 | with tf.name_scope('optimizer'): 67 | optimizer = tf.train.GradientDescentOptimizer(LEARNING_RATE).minimize(loss) 68 | 69 | utils.safe_mkdir('checkpoints') 70 | 71 | with tf.Session() as sess: 72 | sess.run(iterator.initializer) 73 | sess.run(tf.global_variables_initializer()) 74 | 75 | total_loss = 0.0 # we use this to calculate late average loss in the last SKIP_STEP steps 76 | writer = tf.summary.FileWriter('graphs/word2vec_simple', sess.graph) 77 | 78 | for index in range(NUM_TRAIN_STEPS): 79 | try: 80 | loss_batch, _ = sess.run([loss, optimizer]) 81 | total_loss += loss_batch 82 | if (index + 1) % SKIP_STEP == 0: 83 | print('Average loss at step {}: {:5.1f}'.format(index, total_loss / SKIP_STEP)) 84 | total_loss = 0.0 85 | except tf.errors.OutOfRangeError: 86 | sess.run(iterator.initializer) 87 | writer.close() 88 | 89 | def gen(): 90 | yield from word2vec_utils.batch_gen(DOWNLOAD_URL, EXPECTED_BYTES, VOCAB_SIZE, 91 | BATCH_SIZE, SKIP_WINDOW, VISUAL_FLD) 92 | 93 | def main(): 94 | dataset = tf.data.Dataset.from_generator(gen, 95 | (tf.int32, tf.int32), 96 | (tf.TensorShape([BATCH_SIZE]), tf.TensorShape([BATCH_SIZE, 1]))) 97 | word2vec(dataset) 98 | 99 | if __name__ == '__main__': 100 | main() 101 | -------------------------------------------------------------------------------- /examples/04_word2vec_eager.py: -------------------------------------------------------------------------------- 1 | """ starter code for word2vec skip-gram model with NCE loss 2 | Eager execution 3 | CS 20: "TensorFlow for Deep Learning Research" 4 | cs20.stanford.edu 5 | Chip Huyen (chiphuyen@cs.stanford.edu) & Akshay Agrawal (akshayka@cs.stanford.edu) 6 | Lecture 04 7 | """ 8 | 9 | import os 10 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 11 | 12 | import numpy as np 13 | import tensorflow as tf 14 | import tensorflow.contrib.eager as tfe 15 | 16 | import utils 17 | import word2vec_utils 18 | 19 | tfe.enable_eager_execution() 20 | 21 | # Model hyperparameters 22 | VOCAB_SIZE = 50000 23 | BATCH_SIZE = 128 24 | EMBED_SIZE = 128 # dimension of the word embedding vectors 25 | SKIP_WINDOW = 1 # the context window 26 | NUM_SAMPLED = 64 # number of negative examples to sample 27 | LEARNING_RATE = 1.0 28 | NUM_TRAIN_STEPS = 100000 29 | VISUAL_FLD = 'visualization' 30 | SKIP_STEP = 5000 31 | 32 | # Parameters for downloading data 33 | DOWNLOAD_URL = 'http://mattmahoney.net/dc/text8.zip' 34 | EXPECTED_BYTES = 31344016 35 | 36 | class Word2Vec(object): 37 | def __init__(self, vocab_size, embed_size, num_sampled=NUM_SAMPLED): 38 | self.vocab_size = vocab_size 39 | self.num_sampled = num_sampled 40 | self.embed_matrix = tfe.Variable(tf.random_uniform( 41 | [vocab_size, embed_size])) 42 | self.nce_weight = tfe.Variable(tf.truncated_normal( 43 | [vocab_size, embed_size], 44 | stddev=1.0 / (embed_size ** 0.5))) 45 | self.nce_bias = tfe.Variable(tf.zeros([vocab_size])) 46 | 47 | def compute_loss(self, center_words, target_words): 48 | """Computes the forward pass of word2vec with the NCE loss.""" 49 | embed = tf.nn.embedding_lookup(self.embed_matrix, center_words) 50 | loss = tf.reduce_mean(tf.nn.nce_loss(weights=self.nce_weight, 51 | biases=self.nce_bias, 52 | labels=target_words, 53 | inputs=embed, 54 | num_sampled=self.num_sampled, 55 | num_classes=self.vocab_size)) 56 | return loss 57 | 58 | 59 | def gen(): 60 | yield from word2vec_utils.batch_gen(DOWNLOAD_URL, EXPECTED_BYTES, 61 | VOCAB_SIZE, BATCH_SIZE, SKIP_WINDOW, 62 | VISUAL_FLD) 63 | 64 | def main(): 65 | dataset = tf.data.Dataset.from_generator(gen, (tf.int32, tf.int32), 66 | (tf.TensorShape([BATCH_SIZE]), 67 | tf.TensorShape([BATCH_SIZE, 1]))) 68 | optimizer = tf.train.GradientDescentOptimizer(LEARNING_RATE) 69 | model = Word2Vec(vocab_size=VOCAB_SIZE, embed_size=EMBED_SIZE) 70 | grad_fn = tfe.implicit_value_and_gradients(model.compute_loss) 71 | total_loss = 0.0 # for average loss in the last SKIP_STEP steps 72 | num_train_steps = 0 73 | while num_train_steps < NUM_TRAIN_STEPS: 74 | for center_words, target_words in tfe.Iterator(dataset): 75 | if num_train_steps >= NUM_TRAIN_STEPS: 76 | break 77 | loss_batch, grads = grad_fn(center_words, target_words) 78 | total_loss += loss_batch 79 | optimizer.apply_gradients(grads) 80 | if (num_train_steps + 1) % SKIP_STEP == 0: 81 | print('Average loss at step {}: {:5.1f}'.format( 82 | num_train_steps, total_loss / SKIP_STEP)) 83 | total_loss = 0.0 84 | num_train_steps += 1 85 | 86 | 87 | if __name__ == '__main__': 88 | main() 89 | -------------------------------------------------------------------------------- /examples/04_word2vec_eager_starter.py: -------------------------------------------------------------------------------- 1 | """ starter code for word2vec skip-gram model with NCE loss 2 | Eager execution 3 | CS 20: "TensorFlow for Deep Learning Research" 4 | cs20.stanford.edu 5 | Chip Huyen (chiphuyen@cs.stanford.edu) & Akshay Agrawal (akshayka@cs.stanford.edu) 6 | Lecture 04 7 | """ 8 | 9 | import os 10 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 11 | 12 | import numpy as np 13 | import tensorflow as tf 14 | import tensorflow.contrib.eager as tfe 15 | 16 | import utils 17 | import word2vec_utils 18 | 19 | # Enable eager execution! 20 | ############################# 21 | ########## TO DO ############ 22 | ############################# 23 | 24 | # Model hyperparameters 25 | VOCAB_SIZE = 50000 26 | BATCH_SIZE = 128 27 | EMBED_SIZE = 128 # dimension of the word embedding vectors 28 | SKIP_WINDOW = 1 # the context window 29 | NUM_SAMPLED = 64 # number of negative examples to sample 30 | LEARNING_RATE = 1.0 31 | NUM_TRAIN_STEPS = 100000 32 | VISUAL_FLD = 'visualization' 33 | SKIP_STEP = 5000 34 | 35 | # Parameters for downloading data 36 | DOWNLOAD_URL = 'http://mattmahoney.net/dc/text8.zip' 37 | EXPECTED_BYTES = 31344016 38 | 39 | class Word2Vec(object): 40 | def __init__(self, vocab_size, embed_size, num_sampled=NUM_SAMPLED): 41 | self.vocab_size = vocab_size 42 | self.num_sampled = num_sampled 43 | # Create the variables: an embedding matrix, nce_weight, and nce_bias 44 | ############################# 45 | ########## TO DO ############ 46 | ############################# 47 | self.embed_matrix = None 48 | self.nce_weight = None 49 | self.nce_bias = None 50 | 51 | def compute_loss(self, center_words, target_words): 52 | """Computes the forward pass of word2vec with the NCE loss.""" 53 | # Look up the embeddings for the center words 54 | ############################# 55 | ########## TO DO ############ 56 | ############################# 57 | embed = None 58 | 59 | # Compute the loss, using tf.reduce_mean and tf.nn.nce_loss 60 | ############################# 61 | ########## TO DO ############ 62 | ############################# 63 | loss = None 64 | return loss 65 | 66 | 67 | def gen(): 68 | yield from word2vec_utils.batch_gen(DOWNLOAD_URL, EXPECTED_BYTES, 69 | VOCAB_SIZE, BATCH_SIZE, SKIP_WINDOW, 70 | VISUAL_FLD) 71 | 72 | def main(): 73 | dataset = tf.data.Dataset.from_generator(gen, (tf.int32, tf.int32), 74 | (tf.TensorShape([BATCH_SIZE]), 75 | tf.TensorShape([BATCH_SIZE, 1]))) 76 | optimizer = tf.train.GradientDescentOptimizer(LEARNING_RATE) 77 | # Create the model 78 | ############################# 79 | ########## TO DO ############ 80 | ############################# 81 | model = None 82 | 83 | # Create the gradients function, using `tfe.implicit_value_and_gradients` 84 | ############################# 85 | ########## TO DO ############ 86 | ############################# 87 | grad_fn = None 88 | 89 | total_loss = 0.0 # for average loss in the last SKIP_STEP steps 90 | num_train_steps = 0 91 | while num_train_steps < NUM_TRAIN_STEPS: 92 | for center_words, target_words in tfe.Iterator(dataset): 93 | if num_train_steps >= NUM_TRAIN_STEPS: 94 | break 95 | 96 | # Compute the loss and gradients, and take an optimization step. 97 | ############################# 98 | ########## TO DO ############ 99 | ############################# 100 | 101 | if (num_train_steps + 1) % SKIP_STEP == 0: 102 | print('Average loss at step {}: {:5.1f}'.format( 103 | num_train_steps, total_loss / SKIP_STEP)) 104 | total_loss = 0.0 105 | num_train_steps += 1 106 | 107 | 108 | if __name__ == '__main__': 109 | main() 110 | -------------------------------------------------------------------------------- /examples/05_randomization.py: -------------------------------------------------------------------------------- 1 | """ Examples to demonstrate ops level randomization 2 | CS 20: "TensorFlow for Deep Learning Research" 3 | cs20.stanford.edu 4 | Chip Huyen (chiphuyen@cs.stanford.edu) 5 | Lecture 05 6 | """ 7 | import os 8 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 9 | 10 | import tensorflow as tf 11 | 12 | # Example 1: session keeps track of the random state 13 | c = tf.random_uniform([], -10, 10, seed=2) 14 | 15 | with tf.Session() as sess: 16 | print(sess.run(c)) # >> 3.574932 17 | print(sess.run(c)) # >> -5.9731865 18 | 19 | # Example 2: each new session will start the random state all over again. 20 | c = tf.random_uniform([], -10, 10, seed=2) 21 | 22 | with tf.Session() as sess: 23 | print(sess.run(c)) # >> 3.574932 24 | 25 | with tf.Session() as sess: 26 | print(sess.run(c)) # >> 3.574932 27 | 28 | # Example 3: with operation level random seed, each op keeps its own seed. 29 | c = tf.random_uniform([], -10, 10, seed=2) 30 | d = tf.random_uniform([], -10, 10, seed=2) 31 | 32 | with tf.Session() as sess: 33 | print(sess.run(c)) # >> 3.574932 34 | print(sess.run(d)) # >> 3.574932 35 | 36 | # Example 4: graph level random seed 37 | tf.set_random_seed(2) 38 | c = tf.random_uniform([], -10, 10) 39 | d = tf.random_uniform([], -10, 10) 40 | 41 | with tf.Session() as sess: 42 | print(sess.run(c)) # >> 9.123926 43 | print(sess.run(d)) # >> -4.5340395 44 | -------------------------------------------------------------------------------- /examples/05_variable_sharing.py: -------------------------------------------------------------------------------- 1 | """ Examples to demonstrate variable sharing 2 | CS 20: 'TensorFlow for Deep Learning Research' 3 | cs20.stanford.edu 4 | Chip Huyen (chiphuyen@cs.stanford.edu) 5 | Lecture 05 6 | """ 7 | import os 8 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 9 | 10 | import tensorflow as tf 11 | 12 | x1 = tf.truncated_normal([200, 100], name='x1') 13 | x2 = tf.truncated_normal([200, 100], name='x2') 14 | 15 | def two_hidden_layers(x): 16 | assert x.shape.as_list() == [200, 100] 17 | w1 = tf.Variable(tf.random_normal([100, 50]), name='h1_weights') 18 | b1 = tf.Variable(tf.zeros([50]), name='h1_biases') 19 | h1 = tf.matmul(x, w1) + b1 20 | assert h1.shape.as_list() == [200, 50] 21 | w2 = tf.Variable(tf.random_normal([50, 10]), name='h2_weights') 22 | b2 = tf.Variable(tf.zeros([10]), name='2_biases') 23 | logits = tf.matmul(h1, w2) + b2 24 | return logits 25 | 26 | def two_hidden_layers_2(x): 27 | assert x.shape.as_list() == [200, 100] 28 | w1 = tf.get_variable('h1_weights', [100, 50], initializer=tf.random_normal_initializer()) 29 | b1 = tf.get_variable('h1_biases', [50], initializer=tf.constant_initializer(0.0)) 30 | h1 = tf.matmul(x, w1) + b1 31 | assert h1.shape.as_list() == [200, 50] 32 | w2 = tf.get_variable('h2_weights', [50, 10], initializer=tf.random_normal_initializer()) 33 | b2 = tf.get_variable('h2_biases', [10], initializer=tf.constant_initializer(0.0)) 34 | logits = tf.matmul(h1, w2) + b2 35 | return logits 36 | 37 | # logits1 = two_hidden_layers(x1) 38 | # logits2 = two_hidden_layers(x2) 39 | 40 | # logits1 = two_hidden_layers_2(x1) 41 | # logits2 = two_hidden_layers_2(x2) 42 | 43 | # with tf.variable_scope('two_layers') as scope: 44 | # logits1 = two_hidden_layers_2(x1) 45 | # scope.reuse_variables() 46 | # logits2 = two_hidden_layers_2(x2) 47 | 48 | # with tf.variable_scope('two_layers') as scope: 49 | # logits1 = two_hidden_layers_2(x1) 50 | # scope.reuse_variables() 51 | # logits2 = two_hidden_layers_2(x2) 52 | 53 | def fully_connected(x, output_dim, scope): 54 | with tf.variable_scope(scope, reuse=tf.AUTO_REUSE) as scope: 55 | w = tf.get_variable('weights', [x.shape[1], output_dim], initializer=tf.random_normal_initializer()) 56 | b = tf.get_variable('biases', [output_dim], initializer=tf.constant_initializer(0.0)) 57 | return tf.matmul(x, w) + b 58 | 59 | def two_hidden_layers(x): 60 | h1 = fully_connected(x, 50, 'h1') 61 | h2 = fully_connected(h1, 10, 'h2') 62 | 63 | with tf.variable_scope('two_layers') as scope: 64 | logits1 = two_hidden_layers(x1) 65 | # scope.reuse_variables() 66 | logits2 = two_hidden_layers(x2) 67 | 68 | writer = tf.summary.FileWriter('./graphs/cool_variables', tf.get_default_graph()) 69 | writer.close() -------------------------------------------------------------------------------- /examples/07_run_kernels.py: -------------------------------------------------------------------------------- 1 | """ 2 | Simple examples of convolution to do some basic filters 3 | Also demonstrates the use of TensorFlow data readers. 4 | 5 | We will use some popular filters for our image. 6 | It seems to be working with grayscale images, but not with rgb images. 7 | It's probably because I didn't choose the right kernels for rgb images. 8 | 9 | kernels for rgb images have dimensions 3 x 3 x 3 x 3 10 | kernels for grayscale images have dimensions 3 x 3 x 1 x 1 11 | 12 | CS 20: "TensorFlow for Deep Learning Research" 13 | cs20.stanford.edu 14 | Chip Huyen (chiphuyen@cs.stanford.edu) 15 | Lecture 07 16 | """ 17 | import os 18 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 19 | 20 | import sys 21 | sys.path.append('..') 22 | 23 | from matplotlib import gridspec as gridspec 24 | from matplotlib import pyplot as plt 25 | import tensorflow as tf 26 | 27 | import kernels 28 | 29 | def read_one_image(filename): 30 | ''' This method is to show how to read image from a file into a tensor. 31 | The output is a tensor object. 32 | ''' 33 | image_string = tf.read_file(filename) 34 | image_decoded = tf.image.decode_image(image_string) 35 | image = tf.cast(image_decoded, tf.float32) / 256.0 36 | return image 37 | 38 | def convolve(image, kernels, rgb=True, strides=[1, 3, 3, 1], padding='SAME'): 39 | images = [image[0]] 40 | for i, kernel in enumerate(kernels): 41 | filtered_image = tf.nn.conv2d(image, 42 | kernel, 43 | strides=strides, 44 | padding=padding)[0] 45 | if i == 2: 46 | filtered_image = tf.minimum(tf.nn.relu(filtered_image), 255) 47 | images.append(filtered_image) 48 | return images 49 | 50 | def show_images(images, rgb=True): 51 | gs = gridspec.GridSpec(1, len(images)) 52 | for i, image in enumerate(images): 53 | plt.subplot(gs[0, i]) 54 | if rgb: 55 | plt.imshow(image) 56 | else: 57 | image = image.reshape(image.shape[0], image.shape[1]) 58 | plt.imshow(image, cmap='gray') 59 | plt.axis('off') 60 | plt.show() 61 | 62 | def main(): 63 | rgb = False 64 | if rgb: 65 | kernels_list = [kernels.BLUR_FILTER_RGB, 66 | kernels.SHARPEN_FILTER_RGB, 67 | kernels.EDGE_FILTER_RGB, 68 | kernels.TOP_SOBEL_RGB, 69 | kernels.EMBOSS_FILTER_RGB] 70 | else: 71 | kernels_list = [kernels.BLUR_FILTER, 72 | kernels.SHARPEN_FILTER, 73 | kernels.EDGE_FILTER, 74 | kernels.TOP_SOBEL, 75 | kernels.EMBOSS_FILTER] 76 | 77 | kernels_list = kernels_list[1:] 78 | image = read_one_image('data/friday.jpg') 79 | if not rgb: 80 | image = tf.image.rgb_to_grayscale(image) 81 | image = tf.expand_dims(image, 0) # make it into a batch of 1 element 82 | images = convolve(image, kernels_list, rgb) 83 | with tf.Session() as sess: 84 | images = sess.run(images) # convert images from tensors to float values 85 | show_images(images, rgb) 86 | 87 | if __name__ == '__main__': 88 | main() -------------------------------------------------------------------------------- /examples/kernels.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import tensorflow as tf 3 | 4 | a = np.zeros([3, 3, 3, 3]) 5 | a[1, 1, :, :] = 0.25 6 | a[0, 1, :, :] = 0.125 7 | a[1, 0, :, :] = 0.125 8 | a[2, 1, :, :] = 0.125 9 | a[1, 2, :, :] = 0.125 10 | a[0, 0, :, :] = 0.0625 11 | a[0, 2, :, :] = 0.0625 12 | a[2, 0, :, :] = 0.0625 13 | a[2, 2, :, :] = 0.0625 14 | 15 | BLUR_FILTER_RGB = tf.constant(a, dtype=tf.float32) 16 | 17 | a = np.zeros([3, 3, 1, 1]) 18 | # a[1, 1, :, :] = 0.25 19 | # a[0, 1, :, :] = 0.125 20 | # a[1, 0, :, :] = 0.125 21 | # a[2, 1, :, :] = 0.125 22 | # a[1, 2, :, :] = 0.125 23 | # a[0, 0, :, :] = 0.0625 24 | # a[0, 2, :, :] = 0.0625 25 | # a[2, 0, :, :] = 0.0625 26 | # a[2, 2, :, :] = 0.0625 27 | a[1, 1, :, :] = 1.0 28 | a[0, 1, :, :] = 1.0 29 | a[1, 0, :, :] = 1.0 30 | a[2, 1, :, :] = 1.0 31 | a[1, 2, :, :] = 1.0 32 | a[0, 0, :, :] = 1.0 33 | a[0, 2, :, :] = 1.0 34 | a[2, 0, :, :] = 1.0 35 | a[2, 2, :, :] = 1.0 36 | BLUR_FILTER = tf.constant(a, dtype=tf.float32) 37 | 38 | a = np.zeros([3, 3, 3, 3]) 39 | a[1, 1, :, :] = 5 40 | a[0, 1, :, :] = -1 41 | a[1, 0, :, :] = -1 42 | a[2, 1, :, :] = -1 43 | a[1, 2, :, :] = -1 44 | 45 | SHARPEN_FILTER_RGB = tf.constant(a, dtype=tf.float32) 46 | 47 | a = np.zeros([3, 3, 1, 1]) 48 | a[1, 1, :, :] = 5 49 | a[0, 1, :, :] = -1 50 | a[1, 0, :, :] = -1 51 | a[2, 1, :, :] = -1 52 | a[1, 2, :, :] = -1 53 | 54 | SHARPEN_FILTER = tf.constant(a, dtype=tf.float32) 55 | 56 | # a = np.zeros([3, 3, 3, 3]) 57 | # a[:, :, :, :] = -1 58 | # a[1, 1, :, :] = 8 59 | 60 | # EDGE_FILTER_RGB = tf.constant(a, dtype=tf.float32) 61 | 62 | EDGE_FILTER_RGB = tf.constant([ 63 | [[[ -1., 0., 0.], [ 0., -1., 0.], [ 0., 0., -1.]], 64 | [[ -1., 0., 0.], [ 0., -1., 0.], [ 0., 0., -1.]], 65 | [[ -1., 0., 0.], [ 0., -1., 0.], [ 0., 0., -1.]]], 66 | [[[ -1., 0., 0.], [ 0., -1., 0.], [ 0., 0., -1.]], 67 | [[ 8., 0., 0.], [ 0., 8., 0.], [ 0., 0., 8.]], 68 | [[ -1., 0., 0.], [ 0., -1., 0.], [ 0., 0., -1.]]], 69 | [[[ -1., 0., 0.], [ 0., -1., 0.], [ 0., 0., -1.]], 70 | [[ -1., 0., 0.], [ 0., -1., 0.], [ 0., 0., -1.]], 71 | [[ -1., 0., 0.], [ 0., -1., 0.], [ 0., 0., -1.]]] 72 | ]) 73 | 74 | a = np.zeros([3, 3, 1, 1]) 75 | # a[:, :, :, :] = -1 76 | # a[1, 1, :, :] = 8 77 | a[0, 1, :, :] = -1 78 | a[1, 0, :, :] = -1 79 | a[1, 2, :, :] = -1 80 | a[2, 1, :, :] = -1 81 | a[1, 1, :, :] = 4 82 | 83 | EDGE_FILTER = tf.constant(a, dtype=tf.float32) 84 | 85 | a = np.zeros([3, 3, 3, 3]) 86 | a[0, :, :, :] = 1 87 | a[0, 1, :, :] = 2 # originally 2 88 | a[2, :, :, :] = -1 89 | a[2, 1, :, :] = -2 90 | 91 | TOP_SOBEL_RGB = tf.constant(a, dtype=tf.float32) 92 | 93 | a = np.zeros([3, 3, 1, 1]) 94 | a[0, :, :, :] = 1 95 | a[0, 1, :, :] = 2 # originally 2 96 | a[2, :, :, :] = -1 97 | a[2, 1, :, :] = -2 98 | 99 | TOP_SOBEL = tf.constant(a, dtype=tf.float32) 100 | 101 | a = np.zeros([3, 3, 3, 3]) 102 | a[0, 0, :, :] = -2 103 | a[0, 1, :, :] = -1 104 | a[1, 0, :, :] = -1 105 | a[1, 1, :, :] = 1 106 | a[1, 2, :, :] = 1 107 | a[2, 1, :, :] = 1 108 | a[2, 2, :, :] = 2 109 | 110 | EMBOSS_FILTER_RGB = tf.constant(a, dtype=tf.float32) 111 | 112 | a = np.zeros([3, 3, 1, 1]) 113 | a[0, 0, :, :] = -2 114 | a[0, 1, :, :] = -1 115 | a[1, 0, :, :] = -1 116 | a[1, 1, :, :] = 1 117 | a[1, 2, :, :] = 1 118 | a[2, 1, :, :] = 1 119 | a[2, 2, :, :] = 2 120 | EMBOSS_FILTER = tf.constant(a, dtype=tf.float32) -------------------------------------------------------------------------------- /examples/utils.py: -------------------------------------------------------------------------------- 1 | import os 2 | import gzip 3 | import shutil 4 | import struct 5 | import urllib 6 | 7 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 8 | 9 | from matplotlib import pyplot as plt 10 | import numpy as np 11 | import tensorflow as tf 12 | 13 | def huber_loss(labels, predictions, delta=14.0): 14 | residual = tf.abs(labels - predictions) 15 | def f1(): return 0.5 * tf.square(residual) 16 | def f2(): return delta * residual - 0.5 * tf.square(delta) 17 | return tf.cond(residual < delta, f1, f2) 18 | 19 | def safe_mkdir(path): 20 | """ Create a directory if there isn't one already. """ 21 | try: 22 | os.mkdir(path) 23 | except OSError: 24 | pass 25 | 26 | def read_birth_life_data(filename): 27 | """ 28 | Read in birth_life_2010.txt and return: 29 | data in the form of NumPy array 30 | n_samples: number of samples 31 | """ 32 | text = open(filename, 'r').readlines()[1:] 33 | data = [line[:-1].split('\t') for line in text] 34 | births = [float(line[1]) for line in data] 35 | lifes = [float(line[2]) for line in data] 36 | data = list(zip(births, lifes)) 37 | n_samples = len(data) 38 | data = np.asarray(data, dtype=np.float32) 39 | return data, n_samples 40 | 41 | def download_one_file(download_url, 42 | local_dest, 43 | expected_byte=None, 44 | unzip_and_remove=False): 45 | """ 46 | Download the file from download_url into local_dest 47 | if the file doesn't already exists. 48 | If expected_byte is provided, check if 49 | the downloaded file has the same number of bytes. 50 | If unzip_and_remove is True, unzip the file and remove the zip file 51 | """ 52 | if os.path.exists(local_dest) or os.path.exists(local_dest[:-3]): 53 | print('%s already exists' %local_dest) 54 | else: 55 | print('Downloading %s' %download_url) 56 | local_file, _ = urllib.request.urlretrieve(download_url, local_dest) 57 | file_stat = os.stat(local_dest) 58 | if expected_byte: 59 | if file_stat.st_size == expected_byte: 60 | print('Successfully downloaded %s' %local_dest) 61 | if unzip_and_remove: 62 | with gzip.open(local_dest, 'rb') as f_in, open(local_dest[:-3],'wb') as f_out: 63 | shutil.copyfileobj(f_in, f_out) 64 | os.remove(local_dest) 65 | else: 66 | print('The downloaded file has unexpected number of bytes') 67 | 68 | def download_mnist(path): 69 | """ 70 | Download and unzip the dataset mnist if it's not already downloaded 71 | Download from http://yann.lecun.com/exdb/mnist 72 | """ 73 | safe_mkdir(path) 74 | url = 'http://yann.lecun.com/exdb/mnist' 75 | filenames = ['train-images-idx3-ubyte.gz', 76 | 'train-labels-idx1-ubyte.gz', 77 | 't10k-images-idx3-ubyte.gz', 78 | 't10k-labels-idx1-ubyte.gz'] 79 | expected_bytes = [9912422, 28881, 1648877, 4542] 80 | 81 | for filename, byte in zip(filenames, expected_bytes): 82 | download_url = os.path.join(url, filename) 83 | local_dest = os.path.join(path, filename) 84 | download_one_file(download_url, local_dest, byte, True) 85 | 86 | def parse_data(path, dataset, flatten): 87 | if dataset != 'train' and dataset != 't10k': 88 | raise NameError('dataset must be train or t10k') 89 | 90 | label_file = os.path.join(path, dataset + '-labels-idx1-ubyte') 91 | with open(label_file, 'rb') as file: 92 | _, num = struct.unpack(">II", file.read(8)) 93 | labels = np.fromfile(file, dtype=np.int8) #int8 94 | new_labels = np.zeros((num, 10)) 95 | new_labels[np.arange(num), labels] = 1 96 | 97 | img_file = os.path.join(path, dataset + '-images-idx3-ubyte') 98 | with open(img_file, 'rb') as file: 99 | _, num, rows, cols = struct.unpack(">IIII", file.read(16)) 100 | imgs = np.fromfile(file, dtype=np.uint8).reshape(num, rows, cols) #uint8 101 | imgs = imgs.astype(np.float32) / 255.0 102 | if flatten: 103 | imgs = imgs.reshape([num, -1]) 104 | 105 | return imgs, new_labels 106 | 107 | def read_mnist(path, flatten=True, num_train=55000): 108 | """ 109 | Read in the mnist dataset, given that the data is stored in path 110 | Return two tuples of numpy arrays 111 | ((train_imgs, train_labels), (test_imgs, test_labels)) 112 | """ 113 | imgs, labels = parse_data(path, 'train', flatten) 114 | indices = np.random.permutation(labels.shape[0]) 115 | train_idx, val_idx = indices[:num_train], indices[num_train:] 116 | train_img, train_labels = imgs[train_idx, :], labels[train_idx, :] 117 | val_img, val_labels = imgs[val_idx, :], labels[val_idx, :] 118 | test = parse_data(path, 't10k', flatten) 119 | return (train_img, train_labels), (val_img, val_labels), test 120 | 121 | def get_mnist_dataset(batch_size): 122 | # Step 1: Read in data 123 | mnist_folder = 'data/mnist' 124 | download_mnist(mnist_folder) 125 | train, val, test = read_mnist(mnist_folder, flatten=False) 126 | 127 | # Step 2: Create datasets and iterator 128 | train_data = tf.data.Dataset.from_tensor_slices(train) 129 | train_data = train_data.shuffle(10000) # if you want to shuffle your data 130 | train_data = train_data.batch(batch_size) 131 | 132 | test_data = tf.data.Dataset.from_tensor_slices(test) 133 | test_data = test_data.batch(batch_size) 134 | 135 | return train_data, test_data 136 | 137 | def show(image): 138 | """ 139 | Render a given numpy.uint8 2D array of pixel data. 140 | """ 141 | plt.imshow(image, cmap='gray') 142 | plt.show() -------------------------------------------------------------------------------- /examples/word2vec_utils.py: -------------------------------------------------------------------------------- 1 | from collections import Counter 2 | import random 3 | import os 4 | import sys 5 | sys.path.append('..') 6 | import zipfile 7 | 8 | import numpy as np 9 | from six.moves import urllib 10 | import tensorflow as tf 11 | 12 | import utils 13 | 14 | def read_data(file_path): 15 | """ Read data into a list of tokens 16 | There should be 17,005,207 tokens 17 | """ 18 | with zipfile.ZipFile(file_path) as f: 19 | words = tf.compat.as_str(f.read(f.namelist()[0])).split() 20 | return words 21 | 22 | def build_vocab(words, vocab_size, visual_fld): 23 | """ Build vocabulary of VOCAB_SIZE most frequent words and write it to 24 | visualization/vocab.tsv 25 | """ 26 | utils.safe_mkdir(visual_fld) 27 | file = open(os.path.join(visual_fld, 'vocab.tsv'), 'w') 28 | 29 | dictionary = dict() 30 | count = [('UNK', -1)] 31 | index = 0 32 | count.extend(Counter(words).most_common(vocab_size - 1)) 33 | 34 | for word, _ in count: 35 | dictionary[word] = index 36 | index += 1 37 | file.write(word + '\n') 38 | 39 | index_dictionary = dict(zip(dictionary.values(), dictionary.keys())) 40 | file.close() 41 | return dictionary, index_dictionary 42 | 43 | def convert_words_to_index(words, dictionary): 44 | """ Replace each word in the dataset with its index in the dictionary """ 45 | return [dictionary[word] if word in dictionary else 0 for word in words] 46 | 47 | def generate_sample(index_words, context_window_size): 48 | """ Form training pairs according to the skip-gram model. """ 49 | for index, center in enumerate(index_words): 50 | context = random.randint(1, context_window_size) 51 | # get a random target before the center word 52 | for target in index_words[max(0, index - context): index]: 53 | yield center, target 54 | # get a random target after the center wrod 55 | for target in index_words[index + 1: index + context + 1]: 56 | yield center, target 57 | 58 | def most_common_words(visual_fld, num_visualize): 59 | """ create a list of num_visualize most frequent words to visualize on TensorBoard. 60 | saved to visualization/vocab_[num_visualize].tsv 61 | """ 62 | words = open(os.path.join(visual_fld, 'vocab.tsv'), 'r').readlines()[:num_visualize] 63 | words = [word for word in words] 64 | file = open(os.path.join(visual_fld, 'vocab_' + str(num_visualize) + '.tsv'), 'w') 65 | for word in words: 66 | file.write(word) 67 | file.close() 68 | 69 | def batch_gen(download_url, expected_byte, vocab_size, batch_size, 70 | skip_window, visual_fld): 71 | local_dest = 'data/text8.zip' 72 | utils.download_one_file(download_url, local_dest, expected_byte) 73 | words = read_data(local_dest) 74 | dictionary, _ = build_vocab(words, vocab_size, visual_fld) 75 | index_words = convert_words_to_index(words, dictionary) 76 | del words # to save memory 77 | single_gen = generate_sample(index_words, skip_window) 78 | 79 | while True: 80 | center_batch = np.zeros(batch_size, dtype=np.int32) 81 | target_batch = np.zeros([batch_size, 1]) 82 | for index in range(batch_size): 83 | center_batch[index], target_batch[index] = next(single_gen) 84 | yield center_batch, target_batch 85 | -------------------------------------------------------------------------------- /setup/requirements.txt: -------------------------------------------------------------------------------- 1 | tensorflow==1.4.1 2 | scipy==1.0.0 3 | scikit-learn==0.19.1 4 | matplotlib==2.1.1 5 | xlrd==1.1.0 6 | ipdb==0.10.3 7 | Pillow==5.0.0 8 | lxml==4.1.1 -------------------------------------------------------------------------------- /setup/setup_instruction.md: -------------------------------------------------------------------------------- 1 | Please follow the official instruction to install TensorFlow [here](https://www.tensorflow.org/install/). For this course, I will use Python 3.6 and TensorFlow 1.4. You’re welcome to use either Python 2 or Python 3 for the assignments. The starter code, though, will be in Python 3.6. You don't need GPU for most code examples in this course, though having GPU won't hurt. If you install TensorFlow on your local machine, my ecommendation is always set up Tensorflow using virtualenv. 2 | 3 | For the list of dependencies, please consult the file requirements.txt. This list will be updated as the course progresses. 4 | 5 | There are a few things to note: 6 | - As of version 1.2, TensorFlow no longer provides GPU support on macOS. 7 | - On macOS, Python 3.6 might gives warning but still works. 8 | - TensorFlow with GPU support will only work with CUDA® Toolkit 8.0 and cuDNN v6.0, not the newest CUDA and cnDNN version. Make sure that you install the correct CUDA and cuDNN versions to avoid frustrating issues. 9 | - On Windows, TensorFlow supports only 64-bit Python 3.5 anx Python 3.6. 10 | - If you see the warning: 11 | ```bash 12 | Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA 13 | ``` 14 | it's because you didn't install TensorFlow from sources to take advantage of all these settings. You can choose to install TensorFlow from sources -- the process might take up to 30 minutes. To silence the warning, add this before importing TensorFlow:
15 | 16 | ```bash 17 | import os 18 | os.environ['TF_CPP_MIN_LOG_LEVEL']='2' 19 | ``` 20 | 21 | - If you want to install TensorFlow from sources, keep in mind that TensorFlow doesn't officially support building TensorFlow on Windows. On Windows, you may try using the highly experimental Bazel on Windows or TensorFlow CMake build. 22 | 23 | Below is a simpler instruction on how to install TensorFlow on macOS. If you have any problem installing Tensorflow, feel free to post it on [Piazza](piazza.com/stanford/winter2018/cs20) 24 | 25 | If you get “permission denied” error in any command, use “sudo” in front of that command. 26 | 27 | You will need pip3 (or pip if you use Python2), and virtualenv. 28 | 29 | Step 1: install python3 and pip3. Skip this step if you already have both. You can find the official instruction [here](http://docs.python-guide.org/en/latest/starting/install3/osx/) 30 | 31 | Step 2: upgrade six 32 | ```bash 33 | $ sudo easy_install --upgrade six 34 | ``` 35 | 36 | Step 3: install virtualenv. Skip this step if you already have virtualenv 37 | ```bash 38 | $ pip3 install virtualenv 39 | ``` 40 | 41 | Step 4: set up a project directory. You will do all work for this class in this directory 42 | ```bash 43 | $ mkdir cs20 44 | ``` 45 | 46 | Step 5: set up virtual environment with python3 47 | ```bash 48 | $ cd cs20 49 | $ python3 -m venv .env 50 | ``` 51 | These commands create a venv subdirectory in your project where everything is installed. 52 | 53 | Step 6: activate the virtual environment 54 | ```bash 55 | $ source .env/bin/activate 56 | ``` 57 | 58 | If you type: 59 | ```bash 60 | $ pip3 freeze 61 | ``` 62 | 63 | You will see that nothing is shown, which means no package is installed in your virtual environment. So you have to install all packages that you need. For the list of packages you need for this class, you can see/download the list of requirements in [the setup folder of this repository](https://github.com/chiphuyen/stanford-tensorflow-tutorials/blob/master/setup/requirements.txt). 64 | 65 | Step 7: Install Tensorflow and other dependencies 66 | ```bash 67 | $ pip3 install -r requirements.txt 68 | ``` 69 | 70 | Step n: 71 | To exit the virtual environment, use: 72 | ```bash 73 | $ deactivate 74 | ``` 75 | 76 | ### Other options 77 | #### Floydhub 78 | Floydhub has a clean, GitHub-like interface that allows you to create and run TensorFlow projects. 79 | 80 | # Possible set up problems 81 | ## Matplotlib 82 | If you have problem with using Matplotlib in virtual environment, here are two simple ways to fix.
83 | 1. If you installed matplotlib using pip, there is a directory in you root called ~/.matplotlib. 84 | Go there and create a file ~/.matplotlib/matplotlibrc there and add the following code: ```backend: TkAgg``` 85 | 2. After importing matplotlib, simply add: ```matplotlib.use("TkAgg")``` 86 | 87 | If you run into more problems, feel free to post your questions on [Piazza](https://piazza.com/stanford/winter2018/cs20) or email us cs20-win1718-staff@lists.stanford.edu. --------------------------------------------------------------------------------