├── .gitignore
├── README.md
├── code
    ├── ae
    │   ├── __init__.py
    │   ├── autoencoder.py
    │   ├── autoencoder_test.py
    │   └── utils
    │   │   ├── __init__.py
    │   │   ├── data.py
    │   │   ├── eval.py
    │   │   ├── flags.py
    │   │   ├── start_tensorboard.py
    │   │   └── utils.py
    ├── requirements.txt
    └── run.py
├── cpu
    └── Dockerfile
├── filters_1.png
├── setup_linux
├── setup_mac
└── tb_hist.png


/.gitignore:
--------------------------------------------------------------------------------
1 | .idea/
2 | nohup.out
3 | *checkpoint*
4 | *.pyc
5 | tbpid
6 | venv/*
7 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | # Deep Autoencoder with TensorFlow
 2 | 
 3 | <p align="center">
 4 |    <img src="filters_1.png" alt="Some First Layer Filters"/>
 5 | </p>
 6 | <p align="center">
 7 | A selection of first layer weight filters learned during the pretraining
 8 | </p>
 9 | 
10 | ## Introduction
11 | The purpose of this repo is to explore the functionality of Google's recently open-sourced
12 | "sofware library for numerical computation using data flow graphs", 
13 | [TensorFlow](https://www.tensorflow.org/). We use the library to train
14 | a deep autoencoder on the MNIST digit data set. For background and a similar implementation using 
15 | [Theano](http://deeplearning.net/software/theano/) see the tutorial at [http://www.deeplearning.net/tutorial/SdA.html](http://www.deeplearning.net/tutorial/SdA.html).
16 | 
17 | The main training code can be found in [autoencoder.py](https://github.com/cmgreen210/TensorFlowDeepAutoencoder/blob/master/code/ae/autoencoder.py) along with the AutoEncoder class that creates and manages the Variables and Tensors used.
18 | 
19 | ## Docker Setup (CPU version only for the time being)
20 | In order to avoid platform issues it's highly encouraged that you run
21 | the example code in a [Docker](https://www.docker.com/) container. Follow
22 | the Docker installation instructions on the website. Then run:
23 | 
24 | ```bash
25 | $ git clone https://github.com/cmgreen210/TensorFlowDeepAutoencoder
26 | $ cd TensorFlowDeepAutoencoder
27 | $ docker build -t tfdae -f cpu/Dockerfile .
28 | $ docker run -it -p 80:6006 tfdae python run.py
29 | ```
30 | 
31 | Navigate to <a href="http://localhost:80" target="_blank">http://localhost:80</a>
32 | to explore [TensorBoard](https://www.tensorflow.org/versions/master/how_tos/summaries_and_tensorboard/index.html#tensorboard-visualizing-learning) and view the training progress.
33 | <p align="center">
34 |    <img src="tb_hist.png" alt="TensorBoard Histograms"/>
35 | </p>
36 | <p align="center">
37 | View of TensorBoard's display of weight and bias parameter progress.
38 | </p>
39 | ## Customizing
40 | You can play around with the run options, including the neural net size and shape, input corruption, learning rates, etc.
41 | in [flags.py](https://github.com/cmgreen210/TensorFlowDeepAutoencoder/blob/master/code/ae/utils/flags.py).
42 | 
43 | ## Old Setup
44 | It is expected that Python2.7 is installed and your default python version.
45 | ### Ubuntu/Linux
46 | ```bash
47 | $ git clone https://github.com/cmgreen210/TensorFlowDeepAutoencoder
48 | $ cd TensorFlowDeepAutoencoder
49 | $ sudo chmod +x setup_linux
50 | $ sudo ./setup_linux  # If you want GPU version specify -g or --gpu
51 | $ source venv/bin/activate 
52 | ```
53 | ### Mac OS X
54 | ```bash
55 | $ git clone https://github.com/cmgreen210/TensorFlowDeepAutoencoder
56 | $ cd TensorFlowDeepAutoencoder
57 | $ sudo chmod +x setup_mac
58 | $ sudo ./setup_mac
59 | $ source venv/bin/activate 
60 | ```
61 | ## Run
62 | To run the default example execute the following command. 
63 | NOTE: this will take a very long time if you are running on a CPU as opposed to a GPU
64 | ```bash
65 | $ python code/run.py
66 | ```
67 | 
68 | Navigate to <a href="http://localhost:6006" target="_blank">http://localhost:6006</a>
69 | to explore [TensorBoard](https://www.tensorflow.org/versions/master/how_tos/summaries_and_tensorboard/index.html#tensorboard-visualizing-learning) and view training progress.
70 | <p align="center">
71 |    <img src="tb_hist.png" alt="TensorBoard Histograms"/>
72 | </p>
73 | <p align="center">
74 | View of TensorBoard's display of weight and bias parameter progress.
75 | </p>
76 | ## Customizing
77 | You can play around with the run options, including the neural net size and shape, input corruption, learning rates, etc.
78 | in [flags.py](https://github.com/cmgreen210/TensorFlowDeepAutoencoder/blob/master/code/ae/utils/flags.py).
79 | 


--------------------------------------------------------------------------------
/code/ae/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/cmgreen210/TensorFlowDeepAutoencoder/5298ec437689ba7ecb59229599141549ef6a6a1d/code/ae/__init__.py


--------------------------------------------------------------------------------
/code/ae/autoencoder.py:
--------------------------------------------------------------------------------
  1 | from __future__ import division
  2 | from __future__ import print_function
  3 | import time
  4 | from os.path import join as pjoin
  5 | 
  6 | import numpy as np
  7 | import tensorflow as tf
  8 | from utils.data import fill_feed_dict_ae, read_data_sets_pretraining
  9 | from utils.data import read_data_sets, fill_feed_dict
 10 | from utils.flags import FLAGS
 11 | from utils.eval import loss_supervised, evaluation, do_eval_summary
 12 | from utils.utils import tile_raster_images
 13 | 
 14 | 
 15 | class AutoEncoder(object):
 16 |   """Generic deep autoencoder.
 17 | 
 18 |   Autoencoder used for full training cycle, including
 19 |   unsupervised pretraining layers and final fine tuning.
 20 |   The user specifies the structure of the neural net
 21 |   by specifying number of inputs, the number of hidden
 22 |   units for each layer and the number of final output
 23 |   logits.
 24 |   """
 25 |   _weights_str = "weights{0}"
 26 |   _biases_str = "biases{0}"
 27 | 
 28 |   def __init__(self, shape, sess):
 29 |     """Autoencoder initializer
 30 | 
 31 |     Args:
 32 |       shape: list of ints specifying
 33 |               num input, hidden1 units,...hidden_n units, num logits
 34 |       sess: tensorflow session object to use
 35 |     """
 36 |     self.__shape = shape  # [input_dim,hidden1_dim,...,hidden_n_dim,output_dim]
 37 |     self.__num_hidden_layers = len(self.__shape) - 2
 38 | 
 39 |     self.__variables = {}
 40 |     self.__sess = sess
 41 | 
 42 |     self._setup_variables()
 43 | 
 44 |   @property
 45 |   def shape(self):
 46 |     return self.__shape
 47 | 
 48 |   @property
 49 |   def num_hidden_layers(self):
 50 |     return self.__num_hidden_layers
 51 | 
 52 |   @property
 53 |   def session(self):
 54 |     return self.__sess
 55 | 
 56 |   def __getitem__(self, item):
 57 |     """Get autoencoder tf variable
 58 | 
 59 |     Returns the specified variable created by this object.
 60 |     Names are weights#, biases#, biases#_out, weights#_fixed,
 61 |     biases#_fixed.
 62 | 
 63 |     Args:
 64 |      item: string, variables internal name
 65 |     Returns:
 66 |      Tensorflow variable
 67 |     """
 68 |     return self.__variables[item]
 69 | 
 70 |   def __setitem__(self, key, value):
 71 |     """Store a tensorflow variable
 72 | 
 73 |     NOTE: Don't call this explicity. It should
 74 |     be used only internally when setting up
 75 |     variables.
 76 | 
 77 |     Args:
 78 |       key: string, name of variable
 79 |       value: tensorflow variable
 80 |     """
 81 |     self.__variables[key] = value
 82 | 
 83 |   def _setup_variables(self):
 84 |     with tf.name_scope("autoencoder_variables"):
 85 |       for i in xrange(self.__num_hidden_layers + 1):
 86 |         # Train weights
 87 |         name_w = self._weights_str.format(i + 1)
 88 |         w_shape = (self.__shape[i], self.__shape[i + 1])
 89 |         a = tf.mul(4.0, tf.sqrt(6.0 / (w_shape[0] + w_shape[1])))
 90 |         w_init = tf.random_uniform(w_shape, -1 * a, a)
 91 |         self[name_w] = tf.Variable(w_init,
 92 |                                    name=name_w,
 93 |                                    trainable=True)
 94 |         # Train biases
 95 |         name_b = self._biases_str.format(i + 1)
 96 |         b_shape = (self.__shape[i + 1],)
 97 |         b_init = tf.zeros(b_shape)
 98 |         self[name_b] = tf.Variable(b_init, trainable=True, name=name_b)
 99 | 
100 |         if i < self.__num_hidden_layers:
101 |           # Hidden layer fixed weights (after pretraining before fine tuning)
102 |           self[name_w + "_fixed"] = tf.Variable(tf.identity(self[name_w]),
103 |                                                 name=name_w + "_fixed",
104 |                                                 trainable=False)
105 | 
106 |           # Hidden layer fixed biases
107 |           self[name_b + "_fixed"] = tf.Variable(tf.identity(self[name_b]),
108 |                                                 name=name_b + "_fixed",
109 |                                                 trainable=False)
110 | 
111 |           # Pretraining output training biases
112 |           name_b_out = self._biases_str.format(i + 1) + "_out"
113 |           b_shape = (self.__shape[i],)
114 |           b_init = tf.zeros(b_shape)
115 |           self[name_b_out] = tf.Variable(b_init,
116 |                                          trainable=True,
117 |                                          name=name_b_out)
118 | 
119 |   def _w(self, n, suffix=""):
120 |     return self[self._weights_str.format(n) + suffix]
121 | 
122 |   def _b(self, n, suffix=""):
123 |     return self[self._biases_str.format(n) + suffix]
124 | 
125 |   def get_variables_to_init(self, n):
126 |     """Return variables that need initialization
127 | 
128 |     This method aides in the initialization of variables
129 |     before training begins at step n. The returned
130 |     list should be than used as the input to
131 |     tf.initialize_variables
132 | 
133 |     Args:
134 |       n: int giving step of training
135 |     """
136 |     assert n > 0
137 |     assert n <= self.__num_hidden_layers + 1
138 | 
139 |     vars_to_init = [self._w(n), self._b(n)]
140 | 
141 |     if n <= self.__num_hidden_layers:
142 |       vars_to_init.append(self._b(n, "_out"))
143 | 
144 |     if 1 < n <= self.__num_hidden_layers:
145 |       vars_to_init.append(self._w(n - 1, "_fixed"))
146 |       vars_to_init.append(self._b(n - 1, "_fixed"))
147 | 
148 |     return vars_to_init
149 | 
150 |   @staticmethod
151 |   def _activate(x, w, b, transpose_w=False):
152 |     y = tf.sigmoid(tf.nn.bias_add(tf.matmul(x, w, transpose_b=transpose_w), b))
153 |     return y
154 | 
155 |   def pretrain_net(self, input_pl, n, is_target=False):
156 |     """Return net for step n training or target net
157 | 
158 |     Args:
159 |       input_pl:  tensorflow placeholder of AE inputs
160 |       n:         int specifying pretrain step
161 |       is_target: bool specifying if required tensor
162 |                   should be the target tensor
163 |     Returns:
164 |       Tensor giving pretraining net or pretraining target
165 |     """
166 |     assert n > 0
167 |     assert n <= self.__num_hidden_layers
168 | 
169 |     last_output = input_pl
170 |     for i in xrange(n - 1):
171 |       w = self._w(i + 1, "_fixed")
172 |       b = self._b(i + 1, "_fixed")
173 | 
174 |       last_output = self._activate(last_output, w, b)
175 | 
176 |     if is_target:
177 |       return last_output
178 | 
179 |     last_output = self._activate(last_output, self._w(n), self._b(n))
180 | 
181 |     out = self._activate(last_output, self._w(n), self._b(n, "_out"),
182 |                          transpose_w=True)
183 |     out = tf.maximum(out, 1.e-9)
184 |     out = tf.minimum(out, 1 - 1.e-9)
185 |     return out
186 | 
187 |   def supervised_net(self, input_pl):
188 |     """Get the supervised fine tuning net
189 | 
190 |     Args:
191 |       input_pl: tf placeholder for ae input data
192 |     Returns:
193 |       Tensor giving full ae net
194 |     """
195 |     last_output = input_pl
196 | 
197 |     for i in xrange(self.__num_hidden_layers + 1):
198 |       # Fine tuning will be done on these variables
199 |       w = self._w(i + 1)
200 |       b = self._b(i + 1)
201 | 
202 |       last_output = self._activate(last_output, w, b)
203 | 
204 |     return last_output
205 | 
206 | 
207 | loss_summaries = {}
208 | 
209 | 
210 | def training(loss, learning_rate, loss_key=None):
211 |   """Sets up the training Ops.
212 | 
213 |   Creates a summarizer to track the loss over time in TensorBoard.
214 | 
215 |   Creates an optimizer and applies the gradients to all trainable variables.
216 | 
217 |   The Op returned by this function is what must be passed to the
218 |   `sess.run()` call to cause the model to train.
219 | 
220 |   Args:
221 |     loss: Loss tensor, from loss().
222 |     learning_rate: The learning rate to use for gradient descent.
223 |     loss_key: int giving stage of pretraining so we can store
224 |                 loss summaries for each pretraining stage
225 | 
226 |   Returns:
227 |     train_op: The Op for training.
228 |   """
229 |   if loss_key is not None:
230 |     # Add a scalar summary for the snapshot loss.
231 |     loss_summaries[loss_key] = tf.scalar_summary(loss.op.name, loss)
232 |   else:
233 |     tf.scalar_summary(loss.op.name, loss)
234 |     for var in tf.trainable_variables():
235 |       tf.histogram_summary(var.op.name, var)
236 |   # Create the gradient descent optimizer with the given learning rate.
237 |   optimizer = tf.train.GradientDescentOptimizer(learning_rate)
238 |   # Create a variable to track the global step.
239 |   global_step = tf.Variable(0, name='global_step', trainable=False)
240 |   # Use the optimizer to apply the gradients that minimize the loss
241 |   # (and also increment the global step counter) as a single training step.
242 |   train_op = optimizer.minimize(loss, global_step=global_step)
243 |   return train_op, global_step
244 | 
245 | 
246 | def loss_x_entropy(output, target):
247 |   """Cross entropy loss
248 | 
249 |   See https://en.wikipedia.org/wiki/Cross_entropy
250 | 
251 |   Args:
252 |     output: tensor of net output
253 |     target: tensor of net we are trying to reconstruct
254 |   Returns:
255 |     Scalar tensor of cross entropy
256 |   """
257 |   with tf.name_scope("xentropy_loss"):
258 |       net_output_tf = tf.convert_to_tensor(output, name='input')
259 |       target_tf = tf.convert_to_tensor(target, name='target')
260 |       cross_entropy = tf.add(tf.mul(tf.log(net_output_tf, name='log_output'),
261 |                                     target_tf),
262 |                              tf.mul(tf.log(1 - net_output_tf),
263 |                                     (1 - target_tf)))
264 |       return -1 * tf.reduce_mean(tf.reduce_sum(cross_entropy, 1),
265 |                                  name='xentropy_mean')
266 | 
267 | 
268 | def main_unsupervised():
269 |   with tf.Graph().as_default() as g:
270 |     sess = tf.Session()
271 | 
272 |     num_hidden = FLAGS.num_hidden_layers
273 |     ae_hidden_shapes = [getattr(FLAGS, "hidden{0}_units".format(j + 1))
274 |                         for j in xrange(num_hidden)]
275 |     ae_shape = [FLAGS.image_pixels] + ae_hidden_shapes + [FLAGS.num_classes]
276 | 
277 |     ae = AutoEncoder(ae_shape, sess)
278 | 
279 |     data = read_data_sets_pretraining(FLAGS.data_dir)
280 |     num_train = data.train.num_examples
281 | 
282 |     learning_rates = {j: getattr(FLAGS,
283 |                                  "pre_layer{0}_learning_rate".format(j + 1))
284 |                       for j in xrange(num_hidden)}
285 | 
286 |     noise = {j: getattr(FLAGS, "noise_{0}".format(j + 1))
287 |              for j in xrange(num_hidden)}
288 | 
289 |     for i in xrange(len(ae_shape) - 2):
290 |       n = i + 1
291 |       with tf.variable_scope("pretrain_{0}".format(n)):
292 |         input_ = tf.placeholder(dtype=tf.float32,
293 |                                 shape=(FLAGS.batch_size, ae_shape[0]),
294 |                                 name='ae_input_pl')
295 |         target_ = tf.placeholder(dtype=tf.float32,
296 |                                  shape=(FLAGS.batch_size, ae_shape[0]),
297 |                                  name='ae_target_pl')
298 |         layer = ae.pretrain_net(input_, n)
299 | 
300 |         with tf.name_scope("target"):
301 |           target_for_loss = ae.pretrain_net(target_, n, is_target=True)
302 | 
303 |         loss = loss_x_entropy(layer, target_for_loss)
304 |         train_op, global_step = training(loss, learning_rates[i], i)
305 | 
306 |         summary_dir = pjoin(FLAGS.summary_dir, 'pretraining_{0}'.format(n))
307 |         summary_writer = tf.train.SummaryWriter(summary_dir,
308 |                                                 graph_def=sess.graph_def,
309 |                                                 flush_secs=FLAGS.flush_secs)
310 |         summary_vars = [ae["biases{0}".format(n)], ae["weights{0}".format(n)]]
311 | 
312 |         hist_summarries = [tf.histogram_summary(v.op.name, v)
313 |                            for v in summary_vars]
314 |         hist_summarries.append(loss_summaries[i])
315 |         summary_op = tf.merge_summary(hist_summarries)
316 | 
317 |         vars_to_init = ae.get_variables_to_init(n)
318 |         vars_to_init.append(global_step)
319 |         sess.run(tf.initialize_variables(vars_to_init))
320 | 
321 |         print("\n\n")
322 |         print("| Training Step | Cross Entropy |  Layer  |   Epoch  |")
323 |         print("|---------------|---------------|---------|----------|")
324 | 
325 |         for step in xrange(FLAGS.pretraining_epochs * num_train):
326 |           feed_dict = fill_feed_dict_ae(data.train, input_, target_, noise[i])
327 | 
328 |           loss_summary, loss_value = sess.run([train_op, loss],
329 |                                               feed_dict=feed_dict)
330 | 
331 |           if step % 100 == 0:
332 |             summary_str = sess.run(summary_op, feed_dict=feed_dict)
333 |             summary_writer.add_summary(summary_str, step)
334 |             image_summary_op = \
335 |                 tf.image_summary("training_images",
336 |                                  tf.reshape(input_,
337 |                                             (FLAGS.batch_size,
338 |                                              FLAGS.image_size,
339 |                                              FLAGS.image_size, 1)),
340 |                                  max_images=FLAGS.batch_size)
341 | 
342 |             summary_img_str = sess.run(image_summary_op,
343 |                                        feed_dict=feed_dict)
344 |             summary_writer.add_summary(summary_img_str)
345 | 
346 |             output = "| {0:>13} | {1:13.4f} | Layer {2} | Epoch {3}  |"\
347 |                      .format(step, loss_value, n, step // num_train + 1)
348 | 
349 |             print(output)
350 |       if i == 0:
351 |         filters = sess.run(tf.identity(ae["weights1"]))
352 |         np.save(pjoin(FLAGS.chkpt_dir, "filters"), filters)
353 |         filters = tile_raster_images(X=filters.T,
354 |                                      img_shape=(FLAGS.image_size,
355 |                                                 FLAGS.image_size),
356 |                                      tile_shape=(10, 10),
357 |                                      output_pixel_vals=False)
358 |         filters = np.expand_dims(np.expand_dims(filters, 0), 3)
359 |         image_var = tf.Variable(filters)
360 |         image_filter = tf.identity(image_var)
361 |         sess.run(tf.initialize_variables([image_var]))
362 |         img_filter_summary_op = tf.image_summary("first_layer_filters",
363 |                                                  image_filter)
364 |         summary_writer.add_summary(sess.run(img_filter_summary_op))
365 |         summary_writer.flush()
366 | 
367 |   return ae
368 | 
369 | 
370 | def main_supervised(ae):
371 |   with ae.session.graph.as_default():
372 |     sess = ae.session
373 |     input_pl = tf.placeholder(tf.float32, shape=(FLAGS.batch_size,
374 |                                                  FLAGS.image_pixels),
375 |                               name='input_pl')
376 |     logits = ae.supervised_net(input_pl)
377 | 
378 |     data = read_data_sets(FLAGS.data_dir)
379 |     num_train = data.train.num_examples
380 | 
381 |     labels_placeholder = tf.placeholder(tf.int32,
382 |                                         shape=FLAGS.batch_size,
383 |                                         name='target_pl')
384 | 
385 |     loss = loss_supervised(logits, labels_placeholder)
386 |     train_op, global_step = training(loss, FLAGS.supervised_learning_rate)
387 |     eval_correct = evaluation(logits, labels_placeholder)
388 | 
389 |     hist_summaries = [ae['biases{0}'.format(i + 1)]
390 |                       for i in xrange(ae.num_hidden_layers + 1)]
391 |     hist_summaries.extend([ae['weights{0}'.format(i + 1)]
392 |                            for i in xrange(ae.num_hidden_layers + 1)])
393 | 
394 |     hist_summaries = [tf.histogram_summary(v.op.name + "_fine_tuning", v)
395 |                       for v in hist_summaries]
396 |     summary_op = tf.merge_summary(hist_summaries)
397 | 
398 |     summary_writer = tf.train.SummaryWriter(pjoin(FLAGS.summary_dir,
399 |                                                   'fine_tuning'),
400 |                                             graph_def=sess.graph_def,
401 |                                             flush_secs=FLAGS.flush_secs)
402 | 
403 |     vars_to_init = ae.get_variables_to_init(ae.num_hidden_layers + 1)
404 |     vars_to_init.append(global_step)
405 |     sess.run(tf.initialize_variables(vars_to_init))
406 | 
407 |     steps = FLAGS.finetuning_epochs * num_train
408 |     for step in xrange(steps):
409 |       start_time = time.time()
410 | 
411 |       feed_dict = fill_feed_dict(data.train,
412 |                                  input_pl,
413 |                                  labels_placeholder)
414 | 
415 |       _, loss_value = sess.run([train_op, loss],
416 |                                feed_dict=feed_dict)
417 | 
418 |       duration = time.time() - start_time
419 | 
420 |       # Write the summaries and print an overview fairly often.
421 |       if step % 100 == 0:
422 |         # Print status to stdout.
423 |         print('Step %d: loss = %.2f (%.3f sec)' % (step, loss_value, duration))
424 |         # Update the events file.
425 | 
426 |         summary_str = sess.run(summary_op, feed_dict=feed_dict)
427 |         summary_writer.add_summary(summary_str, step)
428 |         summary_img_str = sess.run(
429 |             tf.image_summary("training_images",
430 |                              tf.reshape(input_pl,
431 |                                         (FLAGS.batch_size,
432 |                                          FLAGS.image_size,
433 |                                          FLAGS.image_size, 1)),
434 |                              max_images=FLAGS.batch_size),
435 |             feed_dict=feed_dict
436 |         )
437 |         summary_writer.add_summary(summary_img_str)
438 | 
439 |       if (step + 1) % 1000 == 0 or (step + 1) == steps:
440 |         train_sum = do_eval_summary("training_error",
441 |                                     sess,
442 |                                     eval_correct,
443 |                                     input_pl,
444 |                                     labels_placeholder,
445 |                                     data.train)
446 | 
447 |         val_sum = do_eval_summary("validation_error",
448 |                                   sess,
449 |                                   eval_correct,
450 |                                   input_pl,
451 |                                   labels_placeholder,
452 |                                   data.validation)
453 | 
454 |         test_sum = do_eval_summary("test_error",
455 |                                    sess,
456 |                                    eval_correct,
457 |                                    input_pl,
458 |                                    labels_placeholder,
459 |                                    data.test)
460 | 
461 |         summary_writer.add_summary(train_sum, step)
462 |         summary_writer.add_summary(val_sum, step)
463 |         summary_writer.add_summary(test_sum, step)
464 | 
465 | if __name__ == '__main__':
466 |   ae = main_unsupervised()
467 |   main_supervised(ae)
468 | 


--------------------------------------------------------------------------------
/code/ae/autoencoder_test.py:
--------------------------------------------------------------------------------
 1 | from __future__ import division
 2 | from __future__ import print_function
 3 | 
 4 | import tensorflow as tf
 5 | from autoencoder import AutoEncoder
 6 | 
 7 | 
 8 | class AutoEncoderTest(tf.test.TestCase):
 9 | 
10 |   def test_constructor(self):
11 |     with self.test_session() as sess:
12 | 
13 |       ae_shape = [10, 20, 30, 2]
14 |       self.assertTrue(AutoEncoder(ae_shape, sess))
15 | 
16 |   def test_get_variables(self):
17 |     with self.test_session() as sess:
18 |       ae_shape = [10, 20, 30, 2]
19 |       ae = AutoEncoder(ae_shape, sess)
20 | 
21 |       with self.assertRaises(AssertionError):
22 |         ae.get_variables_to_init(0)
23 |       with self.assertRaises(AssertionError):
24 |         ae.get_variables_to_init(4)
25 | 
26 |       v1 = ae.get_variables_to_init(1)
27 |       self.assertEqual(len(v1), 3)
28 | 
29 |       v2 = ae.get_variables_to_init(2)
30 |       self.assertEqual(len(v2), 5)
31 | 
32 |       v3 = ae.get_variables_to_init(3)
33 |       self.assertEqual(len(v3), 2)
34 | 
35 |   def test_nets(self):
36 |     with self.test_session() as sess:
37 |       ae_shape = [10, 20, 30, 2]
38 |       ae = AutoEncoder(ae_shape, sess)
39 | 
40 |       input_pl = tf.placeholder(tf.float32, shape=(100, 10))
41 |       with self.assertRaises(AssertionError):
42 |         ae.pretrain_net(input_pl, 0)
43 |       with self.assertRaises(AssertionError):
44 |         ae.pretrain_net(input_pl, 3)
45 | 
46 |       net1 = ae.pretrain_net(input_pl, 1)
47 |       net2 = ae.pretrain_net(input_pl, 2)
48 | 
49 |       self.assertEqual(net1.get_shape().dims[1].value, 10)
50 |       self.assertEqual(net2.get_shape().dims[1].value, 20)
51 | 
52 |       net1_target = ae.pretrain_net(input_pl, 1, is_target=True)
53 |       self.assertEqual(net1_target.get_shape().dims[1].value, 10)
54 |       net2_target = ae.pretrain_net(input_pl, 2, is_target=True)
55 |       self.assertEqual(net2_target.get_shape().dims[1].value, 20)
56 | 
57 |       sup_net = ae.supervised_net(input_pl)
58 |       self.assertEqual(sup_net.get_shape().dims[1].value, 2)
59 | 


--------------------------------------------------------------------------------
/code/ae/utils/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/cmgreen210/TensorFlowDeepAutoencoder/5298ec437689ba7ecb59229599141549ef6a6a1d/code/ae/utils/__init__.py


--------------------------------------------------------------------------------
/code/ae/utils/data.py:
--------------------------------------------------------------------------------
  1 | """Functions for downloading and reading MNIST data."""
  2 | from __future__ import division
  3 | from __future__ import print_function
  4 | 
  5 | import gzip
  6 | 
  7 | import numpy
  8 | 
  9 | from six.moves import urllib
 10 | from six.moves import xrange  # pylint: disable=redefined-builtin
 11 | from flags import FLAGS
 12 | import os
 13 | SOURCE_URL = 'http://yann.lecun.com/exdb/mnist/'
 14 | 
 15 | 
 16 | def maybe_download(filename, work_directory):
 17 |   """Download the data from Yann's website, unless it's already here."""
 18 |   if not os.path.exists(work_directory):
 19 |     os.mkdir(work_directory)
 20 |   filepath = os.path.join(work_directory, filename)
 21 |   if not os.path.exists(filepath):
 22 |     filepath, _ = urllib.request.urlretrieve(SOURCE_URL + filename, filepath)
 23 |     statinfo = os.stat(filepath)
 24 |     print('Succesfully downloaded', filename, statinfo.st_size, 'bytes.')
 25 |   return filepath
 26 | 
 27 | 
 28 | def _read32(bytestream):
 29 |   dt = numpy.dtype(numpy.uint32).newbyteorder('>')
 30 |   return numpy.frombuffer(bytestream.read(4), dtype=dt)
 31 | 
 32 | 
 33 | def extract_images(filename):
 34 |   """Extract the images into a 4D uint8 numpy array [index, y, x, depth]."""
 35 |   print('\nExtracting', filename)
 36 |   with gzip.open(filename) as bytestream:
 37 |     magic = _read32(bytestream)
 38 |     if magic != 2051:
 39 |       raise ValueError(
 40 |           'Invalid magic number %d in MNIST image file: %s' %
 41 |           (magic, filename))
 42 |     num_images = _read32(bytestream)
 43 |     rows = _read32(bytestream)
 44 |     cols = _read32(bytestream)
 45 |     buf = bytestream.read(rows * cols * num_images)
 46 |     data = numpy.frombuffer(buf, dtype=numpy.uint8)
 47 |     data = data.reshape(num_images, rows, cols, 1)
 48 |     return data
 49 | 
 50 | 
 51 | def dense_to_one_hot(labels_dense, num_classes=10):
 52 |   """Convert class labels from scalars to one-hot vectors."""
 53 |   num_labels = labels_dense.shape[0]
 54 |   index_offset = numpy.arange(num_labels) * num_classes
 55 |   labels_one_hot = numpy.zeros((num_labels, num_classes))
 56 |   labels_one_hot.flat[index_offset + labels_dense.ravel()] = 1
 57 |   return labels_one_hot
 58 | 
 59 | 
 60 | def extract_labels(filename, one_hot=False):
 61 |   """Extract the labels into a 1D uint8 numpy array [index]."""
 62 |   print('Extracting', filename)
 63 |   with gzip.open(filename) as bytestream:
 64 |     magic = _read32(bytestream)
 65 |     if magic != 2049:
 66 |       raise ValueError(
 67 |           'Invalid magic number %d in MNIST label file: %s' %
 68 |           (magic, filename))
 69 |     num_items = _read32(bytestream)
 70 |     buf = bytestream.read(num_items)
 71 |     labels = numpy.frombuffer(buf, dtype=numpy.uint8)
 72 |     if one_hot:
 73 |       return dense_to_one_hot(labels)
 74 |     return labels
 75 | 
 76 | 
 77 | class DataSet(object):
 78 | 
 79 |   def __init__(self, images, labels, fake_data=False):
 80 |     if fake_data:
 81 |       self._num_examples = 10000
 82 |     else:
 83 |       assert images.shape[0] == labels.shape[0], (
 84 |           "images.shape: %s labels.shape: %s" % (images.shape,
 85 |                                                  labels.shape))
 86 |       self._num_examples = images.shape[0]
 87 | 
 88 |       # Convert shape from [num examples, rows, columns, depth]
 89 |       # to [num examples, rows*columns] (assuming depth == 1)
 90 |       assert images.shape[3] == 1
 91 |       images = images.reshape(images.shape[0],
 92 |                               images.shape[1] * images.shape[2])
 93 |       # Convert from [0, 255] -> [0.0, 1.0].
 94 |       images = images.astype(numpy.float32)
 95 |       images = numpy.multiply(images, 1.0 / 255.0)
 96 |     self._images = images
 97 |     self._labels = labels
 98 |     self._epochs_completed = 0
 99 |     self._index_in_epoch = 0
100 | 
101 |   @property
102 |   def images(self):
103 |     return self._images
104 | 
105 |   @property
106 |   def labels(self):
107 |     return self._labels
108 | 
109 |   @property
110 |   def num_examples(self):
111 |     return self._num_examples
112 | 
113 |   @property
114 |   def epochs_completed(self):
115 |     return self._epochs_completed
116 | 
117 |   def next_batch(self, batch_size):
118 |     """Return the next `batch_size` examples from this data set."""
119 |     start = self._index_in_epoch
120 |     self._index_in_epoch += batch_size
121 |     if self._index_in_epoch > self._num_examples:
122 |       # Finished epoch
123 |       self._epochs_completed += 1
124 |       # Shuffle the data
125 |       perm = numpy.arange(self._num_examples)
126 |       numpy.random.shuffle(perm)
127 |       self._images = self._images[perm]
128 |       self._labels = self._labels[perm]
129 |       # Start next epoch
130 |       start = 0
131 |       self._index_in_epoch = batch_size
132 |       assert batch_size <= self._num_examples
133 |     end = self._index_in_epoch
134 |     return self._images[start:end], self._labels[start:end]
135 | 
136 | 
137 | class DataSetPreTraining(object):
138 | 
139 |   def __init__(self, images):
140 |     self._num_examples = images.shape[0]
141 | 
142 |     # Convert shape from [num examples, rows, columns, depth]
143 |     # to [num examples, rows*columns] (assuming depth == 1)
144 |     assert images.shape[3] == 1
145 |     images = images.reshape(images.shape[0],
146 |                             images.shape[1] * images.shape[2])
147 |     # Convert from [0, 255] -> [0.0, 1.0].
148 |     images = images.astype(numpy.float32)
149 |     images = numpy.multiply(images, 1.0 / 255.0)
150 |     self._images = images
151 |     self._images[self._images < FLAGS.zero_bound] = FLAGS.zero_bound
152 |     self._images[self._images > FLAGS.one_bound] = FLAGS.one_bound
153 |     self._epochs_completed = 0
154 |     self._index_in_epoch = 0
155 | 
156 |   @property
157 |   def images(self):
158 |     return self._images
159 | 
160 |   @property
161 |   def num_examples(self):
162 |     return self._num_examples
163 | 
164 |   @property
165 |   def epochs_completed(self):
166 |     return self._epochs_completed
167 | 
168 |   def next_batch(self, batch_size):
169 |     """Return the next `batch_size` examples from this data set."""
170 |     start = self._index_in_epoch
171 |     self._index_in_epoch += batch_size
172 |     if self._index_in_epoch > self._num_examples:
173 |       # Finished epoch
174 |       self._epochs_completed += 1
175 |       # Shuffle the data
176 |       perm = numpy.arange(self._num_examples)
177 |       numpy.random.shuffle(perm)
178 |       self._images = self._images[perm]
179 |       # Start next epoch
180 |       start = 0
181 |       self._index_in_epoch = batch_size
182 |       assert batch_size <= self._num_examples
183 |     end = self._index_in_epoch
184 | 
185 |     return self._images[start:end], self._images[start:end]
186 | 
187 | 
188 | def read_data_sets(train_dir, fake_data=False, one_hot=False):
189 |   class DataSets(object):
190 |     pass
191 |   data_sets = DataSets()
192 | 
193 |   if fake_data:
194 |     data_sets.train = DataSet([], [], fake_data=True)
195 |     data_sets.validation = DataSet([], [], fake_data=True)
196 |     data_sets.test = DataSet([], [], fake_data=True)
197 |     return data_sets
198 | 
199 |   TRAIN_IMAGES = 'train-images-idx3-ubyte.gz'
200 |   TRAIN_LABELS = 'train-labels-idx1-ubyte.gz'
201 |   TEST_IMAGES = 't10k-images-idx3-ubyte.gz'
202 |   TEST_LABELS = 't10k-labels-idx1-ubyte.gz'
203 |   VALIDATION_SIZE = 5000
204 | 
205 |   local_file = maybe_download(TRAIN_IMAGES, train_dir)
206 |   train_images = extract_images(local_file)
207 | 
208 |   local_file = maybe_download(TRAIN_LABELS, train_dir)
209 |   train_labels = extract_labels(local_file, one_hot=one_hot)
210 | 
211 |   local_file = maybe_download(TEST_IMAGES, train_dir)
212 |   test_images = extract_images(local_file)
213 | 
214 |   local_file = maybe_download(TEST_LABELS, train_dir)
215 |   test_labels = extract_labels(local_file, one_hot=one_hot)
216 | 
217 |   validation_images = train_images[:VALIDATION_SIZE]
218 |   validation_labels = train_labels[:VALIDATION_SIZE]
219 |   train_images = train_images[VALIDATION_SIZE:]
220 |   train_labels = train_labels[VALIDATION_SIZE:]
221 | 
222 |   data_sets.train = DataSet(train_images, train_labels)
223 |   data_sets.validation = DataSet(validation_images, validation_labels)
224 |   data_sets.test = DataSet(test_images, test_labels)
225 | 
226 |   return data_sets
227 | 
228 | 
229 | def read_data_sets_pretraining(train_dir):
230 |   class DataSets(object):
231 |     pass
232 |   data_sets = DataSets()
233 | 
234 |   TRAIN_IMAGES = 'train-images-idx3-ubyte.gz'
235 |   TEST_IMAGES = 't10k-images-idx3-ubyte.gz'
236 |   VALIDATION_SIZE = 5000
237 | 
238 |   local_file = maybe_download(TRAIN_IMAGES, train_dir)
239 |   train_images = extract_images(local_file)
240 | 
241 |   local_file = maybe_download(TEST_IMAGES, train_dir)
242 |   test_images = extract_images(local_file)
243 | 
244 |   validation_images = train_images[:VALIDATION_SIZE]
245 |   train_images = train_images[VALIDATION_SIZE:]
246 | 
247 |   data_sets.train = DataSetPreTraining(train_images)
248 |   data_sets.validation = DataSetPreTraining(validation_images)
249 |   data_sets.test = DataSetPreTraining(test_images)
250 | 
251 |   return data_sets
252 | 
253 | 
254 | def _add_noise(x, rate):
255 |   x_cp = numpy.copy(x)
256 |   pix_to_drop = numpy.random.rand(x_cp.shape[0],
257 |                                   x_cp.shape[1]) < rate
258 |   x_cp[pix_to_drop] = FLAGS.zero_bound
259 |   return x_cp
260 | 
261 | 
262 | def fill_feed_dict_ae(data_set, input_pl, target_pl, noise=None):
263 |     input_feed, target_feed = data_set.next_batch(FLAGS.batch_size)
264 |     if noise:
265 |       input_feed = _add_noise(input_feed, noise)
266 |     feed_dict = {
267 |         input_pl: input_feed,
268 |         target_pl: target_feed
269 |     }
270 |     return feed_dict
271 | 
272 | 
273 | def fill_feed_dict(data_set, images_pl, labels_pl, noise=False):
274 |   """Fills the feed_dict for training the given step.
275 |   A feed_dict takes the form of:
276 |   feed_dict = {
277 |       <placeholder>: <tensor of values to be passed for placeholder>,
278 |       ....
279 |   }
280 |   Args:
281 |     data_set: The set of images and labels, from input_data.read_data_sets()
282 |     images_pl: The images placeholder, from placeholder_inputs().
283 |     labels_pl: The labels placeholder, from placeholder_inputs().
284 |   Returns:
285 |     feed_dict: The feed dictionary mapping from placeholders to values.
286 |   """
287 |   # Create the feed_dict for the placeholders filled with the next
288 |   # `batch size ` examples.
289 |   images_feed, labels_feed = data_set.next_batch(FLAGS.batch_size)
290 |   if noise:
291 |       images_feed = _add_noise(images_feed, FLAGS.drop_out_rate)
292 |   feed_dict = {
293 |       images_pl: images_feed,
294 |       labels_pl: labels_feed,
295 |   }
296 |   return feed_dict
297 | 


--------------------------------------------------------------------------------
/code/ae/utils/eval.py:
--------------------------------------------------------------------------------
  1 | from __future__ import division
  2 | from __future__ import print_function
  3 | 
  4 | from flags import *
  5 | from data import fill_feed_dict as fill_feed_dict
  6 | 
  7 | 
  8 | def loss_supervised(logits, labels):
  9 |   """Calculates the loss from the logits and the labels.
 10 | 
 11 |   Args:
 12 |     logits: Logits tensor, float - [batch_size, NUM_CLASSES].
 13 |     labels: Labels tensor, int32 - [batch_size].
 14 | 
 15 |   Returns:
 16 |     loss: Loss tensor of type float.
 17 |   """
 18 |   # Convert from sparse integer labels in the range [0, NUM_CLASSSES)
 19 |   # to 1-hot dense float vectors (that is we will have batch_size vectors,
 20 |   # each with NUM_CLASSES values, all of which are 0.0 except there will
 21 |   # be a 1.0 in the entry corresponding to the label).
 22 |   batch_size = tf.size(labels)
 23 |   labels = tf.expand_dims(labels, 1)
 24 |   indices = tf.expand_dims(tf.range(0, batch_size), 1)
 25 |   concated = tf.concat(1, [indices, labels])
 26 |   onehot_labels = tf.sparse_to_dense(
 27 |       concated, tf.pack([batch_size, NUM_CLASSES]), 1.0, 0.0)
 28 |   cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits,
 29 |                                                           onehot_labels,
 30 |                                                           name='xentropy')
 31 |   loss = tf.reduce_mean(cross_entropy, name='xentropy_mean')
 32 |   return loss
 33 | 
 34 | 
 35 | def evaluation(logits, labels):
 36 |   """Evaluate the quality of the logits at predicting the label.
 37 | 
 38 |   Args:
 39 |     logits: Logits tensor, float - [batch_size, NUM_CLASSES].
 40 |     labels: Labels tensor, int32 - [batch_size], with values in the
 41 |       range [0, NUM_CLASSES).
 42 | 
 43 |   Returns:
 44 |     A scalar int32 tensor with the number of examples (out of batch_size)
 45 |     that were predicted correctly.
 46 |   """
 47 |   # For a classifier model, we can use the in_top_k Op.
 48 |   # It returns a bool tensor with shape [batch_size] that is true for
 49 |   # the examples where the label's is was in the top k (here k=1)
 50 |   # of all logits for that example.
 51 |   correct = tf.nn.in_top_k(logits, labels, 1)
 52 |   # Return the number of true entries.
 53 |   return tf.reduce_sum(tf.cast(correct, tf.int32))
 54 | 
 55 | 
 56 | def do_eval(sess,
 57 |             eval_correct,
 58 |             images_placeholder,
 59 |             labels_placeholder,
 60 |             data_set):
 61 |   """Runs one evaluation against the full epoch of data.
 62 |   Args:
 63 |     sess: The session in which the model has been trained.
 64 |     eval_correct: The Tensor that returns the number of correct predictions.
 65 |     images_placeholder: The images placeholder.
 66 |     labels_placeholder: The labels placeholder.
 67 |     data_set: The set of images and labels to evaluate, from
 68 |       input_data.read_data_sets().
 69 |   """
 70 |   # And run one epoch of eval.
 71 |   true_count = 0  # Counts the number of correct predictions.
 72 |   steps_per_epoch = data_set.num_examples // FLAGS.batch_size
 73 |   num_examples = steps_per_epoch * FLAGS.batch_size
 74 |   for step in xrange(steps_per_epoch):
 75 |     feed_dict = fill_feed_dict(data_set,
 76 |                                images_placeholder,
 77 |                                labels_placeholder)
 78 |     true_count += sess.run(eval_correct, feed_dict=feed_dict)
 79 |   precision = true_count / num_examples
 80 |   print('  Num examples: %d  Num correct: %d  Precision @ 1: %0.04f' %
 81 |         (num_examples, true_count, precision))
 82 | 
 83 | 
 84 | def do_eval_summary(tag,
 85 |                     sess,
 86 |                     eval_correct,
 87 |                     images_placeholder,
 88 |                     labels_placeholder,
 89 |                     data_set):
 90 |   true_count = 0
 91 |   steps_per_epoch = data_set.num_examples // FLAGS.batch_size
 92 |   num_examples = steps_per_epoch * FLAGS.batch_size
 93 |   for step in xrange(steps_per_epoch):
 94 |     feed_dict = fill_feed_dict(data_set,
 95 |                                images_placeholder,
 96 |                                labels_placeholder)
 97 |     true_count += sess.run(eval_correct, feed_dict=feed_dict)
 98 |   error = 1 - true_count / num_examples
 99 |   return sess.run(tf.scalar_summary(tag, tf.identity(error)))
100 | 


--------------------------------------------------------------------------------
/code/ae/utils/flags.py:
--------------------------------------------------------------------------------
 1 | from __future__ import division
 2 | import os
 3 | from os.path import join as pjoin
 4 | 
 5 | import sys
 6 | 
 7 | import tensorflow as tf
 8 | 
 9 | 
10 | IMAGE_PIXELS = 28 * 28
11 | NUM_CLASSES = 10
12 | 
13 | 
14 | def home_out(path):
15 |   return pjoin(os.environ['HOME'], 'tmp', 'mnist', path)
16 | 
17 | flags = tf.app.flags
18 | FLAGS = flags.FLAGS
19 | 
20 | # Autoencoder Architecture Specific Flags
21 | flags.DEFINE_integer("num_hidden_layers", 3, "Number of hidden layers")
22 | 
23 | flags.DEFINE_integer('hidden1_units', 2000,
24 |                      'Number of units in hidden layer 1.')
25 | flags.DEFINE_integer('hidden2_units', 2000,
26 |                      'Number of units in hidden layer 2.')
27 | flags.DEFINE_integer('hidden3_units', 2000,
28 |                      'Number of units in hidden layer 3.')
29 | 
30 | flags.DEFINE_integer('image_pixels', IMAGE_PIXELS, 'Total number of pixels')
31 | flags.DEFINE_integer('num_classes', 10, 'Number of classes')
32 | 
33 | flags.DEFINE_float('pre_layer1_learning_rate', 0.0001,
34 |                    'Initial learning rate.')
35 | flags.DEFINE_float('pre_layer2_learning_rate', 0.0001,
36 |                    'Initial learning rate.')
37 | flags.DEFINE_float('pre_layer3_learning_rate', 0.0001,
38 |                    'Initial learning rate.')
39 | 
40 | flags.DEFINE_float('noise_1', 0.50, 'Rate at which to set pixels to 0')
41 | flags.DEFINE_float('noise_2', 0.50, 'Rate at which to set pixels to 0')
42 | flags.DEFINE_float('noise_3', 0.50, 'Rate at which to set pixels to 0')
43 | 
44 | # Constants
45 | flags.DEFINE_integer('seed', 1234, 'Random seed')
46 | flags.DEFINE_integer('image_size', 28, 'Image square size')
47 | 
48 | flags.DEFINE_integer('batch_size', 100,
49 |                      'Batch size. Must divide evenly into the dataset sizes.')
50 | 
51 | flags.DEFINE_float('supervised_learning_rate', 0.1,
52 |                    'Supervised initial learning rate.')
53 | 
54 | flags.DEFINE_integer('pretraining_epochs', 60,
55 |                      "Number of training epochs for pretraining layers")
56 | flags.DEFINE_integer('finetuning_epochs', 56,
57 |                      "Number of training epochs for "
58 |                      "fine tuning supervised step")
59 | 
60 | flags.DEFINE_float('zero_bound', 1.0e-9,
61 |                    'Value to use as buffer to avoid '
62 |                    'numerical issues at 0')
63 | flags.DEFINE_float('one_bound', 1.0 - 1.0e-9,
64 |                    'Value to use as buffer to avoid numerical issues at 1')
65 | 
66 | flags.DEFINE_float('flush_secs', 120, 'Number of seconds to flush summaries')
67 | 
68 | # Directories
69 | flags.DEFINE_string('data_dir', home_out('data'),
70 |                     'Directory to put the training data.')
71 | 
72 | flags.DEFINE_string('summary_dir', home_out('summaries'),
73 |                     'Directory to put the summary data')
74 | 
75 | flags.DEFINE_string('chkpt_dir', home_out('chkpts'),
76 |                     'Directory to put the model checkpoints')
77 | 
78 | # TensorBoard
79 | flags.DEFINE_boolean('no_browser', True,
80 |                      'Whether to start browser for TensorBoard')
81 | 
82 | # Python
83 | flags.DEFINE_string('python', sys.executable,
84 |                     'Path to python executable')
85 | 


--------------------------------------------------------------------------------
/code/ae/utils/start_tensorboard.py:
--------------------------------------------------------------------------------
 1 | import os
 2 | import signal
 3 | import subprocess
 4 | 
 5 | from tensorflow import tensorboard as tb
 6 | 
 7 | from flags import FLAGS, home_out
 8 | 
 9 | 
10 | _image_dir = FLAGS.data_dir
11 | _summary_dir = FLAGS.summary_dir
12 | 
13 | _tb_pid_file = home_out(".tbpid")
14 | _tb_path = os.path.join(os.path.dirname(tb.__file__), 'tensorboard.py')
15 | _tb_port = "6006"
16 | 
17 | 
18 | def start():
19 |   if not os.path.exists(_tb_path):
20 |     raise EnvironmentError("tensorboard.py not found!")
21 | 
22 |   if os.path.exists(_tb_pid_file):
23 |     tb_pid = int(open(_tb_pid_file, 'r').readline().strip())
24 |     try:
25 |       os.kill(tb_pid, signal.SIGKILL)
26 |     except OSError:
27 |       pass
28 | 
29 |     os.remove(_tb_pid_file)
30 | 
31 |   devnull = open(os.devnull, 'wb')
32 |   p = subprocess.Popen(['nohup', FLAGS.python,
33 |                         '-u', _tb_path, '--logdir={0}'.format(_summary_dir),
34 |                         '--port=' + _tb_port], stdout=devnull, stderr=devnull)
35 |   with open(_tb_pid_file, 'w') as f:
36 |     f.write(str(p.pid))
37 | 
38 |   if not FLAGS.no_browser:
39 |     subprocess.Popen(['open', 'http://localhost:{0}'.format(_tb_port)])
40 | 
41 | 
42 | if __name__ == '__main__':
43 |   start()
44 | 


--------------------------------------------------------------------------------
/code/ae/utils/utils.py:
--------------------------------------------------------------------------------
  1 | """ This file contains different utility functions that are not connected
  2 | in anyway to the networks presented in the tutorials, but rather help in
  3 | processing the outputs into a more understandable way.
  4 | 
  5 | For example ``tile_raster_images`` helps in generating a easy to grasp
  6 | image from a set of samples or weights.
  7 | """
  8 | 
  9 | 
 10 | import numpy
 11 | 
 12 | 
 13 | def scale_to_unit_interval(ndar, eps=1e-8):
 14 |   """ Scales all values in the ndarray ndar to be between 0 and 1 """
 15 |   ndar = ndar.copy()
 16 |   ndar -= ndar.min()
 17 |   ndar *= 1.0 / (ndar.max() + eps)
 18 |   return ndar
 19 | 
 20 | 
 21 | def tile_raster_images(X, img_shape, tile_shape, tile_spacing=(0, 0),
 22 |                        scale_rows_to_unit_interval=True,
 23 |                        output_pixel_vals=True):
 24 |   """
 25 |   Transform an array with one flattened image per row, into an array in
 26 |   which images are reshaped and layed out like tiles on a floor.
 27 | 
 28 |   This function is useful for visualizing datasets whose rows are images,
 29 |   and also columns of matrices for transforming those rows
 30 |   (such as the first layer of a neural net).
 31 | 
 32 |   :type X: a 2-D ndarray or a tuple of 4 channels, elements of which can
 33 |   be 2-D ndarrays or None;
 34 |   :param X: a 2-D array in which every row is a flattened image.
 35 | 
 36 |   :type img_shape: tuple; (height, width)
 37 |   :param img_shape: the original shape of each image
 38 | 
 39 |   :type tile_shape: tuple; (rows, cols)
 40 |   :param tile_shape: the number of images to tile (rows, cols)
 41 | 
 42 |   :param output_pixel_vals: if output should be pixel values (i.e. int8
 43 |   values) or floats
 44 | 
 45 |   :param scale_rows_to_unit_interval: if the values need to be scaled before
 46 |   being plotted to [0,1] or not
 47 | 
 48 | 
 49 |   :returns: array suitable for viewing as an image.
 50 |   (See:`Image.fromarray`.)
 51 |   :rtype: a 2-d array with same dtype as X.
 52 | 
 53 |   """
 54 | 
 55 |   assert len(img_shape) == 2
 56 |   assert len(tile_shape) == 2
 57 |   assert len(tile_spacing) == 2
 58 | 
 59 |   # The expression below can be re-written in a more C style as
 60 |   # follows :
 61 |   #
 62 |   # out_shape    = [0,0]
 63 |   # out_shape[0] = (img_shape[0]+tile_spacing[0])*tile_shape[0] -
 64 |   #                tile_spacing[0]
 65 |   # out_shape[1] = (img_shape[1]+tile_spacing[1])*tile_shape[1] -
 66 |   #                tile_spacing[1]
 67 |   out_shape = [
 68 |       (ishp + tsp) * tshp - tsp
 69 |       for ishp, tshp, tsp in zip(img_shape, tile_shape, tile_spacing)
 70 |   ]
 71 | 
 72 |   if isinstance(X, tuple):
 73 |       assert len(X) == 4
 74 |       # Create an output numpy ndarray to store the image
 75 |       if output_pixel_vals:
 76 |           out_array = numpy.zeros((out_shape[0], out_shape[1], 4),
 77 |                                   dtype='uint8')
 78 |       else:
 79 |           out_array = numpy.zeros((out_shape[0], out_shape[1], 4),
 80 |                                   dtype=X.dtype)
 81 | 
 82 |       # colors default to 0, alpha defaults to 1 (opaque)
 83 |       if output_pixel_vals:
 84 |           channel_defaults = [0, 0, 0, 255]
 85 |       else:
 86 |           channel_defaults = [0., 0., 0., 1.]
 87 | 
 88 |       for i in xrange(4):
 89 |           if X[i] is None:
 90 |               # if channel is None, fill it with zeros of the correct
 91 |               # dtype
 92 |               dt = out_array.dtype
 93 |               if output_pixel_vals:
 94 |                   dt = 'uint8'
 95 |               out_array[:, :, i] = numpy.zeros(
 96 |                   out_shape,
 97 |                   dtype=dt
 98 |               ) + channel_defaults[i]
 99 |           else:
100 |               # use a recurrent call to compute the channel and store it
101 |               # in the output
102 |               out_array[:, :, i] = tile_raster_images(
103 |                   X[i], img_shape, tile_shape, tile_spacing,
104 |                   scale_rows_to_unit_interval, output_pixel_vals)
105 |       return out_array
106 | 
107 |   else:
108 |       # if we are dealing with only one channel
109 |       H, W = img_shape
110 |       Hs, Ws = tile_spacing
111 | 
112 |       # generate a matrix to store the output
113 |       dt = X.dtype
114 |       if output_pixel_vals:
115 |           dt = 'uint8'
116 |       out_array = numpy.zeros(out_shape, dtype=dt)
117 | 
118 |       for tile_row in xrange(tile_shape[0]):
119 |           for tile_col in xrange(tile_shape[1]):
120 |               if tile_row * tile_shape[1] + tile_col < X.shape[0]:
121 |                   this_x = X[tile_row * tile_shape[1] + tile_col]
122 |                   if scale_rows_to_unit_interval:
123 |                       # if we should scale values to be between 0 and 1
124 |                       # do this by calling the `scale_to_unit_interval`
125 |                       # function
126 |                       this_img = scale_to_unit_interval(
127 |                           this_x.reshape(img_shape))
128 |                   else:
129 |                       this_img = this_x.reshape(img_shape)
130 |                   # add the slice to the corresponding position in the
131 |                   # output array
132 |                   c = 1
133 |                   if output_pixel_vals:
134 |                       c = 255
135 |                   out_array[
136 |                       tile_row * (H + Hs): tile_row * (H + Hs) + H,
137 |                       tile_col * (W + Ws): tile_col * (W + Ws) + W
138 |                   ] = this_img * c
139 |       return out_array
140 | 


--------------------------------------------------------------------------------
/code/requirements.txt:
--------------------------------------------------------------------------------
 1 | argparse==1.2.1
 2 | chardet==2.0.1
 3 | colorama==0.2.5
 4 | html5lib==0.999
 5 | numpy==1.11.2
 6 | requests==2.2.1
 7 | six==1.10.0
 8 | urllib3==1.7.1
 9 | virtualenv==15.0.3
10 | wheel==0.24.0
11 | wsgiref==0.1.2
12 | protobuf==3.0.0b2


--------------------------------------------------------------------------------
/code/run.py:
--------------------------------------------------------------------------------
 1 | import os
 2 | import shutil
 3 | 
 4 | from ae.utils.flags import FLAGS, home_out
 5 | import ae.autoencoder as autoencoder
 6 | from ae.utils.start_tensorboard import start
 7 | 
 8 | 
 9 | _data_dir = FLAGS.data_dir
10 | _summary_dir = FLAGS.summary_dir
11 | _chkpt_dir = FLAGS.chkpt_dir
12 | 
13 | 
14 | def _check_and_clean_dir(d):
15 |   if os.path.exists(d):
16 |     shutil.rmtree(d)
17 |   os.mkdir(d)
18 | 
19 | 
20 | def main():
21 |   home = home_out('')
22 |   if not os.path.exists(home):
23 |     os.makedirs(home)
24 |   if not os.path.exists(_data_dir):
25 |     os.mkdir(_data_dir)
26 | 
27 |   _check_and_clean_dir(_summary_dir)
28 |   _check_and_clean_dir(_chkpt_dir)
29 | 
30 |   os.mkdir(os.path.join(_chkpt_dir, '1'))
31 |   os.mkdir(os.path.join(_chkpt_dir, '2'))
32 |   os.mkdir(os.path.join(_chkpt_dir, '3'))
33 |   os.mkdir(os.path.join(_chkpt_dir, 'fine_tuning'))
34 | 
35 |   start()
36 | 
37 |   ae = autoencoder.main_unsupervised()
38 |   autoencoder.main_supervised(ae)
39 | 
40 | if __name__ == '__main__':
41 |     main()
42 | 


--------------------------------------------------------------------------------
/cpu/Dockerfile:
--------------------------------------------------------------------------------
1 | FROM gcr.io/tensorflow/tensorflow
2 | ENV AE_HOME "/usr/local/src/ae"
3 | ADD code/ "$AE_HOME"
4 | WORKDIR "$AE_HOME"
5 | CMD ["python", "run.py"]
6 | 


--------------------------------------------------------------------------------
/filters_1.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/cmgreen210/TensorFlowDeepAutoencoder/5298ec437689ba7ecb59229599141549ef6a6a1d/filters_1.png


--------------------------------------------------------------------------------
/setup_linux:
--------------------------------------------------------------------------------
 1 | #!/usr/bin/env bash
 2 | 
 3 | VENV="venv"
 4 | CPU_LOC="https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.5.0-cp27-none-linux_x86_64.whl"
 5 | GPU_LOC="https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.5.0-cp27-none-linux_x86_64.whl"
 6 | GPU=0
 7 | 
 8 | function usage
 9 | {
10 |     echo "Usage: sudo setup_linux [--gpu]"
11 |     echo ""
12 |     echo "Help: setup_linux --help"
13 | }
14 | 
15 | if [ $# -gt 0 ]; then
16 |     case $1 in
17 |             -g | --gpu )            shift
18 |                                     GPU=1
19 |                                     ;;
20 |             -h | --help )           usage
21 |                                     exit
22 |                                     ;;
23 |             * )                     usage
24 |                                     exit 1
25 |         esac
26 |         shift
27 | fi
28 | LOC="$CPU_LOC"
29 | if [ "$GPU" -eq 1 ]; then
30 |     LOC="$GPU_LOC"
31 | fi
32 | 
33 | sudo apt-get install python-pip python-dev python-virtualenv
34 | 
35 | virtualenv --system-site-packages "$VENV"
36 | source "$VENV/bin/activate"
37 | pip install -r requirements.txt
38 | pip install --upgrade "$LOC"
39 | 


--------------------------------------------------------------------------------
/setup_mac:
--------------------------------------------------------------------------------
 1 | #!/usr/bin/env bash
 2 | 
 3 | VENV="venv"
 4 | 
 5 | sudo easy_install pip
 6 | sudo pip install --upgrade virtualenv
 7 | virtualenv --system-site-packages "$VENV"
 8 | source "$VENV/bin/activate"
 9 | 
10 | pip install -r requirements.txt
11 | pip install --upgrade https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-0.11.0rc1-py2-none-any.whl
12 | 


--------------------------------------------------------------------------------
/tb_hist.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/cmgreen210/TensorFlowDeepAutoencoder/5298ec437689ba7ecb59229599141549ef6a6a1d/tb_hist.png


--------------------------------------------------------------------------------