└── README.md /README.md: -------------------------------------------------------------------------------- 1 | # Tensorflow Snippets From the Field. 2 | 3 | ## Table of Contents 4 | - [System Setup](#system-setup) 5 | - [On Convolutions](#on-convolutions) 6 | - [Indexing](#indexing) 7 | - [Numerical Stability](#numerical-stability) 8 | - [Shapes](#shapes) 9 | - [Tensor Contraction (More Generalized Matrix Multiplication)](#tensor-contraction-more-generalized-matrix-multiplication) 10 | - [```tf.estimator``` API](#tfestimator-api) 11 | - [Load A saved_model and Run Inference (in Python)](#load-a-saved_model-and-run-inference-in-python) 12 | - [Input Features! ```tf.train.Example``` and ```tf.train.SequenceExample```](#input-features-tftrainexample-and-tftrainsequenceexample) 13 | - [Misc](#misc) 14 | 15 | #### System Setup 16 | - Install CUDA 10.0 on Ubuntu 18.04 LTS GPU server: 17 | ```sh 18 | # 1. Install NVIDIA driver either through "Additional Drivers", or: 19 | $ sudo apt install --no-install-recommends nvidia-driver-430 20 | # Reboot and then check that GPUs are visible using the command: nvidia-smi. 21 | 22 | # 2. Add NVIDIA package repositories 23 | $ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_10.0.130-1_amd64.deb 24 | $ sudo dpkg -i cuda-repo-ubuntu1804_10.0.130-1_amd64.deb 25 | $ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub 26 | $ sudo apt update 27 | $ wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb 28 | $ sudo apt install ./nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb 29 | $ sudo apt update 30 | 31 | # 3. Install development and runtime libraries. 32 | $ sudo apt install --no-install-recommends \ 33 | cuda-10-0 \ 34 | libcudnn7=7.6.2.24-1+cuda10.0 \ 35 | libcudnn7-dev=7.6.2.24-1+cuda10.0 36 | 37 | # 4. Install TensorRT. Requires that libcudnn7 is installed above. 38 | $ sudo apt install -y --no-install-recommends libnvinfer5=5.1.5-1+cuda10.0 \ 39 | libnvinfer-dev=5.1.5-1+cuda10.0 40 | ``` 41 | - Install Tensorflow 2.0 GPU version: 42 | ```sh 43 | $ python3 -m pip install --upgrade pip 44 | $ python3 -m pip install --user tensorflow-gpu 45 | ``` 46 | 47 | - Setup SSH server on GPU server: 48 | * Install OpenSSH server: `$ sudo apt install openssh-server`. 49 | * Add port forwarding rule for port 22. 50 | 51 | - Setup SSH client on our ultra book: 52 | * Create SSH key: `$ ssh-keygen -t rsa -b 4096`. 53 | * Install SSH key on the GPU server as an authorized key: `$ ssh-id-copy @`. 54 | * Now we can connect to the GPU server by: `$ ssh -i @`. 55 | 56 | - Add [PyCharm remote Python interpreter](https://www.jetbrains.com/help/pycharm/configuring-remote-interpreters-via-ssh.html) on GPU server via SSH. 57 | 58 | - Happy machine learning! 59 | 60 | #### On Convolutions 61 | - Typically there are two options for ```padding```: 62 | * ```SAME```: Make sure result has *same* spatial shape as input tensor, this often requires padding 0's to input tensor. 63 | * ```VALID```: No paddings please, only use *valid* points . Result can have different spatial shape. 64 | - Tensorflow by default performs *centered* convolution (kernel is centered around current point). With ```k```: kernel size, ```d```: dilation rate, then extended kernel size ```k' = d * (k - 1) + 1```. For each spatial dimension, convolution at index ```i``` is computed using points between indices (inclusive) ```[i - (k' - 1) // 2, i + k' // 2]```. 65 | - *Causal* convolution uses points between indices ```[i - d * (k - 1), i]```, a simple solution is to pad ```d * (k - 1)``` of 0's at the beginning of that dimension then perform a normal convolution with ```VALID``` padding. 66 | - Convolution kernel has shape ```[spatial_dim[0], ..., spatial_dim[n - 1], num_input_channels, num_output_channels]```. For each output channel ```k```, ```output[..., k] = sum_over_i {input[..., i] * kernel[..., i, k]}```, here ```*``` is convolution operator. 67 | 68 | #### Indexing 69 | - ```tf.gather_nd(params, indices)``` retrieves slices from ```params``` by ```indices```. The rule is simple: *only the last dimension of ```indices``` does slice ```params```, and that dimension is "replaced" with those slices*. It's easy to see that: 70 | * ```indices.shape[-1] <= rank(params)```: The last dimension of ```indices``` must be no greater than the rank of ```params```. 71 | * Result tensor shape is ```indices.shape[:-1] + params.shape[indices.shape[-1]:]```, example: 72 | ```python 73 | # params has shape [4, 5, 6]. 74 | params = tf.reshape(tf.range(0, 120), [4, 5, 6]) 75 | # indices has shape [3, 2]. 76 | indices = tf.constant([[2, 3], [0, 1], [1, 2]], dtype=tf.int32) 77 | # slices has shape [3, 6]. 78 | slices = tf.gather_nd(params, indices) 79 | ``` 80 | - ```tf.gather_nd``` and Numpy fancy indexing: ```x[indices]``` == ```tf.gather_nd(x, zip(*indices))```; ```tf.gather_nd(x, indices)``` == ```x[zip(*indices)]```. Where ```x``` is Numpy array and ```indices``` is indexing array (dim > 1). 81 | 82 | #### Numerical Stability 83 | - ```Inf``` morphs to ```NaN``` while plugged into back-prop (chain rule). 84 | - Watch out! ```tf.where``` Can Spawn NaN in Gradients 85 | If either branch in ```tf.where``` contains Inf/NaN then it produces NaN in gradients, e.g.: 86 | ```python 87 | log_s = tf.constant([-100., 100.], dtype=tf.float32) 88 | # Computes 1.0 / exp(log_s), in a numerically robust way. 89 | inv_s = tf.where(log_s >= 0., 90 | tf.exp(-log_s), # Creates Inf when -log_s is large. 91 |                 1. / (tf.exp(log_s) + 1e-6)) # tf.exp(log_s) is Inf with large log_s. 92 | grad_log_s = tf.gradients(inv_s, [log_s]) 93 | with tf.Session() as sess: 94 | inv_s, grad_log_s = sess.run([inv_s, grad_log_s]) 95 |    print(inv_s) # [ 1.00000000e+06 3.78350585e-44] 96 |    print(grad_log_s) # [array([ nan, nan], dtype=float32)] 97 | ``` 98 | 99 | #### Shapes 100 | - ```tensor.shape``` returns tensor's static shape, while the graph is being built. 101 | - ```tensor.shape.as_list()``` returns the static shape as a integer list. 102 | - ```tensor.shape[i].value``` returns the static shape's i-th dimension size as an integer. 103 | - ```tf.shape(t)``` returns t's run-time shape as a tensor. 104 | - An example: 105 | ```python 106 | x = tf.placeholder(tf.float32, shape=[None, 8]) # x shape is non-deterministic while building the graph. 107 | print(x.shape) # Outputs static shape (?, 8). 108 | shape_t = tf.shape(x) 109 | with tf.Session() as sess: 110 | print(sess.run(shape_t, feed_dict={x: np.random.random(size=[4, 8])})) # Outputs run-time shape (4, 8). 111 | ``` 112 | - [] (empty square brackets) as a shape denotes a scalar (0 dim). E.g. tf.FixedLenFeature([], ..) is a scalar feature. 113 | - Broadcasting on two arrays starts with the _trailing_ dimensions, and works its way _backward_ to the leading dimensions. E.g. 114 | ``` 115 | A (4d array): 8 x 1 x 6 x 1 116 | B (3d array): 7 x 1 x 5 117 | Result (4d array): 8 x 7 x 6 x 5 118 | ``` 119 | 120 | #### Tensor Contraction (More Generalized Matrix Multiplication) 121 | ```python 122 | # Matrix multiplication 123 | tf.einsum('ij,jk->ik', m0, m1) # output[i, k] = sum_j m0[i, j] * m1[j, k] 124 | # Dot product 125 | tf.einsum('i,i->', u, v) # output = sum_i u[i]*v[i] 126 | # Outer product 127 | tf.einsum('i,j->ij', u, v) # output[i, j] = u[i]*v[j] 128 | # Transpose 129 | tf.einsum('ij->ji', m) # output[j, i] = m[i,j] 130 | # Batch matrix multiplication 131 | tf.einsum('aij,jk->aik', s, t) # out[a, i, k] = sum_j s[a, i, j] * t[j, k] 132 | # Batch tensor contraction 133 | tf.einsum('nhwc,nwcd->nhd', s, t) # out[n, h, d] = sum_w_c s[n, h, w, c] * t[n, w, c, d] 134 | ``` 135 | 136 | #### ```tf.estimator``` API 137 | - A typical input_fn (used for train/eval) for tf.estimator API: 138 | ```python 139 | def make_input_fn(mode, ...): 140 | """Return input_fn for train/eval in tf.estimator API. 141 | 142 | Args: 143 | mode: Must be tf.estimator.ModeKeys.TRAIN or tf.estimator.ModeKeys.EVAL. 144 | ... 145 | Returns: 146 | The input_fn. 147 | """ 148 | def _input_fn(): 149 | """The input function. 150 | 151 | Returns: 152 | features: A dict of {'feature_name': feature_tensor}. 153 | labels: A tensor of labels. 154 | """ 155 | if mode == tf.estimator.ModeKeys.TRAIN: 156 | features = ... 157 | labels = ... 158 | elif mode == tf.estimator.ModeKeys.EVAL: 159 | features = ... 160 | labels = ... 161 | else: 162 | raise ValueError(mode) 163 | return features, labels 164 | 165 | return _input_fn 166 | ``` 167 | 168 | - A typical model_fn for tf.estimator API: 169 | ```python 170 | def make_model_fn(...): 171 | """Return model_fn to build a tf.estimator.Estimator. 172 | 173 | Args: 174 | ... 175 | Returns: 176 | The model_fn. 177 | """ 178 | def _model_fn(features, labels, mode): 179 | """Model function. 180 | 181 | Args: 182 | features: The first item returned from the input_fn for train/eval, a dict of {'feature_name': feature_tensor}. If mode is ModeKeys.PREDICT, same as in serving_input_receiver_fn. 183 | labels: The second item returned from the input_fn, a single Tensor or dict. If mode is ModeKeys.PREDICT, labels=None will be passed. 184 | mode: Optional. Specifies if this training, evaluation or prediction. See ModeKeys. 185 | """ 186 | if mode == tf.estimator.ModeKeys.PREDICT: 187 | # Calculate the predictions. 188 | predictions = ... 189 | # For inference/prediction outputs. 190 | export_outputs = { 191 | tf.saved_model.signature_constants.PREDICT_METHOD_NAME: tf.estimator.export.PredictOutput({ 192 | 'output_1': predict_output_1, 193 | 'output_2': predict_output_2, 194 | ... 195 | }), 196 | } 197 | ... 198 | else: 199 | predictions = None 200 | export_outputs = None 201 | 202 | if (mode == tf.estimator.ModeKeys.TRAIN or mode == tf.estimator.ModeKeys.EVAL): 203 | loss = ... 204 | else: 205 | loss = None 206 | 207 | if mode == tf.estimator.ModeKeys.TRAIN: 208 | train_op = ... 209 | # Can use tf.group(..) to group multiple train_op as a single train_op. 210 | else: 211 | train_op = None 212 | 213 | return tf.estimator.EstimatorSpec( 214 | mode=mode, 215 | predictions=predictions, 216 | loss=loss, 217 | train_op=train_op, 218 | export_outputs=export_outputs) 219 | 220 | return _model_fn 221 | ``` 222 | 223 | - Use tf.estimator.Estimator to export a saved_model: 224 | ```python 225 | # serving_features must match features in model_fn when mode == tf.estimator.ModeKeys.PREDICT. 226 | serving_features = {'serving_input_1': tf.placeholder(...), 'serving_input_2': tf.placeholder(...), ...} 227 | estimator.export_savedmodel(export_dir, 228 | tf.estimator.export.build_raw_serving_input_receiver_fn(serving_features)) 229 | ``` 230 | 231 | - Use tf.contrib.learn.Experiment to export a saved_model: 232 | ```python 233 | # serving_features must match features in model_fn when mode == tf.estimator.ModeKeys.PREDICT. 234 | serving_features = {'serving_input_1': tf.placeholder(...), 'serving_input_2': tf.placeholder(...), ...} 235 | export_strategy = tf.contrib.learn.utils.make_export_strategy(tf.estimator.export.build_raw_serving_input_receiver_fn(serving_features)) 236 | expriment = tf.contrib.learn.Experiment(..., export_strategies=[export_strategy], ...) 237 | ``` 238 | 239 | #### Load A saved_model and Run Inference (in Python) 240 | ```python 241 | with tf.Session(...) as sess: 242 | # Load saved_model MetaGraphDef from export_dir. 243 | meta_graph_def = tf.saved_model.loader.load(sess, [tf.saved_model.tag_constants.SERVING], export_dir) 244 | 245 | # Get SignatureDef for serving (here PREDICT_METHOD_NAME is used as export_outputs key in model_fn). 246 | sigs = meta_graph_def.signature_def[tf.saved_model.signature_constants.PREDICT_METHOD_NAME] 247 | 248 | # Get the graph for retrieving input/output tensors. 249 | g = tf.get_default_graph() 250 | 251 | # Retrieve serving input tensors, keys must match keys defined in serving_features (when building input receiver fn). 252 | input_1 = g.get_tensor_by_name(sigs.inputs['input_1'].name) 253 | input_2 = g.get_tensor_by_name(sigs.inputs['input_2'].name) 254 | ... 255 | 256 | # Retrieve serving output tensors, keys must match keys defined in ExportOutput (e.g. PredictOutput) in export_outputs. 257 | output_1 = g.get_tensor_by_name(sigs.outputs['output_1'].name) 258 | output_2 = g.get_tensor_by_name(sigs.outputs['output_2'].name) 259 | ... 260 | 261 | # Run inferences. 262 | outputs_values = sess.run([output_1, output_2, ...], feed_dict={input_1: ..., input_2: ..., ...}) 263 | ``` 264 | 265 | #### Input Features! ```tf.train.Example``` and ```tf.train.SequenceExample``` 266 | - A [tf.train.Example](https://github.com/tensorflow/tensorflow/blob/r1.3/tensorflow/core/example/example.proto#L88) is roughly a map of *{feature_name: value_list}*. 267 | - A [tf.train.SequenceExample](https://github.com/tensorflow/tensorflow/blob/r1.3/tensorflow/core/example/example.proto#L292) is roughly a ```tf.train.Example``` plus a map of *{feature_name: list_of_value_lists}*. 268 | - Build a tf.train.Example in Python: 269 | ```python 270 | # ==================== Build in one line ==================== 271 | example = tf.train.Example(features=tf.train.Features(feature={ 272 | 'bytes_values': tf.train.Feature( 273 | bytes_list=tf.train.BytesList(value=[bytes_feature])), 274 | 'float_values': tf.train.Feature( 275 | float_list=tf.train.FloatList(value=[float_feature])), 276 | 'int64_values': tf.train.Feature( 277 | int64_list=tf.train.Int64List(value=[int64_feature])), 278 | ... 279 | })) 280 | # ==================== OR progressivly ==================== 281 | example = tf.train.Example() 282 | example.features.feature['bytes_feature'].bytes_list.value.extend(bytes_values) 283 | example.features.feature['float_feature'].float_list.value.extend(float_values) 284 | example.features.feature['int64_feature'].int64_list.value.extend(int64_values) 285 | ... 286 | ``` 287 | - Build a tf.train.SequenceExample in Python: 288 | ```python 289 | sequence_example = tf.train.SequenceExample() 290 | 291 | # Populate context data. 292 | sequence_example.context.feature[ 293 | 'context_bytes_values_1'].bytes_list.value.extend(bytes_values) 294 | sequence_example.context.feature[ 295 | 'context_float_values_1'].float_list.value.extend(float_values) 296 | sequence_example.context.feature[ 297 | 'context_int64_values_1'].int64_list.value.extend(int64_values) 298 | ... 299 | 300 | # Populate sequence data. 301 | feature_list_1 = sequence_example.feature_lists.feature_list['feature_list_1'] 302 | # Add tf.train.Feature to feature_list_1. 303 | feature_1 = feature_list_1.feature.add() 304 | # Populate feature_1, e.g. feature_1.float_list.value.extend(float_values) 305 | # Add tf.train.Feature to feature_list_1, if any. 306 | ... 307 | ``` 308 | 309 | - To parse a SequenceExample: 310 | ```python 311 | tf.parse_single_sequence_example(serialized, 312 | context_features={ 313 | 'context_feature_1': tf.FixedLenFeature([], dtype=...), 314 | ... 315 | }, 316 | sequence_features={ 317 | # For 'sequence_features_1' shape, [] results with [?] and [k] results with [?, k], where: 318 | # ?: timesteps, i.e. number of tf.Train.Feature in 'sequence_features_1' list, can be variable. 319 | # k: number of elements in each tf.Train.Feature in 'sequence_features_1'. 320 | 'sequence_features_1': tf.FixedLenSequenceFeature([], dtype=...), 321 | ... 322 | },) 323 | ``` 324 | 325 | - Write tfrecords to sharded files. Reading data from multiple input files can increase I/O throughput in TF: 326 | ```python 327 | import multiprocessing 328 | from concurrent import futures 329 | 330 | def write_tf_records(file_path, tf_records): 331 | """Writes TFRecord (Example or SequenceExample) data to output file. 332 | 333 | :param file_path: the full output file path, can add `@N` at the end that 334 | specifies total number of shards, e.g. /data/training.tfrecords@10. 335 | :param tf_records: a list or tuple of `tf.train.Example` or 336 | `tf.train.SequenceExample` instances. 337 | """ 338 | 339 | def _write_data_to_file(file_path, tf_records): 340 | """Writes data into specified output file path. 341 | 342 | :param file_path: full path of output file. 343 | :param tf_records: a list of Example or SequenceExample instances. 344 | """ 345 | with tf.python_io.TFRecordWriter(file_path) as writer: 346 | for tf_record in tf_records: 347 | writer.write(tf_record.SerializeToString()) 348 | 349 | def _get_shards_paths(file_path): 350 | """Gets (shard) file paths by parsing from provided file path. 351 | 352 | :param file_path: file path that may contain shard syntax "@N". 353 | :return: a list of file paths. 354 | """ 355 | shard_char_idx = file_path.rfind('@') 356 | 357 | # No shards specified. 358 | if shard_char_idx == -1: 359 | return [file_path] 360 | 361 | num_shards = int(file_path[shard_char_idx + 1:]) 362 | if num_shards <= 0: 363 | raise ValueError('Number of shards must be a positive integer.') 364 | prefix = file_path[:shard_char_idx] 365 | return ['{}-{}-of-{}'.format(prefix, i, num_shards) for i 366 | in range(num_shards)] 367 | 368 | if not isinstance(tf_records, list) and not isinstance(tf_records, tuple): 369 | raise TypeError('tf_records must be a list or tuple.') 370 | 371 | tf_records = list(tf_records) 372 | shards_paths = _get_shards_paths(file_path) 373 | 374 | if len(shards_paths) > len(tf_records): 375 | raise ValueError('More data than file shards.') 376 | 377 | with futures.ThreadPoolExecutor( 378 | max_workers=multiprocessing.cpu_count() - 1) as executor: 379 | for shard_id, file_path in enumerate(shards_paths): 380 | executor.submit(_write_data_to_file, file_path, 381 | tf_records[shard_id::len(shards_paths)]) 382 | ``` 383 | 384 | #### Misc 385 | - Don't Forget to Reset Default Graph in Jupyter Notebook 386 | If you forgot to reset default Tensorflow graph (or create a new graph) in a Jupyter notebook cell, and run that cell for a few times then you may get weird results. 387 | - Visualize Tensorflow Graph in Jupyter Notebook 388 | ```python 389 | import numpy as np 390 | from IPython import display 391 | 392 | def strip_consts(graph_def, max_const_size=32): 393 | """Strip large constant values from graph_def.""" 394 | strip_def = tf.GraphDef() 395 | for n0 in graph_def.node: 396 | n = strip_def.node.add() 397 | n.MergeFrom(n0) 398 | if n.op == 'Const': 399 | tensor = n.attr['value'].tensor 400 | size = len(tensor.tensor_content) 401 | if size > max_const_size: 402 | tensor.tensor_content = "".format(size) 403 | return strip_def 404 | 405 | 406 | def show_graph(graph_def, max_const_size=32): 407 | """Visualize TensorFlow graph.""" 408 | if hasattr(graph_def, 'as_graph_def'): 409 | graph_def = graph_def.as_graph_def() 410 | strip_def = strip_consts(graph_def, max_const_size=max_const_size) 411 | code = """ 412 | 417 | 419 |
420 | 421 |
422 | """.format(data=repr(str(strip_def)), id='graph' + str(np.random.rand())) 423 | 424 | iframe = """ 425 | 426 | """.format(code.replace('"', '"')) 427 | display.display(display.HTML(iframe)) 428 | ``` 429 | Then call ```show_graph(tf.get_default_graph())``` to show in your Jupyter/IPython notebook. 430 | --------------------------------------------------------------------------------