Open `cifar10-sagemaker-distributed.ipynb` and run through the cells. The following notebook is located at:
8 | ***distributed-training-workshop > notebooks > part-2-sagemaker > cifar10-sagemaker-distributed.ipynb*** 9 | 10 |  11 | 12 | {{% notice warning %}} 13 | **Stop:** Do this section on JupyterLab. Below is a copy of the jupyter notebook for reference.
14 | {{% /notice %}} 15 | 16 | ---- 17 | 18 | 19 | ## Distributed training with Amazon SageMaker 20 | 21 | In this notebook we use the SageMaker Python SDK to setup and run a distributed training job. 22 | SageMaker makes it easy to train models across a cluster containing a large number of machines, without having to explicitly manage those resources. 23 | 24 | **Step 1:** Import essentials packages, start a sagemaker session and specify the bucket name you created in the pre-requsites section of this workshop. 25 | 26 | 27 | ```python 28 | import os 29 | import time 30 | import numpy as np 31 | import sagemaker 32 | 33 | sagemaker_session = sagemaker.Session() 34 | role = sagemaker.get_execution_role() 35 | bucket_name = 'tfworld2019-
https://tfworld2019.notebook.us-west-2.sagemaker.aws/proxy/6006/ 124 |
Make sure that the name of the notebook instance is correct in the link above. 125 |
Don't forget the slash at the end of the URL 6006/ 126 | -------------------------------------------------------------------------------- /content/sagemaker_dist_training/training_scrip_updates.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Updates required to run on SageMaker" 3 | date: 2019-10-28T13:42:13-07:00 4 | weight: 2 5 | --- 6 | 7 | There are few minor changes required to run a training script on Amazon Sagemaker 8 | 9 | 10 | ##### SageMaker hyperparameters 11 | * SageMaker passes hyperparameters to the training scripts as commandline arguments. Your script must be able to parse these arguments. 12 | 13 | ##### SageMaker environment variables 14 | * SageMaker makes several environment variables available inside the container that a training script can take advantage of for finding location of the training dataset, number of GPU in the instance, dataset channels and others. A full list of environment variables an be found on the [SageMaker container GitHub repository](https://github.com/aws/sagemaker-containers#important-environment-variables) 15 | 16 | ``` 17 | parser = argparse.ArgumentParser() 18 | 19 | # Hyper-parameters 20 | parser.add_argument('--epochs', type=int, default=15) 21 | parser.add_argument('--learning-rate', type=float, default=0.001) 22 | parser.add_argument('--batch-size', type=int, default=256) 23 | parser.add_argument('--weight-decay', type=float, default=2e-4) 24 | parser.add_argument('--momentum', type=float, default='0.9') 25 | parser.add_argument('--optimizer', type=str, default='adam') 26 | 27 | # SageMaker parameters 28 | parser.add_argument('--model_dir', type=str) 29 | parser.add_argument('--model_output_dir', type=str, default=os.environ['SM_MODEL_DIR']) 30 | parser.add_argument('--output_data_dir', type=str, default=os.environ['SM_OUTPUT_DATA_DIR']) 31 | 32 | # Data directories and other options 33 | parser.add_argument('--gpu-count', type=int, default=os.environ['SM_NUM_GPUS']) 34 | parser.add_argument('--train', type=str, default=os.environ['SM_CHANNEL_TRAIN']) 35 | parser.add_argument('--validation', type=str, default=os.environ['SM_CHANNEL_VALIDATION']) 36 | parser.add_argument('--eval', type=str, default=os.environ['SM_CHANNEL_EVAL']) 37 | 38 | args = parser.parse_args() 39 | ``` 40 | 41 | ##### (Optional) TensorBoard callback for real-time monitoring of training 42 | * Using a keras callback we can upload tensorboard files to Amazon S3 so that we can monitor progress in real-time. 43 | tensorboard already comes installed on the SageMaker JupyterLab instance, and has support for reading event files from Amazon S3. 44 | 45 | `tensorboard --logdir s3://{bucket_name}/tensorboard_logs/` 46 | 47 | ``` 48 | class Sync2S3(tf.keras.callbacks.Callback): 49 | def __init__(self, logdir, s3logdir): 50 | super(Sync2S3, self).__init__() 51 | self.logdir = logdir 52 | self.s3logdir = s3logdir 53 | 54 | def on_epoch_end(self, batch, logs={}): 55 | os.system('aws s3 sync '+self.logdir+' '+self.s3logdir) 56 | # ' >/dev/null 2>&1' 57 | ``` 58 | -------------------------------------------------------------------------------- /content/sagemaker_dist_training/workflow.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Workflow" 3 | date: 2019-10-28T12:59:15-07:00 4 | weight: 1 5 | --- 6 | Navigate to 7 | ***distributed-training-workshop > notebooks > part-2-sagemaker*** 8 | You should see the following files: 9 | 10 | ```bash 11 | part-2-sagemaker/ 12 | ├── cifar10-sagemaker-distributed.ipynb 13 | └── code 14 | ├── cifar10-multi-gpu-horovod-sagemaker.py 15 | └── model_def.py 16 | ``` 17 | 18 | |Files/directories|Description| 19 | |-----|-----| 20 | |cifar10-sagemaker-distributed.ipynb |This jupyter notebook contains code to define and kick off a SageMaker training job| 21 | |code |This directory contains the training scrip and other training script dependencies| 22 | 23 |  24 | 25 | SageMaker is a fully-managed service, which means when you kick off a training job using the SageMaker SDK in the `cifar10-sagemaker-distributed.ipynb` notebook, few different things happen behind the scene 26 | 27 | * SageMaker spins up request number of instances in a fully-managed SageMaker cluster 28 | * SageMaker pulls the latest (or specified version) of TensorFlow container images, instantiates it on the new instances and loads the content of the `code` directory into the container 29 | * SageMaker runs the training script on each instance. Since we're running distributed training SageMaker launches an `MPI` job with the right settings so that workers can communicate with each other. 30 | * SageMaker copies the dataset over from Amazon S3 and makes it available inside the container for Training 31 | * SageMaker monitors the training and updates progress on the Amazon SageMaker console 32 | * SageMaker copies all the code and model artifacts to Amazon S3 after the training is finished 33 | 34 | In addition, SageMaker does a lot more to ensure that the jobs run optimally and you get the best perfomance out-of-the box. As a user you don't have to worry about managing machine learning infrastructure. 35 | -------------------------------------------------------------------------------- /content/setup/_index.md: -------------------------------------------------------------------------------- 1 | +++ 2 | title = "Prerequisites" 3 | date = 2019-10-21T13:30:44-07:00 4 | weight = 2 5 | chapter = true 6 | #pre = "0. " 7 | +++ 8 | 9 | # Getting Started 10 | In this section, we'll setup up our development environment. 11 | We'll be using an Amazon SageMaker notebook instance which is a fully managed compute instance running the Jupyter Notebook server. 12 | -------------------------------------------------------------------------------- /content/setup/add_admin_policy.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Update notebook IAM role" 3 | date: 2019-10-27T23:41:36-07:00 4 | weight: 3 5 | --- 6 | 7 | ### Give your notebook instances admin privileges 8 | {{% notice warning %}} 9 | **Note:** We're providing admin privileges to the SageMaker notebook instance only because we'll be using the same instance to launch an Amazon EKS cluster in the later part of the workshop. If you're only going to be using SageMaker managed cluster for training, S3 full access policy should suffice. 10 | {{% /notice %}} 11 | 12 | * Click on the **tfworld2019** and you'll see additional details about the instance. Click on the IAM role link, this should take you to the IAM Management Console. Once there, click attach policy button. 13 |  14 |  15 | * Select **AdministratorAccess** and click on **Attach policy** 16 |  17 | * Close the the IAM Management Console window and head back to the SageMaker console. 18 | -------------------------------------------------------------------------------- /content/setup/download_workshop.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Download the workshop content" 3 | date: 2019-10-28T00:14:06-07:00 4 | weight: 4 5 | --- 6 | ### Launch JupyterLab client and clone the workshop repository 7 | * Your notebook instance should now be ready. Click *JupyterLab* to launch your client. 8 |  9 | 10 | * Click *File > New > Terminal* to launch terminal in your JupyterLab instance. 11 |  12 | 13 | * Download the workshop code and notebooks. Enter bash (optional), change directory to ~/SageMaker, clone the repository 14 | ```bash 15 | bash 16 | cd ~/SageMaker 17 | git clone https://github.com/shashankprasanna/distributed-training-workshop.git 18 | ``` 19 | 20 | * Confirm that you're able to see the contents. Should see 3 parts 21 | ``` 22 | ls distributed-training-workshop/notebooks 23 | ``` 24 | -------------------------------------------------------------------------------- /content/setup/sm_jupyter_instance.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Launch a SageMaker notebook instance" 3 | date: 2019-10-27T22:39:43-07:00 4 | weight: 2 5 | --- 6 | 7 | {{% notice info %}} 8 | **Note:** In this workshop, we'll be using an Amazon SageMaker notebook instance for simplicity and convenience. You can use any local client to perform steps detailed in this and subsequent sections. You'll just need to make sure you have the right privileges to access AWS services such as SageMaker, EKS, S3, ECR and others from your clinet. You'll also need to install AWS Command Line Interface (AWS CLI), python, boto3 and SageMaker SDK installed. The SageMaker Jupyter notebook on the other hand is preconfigured and ready to use. 9 | {{% /notice %}} 10 | 11 | ### Launch an Amazon SageMaker notebook instance 12 | 13 | * Open the [AWS Management Console](https://console.aws.amazon.com/console/home) 14 | {{% notice info %}} 15 | **Note:** This workshop has been tested on the US West (Oregon) (us-west-2) region. Make sure that you see **Oregon** on the top right hand corner of your AWS Management Console. If you see a different region, click the dropdown menu and select US West (Oregon) 16 | {{% /notice %}} 17 | 18 | * In the AWS Console search bar, type SageMaker and select Amazon SageMaker to open the service console. 19 |  20 | * Click on Notebook Instances 21 |  22 | * From the Amazon SageMaker > Notebook instances page, select Create notebook instance. 23 |  24 | * In the Notebook instance name text box, enter a name for the notebook instance. 25 | * For this workshop select **"tfworld2019"** as the instance name 26 | * Choose ml.c5.xlarge. We'll only be using this instance to launch jobs. The training job themselves will run either on a SageMaker managed cluster or an Amazon EKS cluster 27 | * Volume size 50 - this is only needed for building docker containers. During training data is copied directly from Amazon S3 to the training cluster when using SageMaker. When using Amazon EKS, we'll setup an FSx for lustre file system that worker nodes will use to get access to training data. 28 |  29 | * To create an IAM role, from the IAM role drop-down list, select Create a new role. In the Create an IAM role dialog box, select Any S3 bucket. After that select Select **Create role**. Amazon SageMaker creates the **AmazonSageMaker-ExecutionRole-*** ** role. 30 |  31 | * Keep the default settings for the other options and click Create notebook instance. On the **Notebook instances** section you should see the status change from *Pending -> InService* 32 | * While the notebook instance spins up, continue to work on the next section, and we'll come back and launch the instance when it's ready. 33 | -------------------------------------------------------------------------------- /content/update_code_dist_training/_index.md: -------------------------------------------------------------------------------- 1 | +++ 2 | title = "Prepare your training scripts" 3 | date = 2019-10-28T00:51:31-07:00 4 | weight = 3 5 | chapter = true 6 | +++ 7 | 8 | # Prepare your training scripts 9 | 10 | In this section, we'll walk through the process of modifying an existing TensorFlow-Keras training script so that it can perform training in a distributed environment. 11 | -------------------------------------------------------------------------------- /content/update_code_dist_training/distributed_training_script.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Using horovod API for distributed training" 3 | date: 2019-10-28T02:47:30-07:00 4 | weight: 4 5 | --- 6 | 7 | ## Exercise 1: Convert training script to use horovod 8 | 9 |
In this section you'll update the training script with horovod API for run distributed training. 10 | 11 | Open `cifar10-distributed.ipynb` and run through the cells. The following notebook is located at:
12 | ***distributed-training-workshop > notebooks > part-1-horovod*** 13 | 14 |  15 | 16 | {{% notice warning %}} 17 | **Stop:** Do this section on JupyterLab. Below is a copy of the jupyter notebook for reference.
18 | Open `cifar10-distributed.ipynb` and run these cells.
19 | Look for cells that say **Change X** and fill in those cells with the modifications - where **X** is the change number. There are a total of 8 changes. 20 | Click on **> Solution** to see the answers 21 | {{% /notice %}} 22 | 23 | You'll need to make the following modifications to your training script to use horovod for distributed training. 24 | 25 | 1. Run hvd.init() 26 | 2. Pin a server GPU to be used by this process using config.gpu_options.visible_device_list. 27 | 3. Scale the learning rate by the number of workers. 28 | 4. Wrap the optimizer in hvd.DistributedOptimizer. 29 | 5. Add hvd.callbacks.BroadcastGlobalVariablesCallback(0) to broadcast initial variable states from rank 0 to all other processes. 30 | 6. Modify your code to save checkpoints only on worker 0 to prevent other workers from corrupting them. 31 | 32 | 33 | 34 | #### Change 1: Import horovod and keras backend 35 | 36 | 37 | ```python 38 | import tensorflow as tf 39 | 40 | 41 | 42 | ``` 43 | 44 |
Solution
45 |46 | import horovod.tensorflow.keras as hvd 47 | import tensorflow.keras.backend as K 48 |49 |
Solution
176 |177 | hvd.init() 178 | size = hvd.size() 179 |180 |
Solution
194 |195 | config = tf.ConfigProto() 196 | config.gpu_options.allow_growth = True 197 | config.gpu_options.visible_device_list = str(hvd.local_rank()) 198 | K.set_session(tf.Session(config=config)) 199 |200 |
Solution
215 |216 | opt = SGD(lr=lr * size, decay=weight_decay, momentum=momentum) 217 | 218 | You need to scale the learning using the size of the cluster (total number of workers) 219 |220 |
Solution
232 |233 | opt = hvd.DistributedOptimizer(opt) 234 | model.compile(loss='categorical_crossentropy', 235 | optimizer=opt, 236 | metrics=['accuracy']) 237 |238 |
Solution
251 |
252 | callbacks = []
253 | callbacks.append(hvd.callbacks.BroadcastGlobalVariablesCallback(0))
254 | callbacks.append(hvd.callbacks.MetricAverageCallback())
255 | callbacks.append(hvd.callbacks.LearningRateWarmupCallback(warmup_epochs=5, verbose=1))
256 | callbacks.append(tf.keras.callbacks.ReduceLROnPlateau(patience=10, verbose=1))
257 | if hvd.rank() == 0:
258 | callbacks.append(ModelCheckpoint('ckpt_dir' + '/checkpoint-{epoch}.h5'))
259 |
260 | Solution
278 |279 | history = model.fit(x=train_dataset[0], y=train_dataset[1], 280 | steps_per_epoch= (NUM_TRAIN_IMAGES // batch_size)// size, 281 | validation_data=val_dataset, 282 | validation_steps= (NUM_VALID_IMAGES // batch_size)// size, 283 | epochs=epochs, callbacks=callbacks) 284 |285 |
$ jupyter nbconvert --to script notebook_name.ipynb
300 |
--------------------------------------------------------------------------------
/content/update_code_dist_training/prepare_dataset.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: "Prepare training dataset"
3 | date: 2019-10-28T01:48:16-07:00
4 | weight: 2
5 | ---
6 | ### Download the CIFAR10 dataset and upload it to Amazon S3
7 |
8 | On a terminal window in JupyterLab client, navigate to the notebook directory
9 |
10 | ```
11 | cd ~/SageMaker/distributed-training-workshop/notebooks/
12 | ```
13 | Activate the TensorFlow conda environment
14 | ```
15 | source activate tensorflow_p36
16 | ```
17 |
18 | Download CIFAR10 dataset and convert it to TFRecords format
19 | ```
20 | python generate_cifar10_tfrecords.py --data-dir dataset
21 | ```
22 | Confirm that the dataset was downloaded successfully. Run:
23 | ```
24 | sudo yum install tree -y
25 | tree dataset
26 | ```
27 | You should see the following output
28 | ```
29 | dataset
30 | ├── eval
31 | │ └── eval.tfrecords
32 | ├── train
33 | │ └── train.tfrecords
34 | └── validation
35 | └── validation.tfrecords
36 | ```
37 |
38 | Create a new S3 bucket and upload the dataset to it. Be sure to add a unique identifier, such as your name.
39 | ```
40 | aws s3 mb s3://CIFAR-10 dataset includes: 12 | 13 | * 40,000 images for training 14 | * 10,000 images for validation 15 | * 10,000 images for test 16 | 17 | Here are the classes in the dataset, as well as 10 random images from each: 18 |  19 | 20 | {{% notice info %}} 21 | **Note:** Although the dataset is small and this is a simpler problem, all the steps we'll take can easily be applied to large datasets that don't fit in memory. Amazon SageMaker has native pipe-mode support to stream dataset directly from S3 to the training instances. With Amazon EKS, we'll setup an Amazon FSx for lustre file system that's accessible to every worker. 22 | {{% /notice %}} 23 | -------------------------------------------------------------------------------- /content/update_code_dist_training/single_instance_script.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Getting familiar with a single instance training script" 3 | date: 2019-10-28T02:38:32-07:00 4 | weight: 3 5 | --- 6 | 7 | ## Single CPU/GPU training on the local instance 8 | 9 |
In this section you'll get familiar with the training script we'll be converting for distributed training in the next notebook. 10 | 11 | Open `cifar10-single-instance.ipynb` and run through the cells. The following notebook is located at:
12 | ***distributed-training-workshop > notebooks > part-1-horovod*** 13 | 14 |  15 | 16 | {{% notice warning %}} 17 | **Stop:** Do this section on JupyterLab. Below is a copy of the jupyter notebook for reference.
18 | {{% /notice %}} 19 | 20 | 21 | ---- 22 | #### Below is a copy of the Jupyter notebook `cifar10-single-instance.ipynb` 23 | 24 | ---- 25 | 26 | This Jupyter notebook contains code that trains a DNN on the CIFAR10 dataset. 27 | The script was written for local training on a single instance. Run through this notebook either cell-by-cell or by hitting *Run > Run All Cells* 28 | 29 | Once you start to feel comfortable with what the script is doing, we'll then start to make changes to this script so that it can run on a cluster in a distributed fashion. 30 | 31 | **Step 1:** Import essentials packages and define constants 32 | 33 | 34 | ```python 35 | import tensorflow as tf 36 | import argparse 37 | from datetime import datetime 38 | import os 39 | from tensorflow import keras 40 | from tensorflow.keras.layers import Input, Dense, Flatten 41 | from tensorflow.keras.models import Model 42 | from tensorflow.keras.optimizers import Adam, SGD 43 | from tensorflow.keras.callbacks import TensorBoard, ModelCheckpoint 44 | 45 | # Import DNN model definition file 46 | from model_def import get_model 47 | 48 | HEIGHT = 32 49 | WIDTH = 32 50 | DEPTH = 3 51 | NUM_CLASSES = 10 52 | NUM_TRAIN_IMAGES = 40000 53 | NUM_VALID_IMAGES = 10000 54 | NUM_TEST_IMAGES = 10000 55 | ``` 56 | 57 | **Step 2:** Define functions used to load and prepare dataset for training. We incorporate 3 types of data augmentation schemes: random resize, random crop, random flip. Feel free to update this if you're comfortable. Leave the cell as it is if you aren't comfortable making changes. 58 | 59 | 60 | ```python 61 | def train_preprocess_fn(image): 62 | 63 | # Resize the image to add four extra pixels on each side. 64 | image = tf.image.resize_image_with_crop_or_pad(image, HEIGHT + 8, WIDTH + 8) 65 | 66 | # Randomly crop a [HEIGHT, WIDTH] section of the image. 67 | image = tf.random_crop(image, [HEIGHT, WIDTH, DEPTH]) 68 | 69 | # Randomly flip the image horizontally. 70 | image = tf.image.random_flip_left_right(image) 71 | 72 | return image 73 | 74 | def make_batch(filenames, batch_size): 75 | """Read the images and labels from 'filenames'.""" 76 | # Repeat infinitely. 77 | dataset = tf.data.TFRecordDataset(filenames).repeat() 78 | 79 | # Parse records. 80 | dataset = dataset.map(single_example_parser, num_parallel_calls=1) 81 | 82 | # Batch it up. 83 | dataset = dataset.batch(batch_size, drop_remainder=True) 84 | iterator = dataset.make_one_shot_iterator() 85 | 86 | image_batch, label_batch = iterator.get_next() 87 | return image_batch, label_batch 88 | 89 | def single_example_parser(serialized_example): 90 | """Parses a single tf.Example into image and label tensors.""" 91 | # Dimensions of the images in the CIFAR-10 dataset. 92 | # See http://www.cs.toronto.edu/~kriz/cifar.html for a description of the 93 | # input format. 94 | features = tf.parse_single_example( 95 | serialized_example, 96 | features={ 97 | 'image': tf.FixedLenFeature([], tf.string), 98 | 'label': tf.FixedLenFeature([], tf.int64), 99 | }) 100 | image = tf.decode_raw(features['image'], tf.uint8) 101 | image.set_shape([DEPTH * HEIGHT * WIDTH]) 102 | 103 | # Reshape from [depth * height * width] to [depth, height, width]. 104 | image = tf.cast( 105 | tf.transpose(tf.reshape(image, [DEPTH, HEIGHT, WIDTH]), [1, 2, 0]), 106 | tf.float32) 107 | label = tf.cast(features['label'], tf.int32) 108 | 109 | image = train_preprocess_fn(image) 110 | label = tf.one_hot(label, NUM_CLASSES) 111 | 112 | return image, label 113 | ``` 114 | 115 | **Step 3:** 116 | * Define hyperameters, directories for train, validation and test. 117 | * Load model from model_def.py 118 | * Compile model and fit 119 | 120 | 121 | ```python 122 | # Hyper-parameters 123 | epochs = 1 124 | lr = 0.01 125 | batch_size = 128 126 | momentum = 0.9 127 | weight_decay = 2e-4 128 | optimizer = 'sgd' 129 | gpu_count = 1 130 | 131 | # Data directories and other options 132 | checkpoint_dir = '../ckpt_dir' 133 | if not os.path.exists(checkpoint_dir): 134 | os.makedirs(checkpoint_dir) 135 | 136 | train_dir = '../dataset/train' 137 | validation_dir = '../dataset/validation' 138 | eval_dir = '../dataset/eval' 139 | 140 | train_dataset = make_batch(train_dir+'/train.tfrecords', batch_size) 141 | val_dataset = make_batch(validation_dir+'/validation.tfrecords', batch_size) 142 | eval_dataset = make_batch(eval_dir+'/eval.tfrecords', batch_size) 143 | ``` 144 | 145 | 146 | ```python 147 | model = get_model(lr, weight_decay, optimizer, momentum) 148 | opt = SGD(lr=lr, decay=weight_decay, momentum=momentum) 149 | ``` 150 | 151 | 152 | ```python 153 | model.compile(loss='categorical_crossentropy', 154 | optimizer=opt, 155 | metrics=['accuracy']) 156 | ``` 157 | 158 | 159 | ```python 160 | # Compile model 161 | model.compile(optimizer=SGD(lr=lr, decay=weight_decay, momentum=momentum), 162 | loss='categorical_crossentropy', 163 | metrics=['accuracy']) 164 | ``` 165 | 166 | 167 | ```python 168 | %%time 169 | # Train model 170 | history = model.fit(x=train_dataset[0], y=train_dataset[1], 171 | steps_per_epoch=NUM_TRAIN_IMAGES // batch_size, 172 | validation_data=val_dataset, 173 | validation_steps=NUM_VALID_IMAGES // batch_size, 174 | epochs=epochs, 175 | callbacks=[ModelCheckpoint(checkpoint_dir + '/checkpoint-{epoch}.h5')]) 176 | ``` 177 | 178 | 179 | ```python 180 | # Evaluate model performance 181 | score = model.evaluate(eval_dataset[0], 182 | eval_dataset[1], 183 | steps=NUM_TEST_IMAGES // batch_size, 184 | verbose=0) 185 | print('Test loss :', score[0]) 186 | print('Test accuracy:', score[1]) 187 | ``` 188 | 189 | 190 | ---- 191 | ##### Now that you have a successfully working training script, open `cifar10-distributed.ipynb` and start converting it for distributed training 192 | -------------------------------------------------------------------------------- /layouts/404.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | {{ partial "meta.html" . }} {{ partial "favicon.html" . }} {{ .Scratch.Add "title" "" }}{{ if eq .Site.Data.titles .Title }}{{ .Scratch.Set "title" (index .Site.Data.titles .Title).title }}{{ else }}{{ .Scratch.Set "title" .Title}}{{end}} 6 |
44 |
54 |
55 |
45 |
53 | {{T "title-404"}}
46 |47 |
48 |{{T "message-404"}}
49 | 50 | 51 |![]({{ "/images/gopher-404.jpg" | relURL }})
54 |
55 |
56 |
57 |
58 |
59 |
60 |
61 |
62 |
63 |
64 |
65 |
66 |
67 |
68 |
69 |
78 | {{ partial "custom-footer.html" . }}
79 |
80 |