├── .gitignore ├── README.md ├── SGD ├── 1-linear_regression_SGD_TF2-simple.ipynb ├── 2-exercise-linear_regression_SGD_TF2-horovod.ipynb ├── 2-solution-linear_regression_SGD_TF2-horovod.ipynb ├── README.md └── mnist │ ├── 01_mnist.ipynb │ ├── 02_mnist_hvd.ipynb │ ├── 03_mnist_tf.distribute.ipynb │ └── 04_mnist_tf.dist_bn.ipynb ├── imagenet ├── 00_decoding-imagenet.ipynb ├── 01_inceptionv3_tfr.ipynb ├── README.md └── dist │ ├── 02_inceptionv3-hvd_tfr.ipynb │ ├── 03_inceptionv3-hvd_tfr.py │ ├── 04_inceptionv3-tf.dist.ipynb │ ├── 05_inceptionv3-hvd.ipynb │ ├── 06_inceptionv3-hvd_interleave.ipynb │ └── 07_inceptionv3-tf.dist_albumentations.ipynb ├── input_pipelines ├── 1_getting_started_with_tensorflows_dataset_api.ipynb ├── 2_getting_started_with_tensorflows_dataset_api.ipynb ├── 3_getting_started_with_tensorflows_dataset_api.ipynb ├── 4_getting_started_with_tensorflows_dataset_api.ipynb ├── 5_getting_started_with_tensorflows_dataset_api.ipynb ├── 6_getting_started_with_tensorflows_dataset_api.ipynb ├── README.md ├── imgs │ └── timeline.png ├── mnist │ ├── 01_mnist_cnn.ipynb │ ├── 02_mnist_cnn-hvd.ipynb │ ├── 03_mnist_cnn-tf.distribute.ipynb │ └── README.md ├── pipeline-timeline.ipynb └── tfrecords │ ├── convert-MNIST-dataset-to-tfrecords.py │ ├── imgs │ ├── gray-cat.jpg │ └── orange-cat.jpg │ └── read_and_write_TFRecord_files.ipynb ├── keras_benchmark ├── hvd_synthetic_benchmark.py ├── hvd_synthetic_benchmark.sh ├── tfdist_synthetic_benchmark.py └── tfdist_synthetic_benchmark.sh ├── nlp_squad ├── 1_understanding_squad_dataset_for_q-and-a_task.ipynb ├── 2_bert_squad_q-and-a-training.ipynb ├── README.md ├── dataset_utils.py └── dist │ ├── 2_diststr_bert_squad_q-and-a-training-1.py │ ├── 2_diststr_bert_squad_q-and-a-training-2.py │ ├── 2_horovod_bert_squad_q-and-a-training.py │ ├── 3_bert_squad_q-and-a-training-hvd.ipynb │ ├── 4_bert_squad_q-and-a-training-tf.dist.ipynb │ └── dataset_utils.py └── slides ├── distributed_sync_SGD.pdf ├── input-pipelines.pdf └── tf_dist.pdf /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/.gitignore -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/README.md -------------------------------------------------------------------------------- /SGD/1-linear_regression_SGD_TF2-simple.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/SGD/1-linear_regression_SGD_TF2-simple.ipynb -------------------------------------------------------------------------------- /SGD/2-exercise-linear_regression_SGD_TF2-horovod.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/SGD/2-exercise-linear_regression_SGD_TF2-horovod.ipynb -------------------------------------------------------------------------------- /SGD/2-solution-linear_regression_SGD_TF2-horovod.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/SGD/2-solution-linear_regression_SGD_TF2-horovod.ipynb -------------------------------------------------------------------------------- /SGD/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/SGD/README.md -------------------------------------------------------------------------------- /SGD/mnist/01_mnist.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/SGD/mnist/01_mnist.ipynb -------------------------------------------------------------------------------- /SGD/mnist/02_mnist_hvd.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/SGD/mnist/02_mnist_hvd.ipynb -------------------------------------------------------------------------------- /SGD/mnist/03_mnist_tf.distribute.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/SGD/mnist/03_mnist_tf.distribute.ipynb -------------------------------------------------------------------------------- /SGD/mnist/04_mnist_tf.dist_bn.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/SGD/mnist/04_mnist_tf.dist_bn.ipynb -------------------------------------------------------------------------------- /imagenet/00_decoding-imagenet.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/imagenet/00_decoding-imagenet.ipynb -------------------------------------------------------------------------------- /imagenet/01_inceptionv3_tfr.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/imagenet/01_inceptionv3_tfr.ipynb -------------------------------------------------------------------------------- /imagenet/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/imagenet/README.md -------------------------------------------------------------------------------- /imagenet/dist/02_inceptionv3-hvd_tfr.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/imagenet/dist/02_inceptionv3-hvd_tfr.ipynb -------------------------------------------------------------------------------- /imagenet/dist/03_inceptionv3-hvd_tfr.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/imagenet/dist/03_inceptionv3-hvd_tfr.py -------------------------------------------------------------------------------- /imagenet/dist/04_inceptionv3-tf.dist.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/imagenet/dist/04_inceptionv3-tf.dist.ipynb -------------------------------------------------------------------------------- /imagenet/dist/05_inceptionv3-hvd.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/imagenet/dist/05_inceptionv3-hvd.ipynb -------------------------------------------------------------------------------- /imagenet/dist/06_inceptionv3-hvd_interleave.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/imagenet/dist/06_inceptionv3-hvd_interleave.ipynb -------------------------------------------------------------------------------- /imagenet/dist/07_inceptionv3-tf.dist_albumentations.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/imagenet/dist/07_inceptionv3-tf.dist_albumentations.ipynb -------------------------------------------------------------------------------- /input_pipelines/1_getting_started_with_tensorflows_dataset_api.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/input_pipelines/1_getting_started_with_tensorflows_dataset_api.ipynb -------------------------------------------------------------------------------- /input_pipelines/2_getting_started_with_tensorflows_dataset_api.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/input_pipelines/2_getting_started_with_tensorflows_dataset_api.ipynb -------------------------------------------------------------------------------- /input_pipelines/3_getting_started_with_tensorflows_dataset_api.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/input_pipelines/3_getting_started_with_tensorflows_dataset_api.ipynb -------------------------------------------------------------------------------- /input_pipelines/4_getting_started_with_tensorflows_dataset_api.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/input_pipelines/4_getting_started_with_tensorflows_dataset_api.ipynb -------------------------------------------------------------------------------- /input_pipelines/5_getting_started_with_tensorflows_dataset_api.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/input_pipelines/5_getting_started_with_tensorflows_dataset_api.ipynb -------------------------------------------------------------------------------- /input_pipelines/6_getting_started_with_tensorflows_dataset_api.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/input_pipelines/6_getting_started_with_tensorflows_dataset_api.ipynb -------------------------------------------------------------------------------- /input_pipelines/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/input_pipelines/README.md -------------------------------------------------------------------------------- /input_pipelines/imgs/timeline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/input_pipelines/imgs/timeline.png -------------------------------------------------------------------------------- /input_pipelines/mnist/01_mnist_cnn.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/input_pipelines/mnist/01_mnist_cnn.ipynb -------------------------------------------------------------------------------- /input_pipelines/mnist/02_mnist_cnn-hvd.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/input_pipelines/mnist/02_mnist_cnn-hvd.ipynb -------------------------------------------------------------------------------- /input_pipelines/mnist/03_mnist_cnn-tf.distribute.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/input_pipelines/mnist/03_mnist_cnn-tf.distribute.ipynb -------------------------------------------------------------------------------- /input_pipelines/mnist/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/input_pipelines/mnist/README.md -------------------------------------------------------------------------------- /input_pipelines/pipeline-timeline.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/input_pipelines/pipeline-timeline.ipynb -------------------------------------------------------------------------------- /input_pipelines/tfrecords/convert-MNIST-dataset-to-tfrecords.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/input_pipelines/tfrecords/convert-MNIST-dataset-to-tfrecords.py -------------------------------------------------------------------------------- /input_pipelines/tfrecords/imgs/gray-cat.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/input_pipelines/tfrecords/imgs/gray-cat.jpg -------------------------------------------------------------------------------- /input_pipelines/tfrecords/imgs/orange-cat.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/input_pipelines/tfrecords/imgs/orange-cat.jpg -------------------------------------------------------------------------------- /input_pipelines/tfrecords/read_and_write_TFRecord_files.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/input_pipelines/tfrecords/read_and_write_TFRecord_files.ipynb -------------------------------------------------------------------------------- /keras_benchmark/hvd_synthetic_benchmark.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/keras_benchmark/hvd_synthetic_benchmark.py -------------------------------------------------------------------------------- /keras_benchmark/hvd_synthetic_benchmark.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/keras_benchmark/hvd_synthetic_benchmark.sh -------------------------------------------------------------------------------- /keras_benchmark/tfdist_synthetic_benchmark.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/keras_benchmark/tfdist_synthetic_benchmark.py -------------------------------------------------------------------------------- /keras_benchmark/tfdist_synthetic_benchmark.sh: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/keras_benchmark/tfdist_synthetic_benchmark.sh -------------------------------------------------------------------------------- /nlp_squad/1_understanding_squad_dataset_for_q-and-a_task.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/nlp_squad/1_understanding_squad_dataset_for_q-and-a_task.ipynb -------------------------------------------------------------------------------- /nlp_squad/2_bert_squad_q-and-a-training.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/nlp_squad/2_bert_squad_q-and-a-training.ipynb -------------------------------------------------------------------------------- /nlp_squad/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/nlp_squad/README.md -------------------------------------------------------------------------------- /nlp_squad/dataset_utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/nlp_squad/dataset_utils.py -------------------------------------------------------------------------------- /nlp_squad/dist/2_diststr_bert_squad_q-and-a-training-1.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/nlp_squad/dist/2_diststr_bert_squad_q-and-a-training-1.py -------------------------------------------------------------------------------- /nlp_squad/dist/2_diststr_bert_squad_q-and-a-training-2.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/nlp_squad/dist/2_diststr_bert_squad_q-and-a-training-2.py -------------------------------------------------------------------------------- /nlp_squad/dist/2_horovod_bert_squad_q-and-a-training.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/nlp_squad/dist/2_horovod_bert_squad_q-and-a-training.py -------------------------------------------------------------------------------- /nlp_squad/dist/3_bert_squad_q-and-a-training-hvd.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/nlp_squad/dist/3_bert_squad_q-and-a-training-hvd.ipynb -------------------------------------------------------------------------------- /nlp_squad/dist/4_bert_squad_q-and-a-training-tf.dist.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/nlp_squad/dist/4_bert_squad_q-and-a-training-tf.dist.ipynb -------------------------------------------------------------------------------- /nlp_squad/dist/dataset_utils.py: -------------------------------------------------------------------------------- 1 | ../dataset_utils.py -------------------------------------------------------------------------------- /slides/distributed_sync_SGD.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/slides/distributed_sync_SGD.pdf -------------------------------------------------------------------------------- /slides/input-pipelines.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/slides/input-pipelines.pdf -------------------------------------------------------------------------------- /slides/tf_dist.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eth-cscs/tensorflow-training/HEAD/slides/tf_dist.pdf --------------------------------------------------------------------------------