├── .gitignore ├── README.md ├── dataparallel.py ├── dataparallel_apex.py ├── distributed.py ├── distributed_apex.py ├── distributed_gradient_accumulation.py ├── distributed_mp.py ├── figs └── DP.jpg ├── tutorials ├── 0. Basic Theory.md ├── 1. Gradient Accumulation.md ├── 2. DDP Training Details.md ├── 3. DDP Training Tricks.md ├── 4. DDP with apex.md └── 5. DP and DDP.md └── utils ├── config.py ├── dataset.py ├── model.py ├── util.py └── validation.py /.gitignore: -------------------------------------------------------------------------------- 1 | /.idea 2 | /__pycache__ 3 | /log 4 | /ckpts 5 | /runs -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rentainhe/pytorch-distributed-training/HEAD/README.md -------------------------------------------------------------------------------- /dataparallel.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rentainhe/pytorch-distributed-training/HEAD/dataparallel.py -------------------------------------------------------------------------------- /dataparallel_apex.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rentainhe/pytorch-distributed-training/HEAD/dataparallel_apex.py -------------------------------------------------------------------------------- /distributed.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rentainhe/pytorch-distributed-training/HEAD/distributed.py -------------------------------------------------------------------------------- /distributed_apex.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rentainhe/pytorch-distributed-training/HEAD/distributed_apex.py -------------------------------------------------------------------------------- /distributed_gradient_accumulation.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rentainhe/pytorch-distributed-training/HEAD/distributed_gradient_accumulation.py -------------------------------------------------------------------------------- /distributed_mp.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rentainhe/pytorch-distributed-training/HEAD/distributed_mp.py -------------------------------------------------------------------------------- /figs/DP.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rentainhe/pytorch-distributed-training/HEAD/figs/DP.jpg -------------------------------------------------------------------------------- /tutorials/0. Basic Theory.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rentainhe/pytorch-distributed-training/HEAD/tutorials/0. Basic Theory.md -------------------------------------------------------------------------------- /tutorials/1. Gradient Accumulation.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rentainhe/pytorch-distributed-training/HEAD/tutorials/1. Gradient Accumulation.md -------------------------------------------------------------------------------- /tutorials/2. DDP Training Details.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rentainhe/pytorch-distributed-training/HEAD/tutorials/2. DDP Training Details.md -------------------------------------------------------------------------------- /tutorials/3. DDP Training Tricks.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rentainhe/pytorch-distributed-training/HEAD/tutorials/3. DDP Training Tricks.md -------------------------------------------------------------------------------- /tutorials/4. DDP with apex.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rentainhe/pytorch-distributed-training/HEAD/tutorials/4. DDP with apex.md -------------------------------------------------------------------------------- /tutorials/5. DP and DDP.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rentainhe/pytorch-distributed-training/HEAD/tutorials/5. DP and DDP.md -------------------------------------------------------------------------------- /utils/config.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rentainhe/pytorch-distributed-training/HEAD/utils/config.py -------------------------------------------------------------------------------- /utils/dataset.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rentainhe/pytorch-distributed-training/HEAD/utils/dataset.py -------------------------------------------------------------------------------- /utils/model.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rentainhe/pytorch-distributed-training/HEAD/utils/model.py -------------------------------------------------------------------------------- /utils/util.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rentainhe/pytorch-distributed-training/HEAD/utils/util.py -------------------------------------------------------------------------------- /utils/validation.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rentainhe/pytorch-distributed-training/HEAD/utils/validation.py --------------------------------------------------------------------------------