├── README.md ├── accelerate_DDP.py ├── accelerate_DeepSpeed.py ├── accelerate_FSDP.py ├── config ├── classifier_cifar10.yaml ├── default_DDP.yml ├── default_DeepSpeed_stage-2.yml ├── default_FSDP.yml └── zero_stage2_config.json ├── data └── .gitkeep ├── model_export_onnx.py ├── model_inference.py ├── pretrained_models └── .gitkeep ├── pytorch_DDP.py ├── pytorch_DDP_ZeRO.py ├── pytorch_DP.py ├── pytorch_DeepSpeed.py ├── pytorch_FSDP.py ├── pytorch_SingleGPU.py ├── pytorch_auto_mixed_precision.py ├── pytorch_half_precision.py ├── pytorch_torchrun_DDP.py ├── pytorch_torchrun_FSDP.py ├── requirements.txt ├── results ├── FSDP_model.png ├── FairScale.png ├── accelerate_DDP.jpg ├── accelerate_FSDP.jpg ├── half_precision_2.png ├── predict_results.txt ├── pytorch_DDP.jpg ├── pytorch_DDP_ZeRO.jpg ├── pytorch_DP.jpg ├── pytorch_FSDP.jpg ├── pytorch_SingleGPU.jpg ├── pytorch_auto_mixed_precision.jpg ├── pytorch_half_precision.jpg ├── pytorch_torchrun_DDP.jpg └── pytorch_torchrun_FSDP.jpg └── utils.py /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/README.md -------------------------------------------------------------------------------- /accelerate_DDP.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/accelerate_DDP.py -------------------------------------------------------------------------------- /accelerate_DeepSpeed.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/accelerate_DeepSpeed.py -------------------------------------------------------------------------------- /accelerate_FSDP.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/accelerate_FSDP.py -------------------------------------------------------------------------------- /config/classifier_cifar10.yaml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/config/classifier_cifar10.yaml -------------------------------------------------------------------------------- /config/default_DDP.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/config/default_DDP.yml -------------------------------------------------------------------------------- /config/default_DeepSpeed_stage-2.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/config/default_DeepSpeed_stage-2.yml -------------------------------------------------------------------------------- /config/default_FSDP.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/config/default_FSDP.yml -------------------------------------------------------------------------------- /config/zero_stage2_config.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/config/zero_stage2_config.json -------------------------------------------------------------------------------- /data/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /model_export_onnx.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/model_export_onnx.py -------------------------------------------------------------------------------- /model_inference.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/model_inference.py -------------------------------------------------------------------------------- /pretrained_models/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /pytorch_DDP.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/pytorch_DDP.py -------------------------------------------------------------------------------- /pytorch_DDP_ZeRO.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/pytorch_DDP_ZeRO.py -------------------------------------------------------------------------------- /pytorch_DP.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/pytorch_DP.py -------------------------------------------------------------------------------- /pytorch_DeepSpeed.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/pytorch_DeepSpeed.py -------------------------------------------------------------------------------- /pytorch_FSDP.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/pytorch_FSDP.py -------------------------------------------------------------------------------- /pytorch_SingleGPU.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/pytorch_SingleGPU.py -------------------------------------------------------------------------------- /pytorch_auto_mixed_precision.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/pytorch_auto_mixed_precision.py -------------------------------------------------------------------------------- /pytorch_half_precision.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/pytorch_half_precision.py -------------------------------------------------------------------------------- /pytorch_torchrun_DDP.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/pytorch_torchrun_DDP.py -------------------------------------------------------------------------------- /pytorch_torchrun_FSDP.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/pytorch_torchrun_FSDP.py -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/requirements.txt -------------------------------------------------------------------------------- /results/FSDP_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/results/FSDP_model.png -------------------------------------------------------------------------------- /results/FairScale.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/results/FairScale.png -------------------------------------------------------------------------------- /results/accelerate_DDP.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/results/accelerate_DDP.jpg -------------------------------------------------------------------------------- /results/accelerate_FSDP.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/results/accelerate_FSDP.jpg -------------------------------------------------------------------------------- /results/half_precision_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/results/half_precision_2.png -------------------------------------------------------------------------------- /results/predict_results.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/results/predict_results.txt -------------------------------------------------------------------------------- /results/pytorch_DDP.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/results/pytorch_DDP.jpg -------------------------------------------------------------------------------- /results/pytorch_DDP_ZeRO.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/results/pytorch_DDP_ZeRO.jpg -------------------------------------------------------------------------------- /results/pytorch_DP.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/results/pytorch_DP.jpg -------------------------------------------------------------------------------- /results/pytorch_FSDP.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/results/pytorch_FSDP.jpg -------------------------------------------------------------------------------- /results/pytorch_SingleGPU.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/results/pytorch_SingleGPU.jpg -------------------------------------------------------------------------------- /results/pytorch_auto_mixed_precision.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/results/pytorch_auto_mixed_precision.jpg -------------------------------------------------------------------------------- /results/pytorch_half_precision.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/results/pytorch_half_precision.jpg -------------------------------------------------------------------------------- /results/pytorch_torchrun_DDP.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/results/pytorch_torchrun_DDP.jpg -------------------------------------------------------------------------------- /results/pytorch_torchrun_FSDP.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/results/pytorch_torchrun_FSDP.jpg -------------------------------------------------------------------------------- /utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xxcheng0708/pytorch-model-train-template/HEAD/utils.py --------------------------------------------------------------------------------