├── README.md
├── images
└── SVdecay.jpg
├── src
├── README.md
├── codings
│ ├── __init__.py
│ ├── __pycache__
│ │ ├── __init__.cpython-36.pyc
│ │ ├── coding.cpython-36.pyc
│ │ ├── qsgd.cpython-36.pyc
│ │ ├── qsvd.cpython-36.pyc
│ │ └── svd.cpython-36.pyc
│ ├── coding.py
│ ├── qsgd.py
│ ├── svd.py
│ └── utils.py
├── data
│ └── data_prepare.py
├── data_loader_ops
│ ├── __init__.py
│ └── my_data_loader.py
├── data_prepare.sh
├── datasets.py
├── distributed_evaluator.py
├── distributed_nn.py
├── distributed_worker.py
├── evaluate_pytorch.sh
├── model_ops
│ ├── __init__.py
│ ├── alexnet.py
│ ├── densenet.py
│ ├── fc_nn.py
│ ├── lenet.py
│ ├── resnet.py
│ ├── resnet_split.py
│ └── vgg.py
├── nn_ops.py
├── optim
│ ├── __init__.py
│ ├── adam.py
│ └── sgd.py
├── output
│ └── models
│ │ └── README.md
├── run_pytorch.sh
├── single_machine.py
├── sync_replicas_master_nn.py
├── tiny_tuning_parser.py
├── tune.sh
└── utils.py
└── tools
├── config
├── hosts
├── hosts_address
├── hosts_alias
├── install.sh
├── local_script.sh
├── pre_run.sh
├── pytorch_ec2.py
├── remote_script.sh
└── update_git_dir.sh
/README.md:
--------------------------------------------------------------------------------
1 | # Atomo: Communication-efficient Learning via Atomic Sparsification
2 | This repository contains source code for Atomo, a general framework for atomic sparsification of stochastic gradients. Please check [the full paper](http://papers.nips.cc/paper/8191-atomo-communication-efficient-learning-via-atomic-sparsification) for detailed information about this project.
3 |
4 | ## Overview:
5 | ATOMO is a general framework for atomic sparsification of stochastic gradients. Given a gradient, an atomic decomposition,
6 | and a sparsity budget, ATOMO gives a random unbiased sparsification of the atoms minimizing variance. ATOMO sets up and optimally solves a meta-optimization that minimizes the variance of the sparsified gradient, subject to the constraints
7 | that it is sparse on the atomic basis, and also is an unbiased estimator of the input.
8 |
9 | 