└── README.md /README.md: -------------------------------------------------------------------------------- 1 | # awesome-AutoML-and-Lightweight-Models 2 | A list of high-quality (newest) AutoML works and lightweight models including **1.) Neural Architecture Search**, **2.) Lightweight Structures**, **3.) Model Compression, Quantization and Acceleration**, **4.) Hyperparameter Optimization**, **5.) Automated Feature Engineering**. 3 | 4 | This repo is aimed to provide the info for AutoML research (especially for the lightweight models). Welcome to PR the works (papers, repositories) that are missed by the repo. 5 | 6 | ## 1.) Neural Architecture Search 7 | ### **[Papers]** 8 | **Gradient:** 9 | - [When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks](https://arxiv.org/abs/1911.10695) | [**CVPR 2020**] 10 | + [gmh14/RobNets](https://github.com/gmh14/RobNets) | [Pytorch] 11 | 12 | - [Searching for A Robust Neural Architecture in Four GPU Hours](https://xuanyidong.com/publication/cvpr-2019-gradient-based-diff-sampler/) | [**CVPR 2019**] 13 | + [D-X-Y/GDAS](https://github.com/D-X-Y/GDAS) | [Pytorch] 14 | 15 | - [ASAP: Architecture Search, Anneal and Prune](https://arxiv.org/abs/1904.04123) | [2019/04] 16 | 17 | - [Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours](https://arxiv.org/abs/1904.02877#) | [2019/04] 18 | + [dstamoulis/single-path-nas](https://github.com/dstamoulis/single-path-nas) | [Tensorflow] 19 | 20 | - [Automatic Convolutional Neural Architecture Search for Image Classification Under Different Scenes](https://ieeexplore.ieee.org/document/8676019) | [**IEEE Access 2019**] 21 | 22 | - [sharpDARTS: Faster and More Accurate Differentiable Architecture Search](https://arxiv.org/abs/1903.09900) | [2019/03] 23 | 24 | - [Learning Implicitly Recurrent CNNs Through Parameter Sharing](https://arxiv.org/abs/1902.09701) | [**ICLR 2019**] 25 | + [lolemacs/soft-sharing](https://github.com/lolemacs/soft-sharing) | [Pytorch] 26 | 27 | - [Probabilistic Neural Architecture Search](https://arxiv.org/abs/1902.05116) | [2019/02] 28 | 29 | - [Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation](https://arxiv.org/abs/1901.02985) | [2019/01] 30 | 31 | - [SNAS: Stochastic Neural Architecture Search](https://arxiv.org/abs/1812.09926) | [**ICLR 2019**] 32 | 33 | - [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://arxiv.org/abs/1812.03443) | [2018/12] 34 | 35 | - [Neural Architecture Optimization](http://papers.nips.cc/paper/8007-neural-architecture-optimization) | [**NIPS 2018**] 36 | + [renqianluo/NAO](https://github.com/renqianluo/NAO) | [Tensorflow] 37 | 38 | - [DARTS: Differentiable Architecture Search](https://arxiv.org/abs/1806.09055) | [2018/06] 39 | + [quark0/darts](https://github.com/quark0/darts) | [Pytorch] 40 | + [khanrc/pt.darts](https://github.com/khanrc/pt.darts) | [Pytorch] 41 | + [dragen1860/DARTS-PyTorch](https://github.com/dragen1860/DARTS-PyTorch) | [Pytorch] 42 | 43 | **Reinforcement Learning:** 44 | - [Template-Based Automatic Search of Compact Semantic Segmentation Architectures](https://arxiv.org/abs/1904.02365) | [2019/04] 45 | 46 | - [Understanding Neural Architecture Search Techniques](https://arxiv.org/abs/1904.00438) | [2019/03] 47 | 48 | - [Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search](https://arxiv.org/abs/1901.07261) | [2019/01] 49 | + [falsr/FALSR](https://github.com/falsr/FALSR) | [Tensorflow] 50 | 51 | - [Multi-Objective Reinforced Evolution in Mobile Neural Architecture Search](https://arxiv.org/abs/1901.01074) | [2019/01] 52 | + [moremnas/MoreMNAS](https://github.com/moremnas/MoreMNAS) | [Tensorflow] 53 | 54 | - [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://arxiv.org/abs/1812.00332) | [**ICLR 2019**] 55 | + [MIT-HAN-LAB/ProxylessNAS](https://github.com/MIT-HAN-LAB/ProxylessNAS) | [Pytorch, Tensorflow] 56 | 57 | - [Transfer Learning with Neural AutoML](http://papers.nips.cc/paper/8056-transfer-learning-with-neural-automl) | [**NIPS 2018**] 58 | 59 | - [Learning Transferable Architectures for Scalable Image Recognition](https://arxiv.org/abs/1707.07012) | [2018/07] 60 | + [wandering007/nasnet-pytorch](https://github.com/wandering007/nasnet-pytorch) | [Pytorch] 61 | + [tensorflow/models/research/slim/nets/nasnet](https://github.com/tensorflow/models/tree/master/research/slim/nets/nasnet) | [Tensorflow] 62 | 63 | - [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://arxiv.org/abs/1807.11626) | [2018/07] 64 | + [AnjieZheng/MnasNet-PyTorch](https://github.com/AnjieZheng/MnasNet-PyTorch) | [Pytorch] 65 | 66 | - [Practical Block-wise Neural Network Architecture Generation](https://arxiv.org/abs/1708.05552) | [**CVPR 2018**] 67 | 68 | - [Efficient Neural Architecture Search via Parameter Sharing](https://arxiv.org/abs/1802.03268) | [**ICML 2018**] 69 | + [melodyguan/enas](https://github.com/melodyguan/enas) | [Tensorflow] 70 | + [carpedm20/ENAS-pytorch](https://github.com/carpedm20/ENAS-pytorch) | [Pytorch] 71 | 72 | - [Efficient Architecture Search by Network Transformation](https://arxiv.org/abs/1707.04873) | [**AAAI 2018**] 73 | 74 | **Evolutionary Algorithm:** 75 | - [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://arxiv.org/abs/1904.00420) | [2019/04] 76 | 77 | - [DetNAS: Neural Architecture Search on Object Detection](https://arxiv.org/abs/1903.10979) | [2019/03] 78 | 79 | - [The Evolved Transformer](https://arxiv.org/abs/1901.11117) | [2019/01] 80 | 81 | - [Designing neural networks through neuroevolution](https://www.nature.com/articles/s42256-018-0006-z) | [**Nature Machine Intelligence 2019**] 82 | 83 | - [EAT-NAS: Elastic Architecture Transfer for Accelerating Large-scale Neural Architecture Search](https://arxiv.org/abs/1901.05884) | [2019/01] 84 | 85 | - [Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution](https://arxiv.org/abs/1804.09081) | [**ICLR 2019**] 86 | 87 | **SMBO:** 88 | - [MFAS: Multimodal Fusion Architecture Search](https://arxiv.org/abs/1903.06496) | [**CVPR 2019**] 89 | 90 | - [DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures](https://arxiv.org/abs/1806.08198) | [**ECCV 2018**] 91 | 92 | - [Progressive Neural Architecture Search](https://arxiv.org/abs/1712.00559) | [**ECCV 2018**] 93 | + [titu1994/progressive-neural-architecture-search](https://github.com/titu1994/progressive-neural-architecture-search) | [Keras, Tensorflow] 94 | + [chenxi116/PNASNet.pytorch](https://github.com/chenxi116/PNASNet.pytorch) | [Pytorch] 95 | 96 | **Random Search:** 97 | - [Exploring Randomly Wired Neural Networks for Image Recognition](https://arxiv.org/abs/1904.01569) | [2019/04] 98 | 99 | - [Searching for Efficient Multi-Scale Architectures for Dense Image Prediction](http://papers.nips.cc/paper/8087-searching-for-efficient-multi-scale-architectures-for-dense-image-prediction) | [**NIPS 2018**] 100 | 101 | **Hypernetwork:** 102 | - [Graph HyperNetworks for Neural Architecture Search](https://arxiv.org/abs/1810.05749) | [**ICLR 2019**] 103 | 104 | **Bayesian Optimization:** 105 | - [Inductive Transfer for Neural Architecture Optimization](https://arxiv.org/abs/1903.03536) | [2019/03] 106 | 107 | **Partial Order Pruning** 108 | - [Partial Order Pruning: for Best Speed/Accuracy Trade-off in Neural Architecture Search](https://arxiv.org/abs/1903.03777) | [**CVPR 2019**] 109 | + [lixincn2015/Partial-Order-Pruning](https://github.com/lixincn2015/Partial-Order-Pruning) | [Caffe] 110 | 111 | **Knowledge Distillation** 112 | - [Improving Neural Architecture Search Image Classifiers via Ensemble Learning](https://arxiv.org/abs/1903.06236) | [2019/03] 113 | 114 | ### **[Projects]** 115 | - [Microsoft/nni](https://github.com/Microsoft/nni) | [Python] 116 | - [MindsDB](https://github.com/mindsdb/mindsdb) | [Python] 117 | 118 | ## 2.) Lightweight Structures 119 | ### **[Papers]** 120 | **Image Classification:** 121 | - [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](http://proceedings.mlr.press/v97/tan19a.html) | [**ICML 2019**] 122 | + [tensorflow/tpu/models/official/efficientnet/](https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet) | [Tensorflow] 123 | + [lukemelas/EfficientNet-PyTorch](https://github.com/lukemelas/EfficientNet-PyTorch) | [Pytorch] 124 | 125 | - [Searching for MobileNetV3](https://arxiv.org/abs/1905.02244) | [2019/05] 126 | + [kuan-wang/pytorch-mobilenet-v3](https://github.com/kuan-wang/pytorch-mobilenet-v3) | [Pytorch] 127 | + [leaderj1001/MobileNetV3-Pytorch](https://github.com/leaderj1001/MobileNetV3-Pytorch) | [Pytorch] 128 | 129 | **Semantic Segmentation:** 130 | - [CGNet: A Light-weight Context Guided Network for Semantic Segmentation](https://arxiv.org/abs/1811.08201) | [2019/04] 131 | + [wutianyiRosun/CGNet](https://github.com/wutianyiRosun/CGNet) | [Pytorch] 132 | 133 | - [ESPNetv2: A Light-weight, Power Efficient, and General Purpose Convolutional Neural Network](https://arxiv.org/abs/1811.11431) | [2018/11] 134 | + [sacmehta/ESPNetv2](https://github.com/sacmehta/ESPNetv2) | [Pytorch] 135 | 136 | - [ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation](https://sacmehta.github.io/ESPNet/) | [**ECCV 2018**] 137 | + [sacmehta/ESPNet](https://github.com/sacmehta/ESPNet/) | [Pytorch] 138 | 139 | - [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://arxiv.org/abs/1808.00897) | [**ECCV 2018**] 140 | + [ooooverflow/BiSeNet](https://github.com/ooooverflow/BiSeNet) | [Pytorch] 141 | + [ycszen/TorchSeg](https://github.com/ycszen/TorchSeg) | [Pytorch] 142 | 143 | - [ERFNet: Efficient Residual Factorized ConvNet for Real-time Semantic Segmentation](http://www.robesafe.uah.es/personal/eduardo.romera/pdfs/Romera17tits.pdf) | [**T-ITS 2017**] 144 | + [Eromera/erfnet_pytorch](https://github.com/Eromera/erfnet_pytorch) | [Pytorch] 145 | 146 | **Object Detection:** 147 | - [ThunderNet: Towards Real-time Generic Object Detection](https://arxiv.org/abs/1903.11752) | [2019/03] 148 | 149 | - [Pooling Pyramid Network for Object Detection](https://arxiv.org/abs/1807.03284) | [2018/09] 150 | + [tensorflow/models](https://github.com/tensorflow/models/tree/master/research/object_detection/models) | [Tensorflow] 151 | 152 | - [Tiny-DSOD: Lightweight Object Detection for Resource-Restricted Usages](https://arxiv.org/abs/1807.11013) | [**BMVC 2018**] 153 | + [lyxok1/Tiny-DSOD](https://github.com/lyxok1/Tiny-DSOD) | [Caffe] 154 | 155 | - [Pelee: A Real-Time Object Detection System on Mobile Devices](https://arxiv.org/abs/1804.06882) | [**NeurIPS 2018**] 156 | + [Robert-JunWang/Pelee](https://github.com/Robert-JunWang/Pelee) | [Caffe] 157 | + [Robert-JunWang/PeleeNet](https://github.com/Robert-JunWang/PeleeNet) | [Pytorch] 158 | 159 | - [Receptive Field Block Net for Accurate and Fast Object Detection](https://eccv2018.org/openaccess/content_ECCV_2018/papers/Songtao_Liu_Receptive_Field_Block_ECCV_2018_paper.pdf) | [**ECCV 2018**] 160 | + [ruinmessi/RFBNet](https://github.com/ruinmessi/RFBNet) | [Pytorch] 161 | + [ShuangXieIrene/ssds.pytorch](https://github.com/ShuangXieIrene/ssds.pytorch) | [Pytorch] 162 | + [lzx1413/PytorchSSD](https://github.com/lzx1413/PytorchSSD) | [Pytorch] 163 | 164 | - [FSSD: Feature Fusion Single Shot Multibox Detector](https://arxiv.org/abs/1712.00960) | [2017/12] 165 | + [ShuangXieIrene/ssds.pytorch](https://github.com/ShuangXieIrene/ssds.pytorch) | [Pytorch] 166 | + [lzx1413/PytorchSSD](https://github.com/lzx1413/PytorchSSD) | [Pytorch] 167 | + [dlyldxwl/fssd.pytorch](https://github.com/dlyldxwl/fssd.pytorch) | [Pytorch] 168 | 169 | - [Feature Pyramid Networks for Object Detection](https://arxiv.org/abs/1612.03144) | [**CVPR 2017**] 170 | + [tensorflow/models](https://github.com/tensorflow/models/tree/master/research/object_detection/models) | [Tensorflow] 171 | 172 | ## 3.) Model Compression & Acceleration 173 | ### **[Papers]** 174 | **Pruning:** 175 | - [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://arxiv.org/abs/1803.03635) | [**ICLR 2019**] 176 | + [google-research/lottery-ticket-hypothesis](https://github.com/google-research/lottery-ticket-hypothesis) | [Tensorflow] 177 | 178 | - [Rethinking the Value of Network Pruning](https://arxiv.org/abs/1810.05270) | [**ICLR 2019**] 179 | 180 | - [Slimmable Neural Networks](https://openreview.net/pdf?id=H1gMCsAqY7) | [**ICLR 2019**] 181 | + [JiahuiYu/slimmable_networks](https://github.com/JiahuiYu/slimmable_networks) | [Pytorch] 182 | 183 | - [AMC: AutoML for Model Compression and Acceleration on Mobile Devices](https://arxiv.org/abs/1802.03494) | [**ECCV 2018**] 184 | + [AutoML for Model Compression (AMC): Trials and Tribulations](https://github.com/NervanaSystems/distiller/wiki/AutoML-for-Model-Compression-(AMC):-Trials-and-Tribulations) | [Pytorch] 185 | 186 | - [Learning Efficient Convolutional Networks through Network Slimming](https://arxiv.org/abs/1708.06519) | [**ICCV 2017**] 187 | + [foolwood/pytorch-slimming](https://github.com/foolwood/pytorch-slimming) | [Pytorch] 188 | 189 | - [Channel Pruning for Accelerating Very Deep Neural Networks](https://arxiv.org/abs/1707.06168) | [**ICCV 2017**] 190 | + [yihui-he/channel-pruning](https://github.com/yihui-he/channel-pruning) | [Caffe] 191 | 192 | - [Pruning Convolutional Neural Networks for Resource Efficient Inference](https://arxiv.org/abs/1611.06440) | [**ICLR 2017**] 193 | + [jacobgil/pytorch-pruning](https://github.com/jacobgil/pytorch-pruning) | [Pytorch] 194 | 195 | - [Pruning Filters for Efficient ConvNets](https://arxiv.org/abs/1608.08710) | [**ICLR 2017**] 196 | 197 | **Quantization:** 198 | - [Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets](https://arxiv.org/abs/1903.05662) | [**ICLR 2019**] 199 | 200 | - [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](http://openaccess.thecvf.com/content_cvpr_2018/html/Jacob_Quantization_and_Training_CVPR_2018_paper.html) | [**CVPR 2018**] 201 | 202 | - [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://arxiv.org/abs/1806.08342) | [2018/06] 203 | 204 | - [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://arxiv.org/abs/1805.06085) | [2018/05] 205 | 206 | - [Post-training 4-bit quantization of convolution networks for rapid-deployment](https://arxiv.org/abs/1810.05723) | [**ICML 2018**] 207 | 208 | - [WRPN: Wide Reduced-Precision Networks](https://arxiv.org/abs/1709.01134) | [**ICLR 2018**] 209 | 210 | - [Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights](https://arxiv.org/abs/1702.03044) | [**ICLR 2017**] 211 | 212 | - [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://arxiv.org/abs/1606.06160) | [2016/06] 213 | 214 | - [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://arxiv.org/abs/1308.3432) | [2013/08] 215 | 216 | **Knowledge Distillation** 217 | - [Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy](https://arxiv.org/abs/1711.05852) | [**ICLR 2018**] 218 | 219 | - [Model compression via distillation and quantization](https://arxiv.org/abs/1802.05668) | [**ICLR 2018**] 220 | 221 | **Acceleration:** 222 | - [Fast Algorithms for Convolutional Neural Networks](https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Lavin_Fast_Algorithms_for_CVPR_2016_paper.pdf) | [**CVPR 2016**] 223 | + [andravin/wincnn](https://github.com/andravin/wincnn) | [Python] 224 | 225 | ### **[Projects]** 226 | - [NervanaSystems/distiller](https://github.com/NervanaSystems/distiller/) | [Pytorch] 227 | - [Tencent/PocketFlow](https://github.com/Tencent/PocketFlow) | [Tensorflow] 228 | - [aaron-xichen/pytorch-playground](https://github.com/aaron-xichen/pytorch-playground) | [Pytorch] 229 | 230 | ### **[Tutorials/Blogs]** 231 | - [Introducing the CVPR 2018 On-Device Visual Intelligence Challenge](https://research.googleblog.com/search/label/On-device%20Learning) 232 | 233 | ## 4.) Hyperparameter Optimization 234 | ### **[Papers]** 235 | - [Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly](https://arxiv.org/abs/1903.06694) | [2019/03] 236 | + [dragonfly/dragonfly](https://github.com/dragonfly/dragonfly) 237 | 238 | - [Efficient High Dimensional Bayesian Optimization with Additivity and Quadrature Fourier Features](https://papers.nips.cc/paper/8115-efficient-high-dimensional-bayesian-optimization-with-additivity-and-quadrature-fourier-features) | [**NeurIPS 2018**] 239 | 240 | - [Google vizier: A service for black-box optimization](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf) | [**SIGKDD 2017**] 241 | 242 | - [On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice](https://arxiv.org/abs/2007.15745) | [**Neurocomputing 2020**] 243 | + [LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithms](https://github.com/LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithms) 244 | 245 | ### **[Projects]** 246 | - [BoTorch](https://botorch.org/) | [PyTorch] 247 | - [Ax (Adaptive Experimentation Platform)](https://ax.dev/) | [PyTorch] 248 | - [Microsoft/nni](https://github.com/Microsoft/nni) | [Python] 249 | - [dragonfly/dragonfly](https://github.com/dragonfly/dragonfly) | [Python] 250 | - [LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithms](https://github.com/LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithms) | [Python] 251 | 252 | ### **[Tutorials/Blogs]** 253 | - [Hyperparameter tuning in Cloud Machine Learning Engine using Bayesian Optimization](https://cloud.google.com/blog/products/gcp/hyperparameter-tuning-cloud-machine-learning-engine-using-bayesian-optimization) 254 | 255 | - [Overview of Bayesian Optimization](https://soubhikbarari.github.io/blog/2016/09/14/overview-of-bayesian-optimization) 256 | 257 | - [Bayesian optimization](http://krasserm.github.io/2018/03/21/bayesian-optimization/) 258 | + [krasserm/bayesian-machine-learning](https://github.com/krasserm/bayesian-machine-learning) | [Python] 259 | 260 | ## 5.) Automated Feature Engineering 261 | 262 | ## Model Analyzer 263 | - [Netscope CNN Analyzer](https://chakkritte.github.io/netscope/quickstart.html) | [Caffe] 264 | 265 | - [sksq96/pytorch-summary](https://github.com/sksq96/pytorch-summary) | [Pytorch] 266 | 267 | - [Lyken17/pytorch-OpCounter](https://github.com/Lyken17/pytorch-OpCounter) | [Pytorch] 268 | 269 | - [sovrasov/flops-counter.pytorch](https://github.com/sovrasov/flops-counter.pytorch) | [Pytorch] 270 | 271 | ## References 272 | - [LITERATURE ON NEURAL ARCHITECTURE SEARCH](https://www.ml4aad.org/automl/literature-on-neural-architecture-search/) 273 | - [handong1587/handong1587.github.io](https://github.com/handong1587/handong1587.github.io/tree/master/_posts/deep_learning) 274 | - [hibayesian/awesome-automl-papers](https://github.com/hibayesian/awesome-automl-papers) 275 | - [mrgloom/awesome-semantic-segmentation](https://github.com/mrgloom/awesome-semantic-segmentation) 276 | - [amusi/awesome-object-detection](https://github.com/amusi/awesome-object-detection) 277 | --------------------------------------------------------------------------------