└── README.md /README.md: -------------------------------------------------------------------------------- 1 | # A Collection of Neural Architecture Search Benchmarks 2 | 3 | We present a collection of several Neural Architecture Search (NAS) Benchmarks in this repository. Please feel free to open an issue or pull requests to add relevant NAS Benchmark papers in case we missed any or any new NAS Benchmark when it is made public. This repository is maintained by [Prof. Arun Somani's](http://class.ece.iastate.edu/arun/) group at Iowa State University. 4 | 5 | ## NAS Benchmarks for Computer Vision Applications (CNNs) 6 | 7 | 8 | | Benchmark | Paper | GitHub | 9 | |:--------------------------------------------------------------------------------------------------------|:--------------|:-----------------------| 10 | | NAS-Bench-101 | [Paper](https://arxiv.org/pdf/1902.09635.pdf) | [URL](https://github.com/google-research/nasbench) | 11 | | NAS-Bench-201 | [Paper](https://arxiv.org/pdf/2001.00326.pdf) | [URL](https://github.com/D-X-Y/NAS-Bench-201) | 12 | | NATS-Bench | [Paper](https://arxiv.org/pdf/2009.00437.pdf) | [URL](https://github.com/D-X-Y/NATS-Bench) | 13 | | NAS-Bench-301 | [Paper](https://arxiv.org/pdf/2008.09777.pdf) | [URL](https://github.com/automl/nasbench301) | 14 | | NAS-Bench-1Shot1 | [Paper](https://arxiv.org/pdf/2001.10422.pdf) | [URL](https://github.com/automl/nasbench-1shot1) | 15 | | NAS-Bench-MR | [Paper](https://arxiv.org/pdf/2103.13253.pdf) | [URL](https://github.com/dingmyu/NCP) | 16 | | NAS-Bench-Macro | [Paper](https://arxiv.org/pdf/2103.11922.pdf) | [URL](https://github.com/xiusu/NAS-Bench-Macro) | 17 | | TransNAS-Bench-101 | [Paper](https://arxiv.org/pdf/2105.11871.pdf) | [URL](https://github.com/kmdanielduan/TransNASBench) | 18 | | NAS-Bench-360 | [Paper](https://arxiv.org/pdf/2110.05668.pdf) | [URL](https://github.com/rtu715/NAS-Bench-360) | 19 | | NAS-Bench-x11 | [Paper](https://arxiv.org/pdf/2111.03602.pdf) | [URL](https://github.com/automl/nas-bench-x11) | 20 | | NAS-Bench-Suite | [Paper](https://arxiv.org/pdf/2201.13396.pdf) | [URL](https://github.com/automl/NASLib) | 21 | | NAS-Bench-Suite-Zero | [Paper](https://arxiv.org/pdf/2210.03230.pdf) | [URL](https://github.com/automl/naslib/tree/zerocost) | 22 | | BenchENAS | [Paper](https://arxiv.org/pdf/2108.03856.pdf) | [URL](https://benchenas.com/) | 23 | 24 | 25 | 26 | 27 | 28 | 29 | ## NAS Benchmarks for Non Computer Vision Applications 30 | 31 | | Benchmark | Paper | GitHub | 32 | |:--------------------------------------------------------------------------------------------------------|:--------------|:-----------------------| 33 | | NAS-Bench-NLP | [Paper](https://arxiv.org/pdf/2006.07116.pdf) | [URL](https://github.com/fmsnew/nas-bench-nlp-release) | 34 | | NAS-Bench-ASR | [Paper](https://openreview.net/pdf?id=CU0APx9LMaL) | [URL](https://github.com/SamsungLabs/nb-asr) | 35 | | NAS-Bench-Graph | [Paper](https://arxiv.org/pdf/2206.09166.pdf) | [URL](https://github.com/THUMNLab/NAS-Bench-Graph) | 36 | 37 | 38 | 39 | 40 | 41 | ## Hardware-aware NAS Benchmarks 42 | 43 | | Benchmark | Paper | GitHub | 44 | |:--------------------------------------------------------------------------------------------------------|:--------------|:-----------------------| 45 | | LatBench | [Paper](https://arxiv.org/pdf/2007.08668.pdf) | [URL](https://github.com/SamsungLabs/eagle) | 46 | | HW-NAS-Bench | [Paper](https://arxiv.org/pdf/2103.10584.pdf) | [URL](https://github.com/RICE-EIC/HW-NAS-Bench) | 47 | | BLOX | [Paper](https://arxiv.org/pdf/2210.07271.pdf) | [URL](https://github.com/SamsungLabs/blox) | 48 | | EC-NAS-Bench | [Paper](https://arxiv.org/pdf/2210.06015.pdf) | [URL](https://github.com/PedramBakh/EC-NAS-Bench) | 49 | 50 | 51 | 52 | 53 | ## NAS-HPO Benchmarks 54 | 55 | 56 | | Benchmark | Paper | GitHub | 57 | |:--------------------------------------------------------------------------------------------------------|:--------------|:-----------------------| 58 | | NAS-HPO-Bench | [Paper](https://arxiv.org/pdf/1905.04970.pdf) | [URL](https://github.com/automl/nas_benchmarks) | 59 | | NAS-HPO-Bench-II | [Paper](https://arxiv.org/pdf/2110.10165.pdf) | [URL](https://github.com/yoichii/nashpobench2api) | 60 | | LCBench | [Paper](https://arxiv.org/pdf/2006.13799.pdf) | [URL](https://github.com/automl/LCBench) | 61 | 62 | 63 | 64 | ## Survey Papers 65 | 66 | Feel free to go through our previously published NAS survey papers: 67 | 68 | | Survey Title | Paper URL | PDF | 69 | |:--------------------------------------------------------------------------------------------------------|:--------------|:-----------------------| 70 | | Neural Architecture Search Benchmarks: Insights and Survey | [Paper](https://ieeexplore.ieee.org/document/10063950) | [PDF](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10063950) | 71 | | Neural Architecture Search Survey: A Hardware Perspective | [Paper](https://dl.acm.org/doi/full/10.1145/3524500) | [PDF](https://dl.acm.org/doi/pdf/10.1145/3524500) | 72 | | Neural Architecture Search for Transformers: A Survey | [Paper](https://ieeexplore.ieee.org/document/9913476) | [PDF](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9913476) | 73 | 74 | 75 | 76 | 77 | 78 | # Citation 79 | 80 | If you find our papers and the repository useful, please consider citing the following papers: 81 | ``` 82 | @article{chitty2023neural, 83 | title={Neural Architecture Search Benchmarks: Insights and Survey}, 84 | author={Chitty-Venkata, Krishna Teja and Emani, Murali and Vishwanath, Venkatram and Somani, Arun K}, 85 | journal={IEEE Access}, 86 | volume={11}, 87 | pages={25217--25236}, 88 | year={2023}, 89 | publisher={IEEE} 90 | } 91 | 92 | @article{chitty2022neural, 93 | title={Neural architecture search survey: A hardware perspective}, 94 | author={Chitty-Venkata, Krishna Teja and Somani, Arun K}, 95 | journal={ACM Computing Surveys}, 96 | volume={55}, 97 | number={4}, 98 | pages={1--36}, 99 | year={2022}, 100 | publisher={ACM New York, NY} 101 | } 102 | 103 | @article{chitty2022neural_transformers, 104 | title={Neural architecture search for transformers: A survey}, 105 | author={Chitty-Venkata, Krishna Teja and Emani, Murali and Vishwanath, Venkatram and Somani, Arun K}, 106 | journal={IEEE Access}, 107 | volume={10}, 108 | pages={108374--108412}, 109 | year={2022}, 110 | publisher={IEEE} 111 | } 112 | ``` 113 | 114 | 115 | 116 | --------------------------------------------------------------------------------