├── fig1-source.jpg ├── LICENSE └── README.md /fig1-source.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sxzrt/CIFAR-10-W/HEAD/fig1-source.jpg -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 sxzrt 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## CIFAR-10-Warehouse: Towards Broad and More Realistic Testbeds in Model Generalization Analysis 2 | 3 | 4 | [**CIFAR-10-Warehouse**](https://arxiv.org/pdf/2310.04414.pdf), consisting of 180 datasets collected by prompting image search engines (such as Google, Bing, Baidu, 360, Sogou, Pexels and Flickr) and diffusion models in various ways. Generally sized between 300 and 8,000 images, the datasets (domains) contain natural images, cartoons, certain colors, or objects that do not naturally appear. 5 | 6 | 7 | ## Link of the Dataset 8 | 9 | You can access the dataset [here](https://drive.google.com/drive/folders/1b5xhY2C4gcLqPYFZ9qZC9EjMLUEgZaMn?usp=drive_link). 10 | 11 | A linux bash [script](https://github.com/sxzrt/CIFAR-10-W/issues/2) can be used to download the datasets. It is provided by [Adaloglou Nikolas](https://github.com/black0017) 😊. 12 | 13 | 14 | Directories & Files of images 15 | ```shell 16 | CIFAR-10-W 17 | ├── data_360_cartoon_original/ 18 | │   ├── 360-CT-01/ 19 | │   │ ├── airplane/ 20 | │   │ │ ├── 0001.jpg 21 | │   │ │ ├── 0002.jpg 22 | │   │ │ ├── ... 23 | │ │ ├── automobile/ 24 | │   │ │ ├── 0001.jpg 25 | │   │ │ ├── 0002.jpg 26 | │   │ │ ├── ... 27 | │ │ └── ... 28 | │   ├── 360-CT-02/ 29 | │   │ ├── airplane/ 30 | │   │ │ ├── 0001.jpg 31 | │   │ │ ├── 0002.jpg 32 | │   │ │ ├── ... 33 | │ │ ├── automobile/ 34 | │   │ │ ├── 0001.jpg 35 | │   │ │ ├── 0002.jpg 36 | │   │ │ ├── ... 37 | │ │ └── ... 38 | │   └── ... 39 | ├── data_google_original/ 40 | │   └── ... 41 | ├── data_bing_original/ 42 | │   └── ... 43 | ├── ... 44 | └── readme.txt 45 | ``` 46 | 47 | ## Codes of evaluated methods 48 | 49 | The AccP methods evaluated in our study are accessible [here](https://github.com/xingjianleng/autoeval_baselines). 50 | 51 | 52 | For domain generalization, we employed methods enumerated in [DomainBed](https://github.com/facebookresearch/DomainBed/tree/main). It is important to note that datasets in CIFAR-10-W are only utilized as a test set. 53 | 54 | 55 | ## Citation 56 | Please cite this paper if it helps your research: 57 | ```bibtex 58 | @inproceedings{sun2023privacy, 59 | title={Cifar-10-warehouse: Broad and more realistic testbeds in model generalization analysis}, 60 | author={Sun, Xiaoxiao and Leng, Xingjian and Wang, Zijian and Yang, Yang and Huang, Zi and Zheng, Liang}, 61 | booktitle={ICLR}, 62 | year={2024} 63 | } 64 | ``` 65 | 66 | ## Acknowledgement 67 | We express gratitude to the great work [Are Labels Always Necessary for Classifier Accuracy Evaluation?](https://github.com/Simon4Yan/Meta-set) as we benefit a lot from both the paper and code. 68 | 69 | 70 | ## License 71 | This repository is released under the MIT license. 72 | --------------------------------------------------------------------------------