Document not found (404)
159 |This URL is invalid, sorry. Please use the navigation bar or search to continue.
160 | 161 |- Getting Started
- Reference Guide
- Getting Started
- Reference Guide
6 |
7 | CLASS
8 | ddranking.aug.Cutmix(params: dict)
9 | [**[SOURCE]**](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/blob/main/ddranking/aug/cutmix.py)
10 |
11 |
12 |
13 | ### Parameters
14 |
15 | - **params**(dict): Parameters for the cutmix augmentation. We require the parameters to be in the format of `{'param_name': param_value}`. For cutmix, only `beta` (beta distribution parameter) needs to be specified, e.g. `{'beta': 1.0}`.
16 |
17 | ### Example
18 |
19 | ```python
20 | # When intializing an evaluator with cutmix augmentation, and cutmix object will be constructed.
21 | >>> self.aug_func = Cutmix(params={'beta': 1.0})
22 |
23 | # During training, the cutmix object will be used to augment the data.
24 | >>> images = aug_func(images)
25 | ```
--------------------------------------------------------------------------------
/doc/augmentations/dsa.md:
--------------------------------------------------------------------------------
1 | ## Differentiable Siamese Augmentation (DSA)
2 |
3 | DSA is one of differentiable data augmentations, first used in the dataset distillation task by [DSA](https://github.com/VICO-UoE/DatasetCondensation).
4 | Our implementation of DSA is adopted from [DSA](https://github.com/VICO-UoE/DatasetCondensation). It supports the following differentiable augmentations:
5 |
6 | - Random Flip
7 | - Random Rotation
8 | - Random Saturation
9 | - Random Brightness
10 | - Random Contrast
11 | - Random Scale
12 | - Random Crop
13 | - Random Cutout
14 |
15 |
16 |
17 | CLASS
18 | ddranking.aug.DSA(params: dict, seed: int, aug_mode: str)
19 | [**[SOURCE]**](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/blob/main/ddranking/aug/dsa.py)
20 |
21 |
22 |
23 | ### Parameters
24 |
25 | - **params**(dict): Parameters for the DSA augmentations. We require the parameters to be in the format of `{'param_name': param_value}`. For example, `{'flip': 0.5, 'rotate': 15.0, 'scale': 1.2, 'crop': 0.125, 'cutout': 0.5, 'brightness': 1.0, 'contrast': 0.5, 'saturation': 2.0}`.
26 | - **seed**(int): Random seed. Default is `-1`.
27 | - **aug_mode**(str): `S` for randomly selecting one augmentation for each batch. `M` for applying all augmentations for each batch.
28 |
29 | ### Example
30 |
31 | ```python
32 | # When intializing an evaluator with DSA augmentation, and DSA object will be constructed.
33 | >>> self.aug_func = DSA(params={'flip': 0.5, 'rotate': 15.0, 'scale': 1.2, 'crop': 0.125, 'cutout': 0.5, 'brightness': 1.0, 'contrast': 0.5, 'saturation': 2.0}, seed=-1, aug_mode='S')
34 |
35 | # During training, the DSA object will be used to augment the data.
36 | >>> images = aug_func(images)
37 | ```
38 |
--------------------------------------------------------------------------------
/doc/augmentations/mixup.md:
--------------------------------------------------------------------------------
1 | ## Mixup
2 |
3 | Mixup is a data augmentation technique that generates new training samples by linearly interpolating pairs of images. We follow the implementation of mixup in [SRe2L](https://github.com/VILA-Lab/SRe2L/tree/main/SRe2L).
4 |
5 |
6 |
7 | CLASS
8 | ddranking.aug.Mixup(params: dict)
9 | [**[SOURCE]**](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/blob/main/ddranking/aug/mixup.py)
10 |
11 |
12 |
13 | ### Parameters
14 |
15 | - **params**(dict): Parameters for the mixup augmentation. We require the parameters to be in the format of `{'param_name': param_value}`. For mixup, only `lambda` (mixup strength) needs to be specified, e.g. `{'lambda': 0.8}`.
16 |
17 | ### Example
18 |
19 | ```python
20 | # When intializing an evaluator with mixup augmentation, and mixup object will be constructed.
21 | >>> self.aug_func = Mixup(params={'lambda': 0.8})
22 |
23 | # During training, the mixup object will be used to augment the data.
24 | >>> images = aug_func(images)
25 | ```
--------------------------------------------------------------------------------
/doc/augmentations/overview.md:
--------------------------------------------------------------------------------
1 | # Augmentations
2 |
3 | DD-Ranking supports commonly used data augmentations in existing methods. A list of augmentations is provided below:
4 |
5 | - [Torchvision transforms](https://pytorch.org/vision/stable/transforms.html)
6 | - [DSA](datm.md)
7 | - [Mixup](mixup.md)
8 | - [Cutmix](cutmix.md)
9 |
10 | In DD-Ranking, data augmentations are specified when initializing an evaluator.
11 | The following arguments are related to data augmentations:
12 |
13 | - **data_aug_func**(str): The name of the data augmentation function used during training. Currently, we support `dsa`, `mixup`, `cutmix`.
14 | - **aug_params**(dict): The parameters for the data augmentation function.
15 | - **custom_train_trans**(torchvision.transforms.Compose): The custom train transform used to load the synthetic data when it's in '.jpg' or '.png' format.
16 | - **custom_val_trans**(torchvision.transforms.Compose): The custom val transform used to load the test dataset.
17 | - **use_zca**(bool): Whether to use ZCA whitening for the data augmentation. This is only applicable to methods that use ZCA whitening during distillation.
18 |
19 | ```python
20 | # When initializing an evaluator, the data augmentation function is specified.
21 | >>> evaluator = SoftLabelEvaluator(
22 | ...
23 | data_aug_func=..., # Specify the data augmentation function
24 | aug_params=..., # Specify the parameters for the data augmentation function
25 | custom_train_trans=..., # Specify the custom train transform
26 | custom_val_trans=..., # Specify the custom val transform
27 | use_zca=..., # Specify whether to use ZCA whitening
28 | ...
29 | )
30 | ```
--------------------------------------------------------------------------------
/doc/augmentations/torchvision.md:
--------------------------------------------------------------------------------
1 | ## Torchvision Transfoms
2 |
3 | We notice that some methods use jpg or png format images instead of image tensors during evaluation, and apply an additional torchvision-based tranformation to preprocess these images. Also, they may apply different augmentations to the test dataset. Thus, we support the torchvision-based transformations in DD-Ranking for both synthetic and real data.
4 |
5 | We require the torchvision-based transformations to be a `torchvision.transforms.Compose` object. If you have customized transformations, please make sure they have a `__call__` method. For the list of torchvision transformations, please refer to [torchvision-transforms](https://pytorch.org/vision/stable/transforms.html).
6 |
7 | ### Example
8 |
9 | ```python
10 | # Define a custom transformation
11 | class MyTransform:
12 | def __init__(self):
13 | pass
14 |
15 | def __call__(self, x):
16 | return x
17 |
18 | custom_train_trans = torchvision.transforms.Compose([
19 | MyTransform(),
20 | torchvision.transforms.RandomResizedCrop(32),
21 | torchvision.transforms.ToTensor(),
22 | torchvision.transforms.Normalize(mean, std)
23 | ])
24 | custom_val_trans = torchvision.transforms.Compose([
25 | torchvision.transforms.RandomCrop(32, padding=4),
26 | torchvision.transforms.ToTensor(),
27 | torchvision.transforms.Normalize(mean, std)
28 | ])
29 |
30 | evaluator = DD_Ranking(
31 | ...
32 | custom_train_trans=custom_train_trans,
33 | custom_val_trans=custom_val_trans,
34 | ...
35 | )
36 | ```
37 |
--------------------------------------------------------------------------------
/doc/config/overview.md:
--------------------------------------------------------------------------------
1 | # Config
2 |
3 | To ease the usage of DD-Ranking, we allow users to specify the parameters of the evaluator in a config file. The config file is a YAML file that contains the parameters of the evaluator. We illustrate the config file with the following examples.
4 |
5 | ## LRS
6 |
7 | ```yaml
8 | dataset: CIFAR100 # dataset name
9 | real_data_path: ./dataset/ # path to the real dataset
10 | ipc: 10 # image per class
11 | im_size: [32, 32] # image size
12 | model_name: ResNet-18-BN # model name
13 | stu_use_torchvision: true # whether to use torchvision to load student model
14 |
15 | tea_use_torchvision: true # whether to use torchvision to load teacher model
16 |
17 | teacher_dir: ./teacher_models # path to the pretrained teacher model
18 | teacher_model_names: [ResNet-18-BN] # the list of teacher models being used for evaluation
19 |
20 | data_aug_func: mixup # data augmentation function
21 | aug_params:
22 | lambda: 0.8 # data augmentation parameter; please follow this format for other parameters
23 |
24 | use_zca: false # whether to use ZCA whitening
25 | use_aug_for_hard: false # whether to use data augmentation for hard label evaluation
26 |
27 | custom_train_trans: # custom torchvision-based transformations to process training data; please follow this format for your own transformations
28 | - name: RandomCrop
29 | args:
30 | size: 32
31 | padding: 4
32 | - name: RandomHorizontalFlip
33 | args:
34 | p: 0.5
35 | - name: ToTensor
36 | - name: Normalize
37 | args:
38 | mean: [0.4914, 0.4822, 0.4465]
39 | std: [0.2023, 0.1994, 0.2010]
40 |
41 | custom_val_trans: null # custom torchvision-based transformations to process validation data; please follow the format above for your own transformations
42 |
43 | soft_label_mode: M # soft label mode
44 | soft_label_criterion: kl # soft label criterion
45 | loss_fn_kwargs:
46 | temperature: 30.0 # temperature for soft label
47 | scale_loss: false # whether to scale the loss
48 |
49 | optimizer: adamw # optimizer
50 | lr_scheduler: cosine # learning rate scheduler
51 | weight_decay: 0.01 # weight decay
52 | momentum: 0.9 # momentum
53 | num_eval: 5 # number of evaluations
54 | eval_full_data: false # whether to compute the test accuracy on the full dataset
55 | num_epochs: 400 # number of training epochs
56 | num_workers: 4 # number of workers
57 | device: cuda # device
58 | dist: true # whether to use distributed training
59 | syn_batch_size: 256 # batch size for synthetic data
60 | real_batch_size: 256 # batch size for real data
61 | save_path: ./results.csv # path to save the results
62 |
63 | random_data_format: tensor # format of the random data, tensor or image
64 | random_data_path: ./random_data # path to the save the random data
65 |
66 | ```
67 |
68 | To use config file, you can follow the example below.
69 |
70 | ```python
71 | from dd_ranking.metrics import LabelRobustScoreSoft
72 |
73 | config = Config(config_path='./config.yaml')
74 | evaluator = LabelRobustScoreSoft(config)
75 | ```
76 |
77 |
78 | ## ARS
79 |
80 | ```yaml
81 |
82 | ```
83 |
84 |
--------------------------------------------------------------------------------
/doc/contributing.md:
--------------------------------------------------------------------------------
1 | # Contributing
2 |
3 | Welcome! We are glad that you by willing to contribute to the field of dataset distillation.
4 |
5 | - **New Baselines**: If you would like to report new baselines, please submit them by creating a pull request. The exact format is below: name of the baseline, code link, [paper link and score run using this tool].
6 |
7 | - **New Components**: If you would like to integrate new components, such as new model architectures, new data augmentation methods, and new soft label strategies, please submit them by creating a pull request.
8 |
9 | - **Issues**: If you want to submit issues, you are encouraged to submit yes directly in issues.
10 |
11 | - **Appeal**: If you want to appeal for the score of your method, please submit an issue with your code and a detailed readme file of how to reproduce your results. We tried our best to replicate all methods in the leaderboard based on their papers and open-source code. We are sorry if we miss some details and will be grateful if you can help us improve the leaderboard.
12 |
--------------------------------------------------------------------------------
/doc/datasets/overview.md:
--------------------------------------------------------------------------------
1 | # Datasets
2 |
3 | DD-Ranking provides a set of commonly used datasets in existing dataset distillation methods. Users can flexibly use these datasets for evaluation. The interface to load datasets is as follows:
4 |
5 |
6 |
7 | ddranking.utils.get_dataset(dataset: str, data_path: str, im_size: tuple, use_zca: bool, custom_val_trans: Optional[Callable], device: str)
8 | [**[SOURCE]**](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/blob/main/ddranking/utils/data.py)
9 |
10 |
11 | ### Parameters
12 |
13 | - **dataset**(str): Name of the dataset.
14 | - **data_path**(str): Path to the dataset.
15 | - **im_size**(tuple): Image size.
16 | - **use_zca**(bool): Whether to use ZCA whitening. When set to True, the dataset will **not be** normalized using the mean and standard deviation of the training set.
17 | - **custom_train_trans**(Optional[Callable]): Custom transformation on the training set.
18 | - **custom_val_trans**(Optional[Callable]): Custom transformation on the validation set.
19 | - **device**(str): Device for performing ZCA whitening.
20 |
21 | Currently, we support the following datasets with default settings. We will keep updating this section with more datasets.
22 |
23 | - **CIFAR10**
24 | - **channels**: `3`
25 | - **im_size**: `(32, 32)`
26 | - **num_classes**: `10`
27 | - **mean**: `[0.4914, 0.4822, 0.4465]`
28 | - **std**: `[0.2023, 0.1994, 0.2010]`
29 | - **CIFAR100**
30 | - **channels**: `3`
31 | - **im_size**: `(32, 32)`
32 | - **num_classes**: `100`
33 | - **mean**: `[0.4914, 0.4822, 0.4465]`
34 | - **std**: `[0.2023, 0.1994, 0.2010]`
35 | - **TinyImageNet**
36 | - **channels**: `3`
37 | - **im_size**: `(64, 64)`
38 | - **num_classes**: `200`
39 | - **mean**: `[0.485, 0.456, 0.406]`
40 | - **std**: `[0.229, 0.224, 0.225]`
41 | - **ImageNet1K**
42 | - **channels**: `3`
43 | - **im_size**: `(224, 224)`
44 | - **num_classes**: `1000`
45 | - **mean**: `[0.485, 0.456, 0.406]`
46 | - **std**: `[0.229, 0.224, 0.225]`
47 |
--------------------------------------------------------------------------------
/doc/getting-started/installation.md:
--------------------------------------------------------------------------------
1 | ## Installation
2 |
3 | From pip
4 |
5 | ```bash
6 | pip install ddranking
7 | ```
8 |
9 | From source
10 |
11 | ```bash
12 | python setup.py install
13 | ```
--------------------------------------------------------------------------------
/doc/getting-started/quick-start.md:
--------------------------------------------------------------------------------
1 | ## Quick Start
2 |
3 | Below is a step-by-step guide on how to use our `dd_ranking`. This demo is for label-robust score (LRS) on soft labels (source code can be found in `demo_lrs_soft.py`). You can find the demo for LRS on hard label demo in `demo_lrs_hard.py` and the demo for augmentation-robust score (ARS) in `demo_ars.py`.
4 | DD-Ranking supports multi-GPU Distributed evaluation. You can simply use `torchrun` to launch the evaluation.
5 |
6 | **Step1**: Intialize a soft-label metric evaluator object. Config files are recommended for users to specify hyper-parameters. Sample config files are provided [here](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/tree/main/configs).
7 |
8 | ```python
9 | from ddranking.metrics import LabelRobustScoreSoft
10 | from ddranking.config import Config
11 |
12 | >>> config = Config.from_file("./configs/Demo_LRS_Soft_Label.yaml")
13 | >>> lrs_soft_metric = LabelRobustScoreSoft(config)
14 | ```
15 |
16 |
17 |
69 |
70 | For detailed explanation for hyper-parameters, please refer to our documentation.
71 |
72 | **Step 2:** Load your synthetic data, labels (if any), and learning rate (if any).
73 |
74 | ```python
75 | >>> syn_images = torch.load('/your/path/to/syn/images.pt')
76 | # You must specify your soft labels if your soft label mode is 'S'
77 | >>> soft_labels = torch.load('/your/path/to/syn/labels.pt')
78 | >>> syn_lr = torch.load('/your/path/to/syn/lr.pt')
79 | ```
80 |
81 | **Step 3:** Compute the metric.
82 |
83 | ```python
84 | >>> lrs_soft_metric.compute_metrics(image_tensor=syn_images, soft_labels=soft_labels, syn_lr=syn_lr)
85 | # alternatively, you can specify the image folder path to compute the metric
86 | >>> lrs_soft_metric.compute_metrics(image_path='./your/path/to/syn/images', soft_labels=soft_labels, syn_lr=syn_lr)
87 | ```
88 |
89 | The following results will be printed and saved to `save_path`:
90 | - `HLR mean`: The mean of hard label recovery over `num_eval` runs.
91 | - `HLR std`: The standard deviation of hard label recovery over `num_eval` runs.
92 | - `IOR mean`: The mean of improvement over random over `num_eval` runs.
93 | - `IOR std`: The standard deviation of improvement over random over `num_eval` runs.
94 | - `LRS mean`: The mean of Label-Robust Score over `num_eval` runs.
95 | - `LRS std`: The standard deviation of Label-Robust Score over `num_eval` runs.
96 |
98 |
--------------------------------------------------------------------------------
/doc/introduction.md:
--------------------------------------------------------------------------------
1 |
2 | You can also pass keyword arguments.
18 | 19 | ```python 20 | device = "cuda" 21 | method_name = "DATM" # Specify your method name 22 | ipc = 10 # Specify your IPC 23 | dataset = "CIFAR100" # Specify your dataset name 24 | syn_data_dir = "./data/CIFAR100/IPC10/" # Specify your synthetic data path 25 | real_data_dir = "./datasets" # Specify your dataset path 26 | model_name = "ConvNet-3" # Specify your model name 27 | teacher_dir = "./teacher_models" # Specify your path to teacher model chcekpoints 28 | teacher_model_names = ["ConvNet-3"] # Specify your teacher model names 29 | im_size = (32, 32) # Specify your image size 30 | dsa_params = { # Specify your data augmentation parameters 31 | "prob_flip": 0.5, 32 | "ratio_rotate": 15.0, 33 | "saturation": 2.0, 34 | "brightness": 1.0, 35 | "contrast": 0.5, 36 | "ratio_scale": 1.2, 37 | "ratio_crop_pad": 0.125, 38 | "ratio_cutout": 0.5 39 | } 40 | random_data_format = "tensor" # Specify your random data format (tensor or image) 41 | random_data_path = "./random_data" # Specify your random data path 42 | save_path = f"./results/{dataset}/{model_name}/IPC{ipc}/dm_hard_scores.csv" 43 | 44 | """ We only list arguments that usually need specifying""" 45 | lrs_soft_metric = LabelRobustScoreSoft( 46 | dataset=dataset, 47 | real_data_path=real_data_dir, 48 | ipc=ipc, 49 | model_name=model_name, 50 | soft_label_criterion='sce', # Use Soft Cross Entropy Loss 51 | soft_label_mode='S', # Use one-to-one image to soft label mapping 52 | loss_fn_kwargs={'temperature': 1.0, 'scale_loss': False}, 53 | data_aug_func='dsa', # Use DSA data augmentation 54 | aug_params=dsa_params, # Specify dsa parameters 55 | im_size=im_size, 56 | random_data_format=random_data_format, 57 | random_data_path=random_data_path, 58 | stu_use_torchvision=False, 59 | tea_use_torchvision=False, 60 | teacher_dir=teacher_dir, 61 | teacher_model_names=teacher_model_names, 62 | num_eval=5, 63 | device=device, 64 | dist=True, 65 | save_path=save_path 66 | ) 67 | ``` 68 |
3 |
4 |
5 |
6 |
7 | [](https://github.com/NUS-HPC-AI-Lab/DD-Ranking)
8 | [](https://huggingface.co/spaces/logits/DD-Ranking)
9 | [](https://x.com/Richard91316073/status/1890296645486801230)
10 |
11 |
12 |
13 | Welcome to **DD-Ranking** (DD, *i.e.*, Dataset Distillation), an integrated and easy-to-use evaluation benchmark for dataset distillation! It aims to provide a fair evaluation scheme for DD methods that can decouple the impacts from knowledge distillation and data augmentation to reflect the real informativeness of the distilled data.
14 |
15 | ## Motivation
16 | Dataset Distillation (DD) aims to condense a large dataset into a much smaller one, which allows a model to achieve comparable performance after training on it. DD has gained extensive attention since it was proposed. With some foundational methods such as DC, DM, and MTT, various works have further pushed this area to a new standard with their novel designs.
17 |
18 | 
19 |
20 | Notebaly, more and more methods are transitting from "hard label" to "soft label" in dataset distillation, especially during evaluation. **Hard labels** are categorical, having the same format of the real dataset. **Soft labels** are outputs of a pre-trained teacher model.
21 | Recently, Deng et al., pointed out that "a label is worth a thousand images". They showed analytically that soft labels are exetremely useful for accuracy improvement.
22 |
23 | However, since the essence of soft labels is **knowledge distillation**, we find that when applying the same evaluation method to randomly selected data, the test accuracy also improves significantly (see the figure above).
24 |
25 | This makes us wonder: **Can the test accuracy of the model trained on distilled data reflect the real informativeness of the distilled data?**
26 |
27 | We summaize the evaluation configurations of existing works in the following table, with different colors highlighting different values for each configuration.
28 | 
29 | As can be easily seen, the evaluation configurations are diverse, leading to unfairness of using only test accuracy to demonstrate one's performance.
30 | Among these inconsistencies, two critical factors significantly undermine the fairness of current evaluation protocols: label representation (including the corresponding loss function) and data augmentation techniques.
31 |
32 | Motivated by this, we propose DD-Ranking, a new benchmark for DD evaluation. DD-Ranking provides a fair evaluation scheme for DD methods that can decouple the impacts from knowledge distillation and data augmentation to reflect the real informativeness of the distilled data.
33 |
34 | ## Features
35 |
36 | - **Fair Evaluation**: DD-Ranking provides a fair evaluation scheme for DD methods that can decouple the impacts from knowledge distillation and data augmentation to reflect the real informativeness of the distilled data.
37 | - **Easy-to-use**: DD-Ranking provides a unified interface for dataset distillation evaluation.
38 | - **Extensible**: DD-Ranking supports various datasets and models.
39 | - **Customizable**: DD-Ranking supports various data augmentations and soft label strategies.
40 |
41 | ## DD-Ranking Benchmark
42 |
43 | Revisit the original goal of dataset distillation:
44 | > The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. (Wang et al., 2020)
45 | >
46 |
47 | ### Label-Robust Score (LRS)
48 | For the label representation, we introduce the Label-Robust Score (LRS) to evaluate the informativeness of the synthesized data using the following two aspects:
49 | 1. The degree to which the real dataset is recovered under hard labels (hard label recovery): \\( \text{HLR}=\text{Acc.}{\text{real-hard}}-\text{Acc.}{\text{syn-hard}} \\).
50 |
51 | 2. The improvement over random selection when using personalized evaluation methods (improvement over random): \\( \text{IOR}=\text{Acc.}{\text{syn-any}}-\text{Acc.}{\text{rdm-any}} \\).
52 | \\(\text{Acc.}\\) is the accuracy of models trained on different samples. Samples' marks are as follows:
53 | - \\(\text{real-hard}\\): Real dataset with hard labels;
54 | - \\(\text{syn-hard}\\): Synthetic dataset with hard labels;
55 | - \\(\text{syn-any}\\): Synthetic dataset with personalized evaluation methods (hard or soft labels);
56 | - \\(\text{rdm-any}\\): Randomly selected dataset (under the same compression ratio) with the same personalized evaluation methods.
57 |
58 | LRS is defined as a weight sum of \\(\text{IOR}\\) and \\(-\text{HLR}\\) to rank different methods:
59 | \\[
60 | \alpha = w\text{IOR}-(1-w)\text{HLR}, \quad w \in [0, 1]
61 | \\]
62 | Then, the LRS is normalized to \\([0, 1]\\) as follows:
63 | \\[
64 | \text{LRS} = 100\% \times (e^{\alpha}-e^{-1}) / (e - e^{-1})
65 | \\]
66 |
67 | By default, we set \\(w = 0.5\\) on the leaderboard, meaning that both \\(\text{IOR}\\) and \\(\text{HLR}\\) are equally important. Users can adjust the weights to emphasize one aspect on the leaderboard.
68 |
69 | ### Augmentation-Robust Score (ARS)
70 | To disentangle data augmentation’s impact, we introduce the augmentation-robust score (ARS) which continues to leverage the relative improvement over randomly selected data. Specifically, we first evaluate synthetic data and a randomly selected subset under the same setting to obtain \\(\text{Acc.}{\text{syn-aug}}\\) and \\(\text{Acc.}{\text{rdm-aug}}\\) (same as IOR). Next, we evaluate both synthetic data and random data again without the data augmentation, and results are denoted as \\(\text{Acc.}{\text{syn-naug}}\\) and \\(\text{Acc.}{\text{rdm-naug}}\\).
71 | Both differences, \\(\text{Acc.syn-aug} - \text{Acc.rdm-aug}\\) and \\(\text{Acc.syn-naug} - \text{Acc.rdm-naug}\\), are positively correlated to the real informativeness of the distilled dataset.
72 |
73 | ARS is a weighted sum of the two differences:
74 | \\[
75 | \beta = \gamma(\text{Acc.syn-aug} - \text{Acc.rdm-aug}) + (1 - \gamma)(\text{Acc.syn-naug} - \text{Acc.rdm-naug})
76 | \\]
77 | and normalized similarly.
78 |
79 |
--------------------------------------------------------------------------------
/doc/metrics/ars.md:
--------------------------------------------------------------------------------
1 | ## AugmentationRobustScore
2 |
3 |
4 |
5 | CLASS
6 | dd_ranking.metrics.AugmentationRobustScore(config: Optional[Config] = None,
7 | dataset: str = 'ImageNet1K',
8 | real_data_path: str = './dataset/',
9 | ipc: int = 10,
10 | model_name: str = 'ResNet-18-BN',
11 | label_type: str = 'soft',
12 | soft_label_mode: str='S',
13 | soft_label_criterion: str='kl',
14 | loss_fn_kwargs: dict=None,
15 | data_aug_func: str='cutmix',
16 | aug_params: dict={'cutmix_p': 1.0},
17 | optimizer: str='sgd',
18 | lr_scheduler: str='step',
19 | weight_decay: float=0.0005,
20 | momentum: float=0.9,
21 | step_size: int=None,
22 | num_eval: int=5,
23 | im_size: tuple=(224, 224),
24 | num_epochs: int=300,
25 | use_zca: bool=False,
26 | random_data_format: str='image',
27 | random_data_path: str=None,
28 | batch_size: int=256,
29 | save_path: str=None,
30 | stu_use_torchvision: bool=False,
31 | tea_use_torchvision: bool=False,
32 | num_workers: int=4,
33 | teacher_dir: str='./teacher_models',
34 | teacher_model_names: list=None,
35 | custom_train_trans: Optional[Callable]=None,
36 | custom_val_trans: Optional[Callable]=None,
37 | device: str="cuda",
38 | dist: bool=False
39 | )
40 | [**[SOURCE]**](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/blob/main/ddranking/metrics/aug_robust.py)
41 |
42 |
43 | A class for evaluating the performance of a dataset distillation method with soft labels. User is able to modify the attributes as needed.
44 |
45 | ### Parameters
46 |
47 | - **config**(Optional[Config]): Config object for specifying all attributes. See [config](../config/overview.md) for more details.
48 | - **dataset**(str): Name of the real dataset.
49 | - **real_data_path**(str): Path to the real dataset.
50 | - **ipc**(int): Images per class.
51 | - **model_name**(str): Name of the surrogate model. See [models](../models/overview.md) for more details.
52 | - **label_type**(str): Type of label representation. `soft` for soft labels, `hard` for hard labels.
53 | - **soft_label_mode**(str): Number of soft labels per image. `S` for single soft label, `M` for multiple soft labels.
54 | - **soft_label_criterion**(str): Loss function for using soft labels. Currently supports `kl` for KL divergence, `sce` for soft cross-entropy, and `mse_gt` for MSEGT loss introduced in EDC.
55 | - **loss_fn_kwargs**(dict): Keyword arguments for the loss function, e.g. `temperature` and `scale_loss` for KL and SCE loss, and `mse_weight` and `ce_weight` for MSEGT loss.
56 | - **data_aug_func**(str): Data augmentation function used during training. Currently supports `dsa`, `cutmix`, `mixup`. See [augmentations](../augmentations/overview.md) for more details.
57 | - **aug_params**(dict): Parameters for the data augmentation function.
58 | - **optimizer**(str): Name of the optimizer. Currently supports torch-based optimizers - `sgd`, `adam`, and `adamw`.
59 | - **lr_scheduler**(str): Name of the learning rate scheduler. Currently supports torch-based schedulers - `step`, `cosine`, `lambda_step`, and `cosineannealing`.
60 | - **weight_decay**(float): Weight decay for the optimizer.
61 | - **momentum**(float): Momentum for the optimizer.
62 | - **step_size**(int): Step size for the learning rate scheduler.
63 | - **use_zca**(bool): Whether to use ZCA whitening.
64 | - **num_eval**(int): Number of evaluations to perform.
65 | - **im_size**(tuple): Size of the images.
66 | - **num_epochs**(int): Number of epochs to train.
67 | - **batch_size**(int): Batch size for the model training.
68 | - **stu_use_torchvision**(bool): Whether to use torchvision to initialize the student model.
69 | - **tea_use_torchvision**(bool): Whether to use torchvision to initialize the teacher model.
70 | - **teacher_dir**(str): Path to the teacher model.
71 | - **teacher_model_names**(list): List of teacher model names.
72 | - **random_data_format**(str): Format of the random data, `tensor` or `image`.
73 | - **random_data_path**(str): Path to save the random data.
74 | - **num_workers**(int): Number of workers for data loading.
75 | - **save_path**(Optional[str]): Path to save the results.
76 | - **custom_train_trans**(Optional[Callable]): Custom transformation function when loading synthetic data. Only support torchvision transformations. See [torchvision-based transformations](../augmentations/torchvision.md) for more details.
77 | - **custom_val_trans**(Optional[Callable]): Custom transformation function when loading test dataset. Only support torchvision transformations. See [torchvision-based transformations](../augmentations/torchvision.md) for more details.
78 | - **device**(str): Device to use for evaluation, `cuda` or `cpu`.
79 | - **dist**(bool): Whether to use distributed training.
80 |
81 | ### Methods
82 |
83 |
84 | compute_metrics(image_tensor: Tensor = None, image_path: str = None, soft_labels: Tensor = None, syn_lr: float = None, ars_lambda: float = 0.5)
85 |
86 |
87 |
88 | This method computes the ARS score for the given image and soft labels (if provided). In each evaluation round, we set a different random seed and perform the following steps:
89 |
90 | 1. Compute the test accuracy of the surrogate model on the synthetic dataset without data augmentation.
91 | 2. Compute the test accuracy of the surrogate model on the synthetic dataset with data augmentation.
92 | 3. Compute the test accuracy of the surrogate model on the randomly selected dataset without data augmentation. We perform learning rate tuning for the best performance.
93 | 4. Compute the test accuracy of the surrogate model on the randomly selected dataset with data augmentation. We perform learning rate tuning for the best performance.
94 | 5. Compute the ARS score.
95 |
96 | The final scores are the average of the scores from `num_eval` rounds.
97 |
98 | #### Parameters
99 |
100 | - **image_tensor**(Tensor): Image tensor. Must specify when `image_path` is not provided. We require the shape to be `(N x IPC, C, H, W)` where `N` is the number of classes.
101 | - **image_path**(str): Path to the image. Must specify when `image_tensor` is not provided.
102 | - **soft_labels**(Tensor): Soft label tensor. Must specify when `soft_label_mode` is `S`. The first dimension must be the same as `image_tensor`.
103 | - **syn_lr**(float): Learning rate for the synthetic dataset. If not specified, the learning rate will be tuned automatically.
104 | - **ars_lambda**(float): Weighting parameter for the ARS.
105 |
106 | #### Returns
107 |
108 | A dictionary with the following keys:
109 |
110 | - **with_aug_mean**: Mean of test accuracy scores with data augmentation from `num_eval` rounds.
111 | - **with_aug_std**: Standard deviation of test accuracy scores with data augmentation from `num_eval` rounds.
112 | - **without_aug_mean**: Mean of test accuracy scores without data augmentation from `num_eval` rounds.
113 | - **without_aug_std**: Standard deviation of test accuracy scores without data augmentation from `num_eval` rounds.
114 | - **augmentation_robust_score_mean**: Mean of ARS scores from `num_eval` rounds.
115 | - **augmentation_robust_score_std**: Standard deviation of ARS scores from `num_eval` rounds.
116 |
117 |
118 |
119 | ### Examples
120 |
121 | with config file:
122 | ```python
123 | >>> config = Config('/path/to/config.yaml')
124 | >>> evaluator = AugmentationRobustScore(config=config)
125 | # load image and soft labels
126 | >>> image_tensor, soft_labels = ...
127 | # compute metrics
128 | >>> evaluator.compute_metrics(image_tensor=image_tensor, soft_labels=soft_labels)
129 | # alternatively, provide image path
130 | >>> evaluator.compute_metrics(image_path='path/to/image/folder/', soft_labels=soft_labels)
131 | ```
132 |
133 | with keyword arguments:
134 | ```python
135 | >>> evaluator = AugmentationRobustScore(
136 | ... dataset='ImageNet1K',
137 | ... real_data_path='./dataset/',
138 | ... ipc=10,
139 | ... model_name='ResNet-18-BN',
140 | ... label_type='soft',
141 | ... soft_label_mode='M',
142 | ... soft_label_criterion='kl',
143 | ... loss_fn_kwargs={
144 | ... "temperature": 30.0,
145 | ... "scale_loss": False,
146 | ... },
147 | ... data_aug_func='mixup',
148 | ... aug_params={
149 | ... "mixup_p": 0.8,
150 | ... },
151 | ... optimizer='adamw',
152 | ... lr_scheduler='cosine',
153 | ... num_epochs=300,
154 | ... weight_decay=0.0005,
155 | ... momentum=0.9,
156 | ... use_zca=False,
157 | ... stu_use_torchvision=True,
158 | ... tea_use_torchvision=True,
159 | ... num_workers=4,
160 | ... save_path='./results',
161 | ... random_data_format='image',
162 | ... random_data_path='./random_data',
163 | ... teacher_dir='./teacher_models',
164 | ... teacher_model_names=['ResNet-18-BN'],
165 | ... num_eval=5,
166 | ... device='cuda'
167 | ... )
168 | # load image and soft labels
169 | >>> image_tensor, soft_labels = ...
170 | # compute metrics
171 | >>> evaluator.compute_metrics(image_tensor=image_tensor, soft_labels=soft_labels)
172 | # alternatively, provide image path
173 | >>> evaluator.compute_metrics(image_path='path/to/image/folder/', soft_labels=soft_labels)
174 | ```
--------------------------------------------------------------------------------
/doc/metrics/general.md:
--------------------------------------------------------------------------------
1 | ## GeneralEvaluator
2 |
3 |
4 |
5 | CLASS
6 | dd_ranking.metrics.GeneralEvaluator(config: Optional[Config] = None,
7 | dataset: str = 'CIFAR10',
8 | real_data_path: str = './dataset/',
9 | ipc: int = 10,
10 | model_name: str = 'ConvNet-3',
11 | soft_label_mode: str='S',
12 | soft_label_criterion: str='kl',
13 | temperature: float=1.0,
14 | data_aug_func: str='cutmix',
15 | aug_params: dict={'cutmix_p': 1.0},
16 | optimizer: str='sgd',
17 | lr_scheduler: str='step',
18 | weight_decay: float=0.0005,
19 | momentum: float=0.9,
20 | num_eval: int=5,
21 | im_size: tuple=(32, 32),
22 | num_epochs: int=300,
23 | use_zca: bool=False,
24 | real_batch_size: int=256,
25 | syn_batch_size: int=256,
26 | default_lr: float=0.01,
27 | save_path: str=None,
28 | stu_use_torchvision: bool=False,
29 | tea_use_torchvision: bool=False,
30 | num_workers: int=4,
31 | teacher_dir: str='./teacher_models',
32 | custom_train_trans: Optional[Callable]=None,
33 | custom_val_trans: Optional[Callable]=None,
34 | device: str="cuda"
35 | )
36 | [**[SOURCE]**](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/blob/main/ddranking/metrics/general.py)
37 |
38 |
39 | A class for evaluating the traditional test accuracy of a surrogate model on the synthetic dataset under various settings (label type, data augmentation, etc.).
40 |
41 | ### Parameters
42 | Same as [Soft Label Evaluator](soft-label.md).
43 |
44 | ### Methods
45 |
46 |
47 | compute_metrics(image_tensor: Tensor = None, image_path: str = None, labels: Tensor = None, syn_lr: float = None)
48 |
49 |
50 |
51 | This method computes the test accuracy of the surrogate model on the synthetic dataset under various settings (label type, data augmentation, etc.).
52 |
53 | #### Parameters
54 |
55 | - **image_tensor**(Tensor): Image tensor. Must specify when `image_path` is not provided. We require the shape to be `(N x IPC, C, H, W)` where `N` is the number of classes.
56 | - **image_path**(str): Path to the image. Must specify when `image_tensor` is not provided.
57 | - **labels**(Tensor): Label tensor. It can be either hard labels or soft labels. When `soft_label_mode=S`, the label tensor must be provided.
58 | - **syn_lr**(float): Learning rate for the synthetic dataset. If not specified, the learning rate will be tuned automatically.
59 |
60 | #### Returns
61 | A dictionary with the following keys:
62 | - **acc_mean**: Mean of test accuracy from `num_eval` rounds.
63 | - **acc_std**: Standard deviation of test accuracy from `num_eval` rounds.
64 |
65 |
66 | ### Examples
67 |
68 | with config file:
69 | ```python
70 | >>> config = Config('/path/to/config.yaml')
71 | >>> evaluator = GeneralEvaluator(config=config)
72 | # load image and labels
73 | >>> image_tensor, labels = ...
74 | # compute metrics
75 | >>> evaluator.compute_metrics(image_tensor=image_tensor, labels=labels)
76 | # alternatively, provide image path
77 | >>> evaluator.compute_metrics(image_path='path/to/image.jpg', labels=labels)
78 | ```
79 |
80 | with keyword arguments:
81 | ```python
82 | >>> evaluator = GeneralEvaluator(
83 | ... dataset='CIFAR10',
84 | ... model_name='ConvNet-3',
85 | ... soft_label_mode='S',
86 | ... soft_label_criterion='sce',
87 | ... temperature=1.0,
88 | ... data_aug_func='cutmix',
89 | ... aug_params={
90 | ... "cutmix_p": 1.0,
91 | ... },
92 | ... optimizer='sgd',
93 | ... lr_scheduler='step',
94 | ... weight_decay=0.0005,
95 | ... momentum=0.9,
96 | ... stu_use_torchvision=False,
97 | ... tea_use_torchvision=False,
98 | ... num_eval=5,
99 | ... device='cuda'
100 | ... )
101 | # load image and labels
102 | >>> image_tensor, labels = ...
103 | # compute metrics
104 | >>> evaluator.compute_metrics(image_tensor=image_tensor, labels=labels)
105 | # alternatively, provide image path
106 | >>> evaluator.compute_metrics(image_path='path/to/image.jpg', labels=labels)
107 | ```
--------------------------------------------------------------------------------
/doc/metrics/lrs-hard-label.md:
--------------------------------------------------------------------------------
1 | ## LabelRobustScoreHard
2 |
3 |
4 |
5 | CLASS
6 | dd_ranking.metrics.LabelRobustScoreHard(config=None,
7 | dataset: str = 'CIFAR10',
8 | real_data_path: str = './dataset/',
9 | ipc: int = 10,
10 | model_name: str = 'ConvNet-3',
11 | data_aug_func: str = 'cutmix',
12 | aug_params: dict = {'cutmix_p': 1.0},
13 | optimizer: str = 'sgd',
14 | lr_scheduler: str = 'step',
15 | step_size: int = None,
16 | weight_decay: float = 0.0005,
17 | momentum: float = 0.9,
18 | use_zca: bool = False,
19 | num_eval: int = 5,
20 | im_size: tuple = (32, 32),
21 | num_epochs: int = 300,
22 | real_batch_size: int = 256,
23 | syn_batch_size: int = 256,
24 | use_torchvision: bool = False,
25 | eval_full_data: bool = False,
26 | random_data_format: str = 'tensor',
27 | random_data_path: str = './dataset/',
28 | num_workers: int = 4,
29 | save_path: Optional[str] = None,
30 | custom_train_trans: Optional[Callable] = None,
31 | custom_val_trans: Optional[Callable] = None,
32 | device: str = "cuda",
33 | dist: bool = False
34 | )
35 | [**[SOURCE]**](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/blob/main/ddranking/metrics/hard_label.py)
36 |
37 |
38 | A class for evaluating the performance of a dataset distillation method with hard labels. User is able to modify the attributes as needed.
39 |
40 | ### Parameters
41 |
42 | - **config**(Optional[Config]): Config object for specifying all attributes. See [config](../config/overview.md) for more details.
43 | - **dataset**(str): Name of the real dataset.
44 | - **real_data_path**(str): Path to the real dataset.
45 | - **ipc**(int): Images per class.
46 | - **model_name**(str): Name of the surrogate model. See [models](../models/overview.md) for more details.
47 | - **data_aug_func**(str): Data augmentation function used during training. Currently supports `dsa`, `cutmix`, `mixup`. See [augmentations](../augmentations/overview.md) for more details.
48 | - **aug_params**(dict): Parameters for the data augmentation function.
49 | - **optimizer**(str): Name of the optimizer. Currently supports torch-based optimizers - `sgd`, `adam`, and `adamw`.
50 | - **lr_scheduler**(str): Name of the learning rate scheduler. Currently supports torch-based schedulers - `step`, `cosine`, `lambda_step`, and `cosineannealing`.
51 | - **weight_decay**(float): Weight decay for the optimizer.
52 | - **momentum**(float): Momentum for the optimizer.
53 | - **step_size**(int): Step size for the learning rate scheduler.
54 | - **use_zca**(bool): Whether to use ZCA whitening.
55 | - **num_eval**(int): Number of evaluations to perform.
56 | - **im_size**(tuple): Size of the images.
57 | - **num_epochs**(int): Number of epochs to train.
58 | - **real_batch_size**(int): Batch size for the real dataset.
59 | - **syn_batch_size**(int): Batch size for the synthetic dataset.
60 | - **use_torchvision**(bool): Whether to use torchvision to initialize the model.
61 | - **eval_full_data**(bool): Whether to evaluate on the full dataset.
62 | - **random_data_format**(str): Format of the randomly selected dataset. Currently supports `tensor` and `image`.
63 | - **random_data_path**(str): Path to the randomly selected dataset.
64 | - **num_workers**(int): Number of workers for data loading.
65 | - **save_path**(Optional[str]): Path to save the results.
66 | - **custom_train_trans**(Optional[Callable]): Custom transformation function when loading synthetic data. Only support torchvision transformations. See [torchvision-based transformations](../augmentations/torchvision.md) for more details.
67 | - **custom_val_trans**(Optional[Callable]): Custom transformation function when loading test dataset. Only support torchvision transformations. See [torchvision-based transformations](../augmentations/torchvision.md) for more details.
68 | - **device**(str): Device to use for evaluation, `cuda` or `cpu`.
69 | - **dist**(bool): Whether to use distributed evaluation.
70 |
71 | ### Methods
72 |
73 |
74 | compute_metrics(image_tensor: Tensor = None, image_path: str = None, hard_labels: Tensor = None, syn_lr: float = None, lrs_lambda: float = 0.5)
75 |
76 |
77 | This method computes the HLR, IOR, and LRS for the given image and hard labels (if provided). In each evaluation round, we set a different random seed and perform the following steps:
78 |
79 | 1. Compute the test accuracy of the surrogate model on the synthetic dataset under hard labels. We tune the learning rate for the best performance if `syn_lr` is not provided.
80 | 2. Compute the test accuracy of the surrogate model on the real dataset under the same setting as step 1.
81 | 3. Compute the test accuracy of the surrogate model on the randomly selected dataset under the same setting as step 1.
82 | 4. Compute the HLR and IOR scores.
83 | 5. Compute the LRS.
84 |
85 | The final scores are the average of the scores from `num_eval` rounds.
86 |
87 | #### Parameters
88 |
89 | - **image_tensor**(Tensor): Image tensor. Must specify when `image_path` is not provided. We require the shape to be `(N x IPC, C, H, W)` where `N` is the number of classes.
90 | - **image_path**(str): Path to the image. Must specify when `image_tensor` is not provided.
91 | - **hard_labels**(Tensor): Hard label tensor. The first dimension must be the same as `image_tensor`.
92 | - **syn_lr**(float): Learning rate for the synthetic dataset. If not specified, the learning rate will be tuned automatically.
93 | - **lrs_lambda**(float): Weighting parameter for the LRS.
94 |
95 | #### Returns
96 |
97 | A dictionary with the following keys:
98 |
99 | - **hard_label_recovery_mean**: Mean of HLR scores from `num_eval` rounds.
100 | - **hard_label_recovery_std**: Standard deviation of HLR scores from `num_eval` rounds.
101 | - **improvement_over_random_mean**: Mean of improvement over random scores from `num_eval` rounds.
102 | - **improvement_over_random_std**: Standard deviation of improvement over random scores from `num_eval` rounds.
103 | - **label_robust_score_mean**: Mean of LRS scores from `num_eval` rounds.
104 | - **label_robust_score_std**: Standard deviation of LRS scores from `num_eval` rounds.
105 |
106 | **Examples:**
107 |
108 | with config file:
109 | ```python
110 | >>> config = Config('/path/to/config.yaml')
111 | >>> evaluator = LabelRobustScoreHard(config=config)
112 | # load the image and hard labels
113 | >>> image_tensor, hard_labels = ...
114 | # compute the metrics
115 | >>> evaluator.compute_metrics(image_tensor=image_tensor, hard_labels=hard_labels)
116 | # alternatively, you can provide the image path
117 | >>> evaluator.compute_metrics(image_path='path/to/image/folder/', hard_labels=hard_labels)
118 | ```
119 |
120 | with keyword arguments:
121 | ```python
122 | >>> evaluator = LabelRobustScoreHard(
123 | ... dataset='CIFAR10',
124 | ... real_data_path='./dataset/',
125 | ... ipc=10,
126 | ... model_name='ConvNet-3',
127 | ... data_aug_func='dsa',
128 | ... aug_params={
129 | ... "prob_flip": 0.5,
130 | ... "ratio_rotate": 15.0,
131 | ... "saturation": 2.0,
132 | ... "brightness": 1.0,
133 | ... "contrast": 0.5,
134 | ... "ratio_scale": 1.2,
135 | ... "ratio_crop_pad": 0.125,
136 | ... "ratio_cutout": 0.5
137 | ... },
138 | ... optimizer='sgd',
139 | ... lr_scheduler='step',
140 | ... weight_decay=0.0005,
141 | ... momentum=0.9,
142 | ... step_size=500,
143 | ... num_epochs=1000,
144 | ... real_batch_size=256,
145 | ... syn_batch_size=256,
146 | ... use_torchvision=False,
147 | ... eval_full_data=True,
148 | ... random_data_format='tensor',
149 | ... random_data_path='./random_data/',
150 | ... num_workers=4,
151 | ... save_path='./results/',
152 | ... use_zca=False,
153 | ... num_eval=5,
154 | ... device='cuda',
155 | ... dist=True
156 | ... )
157 | # load the image and hard labels
158 | >>> image_tensor, hard_labels = ...
159 | # compute the metrics
160 | >>> evaluator.compute_metrics(image_tensor=image_tensor, hard_labels=hard_labels)
161 | # alternatively, you can provide the image path
162 | >>> evaluator.compute_metrics(image_path='path/to/image/folder/', hard_labels=hard_labels)
163 | ```
164 |
--------------------------------------------------------------------------------
/doc/metrics/lrs-soft-label.md:
--------------------------------------------------------------------------------
1 | ## LabelRobustScoreSoft
2 |
3 |
4 |
5 | CLASS
6 | dd_ranking.metrics.LabelRobustScoreSoft(config: Optional[Config] = None,
7 | dataset: str = 'CIFAR10',
8 | real_data_path: str = './dataset/',
9 | ipc: int = 10,
10 | model_name: str = 'ConvNet-3',
11 | soft_label_mode: str='S',
12 | soft_label_criterion: str='kl',
13 | loss_fn_kwargs: dict=None,
14 | data_aug_func: str='cutmix',
15 | aug_params: dict={'cutmix_p': 1.0},
16 | optimizer: str='sgd',
17 | lr_scheduler: str='step',
18 | weight_decay: float=0.0005,
19 | momentum: float=0.9,
20 | step_size: int=None,
21 | num_eval: int=5,
22 | im_size: tuple=(32, 32),
23 | num_epochs: int=300,
24 | use_zca: bool=False,
25 | use_aug_for_hard: bool=False,
26 | random_data_format: str='tensor',
27 | random_data_path: str=None,
28 | real_batch_size: int=256,
29 | syn_batch_size: int=256,
30 | save_path: str=None,
31 | eval_full_data: bool=False,
32 | stu_use_torchvision: bool=False,
33 | tea_use_torchvision: bool=False,
34 | num_workers: int=4,
35 | teacher_dir: str='./teacher_models',
36 | teacher_model_names: list=None,
37 | custom_train_trans: Optional[Callable]=None,
38 | custom_val_trans: Optional[Callable]=None,
39 | device: str="cuda",
40 | dist: bool=False
41 | )
42 | [**[SOURCE]**](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/blob/main/ddranking/metrics/soft_label.py)
43 |
44 |
45 | A class for evaluating the performance of a dataset distillation method with soft labels. User is able to modify the attributes as needed.
46 |
47 | ### Parameters
48 |
49 | - **config**(Optional[Config]): Config object for specifying all attributes. See [config](../config/overview.md) for more details.
50 | - **dataset**(str): Name of the real dataset.
51 | - **real_data_path**(str): Path to the real dataset.
52 | - **ipc**(int): Images per class.
53 | - **model_name**(str): Name of the surrogate model. See [models](../models/overview.md) for more details.
54 | - **soft_label_mode**(str): Number of soft labels per image. `S` for single soft label, `M` for multiple soft labels.
55 | - **soft_label_criterion**(str): Loss function for using soft labels. Currently supports `kl` for KL divergence, `sce` for soft cross-entropy, and `mse_gt` for MSEGT loss introduced in EDC.
56 | - **loss_fn_kwargs**(dict): Keyword arguments for the loss function. `temperature` and `scale_loss` for KL and SCE loss, and `mse_weight` and `ce_weight` for MSE and CE loss.
57 | - **data_aug_func**(str): Data augmentation function used during training. Currently supports `dsa`, `cutmix`, `mixup`. See [augmentations](../augmentations/overview.md) for more details.
58 | - **aug_params**(dict): Parameters for the data augmentation function.
59 | - **use_aug_for_hard**(bool): Whether to use the data augmentation specified in `data_aug_func` for hard label evaluation.
60 | - **optimizer**(str): Name of the optimizer. Currently supports torch-based optimizers - `sgd`, `adam`, and `adamw`.
61 | - **lr_scheduler**(str): Name of the learning rate scheduler. Currently supports torch-based schedulers - `step`, `cosine`, `lambda_step`, and `cosineannealing`.
62 | - **weight_decay**(float): Weight decay for the optimizer.
63 | - **momentum**(float): Momentum for the optimizer.
64 | - **step_size**(int): Step size for the learning rate scheduler.
65 | - **use_zca**(bool): Whether to use ZCA whitening.
66 | - **num_eval**(int): Number of evaluations to perform.
67 | - **im_size**(tuple): Size of the images.
68 | - **num_epochs**(int): Number of epochs to train.
69 | - **real_batch_size**(int): Batch size for the real dataset.
70 | - **syn_batch_size**(int): Batch size for the synthetic dataset.
71 | - **stu_use_torchvision**(bool): Whether to use torchvision to initialize the student model.
72 | - **tea_use_torchvision**(bool): Whether to use torchvision to initialize the teacher model.
73 | - **teacher_dir**(str): Path to the teacher model.
74 | - **teacher_model_names**(list): List of teacher model names.
75 | - **random_data_format**(str): Format of the random data, `tensor` or `image`.
76 | - **random_data_path**(str): Path to save the random data.
77 | - **eval_full_data**(bool): Whether to compute the test accuracy on the full dataset (might be time-consuming on large datasets such as ImageNet1K, so we have provided a full dataset performance cache).
78 | - **num_workers**(int): Number of workers for data loading.
79 | - **save_path**(Optional[str]): Path to save the results.
80 | - **custom_train_trans**(Optional[Callable]): Custom transformation function when loading synthetic data. Only support torchvision transformations. See [torchvision-based transformations](../augmentations/torchvision.md) for more details.
81 | - **custom_val_trans**(Optional[Callable]): Custom transformation function when loading test dataset. Only support torchvision transformations. See [torchvision-based transformations](../augmentations/torchvision.md) for more details.
82 | - **device**(str): Device to use for evaluation, `cuda` or `cpu`.
83 | - **dist**(bool): Whether to use distributed training.
84 |
85 | ### Methods
86 |
87 |
88 | compute_metrics(image_tensor: Tensor = None, image_path: str = None, soft_labels: Tensor = None, syn_lr: float = None, lrs_lambda: float = 0.5)
89 |
90 |
91 |
92 | This method computes the HLR, IOR, and LRS for the given image and soft labels (if provided). In each evaluation round, we set a different random seed and perform the following steps:
93 |
94 | 1. Compute the test accuracy of the surrogate model on the synthetic dataset under hard labels. We perform learning rate tuning for the best performance.
95 | 2. Compute the test accuracy of the surrogate model on the real dataset under the same setting as step 1.
96 | 3. Compute the test accuracy of the surrogate model on the synthetic dataset under soft labels.
97 | 4. Compute the test accuracy of the surrogate model on the randomly selected dataset under the same setting as step 3.
98 | 5. Compute the HLR and IOR scores.
99 | 6. Compute the LRS.
100 |
101 | The final scores are the average of the scores from `num_eval` rounds.
102 |
103 | #### Parameters
104 |
105 | - **image_tensor**(Tensor): Image tensor. Must specify when `image_path` is not provided. We require the shape to be `(N x IPC, C, H, W)` where `N` is the number of classes.
106 | - **image_path**(str): Path to the image. Must specify when `image_tensor` is not provided.
107 | - **soft_labels**(Tensor): Soft label tensor. Must specify when `soft_label_mode` is `S`. The first dimension must be the same as `image_tensor`.
108 | - **syn_lr**(float): Learning rate for the synthetic dataset. If not specified, the learning rate will be tuned automatically.
109 | - **lrs_lambda**(float): Weighting parameter for the LRS.
110 |
111 | #### Returns
112 |
113 | A dictionary with the following keys:
114 |
115 | - **hard_label_recovery_mean**: Mean of HLR scores from `num_eval` rounds.
116 | - **hard_label_recovery_std**: Standard deviation of HLR scores from `num_eval` rounds.
117 | - **improvement_over_random_mean**: Mean of improvement over random scores from `num_eval` rounds.
118 | - **improvement_over_random_std**: Standard deviation of improvement over random scores from `num_eval` rounds.
119 | - **label_robust_score_mean**: Mean of LRS from `num_eval` rounds.
120 | - **label_robust_score_std**: Standard deviation of LRS from `num_eval` rounds.
121 |
122 |
123 |
124 | ### Examples
125 |
126 | with config file:
127 | ```python
128 | >>> config = Config('/path/to/config.yaml')
129 | >>> evaluator = LabelRobustScoreSoft(config=config)
130 | # load image and soft labels
131 | >>> image_tensor, soft_labels = ...
132 | # compute metrics
133 | >>> evaluator.compute_metrics(image_tensor=image_tensor, soft_labels=soft_labels)
134 | # alternatively, provide image path
135 | >>> evaluator.compute_metrics(image_path='path/to/image/folder/', soft_labels=soft_labels)
136 | ```
137 |
138 | with keyword arguments:
139 | ```python
140 | >>> evaluator = LabelRobustScoreSoft(
141 | ... dataset='TinyImageNet',
142 | ... real_data_path='./dataset/',
143 | ... ipc=10,
144 | ... model_name='ResNet-18-BN',
145 | ... soft_label_mode='M',
146 | ... soft_label_criterion='kl',
147 | ... loss_fn_kwargs={
148 | ... "temperature": 30.0,
149 | ... "scale_loss": False,
150 | ... },
151 | ... data_aug_func='mixup',
152 | ... aug_params={
153 | ... "mixup_p": 0.8,
154 | ... },
155 | ... optimizer='sgd',
156 | ... lr_scheduler='step',
157 | ... num_epochs=300,
158 | ... step_size=100,
159 | ... weight_decay=0.0005,
160 | ... momentum=0.9,
161 | ... use_zca=False,
162 | ... use_aug_for_hard=False,
163 | ... stu_use_torchvision=True,
164 | ... tea_use_torchvision=True,
165 | ... num_workers=4,
166 | ... save_path='./results',
167 | ... eval_full_data=False,
168 | ... random_data_format='image',
169 | ... random_data_path='./random_data',
170 | ... num_eval=5,
171 | ... device='cuda'
172 | ... )
173 | # load image and soft labels
174 | >>> image_tensor, soft_labels = ...
175 | # compute metrics
176 | >>> evaluator.compute_metrics(image_tensor=image_tensor, soft_labels=soft_labels)
177 | # alternatively, provide image path
178 | >>> evaluator.compute_metrics(image_path='path/to/image/folder/', soft_labels=soft_labels)
179 | ```
--------------------------------------------------------------------------------
/doc/metrics/overview.md:
--------------------------------------------------------------------------------
1 | # DD-Ranking Metrics
2 |
3 | DD-Ranking provides a set of metrics to evaluate the real informativeness of datasets distilled by different methods. The unfairness of existing evaluation is mainly caused by two factors, the label representation and the data augmentation. We design the label-robust score (LRS) and the augmentation robust score (ARS) to disentangle the impact of label representation and data augmentation on the evaluation, respectively.
4 |
5 | ## Evaluation Classes
6 | * [LabelRobustScoreHard](lrs-hard-label.md) computes HLR, IOR, and LRS for methods using hard labels.
7 | * [LabelRobustScoreSoft](lrs-soft-label.md) computes HLR, IOR, and LRS for methods using soft labels.
8 | * [AugmentationRobustScore](ars.md) computes the ARS for methods using soft labels.
9 | * [GeneralEvaluator](general.md) computes the traditional test accuracy for existing methods.
10 |
--------------------------------------------------------------------------------
/doc/models/alexnet.md:
--------------------------------------------------------------------------------
1 | ## AlexNet
2 |
3 | Our [implementation](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/blob/main/ddranking/utils/networks.py) of ConvNet is based on [DC](https://github.com/VICO-UoE/DatasetCondensation).
4 |
5 | We provide the following interface to initialize a AlexNet model:
6 |
7 |
8 |
9 | ddranking.utils.get_alexnet(model_name: str, im_size: tuple, channel: int, num_classes: int, pretrained: bool, model_path: str)
10 | [**[SOURCE]**](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/blob/main/ddranking/utils/model.py)
11 |
12 |
13 | ### Parameters
14 |
15 | - **model_name**(str): Name of the model. Please navigate to [models](models/overview.md) for the model naming convention in DD-Ranking.
16 | - **im_size**(tuple): Image size.
17 | - **channel**(int): Number of channels of the input image.
18 | - **num_classes**(int): Number of classes.
19 | - **pretrained**(bool): Whether to load pretrained weights.
20 | - **model_path**(str): Path to the pretrained model weights.
21 |
--------------------------------------------------------------------------------
/doc/models/convnet.md:
--------------------------------------------------------------------------------
1 | ## ConvNet
2 |
3 | Our [implementation](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/blob/main/ddranking/utils/networks.py) of ConvNet is based on [DC](https://github.com/VICO-UoE/DatasetCondensation).
4 |
5 | By default, we use width 128, average pooling, and ReLU activation. We provide the following interface to initialize a ConvNet model:
6 |
7 |
8 |
9 | dd_ranking.utils.get_convnet(model_name: str,
10 | im_size: tuple, channel: int, num_classes: int, net_depth: int, net_norm: str, pretrained: bool, model_path: str)
11 | [**[SOURCE]**](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/blob/main/ddranking/utils/model.py)
12 |
13 |
14 | ### Parameters
15 |
16 | - **model_name**(str): Name of the model. Please navigate to [models](models/overview.md) for the model naming convention in DD-Ranking.
17 | - **im_size**(tuple): Image size.
18 | - **channel**(int): Number of channels of the input image.
19 | - **num_classes**(int): Number of classes.
20 | - **net_depth**(int): Depth of the network.
21 | - **net_norm**(str): Normalization method. In ConvNet, we support `instance`, `batch`, and `group` normalization.
22 | - **pretrained**(bool): Whether to load pretrained weights.
23 | - **model_path**(str): Path to the pretrained model weights.
24 |
25 | To load a ConvNet model with different width or activation function or pooling method, you can use the following interface:
26 |
27 |
28 |
29 | dd_ranking.utils.networks.ConvNet(channel, num_classes, net_width, net_depth, net_act, net_norm, net_pooling, im_size)
30 | [**[SOURCE]**](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/blob/main/dd_ranking/utils/networks.py)
31 |
32 |
33 | ### Parameters
34 | We only list the parameters that are not present in `get_convnet`.
35 | - **net_width**(int): Width of the network.
36 | - **net_act**(str): Activation function. We support `relu`, `leakyrelu`, and `sigmoid`.
37 | - **net_pooling**(str): Pooling method. We support `avgpooling`, `maxpooling`, and `none`.
38 |
39 |
--------------------------------------------------------------------------------
/doc/models/lenet.md:
--------------------------------------------------------------------------------
1 | ## LeNet
2 |
3 | Our [implementation](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/blob/main/ddranking/utils/networks.py) of LeNet is based on [DC](https://github.com/VICO-UoE/DatasetCondensation).
4 |
5 | We provide the following interface to initialize a LeNet model:
6 |
7 |
8 |
9 | ddranking.utils.get_lenet(model_name: str, im_size: tuple, channel: int, num_classes: int, pretrained: bool, model_path: str)
10 | [**[SOURCE]**](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/blob/main/ddranking/utils/model.py)
11 |
12 |
13 | ### Parameters
14 |
15 | - **model_name**(str): Name of the model. Please navigate to [models](models/overview.md) for the model naming convention in DD-Ranking.
16 | - **im_size**(tuple): Image size.
17 | - **channel**(int): Number of channels of the input image.
18 | - **num_classes**(int): Number of classes.
19 | - **pretrained**(bool): Whether to load pretrained weights.
20 | - **model_path**(str): Path to the pretrained model weights.
21 |
--------------------------------------------------------------------------------
/doc/models/mlp.md:
--------------------------------------------------------------------------------
1 | ## MLP
2 |
3 | Our [implementation](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/blob/main/ddranking/utils/networks.py) of MLP is based on [DC](https://github.com/VICO-UoE/DatasetCondensation).
4 |
5 | We provide the following interface to initialize a MLP model:
6 |
7 |
8 |
9 | ddranking.utils.get_mlp(model_name: str, im_size: tuple, channel: int, num_classes: int, pretrained: bool, model_path: str)
10 | [**[SOURCE]**](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/blob/main/ddranking/utils/model.py)
11 |
12 |
13 | ### Parameters
14 |
15 | - **model_name**(str): Name of the model. Please navigate to [models](models/overview.md) for the model naming convention in DD-Ranking.
16 | - **im_size**(tuple): Image size.
17 | - **channel**(int): Number of channels of the input image.
18 | - **num_classes**(int): Number of classes.
19 | - **pretrained**(bool): Whether to load pretrained weights.
20 | - **model_path**(str): Path to the pretrained model weights.
21 |
--------------------------------------------------------------------------------
/doc/models/overview.md:
--------------------------------------------------------------------------------
1 | # Models
2 |
3 | DD-Ranking provides the implementation of a set of commonly used model architectures in existing dataset distillation methods. Users can flexibly use these models for main evaluation or cross-architecture evaluation. We will keep updating this section with more models.
4 |
5 | - [ConvNet](convnet.md)
6 | - [ResNet](resnet.md)
7 | - [VGG](vgg.md)
8 | - [LeNet](lenet.md)
9 | - [AlexNet](alexnet.md)
10 | - [MLP](mlp.md)
11 |
12 | Users can also define any model with `torchvision`.
13 |
14 | ## Naming Convention
15 |
16 | We use the following naming conventions for models in DD-Ranking:
17 |
18 | - `model name - model depth - norm type` (for DD-Ranking implemented models)
19 | - torchvision model names, e.g. `vgg11` and `vit_b_16`
20 |
21 | Model name and depth are required when **not using tochvision**. When norm type is not specified, we use default normalization for the model. For example, `ResNet-18-BN` means ResNet18 with batch normalization. `ConvNet-4` means ConvNet with depth 4 and default instance normalization.
22 |
23 | ## Pretrained Model Weights
24 |
25 | For users' convenience, we provide pretrained model weights on CIFAR10, CIFAR100, and TinyImageNet for the following models:
26 | - ConvNet-3 (CIFAR10, CIFAR100)
27 | - ConvNet-3-BN (CIFAR10, CIFAR100)
28 | - ConvNet-4 (TinyImageNet)
29 | - ConvNet-4-BN (TinyImageNet)
30 | - ResNet-18-BN (CIFAR10, CIFAR100, TinyImageNet, ImageNet1K)
31 |
32 | Users can download the weights from the following links: [Pretrained Model Weights](https://drive.google.com/drive/folders/19OnR85PRs3TZk8xS8XNr9hiokfsML4m2?usp=sharing).
33 |
34 | Users can also feel free to use `torchvision` pretrained models.
35 |
--------------------------------------------------------------------------------
/doc/models/resnet.md:
--------------------------------------------------------------------------------
1 | ## ResNet
2 |
3 | DD-Ranking supports implementation of ResNet in both [DC](https://github.com/VICO-UoE/DatasetCondensation) and [torchvision](https://pytorch.org/vision/main/models/resnet.html).
4 |
5 | We provide the following interface to initialize a ConvNet model:
6 |
7 |
8 |
9 | ddranking.utils.get_resnet(model_name: str,
10 | im_size: tuple, channel: int, num_classes: int, depth: int, batchnorm: bool, use_torchvision: bool, pretrained: bool, model_path: str)
11 | [**[SOURCE]**](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/blob/main/ddranking/utils/model.py)
12 |
13 |
14 | ### Parameters
15 |
16 | - **model_name**(str): Name of the model. Please navigate to [models](models/overview.md) for the model naming convention in DD-Ranking.
17 | - **im_size**(tuple): Image size.
18 | - **channel**(int): Number of channels of the input image.
19 | - **num_classes**(int): Number of classes.
20 | - **depth**(int): Depth of the network.
21 | - **batchnorm**(bool): Whether to use batch normalization.
22 | - **use_torchvision**(bool): Whether to use torchvision to initialize the model. When using torchvision, the ResNet model uses batch normalization by default.
23 | - **pretrained**(bool): Whether to load pretrained weights.
24 | - **model_path**(str): Path to the pretrained model weights.
25 |
26 | NOTE
27 |
28 |
29 | When using torchvision ResNet on image size smaller than 224 x 224, we make the following modifications:
30 |
31 | ```python
32 | model.conv1 = torch.nn.Conv2d(3, 64, kernel_size=(3,3), stride=(1,1), padding=(1,1), bias=False)
33 | model.maxpool = torch.nn.Identity()
34 | ```
35 |
36 |
--------------------------------------------------------------------------------
/doc/models/vgg.md:
--------------------------------------------------------------------------------
1 | ## VGG
2 |
3 | DD-Ranking supports implementation of VGG in both [DC](https://github.com/VICO-UoE/DatasetCondensation) and [torchvision](https://pytorch.org/vision/main/models/vgg.html).
4 |
5 | We provide the following interface to initialize a ConvNet model:
6 |
7 |
8 |
9 | ddranking.utils.get_vgg(model_name: str,
10 | im_size: tuple, channel: int, num_classes: int, depth: int, batchnorm: bool, use_torchvision: bool, pretrained: bool, model_path: str)
11 | [**[SOURCE]**](https://github.com/NUS-HPC-AI-Lab/DD-Ranking/blob/main/ddranking/utils/model.py)
12 |
13 |
14 | ### Parameters
15 |
16 | - **model_name**(str): Name of the model. Please navigate to [models](models/overview.md) for the model naming convention in DD-Ranking.
17 | - **im_size**(tuple): Image size.
18 | - **channel**(int): Number of channels of the input image.
19 | - **num_classes**(int): Number of classes.
20 | - **depth**(int): Depth of the network.
21 | - **batchnorm**(bool): Whether to use batch normalization.
22 | - **use_torchvision**(bool): Whether to use torchvision to initialize the model.
23 | - **pretrained**(bool): Whether to load pretrained weights.
24 | - **model_path**(str): Path to the pretrained model weights.
25 |
26 | NOTE
27 |
28 |
29 | When using torchvision VGG on image size smaller than 224 x 224, we make the following modifications:
30 |
31 | For 32x32 image size:
32 | ```python
33 | model.classifier = nn.Sequential(OrderedDict([
34 | ('fc1', nn.Linear(512 * 1 * 1, 4096)),
35 | ('relu1', nn.ReLU(True)),
36 | ('drop1', nn.Dropout()),
37 | ('fc2', nn.Linear(4096, 4096)),
38 | ('relu2', nn.ReLU(True)),
39 | ('drop2', nn.Dropout()),
40 | ('fc3', nn.Linear(4096, num_classes)),
41 | ]))
42 | ```
43 |
44 | For 64x64 image size:
45 | ```python
46 | model.classifier = nn.Sequential(OrderedDict([
47 | ('fc1', nn.Linear(512 * 2 * 2, 4096)),
48 | ('relu1', nn.ReLU(True)),
49 | ('drop1', nn.Dropout()),
50 | ('fc2', nn.Linear(4096, 4096)),
51 | ('relu2', nn.ReLU(True)),
52 | ('drop2', nn.Dropout()),
53 | ('fc3', nn.Linear(4096, num_classes)),
54 | ]))
55 | ```
56 |
--------------------------------------------------------------------------------
/doc/static/configurations.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/NUS-HPC-AI-Lab/DD-Ranking/c1ea0b3f251b7927feb25c4dafc76d5312011caf/doc/static/configurations.png
--------------------------------------------------------------------------------
/doc/static/history.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/NUS-HPC-AI-Lab/DD-Ranking/c1ea0b3f251b7927feb25c4dafc76d5312011caf/doc/static/history.png
--------------------------------------------------------------------------------
/doc/static/logo.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/NUS-HPC-AI-Lab/DD-Ranking/c1ea0b3f251b7927feb25c4dafc76d5312011caf/doc/static/logo.png
--------------------------------------------------------------------------------
/doc/static/team/zekai.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/NUS-HPC-AI-Lab/DD-Ranking/c1ea0b3f251b7927feb25c4dafc76d5312011caf/doc/static/team/zekai.jpg
--------------------------------------------------------------------------------
/pyproject.toml:
--------------------------------------------------------------------------------
1 | [build-system]
2 | requires = [
3 | "setuptools",
4 | "wheel",
5 | "torch >= 2.0.0"
6 | ]
7 | build-backend = "setuptools.build_meta"
8 |
9 | [project]
10 | name = "ddranking"
11 | version = "0.2.0"
12 | description = "DD-Ranking: Rethinking the Evaluation of Dataset Distillation"
13 | readme = "README.md"
14 | requires-python = ">=3.8"
15 | dependencies = [
16 | "torch",
17 | "numpy<2.0.0",
18 | "torchvision",
19 | "tqdm",
20 | "scipy<1.16.0",
21 | "kornia",
22 | "timm",
23 | "pandas"
24 | ]
25 |
--------------------------------------------------------------------------------
/setup.py:
--------------------------------------------------------------------------------
1 | from setuptools import setup, find_packages
2 |
3 | packages = find_packages()
4 |
5 | setup(
6 | name='ddranking', # Package name
7 | version='0.2.0', # Version number
8 | description='DD-Ranking: Rethinking the Evaluation of Dataset Distillation',
9 | long_description=open('README.md').read(), # Use your README as the long description
10 | long_description_content_type='text/markdown',
11 | author='DD-Ranking Team',
12 | author_email='lizekai@u.nus.edu',
13 | include_dirs=['ddranking', 'configs', 'static'],
14 | include_package_data=True,
15 | packages=packages, # Automatically discover submodules
16 | install_requires=[
17 | 'torch',
18 | 'numpy',
19 | 'torchvision',
20 | 'tqdm',
21 | 'scipy<1.16.0',
22 | 'pandas',
23 | 'kornia',
24 | 'timm'
25 | ],
26 | classifiers=[
27 | "Programming Language :: Python :: 3.8",
28 | "Programming Language :: Python :: 3.9",
29 | "Programming Language :: Python :: 3.10",
30 | "Programming Language :: Python :: 3.11",
31 | "Programming Language :: Python :: 3.12",
32 | "Intended Audience :: Developers",
33 | "Intended Audience :: Information Technology",
34 | "Intended Audience :: Science/Research",
35 | "Topic :: Scientific/Engineering :: Artificial Intelligence",
36 | "Topic :: Scientific/Engineering :: Information Analysis",
37 | 'License :: OSI Approved :: MIT License',
38 | 'Operating System :: OS Independent',
39 | ],
40 | python_requires='>=3.8',
41 | )
42 |
--------------------------------------------------------------------------------
/static/configurations.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/NUS-HPC-AI-Lab/DD-Ranking/c1ea0b3f251b7927feb25c4dafc76d5312011caf/static/configurations.png
--------------------------------------------------------------------------------
/static/history.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/NUS-HPC-AI-Lab/DD-Ranking/c1ea0b3f251b7927feb25c4dafc76d5312011caf/static/history.png
--------------------------------------------------------------------------------
/static/logo.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/NUS-HPC-AI-Lab/DD-Ranking/c1ea0b3f251b7927feb25c4dafc76d5312011caf/static/logo.png
--------------------------------------------------------------------------------