50 |
51 |
52 | Princeton University1
53 |
54 |
55 | Google Research2
56 |
57 |
58 |
83 |
84 |
85 |
90 |
91 |
92 | Abstract
93 |
94 |
96 |
97 |
98 |
99 |
100 | Dataset distillation methods offer the promise of reducing a large-scale dataset down to a significantly smaller set of (potentially synthetic) training examples, which preserve sufficient information for training a new model from scratch. So far dataset distillation methods have been developed for image classification . However, with the rise in capabilities of vision-language models , and especially given the scale of datasets necessary to train these models, the time is ripe to expand dataset distillation methods beyond image classification. In this work, we take the first steps towards this goal by expanding on the idea of trajectory matching to create a distillation method for vision-language datasets. The key challenge is that vision-language datasets do not have a set of discrete classes. To overcome this, our proposed vision-and-language dataset distillation method jointly distill the images and their corresponding language descriptions in a contrastive formulation. Since there are no existing baselines, we compare our approach to three coreset selection methods (strategic subsampling of the training dataset), which we adapt to the vision-language setting. We demonstrate significant improvements on the challenging Flickr30K and COCO retrieval benchmarks: for example, on Flickr30K the best coreset selection method which selects 1000 image-text pairs for training is able to achieve only 5.6% image-to-text retrieval accuracy (i.e., recall@1); in contrast, our dataset distillation approach almost doubles that to 9.9% with just 100 (an order of magnitude fewer) training pairs.
101 |
102 |
105 | Bi-Trajectory-Guided Co-Distillation
108 | Dataset distillation traditionally focuses on classification tasks with distinct labels, creating compact distilled datasets for efficient learning.
109 | We've expanded this to a multimodal approach, distilling both vision and language data, emphasizing their interrelation.
110 | Unlike simple classification, our method captures complex connections between image and text data.
111 | It is worth noting that this would be impossible if we solely optimize a single modality, which is supported by our single-modality distillation results.
112 |
113 | The approach consists of two stages:
114 |
115 |
116 |
117 | Obtaining the expert training trajectories \( \{\tau^*\} \), with each trajectory \( \tau^* = \{\theta^*_t\}_{t=0}^T \), by training multiple models for \( T \) epochs on the full dataset \( \mathbf{D} \). For our multimodal setting, the models are trained using bidirectional contrastive loss .
118 |
119 |
120 | Training a set of student models on the current distilled dataset \( \hat{\mathbf{D}} \) using the same bidirectional contrastive loss, and then updating the distilled dataset \( \hat{\mathbf{D}} \) based on the multimodal trajectory matching loss of the student models' parameters and the optimal \( \theta^* \).
121 |
122 |
123 |
124 |
125 |
126 |
128 |
129 |
130 |
132 |
133 |
134 | Vision-Language Bi-Trajectory Matching
135 |
136 |
137 |
138 |
139 |
140 | Following the MTT formulation, we randomly sample \( M \) image-text pairs from \( \mathbf{D} \) to initialize the distilled dataset \( \mathbf{\hat{D}} \) (more details can be found elsewhere). We sample an expert trajectory (i.e., the trajectory of a model trained on the full dataset) \( \tau^* = \{\theta^*_t\}_{t=0}^T \) and a random starting epoch \( s \) to initialize \( \hat{\theta}_s = \theta^*_s \).
141 |
150 |
151 |
152 |
154 |
155 |
156 |
157 |
158 |
159 |
161 |
162 | Results
163 |
164 |
165 | We compare our distillation method to four coreset selection methods: random selection of training examples, herding, k-center and forgetting. We consider different selected sizes (100, 200, 500, and 1000) and report the image-to-text (TR) and text-to-image (IR) retrieval performance on the Flickr30K dataset in Table A.
166 |
We also provide ablation study on the selection of vision (Table B) and language (Table C) backbones. We introduce the Performance Recovery Ratio (PRR) to evaluate the effectiveness of dataset distillation. It quantifies the percentage of performance retained from the original data. The performance for various backbone combinations is shown in Table D.
168 |
169 |
170 |
176 |
177 |
178 |
179 |
180 |
184 | Visualization Left Right
Here we include a number of visualizations of the data we distilled from the multimodal dataset (both Flickr30K and COCO) for a more intuitive understanding of the distilled set.
189 | We provide 50 distilled image-text paired examples including their visualization before the distillation process.
190 | Those experiments are conducted using 100 distilled pairs, with pretrained NFNet and BERT as backbones and the synthetic step is set to 8 during distillation.
191 |
192 |
193 |
202 |
203 |
207 |
208 | Conclusion
209 |
210 |
211 | In this work, we propose a multimodal dataset distillation method for the image-text retrieval task.
212 | By co-distilling both the vision and language modalities, we can progressively optimize and distill the most critical information.
213 | Our experiments show that co-distilling different modalities via trajectory matching holds promise.
214 | We hope that the insights we gathered can be a roadmap for future studies exploring more complex settings, and that our work lays the groundwork for future research aimed at understanding what is the minimum information required for a vision-language model to achieve comparable performance quickly, thereby building a better understanding of the compositionality of compact visual-linguistic knowledge.
215 |
216 |
222 |
223 |
226 | Paper
228 | Xindi Wu, Byron Zhang, Zhiwei Deng, Olga Russakovsky.Vision-Language Dataset Distillation [Arxiv]
233 | [OpenReview]
234 | [Code]
235 | [Video]
236 | [Slides]
237 | [Poster]
238 |
240 |
241 |
242 |
243 |
251 |
252 |
253 |
254 | Copy to Clipboard
255 |
256 |
257 |
258 |
259 |
265 |
266 |
267 |
268 |
269 | Acknowledgements
270 |
271 | This material is based upon work supported by the National Science Foundation under Grant No. 2107048. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. We thank many people from Princeton Visual AI lab (Allison Chen, Jihoon Chung, Tyler Zhu, Ye Zhu, William Yang and Kaiqu Liang) and Princeton NLP group (Carlos E. Jimenez, John Yang), Tiffany Ling and George Cazenavette for their helpful feedback on this work.
272 |
273 |
274 |
275 |
276 |
277 |
16 | 
153 | 
