├── figures
└── fig_teaser_combined.jpg
└── README.md
/figures/fig_teaser_combined.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ziqihuangg/Awesome-Evaluation-of-Visual-Generation/HEAD/figures/fig_teaser_combined.jpg
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Awesome Evaluation of Visual Generation
2 |
3 |
4 | *This repository collects methods for evaluating visual generation.*
5 |
6 | 
7 |
8 | ## Overview
9 |
10 | ### What You'll Find Here
11 |
12 | Within this repository, we collect works that aim to answer some critical questions in the field of evaluating visual generation, such as:
13 |
14 | - **Model Evaluation**: How does one determine the quality of a specific image or video generation model?
15 | - **Sample/Content Evaluation**: What methods can be used to evaluate the quality of a particular generated image or video?
16 | - **User Control Consistency Evaluation**: How to tell how well the generated images and videos align with the user controls or inputs?
17 |
18 | ### Updates
19 |
20 | This repository is updated periodically. If you have suggestions for additional resources, updates on methodologies, or fixes for expiring links, please feel free to do any of the following:
21 | - raise an [Issue](https://github.com/ziqihuangg/Awesome-Evaluation-of-Visual-Generation/issues),
22 | - nominate awesome related works with [Pull Requests](https://github.com/ziqihuangg/Awesome-Evaluation-of-Visual-Generation/pulls),
23 | - We are also contactable via email (`ZIQI002 at e dot ntu dot edu dot sg`).
24 |
25 | ### Table of Contents
26 | - [1. Evaluation Metrics of Generative Models](#1.)
27 | - [1.1. Evaluation Metrics of Image Generation](#1.1.)
28 | - [1.2. Evaluation Metrics of Video Generation](#1.2.)
29 | - [1.3. Evaluation Metrics for Latent Representation](#1.3.)
30 | - [2. Evaluation Metrics of Condition Consistency](#2.)
31 | - [2.1 Evaluation Metrics of Multi-Modal Condition Consistency](#2.1.)
32 | - [2.2. Evaluation Metrics of Image Similarity](#2.2.)
33 | - [3. Evaluation Systems of Generative Models](#3.)
34 | - [3.1. Evaluation of Unconditional Image Generation](#3.1.)
35 | - [3.2. Evaluation of Text-to-Image Generation](#3.2.)
36 | - [3.3. Evaluation of Text-Based Image Editing](#3.3.)
37 | - [3.4. Evaluation of Neural Style Transfer](#3.4.)
38 | - [3.5. Evaluation of Video Generation](#3.5.)
39 | - [3.6. Evaluation of Text-to-Motion Generation](#3.6.)
40 | - [3.7. Evaluation of Model Trustworthiness](#3.7.)
41 | - [3.8. Evaluation of Entity Relation](#3.8.)
42 | - [3.9. Agentic Evaluation](#3.9.)
43 | - [4. Improving Visual Generation with Evaluation / Feedback / Reward](#4.)
44 | - [5. Quality Assessment for AIGC](#5.)
45 | - [6. Study and Rethinking](#6.)
46 | - [7. Other Useful Resources](#7.)
47 |
48 |
49 | ## 1. Evaluation Metrics of Generative Models
50 |
51 | ### 1.1. Evaluation Metrics of Image Generation
52 |
53 |
54 | | Metric | Paper | Code |
55 | | -------- | -------- | ------- |
56 | | Inception Score (IS) | [Improved Techniques for Training GANs](https://arxiv.org/abs/1606.03498) (NeurIPS 2016) | |
57 | | Fréchet Inception Distance (FID) | [GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium](https://arxiv.org/abs/1706.08500) (NeurIPS 2017) | [](https://github.com/bioinf-jku/TTUR) [](https://github.com/mseitzer/pytorch-fid) |
58 | | Kernel Inception Distance (KID) | [Demystifying MMD GANs](https://arxiv.org/abs/1801.01401) (ICLR 2018) | [](https://github.com/toshas/torch-fidelity) [](https://github.com/NVlabs/stylegan2-ada-pytorch/blob/main/metrics/kernel_inception_distance.py)
59 | | CLIP-FID | [The Role of ImageNet Classes in Fréchet Inception Distance](https://arxiv.org/abs/2203.06026) (ICLR 2023) | [](https://github.com/kynkaat/role-of-imagenet-classes-in-fid) [](https://github.com/GaParmar/clean-fid?tab=readme-ov-file#computing-clip-fid) |
60 | | Precision-and-Recall |[Assessing Generative Models via Precision and Recall](https://arxiv.org/abs/1806.00035) (2018-05-31, NeurIPS 2018)
[Improved Precision and Recall Metric for Assessing Generative Models](https://arxiv.org/abs/1904.06991) (NeurIPS 2019) | [](https://github.com/msmsajjadi/precision-recall-distributions) [](https://github.com/kynkaat/improved-precision-and-recall-metric) |
61 | | Renyi Kernel Entropy (RKE) | [An Information-Theoretic Evaluation of Generative Models in Learning Multi-modal Distributions](https://openreview.net/forum?id=PdZhf6PiAb) (NeurIPS 2023) | [](https://github.com/mjalali/renyi-kernel-entropy) |
62 | | CLIP Maximum Mean Discrepancy (CMMD) | [Rethinking FID: Towards a Better Evaluation Metric for Image Generation](https://arxiv.org/abs/2401.09603) (CVPR 2024) | [](https://github.com/google-research/google-research/tree/master/cmmd) |
63 | | Fréchet Wavelet Distance (FWD) | [Fréchet Wavelet Distance: A Domain-Agnostic Metric For Image Generation](https://openreview.net/pdf?id=QinkNNKZ3b) (ICLR 2025) | [](https://github.com/BonnBytes/PyTorch-FWD) |
64 |
65 |
66 | + [Towards a Scalable Reference-Free Evaluation of Generative Models](https://arxiv.org/abs/2407.02961) (2024-07-03)
67 |
68 | + [FaceScore: Benchmarking and Enhancing Face Quality in Human Generation](https://arxiv.org/abs/2406.17100) (2024-06-24)
69 | >Note: Face Score introduced
70 |
71 | + [Global-Local Image Perceptual Score (GLIPS): Evaluating Photorealistic Quality of AI-Generated Images](https://arxiv.org/abs/2405.09426) (2024-05-15)
72 |
73 | + [Unifying and extending Precision Recall metrics for assessing generative models](https://arxiv.org/abs/2405.01611) (2024-05-02)
74 |
75 | + [Enhancing Plausibility Evaluation for Generated Designs with Denoising Autoencoder](https://arxiv.org/abs/2403.05352) (2024-03-08)
76 | >Note: Fréchet Denoised Distance introduced
77 |
78 | + Virtual Classifier Error (VCE) from [Virtual Classifier: A Reversed Approach for Robust Image Evaluation](https://openreview.net/forum?id=IE6FbueT47) (2024-03-04)
79 |
80 | + [An Interpretable Evaluation of Entropy-based Novelty of Generative Models](https://arxiv.org/abs/2402.17287) (2024-02-27)
81 |
82 | + Semantic Shift Rate from [Discovering Universal Semantic Triggers for Text-to-Image Synthesis](https://arxiv.org/abs/2402.07562) (2024-02-12)
83 |
84 | + [Optimizing Prompts Using In-Context Few-Shot Learning for Text-to-Image Generative Models](https://ieeexplore.ieee.org/document/10378642) (2024-01-01)
85 | >Note: Quality Loss introduced
86 |
87 | + [Attribute Based Interpretable Evaluation Metrics for Generative Models](https://arxiv.org/abs/2310.17261) (2023-10-26)
88 |
89 | + [On quantifying and improving realism of images generated with diffusion](https://arxiv.org/abs/2309.14756) (2023-09-26)
90 | >Note: Image Realism Score introduced
91 |
92 | + [Probabilistic Precision and Recall Towards Reliable Evaluation of Generative Models](https://arxiv.org/abs/2309.01590) (2023-09-04)
93 | [](https://github.com/kdst-team/Probablistic_precision_recall)
94 | >Note: P-precision and P-recall introduced
95 |
96 | + [Learning to Evaluate the Artness of AI-generated Images](https://arxiv.org/abs/2305.04923) (2023-05-08)
97 | >Note: ArtScore, metric for images resembling authentic artworks by artists
98 |
99 | + [Training-Free Location-Aware Text-to-Image Synthesis](https://arxiv.org/abs/2304.13427) (2023-04-26)
100 | > Note: New evaluation metric for control capability of location aware generation task
101 |
102 | + [Feature Likelihood Divergence: Evaluating the Generalization of Generative Models Using Samples](https://arxiv.org/abs/2302.04440) (2023-02-09)
103 | [](https://github.com/marcojira/fld)
104 |
105 | + [LGSQE: Lightweight Generated Sample Quality Evaluatoin](https://arxiv.org/abs/2211.04590) (2022-11-08)
106 |
107 | + [SSD: Towards Better Text-Image Consistency Metric in Text-to-Image Generation](https://arxiv.org/abs/2210.15235) (2022-10-27)
108 | > Note: Semantic Similarity Distance introduced
109 |
110 | + [Layout-Bridging Text-to-Image Synthesis](https://arxiv.org/abs/2208.06162) (2022-08-12)
111 | > Note: Layout Quality Score (LQS), new metric for evaluating the generated layout
112 |
113 | + [Rarity Score: A New Metric to Evaluate the Uncommonness of Synthesized Images](https://arxiv.org/abs/2206.08549) (2022-06-17)
114 | [](https://github.com/hichoe95/Rarity-Score)
115 |
116 | + [Mutual Information Divergence: A Unified Metric for Multimodal Generative Models](https://arxiv.org/abs/2205.13445) (2022-05-25)
117 | [](https://github.com/naver-ai/mid.metric)
118 | >Note: evaluates text to image and utilizes vision language models (VLM)
119 |
120 |
121 | + [TREND: Truncated Generalized Normal Density Estimation of Inception Embeddings for GAN Evaluation](https://arxiv.org/abs/2104.14767) (2021-04-30, ECCV 2022)
122 |
123 | + CFID from [Conditional Frechet Inception Distance](https://arxiv.org/abs/2103.11521) (2021-03-21)
124 | [](https://github.com/Michael-Soloveitchik/CFID/)
125 | [](https://michael-soloveitchik.github.io/CFID/)
126 |
127 | + [On Self-Supervised Image Representations for GAN Evaluation](https://openreview.net/forum?id=NeRdBeTionN) (2021-01-12)
128 | [](https://github.com/stanis-morozov/self-supervised-gan-eval)
129 | > Note: SwAV, self-supervised image representation model
130 |
131 | + [Random Network Distillation as a Diversity Metric for Both Image and Text Generation](https://arxiv.org/abs/2010.06715) (2020-10-13)
132 | >Note: RND metric introduced
133 |
134 | + [The Vendi Score: A Diversity Evaluation Metric for Machine Learning](https://arxiv.org/abs/2210.02410) (2022-10-05)
135 | [](https://github.com/vertaix/Vendi-Score)
136 |
137 | + CIS from [Evaluation Metrics for Conditional Image Generation](https://arxiv.org/abs/2004.12361) (2020-04-26)
138 |
139 | + [Text-To-Image Synthesis Method Evaluation Based On Visual Patterns](https://arxiv.org/abs/1911.00077) (2020-04-09)
140 |
141 | + [Cscore: A Novel No-Reference Evaluation Metric for Generated Images](https://dl.acm.org/doi/abs/10.1145/3373509.3373546) (2020-03-25)
142 |
143 |
144 | + SceneFID from [Object-Centric Image Generation from Layouts](https://arxiv.org/abs/2003.07449) (2020-03-16)
145 |
146 | + [Reliable Fidelity and Diversity Metrics for Generative Models](https://arxiv.org/abs/2002.09797) (2020-02-23, ICML 2020)
147 | [](https://github.com/clovaai/generative-evaluation-prdc)
148 |
149 | + [Effectively Unbiased FID and Inception Score and where to find them](https://arxiv.org/abs/1911.07023) (2019-11-16, CVPR 2020)
150 | [](https://github.com/mchong6/FID_IS_infinity)
151 |
152 | + [On the Evaluation of Conditional GANs](https://arxiv.org/abs/1907.08175) (2019-07-11)
153 | >Note:Fréchet Joint Distance (FJD), which is able to assess image quality, conditional consistency, and intra-conditioning diversity within a single metric.
154 |
155 | + [Quality Evaluation of GANs Using Cross Local Intrinsic Dimensionality](https://arxiv.org/abs/1905.00643) (2019-05-02)
156 | > CrossLID, assesses the local intrinsic dimensionality
157 |
158 | + [A domain agnostic measure for monitoring and evaluating GANs](https://arxiv.org/abs/1811.05512) (2018-11-13)
159 |
160 | + [Learning to Generate Images with Perceptual Similarity Metrics](https://arxiv.org/abs/1511.06409) (2015-11-19)
161 | > Multiscale structural-similarity score introduced
162 |
163 | + [A No-Reference Image Blur Metric Based on the Cumulative Probability of Blur Detection (CPBD)](https://ieeexplore.ieee.org/document/5739529) (2011-03-28)
164 |
165 |
166 |
167 | ### 1.2. Evaluation Metrics of Video Generation
168 |
169 |
170 | | Metric | Paper | Code |
171 | | -------- | -------- | ------- |
172 | | FID-vid | [GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium](https://arxiv.org/abs/1706.08500) (NeurIPS 2017) | |
173 | | Fréchet Video Distance (FVD) | [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://arxiv.org/abs/1812.01717) (arXiv 2018)
[FVD: A new Metric for Video Generation](https://openreview.net/forum?id=rylgEULtdN) (2019-05-04) (Note: ICLR 2019 Workshop DeepGenStruct Program Chairs)| [](https://github.com/songweige/TATS/blob/main/tats/fvd/fvd.py) |
174 |
175 |
176 | ### 1.3. Evaluation Metrics for Latent Representation
177 |
178 | + Linear Separability & Perceptual Path Length (PPL) from [A Style-Based Generator Architecture for Generative Adversarial Networks](https://arxiv.org/abs/1812.04948) (2020-01-09)
179 | [](https://github.com/NVlabs/stylegan?tab=readme-ov-file)
180 | [](https://github.com/NVlabs/ffhq-dataset)
181 |
182 |
183 |
184 | ## 2. Evaluation Metrics of Condition Consistency
185 |
186 | ### 2.1 Evaluation Metrics of Multi-Modal Condition Consistency
187 |
188 |
189 | | Metric | Condition | Pipeline | Code | References |
190 | | -------- | -------- | ------- | -------- | -------- |
191 | | CLIP Score (`a.k.a.` CLIPSIM) | Text | cosine similarity between the CLIP image and text embeddings | [](https://github.com/openai/CLIP) [PyTorch Lightning](https://lightning.ai/docs/torchmetrics/stable/multimodal/clip_score.html) | [CLIP Paper](https://arxiv.org/abs/2103.00020) (ICML 2021). Metrics first used in [CLIPScore Paper](https://arxiv.org/abs/2104.08718) (arXiv 2021) and [GODIVA Paper](https://arxiv.org/abs/2104.14806) (arXiv 2021) applies it in video evaluation. |
192 | | Mask Accuracy | Segmentation Mask | predict the segmentatio mask, and compute pixel-wise accuracy against the ground-truth segmentation mask | any segmentation method for your setting |
193 | | DINO Similarity | Image of a Subject (human / object *etc*) | cosine similarity between the DINO embeddings of the generated image and the condition image | [](https://github.com/facebookresearch/dino) | [DINO paper](https://arxiv.org/abs/2104.14294). Metric is proposed in [DreamBooth](https://arxiv.org/abs/2208.12242).
194 |
195 |
196 |
197 |
200 |
201 | + NexusScore, NaturalScore and GmeScore from [OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation](https://arxiv.org/abs/2505.20292) (2025-06-03)
202 | >Note: NexusScore - Identity Consistency - image retrieval + cosine similarity; NaturalScore - Identity Naturalness - prompting gpt4o; GmeScore - Text - cosine similarity between the GME image and text embeddings.
203 |
204 | + FaceSim-Cur from [Identity-Preserving Text-to-Video Generation by Frequency Decomposition](https://arxiv.org/abs/2411.17440) (2024-11-26)
205 | >Note: NFaceSim-Cur - Face image of human - cosine similarity between the curricularface embeddings of the generated face and the input face.
206 |
207 | + Manipulation Direction (MD) from [Manipulation Direction: Evaluating Text-Guided Image Manipulation Based on Similarity between Changes in Image and Text Modalities](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10675000/) (2023-11-20)
208 |
209 | + [Semantic Similarity Distance: Towards better text-image consistency metric in text-to-image generation](https://www-sciencedirect-com.remotexs.ntu.edu.sg/science/article/pii/S0031320323005812?via%3Dihub) (2022-12-02)
210 |
211 | + [On the Evaluation of Conditional GANs](https://arxiv.org/abs/1907.08175) (2019-07-11)
212 | >Note: Fréchet Joint Distance (FJD), which is able to assess image quality, conditional consistency, and intra-conditioning diversity within a single metric.
213 |
214 | + [Classification Accuracy Score for Conditional Generative Models](https://arxiv.org/abs/1905.10887) (2019-05-26)
215 | > Note: New metric Classification Accuracy Score (CAS)
216 |
217 |
218 | + Visual-Semantic (VS) Similarity from [Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Network](https://arxiv.org/abs/1802.09178v2) (2018-12-26)
219 | [](https://github.com/ypxie/HDGan)
220 | [](https://alexhex7.github.io/2018/05/30/Photographic%20Text-to-Image%20Synthesis%20with%20a%20Hierarchically-nested%20Adversarial%20Network/)
221 |
222 |
223 | + [Semantically Invariant Text-to-Image Generation](https://arxiv.org/abs/1809.10274) (2018-09-06)
224 | [](https://github.com/sxs4337/MMVR)
225 | > Note: They evaluate image-text similarity via image captioning
226 |
227 | + [Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis](https://arxiv.org/abs/1801.05091v2) (2018-01-16)
228 | > Note: An object detector based metric is proposed.
229 |
230 |
231 |
232 |
233 | ### 2.2. Evaluation Metrics of Image Similarity
234 |
235 | | Metrics | Paper | Code |
236 | | -------- | -------- | ------- |
237 | | Learned Perceptual Image Patch Similarity (LPIPS) | [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://arxiv.org/abs/1801.03924) (2018-01-11) (CVPR 2018) | [](https://github.com/richzhang/PerceptualSimilarity) [](https://richzhang.github.io/PerceptualSimilarity/) |
238 | | Structural Similarity Index (SSIM) | [Image quality assessment: from error visibility to structural similarity](https://ieeexplore.ieee.org/document/1284395) (TIP 2004) | [](https://github.com/open-mmlab/mmagic/blob/main/tests/test_evaluation/test_metrics/test_ssim.py) [](https://github.com/Po-Hsun-Su/pytorch-ssim) |
239 | | Peak Signal-to-Noise Ratio (PSNR) | - | [](https://github.com/open-mmlab/mmagic/blob/main/tests/test_evaluation/test_metrics/test_psnr.py) |
240 | | Multi-Scale Structural Similarity Index (MS-SSIM) | [Multiscale structural similarity for image quality assessment](https://ieeexplore.ieee.org/document/1292216) (SSC 2004) | [PyTorch-Metrics](https://lightning.ai/docs/torchmetrics/stable/image/multi_scale_structural_similarity.html#:~:text=Compute%20MultiScaleSSIM%2C%20Multi%2Dscale%20Structural,details%20at%20different%20resolution%20scores.&text=a%20method%20to%20reduce%20metric%20score%20over%20labels.) |
241 | | Feature Similarity Index (FSIM) | [FSIM: A Feature Similarity Index for Image Quality Assessment](https://ieeexplore.ieee.org/document/5705575) (TIP 2011) | [](https://github.com/mikhailiuk/pytorch-fsim)
242 |
243 |
244 |
245 | The community has also been using [DINO](https://arxiv.org/abs/2104.14294) or [CLIP](https://arxiv.org/abs/2103.00020) features to measure the semantic similarity of two images / frames.
246 |
247 |
248 | There are also recent works on new methods to measure visual similarity (more will be added):
249 |
250 | + [DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data](https://arxiv.org/abs/2306.09344) (2023-06-15)
251 | [](https://github.com/ssundaram21/dreamsim)
252 | [](https://dreamsim-nights.github.io)
253 |
254 |
255 | ## 3. Evaluation Systems of Generative Models
256 |
257 |
258 | ### 3.1. Evaluation of Unconditional Image Generation
259 |
260 | + [AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perception](https://arxiv.org/abs/2401.08276) (2024-01-16)
261 |
262 | + [A Lightweight Generalizable Evaluation and Enhancement Framework for Generative Models and Generated Samples](https://ieeexplore.ieee.org/document/10495634) (2024-04-16)
263 |
264 | + [Anomaly Score: Evaluating Generative Models and Individual Generated Images based on Complexity and Vulnerability](https://arxiv.org/abs/2312.10634) (2023-12-17, CVPR 2024)
265 |
266 | + [Using Skew to Assess the Quality of GAN-generated Image Features](https://arxiv.org/abs/2310.20636) (2023-10-31)
267 | > Note: Skew Inception Distance introduced
268 |
269 | + [StudioGAN: A Taxonomy and Benchmark of GANs for Image Synthesis](https://arxiv.org/abs/2206.09479) (2022-06-19)
270 | [](https://github.com/POSTECH-CVLab/PyTorch-StudioGAN) [](https://huggingface.co/Mingguksky/PyTorch-StudioGAN/tree/main)
271 |
272 |
273 |
274 | + [HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models](https://arxiv.org/abs/1904.01121) (2019-04-01)
275 | [](https://stanfordhci.github.io/gen-eval/)
276 |
277 | + [An Improved Evaluation Framework for Generative Adversarial Networks](https://arxiv.org/abs/1803.07474) (2018-03-20)
278 | > Note: Class-Aware Frechet Distance introduced
279 |
280 |
281 |
282 | ### 3.2. Evaluation of Text-to-Image Generation
283 | + [GenExam: A Multidisciplinary Text-to-Image Exam](https://arxiv.org/abs/2509.14232) (2025-09-18)
284 | [](https://github.com/OpenGVLab/GenExam)
285 |
286 | + [What Makes a Scene ? Scene Graph-based Evaluation and Feedback for Controllable Generation](https://arxiv.org/abs/2411.15435) (2024-05-26)
287 |
288 | + [Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense?](https://arxiv.org/abs/2406.07546) (2024-08-12)
289 | [](https://zeyofu.github.io/CommonsenseT2I/)
290 |
291 | + [WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation](https://arxiv.org/abs/2503.07265) (2025-05-27)
292 | [](https://github.com/PKU-YuanGroup/WISE)
293 |
294 | + [Why Settle for One? Text-to-ImageSet Generation and Evaluation](https://arxiv.org/abs/2506.23275) (2025-06-29)
295 |
296 | + [LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMs](https://arxiv.org/abs/2504.08358) (2025-04-11)
297 | [](https://github.com/IntMeGroup/LMM4LMM)
298 |
299 | + [Robust and Discriminative Speaker Embedding via Intra-Class Distance Variance Regularization](https://www.isca-archive.org/interspeech_2018/le18_interspeech.html) (2018-09)
300 | >Note: IntraClass Average Distance(ICAD) - Diversity
301 |
302 | + [REAL: Realism Evaluation of Text-to-Image Generation Models for Effective Data Augmentation](https://arxiv.org/abs/2502.10663). (2025-02-15)
303 |
304 | + [Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models](https://arxiv.org/abs/2412.09645) (2024-12-16)
305 | [](https://github.com/Vchitect/Evaluation-Agent)
306 | [](https://vchitect.github.io/Evaluation-Agent-project/)
307 | >Note: focus on efficient and dynamic evaluation.
308 |
309 |
310 | + [ABHINAW: A method for Automatic Evaluation of Typography within AI-Generated Images](https://arxiv.org/abs/2409.11874) (2024-09-18)
311 | [](https://github.com/Abhinaw3906/ABHINAW-MATRIX)
312 |
313 | + [Finding the Subjective Truth: Collecting 2 Million Votes for Comprehensive Gen-AI Model Evaluation](https://arxiv.org/abs/2409.11904) (2024-09-18)
314 |
315 | + [Beyond Aesthetics: Cultural Competence in Text-to-Image Models](https://arxiv.org/abs/2407.06863) (2024-07-09)
316 | > Note: CUBE benchmark introduced
317 |
318 | + [MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?](https://arxiv.org/abs/2407.04842) (2024-07-05)
319 | > Note: MJ-Bench introduced
320 |
321 | + [MIGC++: Advanced Multi-Instance Generation Controller for Image Synthesis](https://arxiv.org/abs/2407.02329) (2024-07-02)
322 | [](https://github.com/limuloo/MIGC)
323 | [](https://migcproject.github.io/)
324 | > Note: Benchmark COCO-MIG and Multimodal-MIG introduced
325 |
326 |
327 | + [Analyzing Quality, Bias, and Performance in Text-to-Image Generative Models](https://arxiv.org/abs/2407.00138) (2024-06-28)
328 |
329 |
330 | + [EvalAlign: Evaluating Text-to-Image Models through Precision Alignment of Multimodal Large Models with Supervised Fine-Tuning to Human Annotations](https://arxiv.org/abs/2406.16562) (2024-06-24)
331 | [](https://github.com/SAIS-FUXI/EvalAlign)
332 | [](https://huggingface.co/Fudan-FUXI/evalalign-v1.0-13b)
333 |
334 |
335 | + [DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation](https://arxiv.org/abs/2406.16855) (2024-06-24)
336 | [](https://github.com/yuangpeng/dreambench_plus)
337 | [](https://dreambenchplus.github.io/)
338 |
339 |
340 | + [Six-CD: Benchmarking Concept Removals for Benign Text-to-image Diffusion Models](https://arxiv.org/pdf/2406.14855) (2024-06-21)
341 | [](https://github.com/Artanisax/Six-CD)
342 |
343 | + [Evaluating Numerical Reasoning in Text-to-Image Models](https://arxiv.org/abs/2406.14774) (2024-06-20)
344 | > Note: GeckoNum introduced
345 |
346 | + [Holistic Evaluation for Interleaved Text-and-Image Generation](https://arxiv.org/abs/2406.14643) (2024-06-20)
347 | > Note: InterleavedBench and InterleavedEval metric introduced
348 |
349 | + [GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation](https://arxiv.org/abs/2406.13743) (2024-06-19)
350 |
351 | + [Decomposed evaluations of geographic disparities in text-to-image models](https://arxiv.org/abs/2406.11988) (2024-06-17)
352 | [](https://ai.meta.com/research/publications/decomposed-evaluations-of-geographic-disparities-in-text-to-image-models/)
353 | > Note: new metric Decomposed Indicators of Disparities introduced
354 |
355 | + [PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Models](https://arxiv.org/abs/2406.11802) (2024-06-17)
356 | [](https://github.com/OpenGVLab/PhyBench)
357 | > Note: PhyBench introduced
358 |
359 | + [Make It Count: Text-to-Image Generation with an Accurate Number of Objects](https://arxiv.org/abs/2406.10210) (2024-06-14)
360 | [](https://github.com/Litalby1/make-it-count)
361 | [](https://make-it-count-paper.github.io/)
362 |
363 | + [Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense?](https://arxiv.org/abs/2406.07546) (2024-06-11)
364 | [](https://github.com/zeyofu/Commonsense-T2I)
365 | [](https://zeyofu.github.io/CommonsenseT2I/)
366 | [](https://huggingface.co/datasets/CommonsenseT2I/CommonsensenT2I)
367 | > Note: Commonsense-T2I, benchmark for real-life commonsense reasoning capabilities of T2I models
368 |
369 | + [Unified Text-to-Image Generation and Retrieval](https://arxiv.org/abs/2406.05814) (2024-06-09)
370 | > Note: TIGeR-Bench, benchmark for evaluation of unified text-to-image generation and retrieval.
371 |
372 | + [PQPP: A Joint Benchmark for Text-to-Image Prompt and Query Performance Prediction](https://arxiv.org/abs/2406.04746) (2024-06-07)
373 | [](https://github.com/Eduard6421/PQPP)
374 |
375 | + [GenAI Arena: An Open Evaluation Platform for Generative Models](https://arxiv.org/abs/2406.04485) (2024-06-06)
376 | [](https://github.com/TIGER-AI-Lab/VideoGenHub?tab=readme-ov-file)
377 |
378 | + [A-Bench: Are LMMs Masters at Evaluating AI-generated Images?](https://arxiv.org/abs/2406.03070) (2024-06-05)
379 | [](https://github.com/Q-Future/A-Bench) [](https://a-bench-sjtu.github.io/) [](https://huggingface.co/datasets/q-future/A-Bench)
380 |
381 | + Multidimensional Preference Score from [Learning Multi-dimensional Human Preference for Text-to-Image Generation](https://arxiv.org/abs/2405.14705) (2024-05-23)
382 |
383 | + [Evolving Storytelling: Benchmarks and Methods for New Character Customization with Diffusion Models](https://arxiv.org/abs/2405.11852) (2024-05-20)
384 | >Note: NewEpisode benchmark introduced
385 |
386 | + [Training-free Subject-Enhanced Attention Guidance for Compositional Text-to-image Generation](https://arxiv.org/abs/2405.06948) (2024-05-11)
387 | >Note: GroundingScore metric introduced
388 |
389 | + [TheaterGen: Character Management with LLM for Consistent Multi-turn Image Generation](https://arxiv.org/abs/2404.18919) (2024-04-29)
390 | [](https://github.com/donahowe/Theatergen)
391 | [](https://howe140.github.io/theatergen.io/)
392 | >Note: consistent score r introduced
393 |
394 | + [Exposing Text-Image Inconsistency Using Diffusion Models](https://arxiv.org/abs/2404.18033) (2024-04-28)
395 |
396 | + [Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratings](https://arxiv.org/abs/2404.16820) (2024-04-25)
397 |
398 | + [Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation](https://arxiv.org/abs/2404.15100) (2024-04-23)
399 |
400 | + [Infusion: Preventing Customized Text-to-Image Diffusion from Overfitting](https://arxiv.org/abs/2404.14007) (2024-04-22)
401 | >Note: Latent Fisher divergence and Wasserstein metric introduced
402 |
403 | + [TAVGBench: Benchmarking Text to Audible-Video Generation](https://arxiv.org/abs/2404.14381) (2024-04-22)
404 | [](https://github.com/OpenNLPLab/TAVGBench)
405 |
406 | + [Object-Attribute Binding in Text-to-Image Generation: Evaluation and Control](https://arxiv.org/abs/2404.13766) (2024-04-21)
407 |
408 | + [Magic Clothing: Controllable Garment-Driven Image Synthesis](https://arxiv.org/abs/2404.09512) (2024-04-15)
409 | [](https://github.com/ShineChen1024/MagicClothing)
410 | [](https://huggingface.co/ShineChen1024/MagicClothing)
411 | > Note: new metric Matched-Points-LPIPS introduced
412 |
413 | + [GenAI-Bench: A Holistic Benchmark for Compositional Text-to-Visual Generation](https://openreview.net/forum?id=hJm7qnW3ym) (2024-04-09)
414 | > Note: GenAI-Bench was introduced in a previous paper 'Evaluating Text-to-Visual Generation with Image-to-Text Generation'
415 |
416 | + Detect-and-Compare from [Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models](https://arxiv.org/abs/2404.04243) (2024-04-05)
417 | [](https://github.com/agwmon/MuDI)
418 | [](https://mudi-t2i.github.io/)
419 |
420 | + [Enhancing Text-to-Image Model Evaluation: SVCS and UCICM](https://ieeexplore.ieee.org/abstract/document/10480770) (2024-04-02)
421 | > Note: Evaluation metrics: Semantic Visual Consistency Score and User-Centric Image Coherence Metric
422 |
423 | + [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://arxiv.org/abs/2404.01291) (2024-04-01)
424 | [](https://github.com/linzhiqiu/t2v_metrics)
425 | [](https://linzhiqiu.github.io/papers/vqascore)
426 |
427 | + [Measuring Style Similarity in Diffusion Models](https://arxiv.org/abs/2404.01292) (2024-04-01)
428 | [](https://github.com/learn2phoenix/CSD)
429 |
430 | + [AAPMT: AGI Assessment Through Prompt and Metric Transformer](https://arxiv.org/abs/2403.19101) (2024-03-28)
431 | [](https://github.com/huskydoge/CS3324-Digital-Image-Processing/tree/main/Assignment1)
432 |
433 |
434 | + [FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models](https://arxiv.org/abs/2403.16379) (2024-03-25)
435 |
436 | + [Refining Text-to-Image Generation: Towards Accurate Training-Free Glyph-Enhanced Image Generation](https://arxiv.org/abs/2403.16422) (2024-03-25)
437 | > Note: LenCom-Eval introduced
438 |
439 | + [Exploring GPT-4 Vision for Text-to-Image Synthesis Evaluation](https://openreview.net/forum?id=xmQoodG82a) (2024-03-20)
440 |
441 | + [DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generation](https://arxiv.org/abs/2403.08857) (2024-03-13)
442 | [](https://github.com/Centaurusalpha/DialogGen)
443 | > Note: DialogBen introduced
444 |
445 | + [Evaluating Text-to-Image Generative Models: An Empirical Study on Human Image Synthesis](https://arxiv.org/abs/2403.05125) (2024-03-08)
446 |
447 | + [An Information-Theoretic Evaluation of Generative Models in Learning Multi-modal Distributions](https://openreview.net/forum?id=PdZhf6PiAb) (2024-02-13)
448 | [](https://github.com/mjalali/renyi-kernel-entropy)
449 |
450 | + [MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis](https://arxiv.org/abs/2402.05408) (2024-02-08)
451 | [](https://github.com/limuloo/MIGC)
452 | [](https://migcproject.github.io/)
453 | > Note: COCO-MIG benchmark introduced
454 |
455 | + [CAS: A Probability-Based Approach for Universal Condition Alignment Score](https://openreview.net/forum?id=E78OaH2s3f) (2024-01-16)
456 | [](https://github.com/unified-metric/unified_metric) [](https://unified-metric.github.io/)
457 | > Note: Condition alignment of text-to-image, {instruction, image}-to-image, edge-/scribble-to-image, and text-to-audio
458 |
459 | + [EmoGen: Emotional Image Content Generation with Text-to-Image Diffusion Models](https://arxiv.org/abs/2401.04608) (2024-01-09)
460 | [](https://github.com/JingyuanYY/EmoGen)
461 | >Note: emotion accuracy, semantic clarity and semantic diversity are not core contributions of this paper
462 |
463 | + [VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation](https://arxiv.org/abs/2312.14867) (2023-12-22)
464 | [](https://github.com/TIGER-AI-Lab/VIEScore) [](https://tiger-ai-lab.github.io/VIEScore/)
465 |
466 | + [PIA: Your Personalized Image Animator via Plug-and-Play Modules in Text-to-Image Models](https://arxiv.org/abs/2312.13964) (2023-12-21)
467 | [](https://github.com/open-mmlab/PIA) [](https://pi-animator.github.io/)
468 | > Note: AnimateBench, benchmark for comparisons in the field of personalized image animation
469 |
470 | + [Stellar: Systematic Evaluation of Human-Centric Personalized Text-to-Image Methods](https://arxiv.org/abs/2312.06116) (2023-12-11)
471 | [](https://github.com/stellar-gen-ai/stellar-metrics)
472 | [](https://stellar-gen-ai.github.io/)
473 |
474 |
475 | + [A Contrastive Compositional Benchmark for Text-to-Image Synthesis: A Study with Unified Text-to-Image Fidelity Metrics](https://arxiv.org/abs/2312.02338) (2023-12-04)
476 | [](https://github.com/zhuxiangru/Winoground-T2I)
477 |
478 | + [The Challenges of Image Generation Models in Generating Multi-Component Images](https://arxiv.org/abs/2311.13620) (2023-11-22)
479 |
480 |
481 | + [SelfEval: Leveraging the discriminative nature of generative models for evaluation](https://arxiv.org/abs/2311.10708) (2023-11-17)
482 |
483 |
484 | + [GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks](https://arxiv.org/abs/2311.01361) (2023-11-02)
485 |
486 | + [Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation](https://arxiv.org/abs/2310.18235) (2023-10-27, ICLR 2024)
487 | [](https://github.com/j-min/DSG)
488 | [](https://google.github.io/dsg/)
489 |
490 | + [DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual Design](https://arxiv.org/abs/2310.15144) (2023-10-23)
491 | [](https://design-bench.github.io)
492 |
493 |
494 | + [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://arxiv.org/abs/2310.11513) (2023-10-17)
495 | [](https://github.com/djghosh13/geneval)
496 |
497 | + [Hypernymy Understanding Evaluation of Text-to-Image Models via WordNet Hierarchy](https://arxiv.org/abs/2310.09247) (2023-10-13)
498 | [](https://github.com/yandex-research/text-to-img-hypernymy)
499 |
500 | + [SingleInsert: Inserting New Concepts from a Single Image into Text-to-Image Models for Flexible Editing](https://arxiv.org/abs/2310.08094) (2023-10-12)
501 | [](https://github.com/JarrentWu1031/SingleInsert) [](https://jarrentwu1031.github.io/SingleInsert-web/)
502 | > Note: New Metric: Editing Success Rate
503 |
504 | + [ImagenHub: Standardizing the evaluation of conditional image generation models](https://arxiv.org/abs/2310.01596) (2023-10-02)
505 | [](https://github.com/TIGER-AI-Lab/ImagenHub) [](https://tiger-ai-lab.github.io/ImagenHub/)
506 | [](https://huggingface.co/ImagenHub)
507 |
508 | + [Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation](https://arxiv.org/abs/2309.14859) (2023-09-26, ICLR 2024)
509 | [](https://github.com/KohakuBlueleaf/LyCORIS)
510 |
511 | + Concept Score from [Text-to-Image Generation for Abstract Concepts](https://paperswithcode.com/paper/text-to-image-generation-for-abstract) (2023-09-26)
512 |
513 | + [OpenLEAF: Open-Domain Interleaved Image-Text Generation and Evaluation](https://openreview.net/forum?id=SeiL55hCnu) (2023-09-23)
514 | [](https://huggingface.co/ImagenHub) [GenAI-Arena](https://huggingface.co/papers/2310.07749)
515 | > Note: Evaluates task of image and text generation
516 |
517 | + [Progressive Text-to-Image Diffusion with Soft Latent Direction](https://arxiv.org/abs/2309.09466) (2023-09-18)
518 | [](https://github.com/babahui/Progressive-Text-to-Image)
519 | >Note: Benchmark for text-to-image generation tasks
520 |
521 |
522 | + [AltDiffusion: A Multilingual Text-to-Image Diffusion Model](https://arxiv.org/abs/2308.09991) (2023-08-19, AAAI 2024)
523 | [](https://github.com/superhero-7/AltDiffusion)
524 | >Note: Benchmark with focus on multilingual generation aspect
525 |
526 |
527 |
528 |
529 |
532 |
533 |
534 | + LEICA from [Likelihood-Based Text-to-Image Evaluation with Patch-Level Perceptual and Semantic Credit Assignment](https://arxiv.org/abs/2308.08525) (2023-08-16)
535 |
536 | + [Let's ViCE! Mimicking Human Cognitive Behavior in Image Generation Evaluation](https://arxiv.org/abs/2307.09416) (2023-07-18)
537 |
538 | + [T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation](https://arxiv.org/abs/2307.06350) (2023-07-12)
539 | [](https://github.com/Karine-Huang/T2I-CompBench)
540 | [](https://karine-h.github.io/T2I-CompBench/)
541 |
542 | + [TIAM -- A Metric for Evaluating Alignment in Text-to-Image Generation](https://arxiv.org/abs/2307.05134) (2023-07-11, WACV 2024)
543 | [](https://github.com/grimalPaul/TIAM)
544 |
545 | + [Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image Alignment with Iterative VQA Feedback](https://arxiv.org/abs/2307.04749) (2023-07-10, NeurIPS 2023)
546 | [](https://github.com/1jsingh/Divide-Evaluate-and-Refine) [](https://1jsingh.github.io/divide-evaluate-and-refine)
547 |
548 | + [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://arxiv.org/abs/2306.09341) (2023-06-15)
549 | [](https://github.com/tgxs002/HPSv2)
550 |
551 | + [ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models](https://arxiv.org/abs/2306.04695) (2023-06-07, AAAI 2024)
552 | [](https://github.com/ConceptBed/evaluations) [](https://conceptbed.github.io/) [](https://huggingface.co/spaces/mpatel57/ConceptBed)
553 |
554 | + [Visual Programming for Text-to-Image Generation and Evaluation](https://arxiv.org/abs/2305.15328) (2023-05-24, NeurIPS 2023)
555 | [](https://github.com/aszala/VPEval) [](https://vp-t2i.github.io/)
556 |
557 | + [LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation](https://arxiv.org/abs/2305.11116) (2023-05-18, NeurIPS 2023)
558 | [](https://github.com/YujieLu10/LLMScore)
559 |
560 | + [X-IQE: eXplainable Image Quality Evaluation for Text-to-Image Generation with Visual Large Language Models](https://arxiv.org/abs/2305.10843) (2023-05-18)
561 | [](https://github.com/Schuture/Benchmarking-Awesome-Diffusion-Models)
562 |
563 | + [What You See is What You Read? Improving Text-Image Alignment Evaluation](https://arxiv.org/abs/2305.10400) (2023-05-17, NeurIPS 2023)
564 | [](https://github.com/yonatanbitton/wysiwyr) [](https://wysiwyr-itm.github.io/) [](https://huggingface.co/datasets/yonatanbitton/SeeTRUE)
565 |
566 | + [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://arxiv.org/abs/2305.01569) (2023-05-02)
567 | [](https://github.com/yuvalkirstain/PickScore)
568 |
569 | + [Analysis of Appeal for Realistic AI-Generated Photos](https://ieeexplore.ieee.org/document/10103686) (2023-04-17) [](https://github.com/Telecommunication-Telemedia-Assessment/avt_ai_images)
570 |
571 | + [Appeal and quality assessment for AI-generated images](https://ieeexplore.ieee.org/document/10178486) (2023-06-22) [](https://github.com/Telecommunication-Telemedia-Assessment/avt_ai_images)
572 |
573 | + [Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation](https://arxiv.org/abs/2304.06671) (2023-04-13)
574 | [](https://github.com/j-min/IterInpaint)
575 | [](https://layoutbench.github.io/)
576 |
577 | + [HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models](https://arxiv.org/abs/2304.05390) (2023-04-11, ICCV 2023)
578 | [](https://github.com/eslambakr/HRS_benchmark) [](https://eslambakr.github.io/hrsbench.github.io/)
579 |
580 | + [Human Preference Score: Better Aligning Text-to-Image Models with Human Preference](https://arxiv.org/abs/2303.14420) (2023-03-25, ICCV 2023)
581 | [](https://github.com/tgxs002/align_sd)
582 | [](https://tgxs002.github.io/align_sd_web/)
583 |
584 | + [A study of the evaluation metrics for generative images containing combinational creativity](https://www-cambridge-org.remotexs.ntu.edu.sg/core/journals/ai-edam/article/study-of-the-evaluation-metrics-for-generative-images-containing-combinational-creativity/FBB623857EE474ED8CD2114450EA8484) (2023-03-23)
585 | >Note: Consensual Assessment Technique and Turing Test used in T2I evaluation
586 |
587 | + [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](https://arxiv.org/abs/2303.11897) (2023-03-21, ICCV 2023)
588 | [](https://github.com/Yushi-Hu/tifa) [](https://tifa-benchmark.github.io/)
589 |
590 | + [Is This Loss Informative? Faster Text-to-Image Customization by Tracking Objective Dynamics](https://arxiv.org/abs/2302.04841) (2023-02-09)
591 | [](https://github.com/yandex-research/DVAR)
592 | >Note: an evaluation approach for early stopping criterion in T2I customization
593 |
594 | + [Benchmarking Spatial Relationships in Text-to-Image Generation](https://arxiv.org/abs/2212.10015) (2022-12-20)
595 | [](https://github.com/microsoft/VISOR)
596 |
597 | + MMI and MOR from from [Benchmarking Robustness of Multimodal Image-Text Models under Distribution Shift](https://arxiv.org/abs/2212.08044) (2022-12-15)
598 | [](https://mmrobustness.github.io/)
599 |
600 | + [TeTIm-Eval: a novel curated evaluation data set for comparing text-to-image models](https://arxiv.org/abs/2212.07839) (2022-12-15)
601 |
602 |
603 | + [Human Evaluation of Text-to-Image Models on a Multi-Task Benchmark](https://arxiv.org/abs/2211.12112) (2022-11-22)
604 |
605 | + [UPainting: Unified Text-to-Image Diffusion Generation with Cross-modal Guidance](https://arxiv.org/abs/2210.16031) (2022-10-28)
606 | [](https://upainting.github.io/)
607 | > Note: UniBench, benchmark contains prompts for simple-scene images and complex-scene images in Chinese and English
608 |
609 | + [Re-Imagen: Retrieval-Augmented Text-to-Image Generator](https://arxiv.org/abs/2209.14491) (2022-09-29)
610 | > Note: EntityDrawBench, benchmark to evaluates image generation for diverse entities
611 |
612 | + [Vision-Language Matching for Text-to-Image Synthesis via Generative Adversarial Networks](https://arxiv.org/abs/2208.09596) (2022-08-20)
613 | > Note: new metric, Vision-Language Matching Score (VLMS)
614 |
615 | + [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://arxiv.org/abs/2206.10789) (2022-06-22)
616 | [](https://github.com/google-research/parti) [](https://sites.research.google/parti/)
617 |
618 | + [GR-GAN: Gradual Refinement Text-to-image Generation](https://arxiv.org/abs/2205.11273) (2022-05-23)
619 | [](https://github.com/BoO-18/GR-GAN)
620 | > Note: new metric Cross-Model Distance introduced
621 |
622 | + [DrawBench from Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://arxiv.org/abs/2205.11487) (2022-05-23)
623 | [](https://imagen.research.google/)
624 |
625 | + [StyleT2I: Toward Compositional and High-Fidelity Text-to-Image Synthesis](https://arxiv.org/abs/2203.15799) (2022-03-29, CVPR 2024)
626 | [](https://github.com/zhihengli-UR/StyleT2I)
627 | > Note: Evaluation metric for compositionality of T2I models
628 |
629 |
630 | + [Benchmark for Compositional Text-to-Image Synthesis](https://openreview.net/forum?id=bKBhQhPeKaF) (2021-07-29)
631 | [](https://github.com/Seth-Park/comp-t2i-dataset)
632 |
633 | + [TISE: Bag of Metrics for Text-to-Image Synthesis Evaluation](https://arxiv.org/abs/2112.01398) (2021-12-02, ECCV 2022)
634 | [](https://github.com/VinAIResearch/tise-toolbox)
635 |
636 | + [Improving Generation and Evaluation of Visual Stories via Semantic Consistency](https://arxiv.org/abs/2105.10026) (2021-05-20)
637 | [](https://github.com/adymaharana/StoryViz)
638 |
639 | + [Leveraging Visual Question Answering to Improve Text-to-Image Synthesis](https://arxiv.org/abs/2010.14953) (2020-10-28)
640 |
641 | + [Image Synthesis from Locally Related Texts](https://dl.acm.org/doi/abs/10.1145/3372278.3390684) (2020-06-08)
642 | > Note: VQA accuracy as a new evaluation metric.
643 |
644 | + [Semantic Object Accuracy for Generative Text-to-Image Synthesis](https://arxiv.org/abs/1910.13321) (2019-10-29)
645 | [](https://github.com/tohinz/semantic-object-accuracy-for-generative-text-to-image-synthesis?tab=readme-ov-file) [](https://www.tobiashinz.com/2019/10/30/semantic-object-accuracy-for-generative-text-to-image-synthesis)
646 | > Note: new evaluation metric, Semantic Object Accuracy (SOA)
647 |
648 | + [GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generation](https://arxiv.org/abs/2504.02782) (2025-04-03)
649 | [](https://github.com/PicoTrex/GPT-ImgEval)
650 |
651 | + [R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation](https://arxiv.org/abs/2505.23493) (2025-05-28)
652 |
653 |
654 | ### 3.3. Evaluation of Text-Based Image Editing
655 |
656 | + [Learning Action and Reasoning-Centric Image Editing from Videos and Simulations](https://arxiv.org/abs/2407.03471) (2024-07-03)
657 | > Note: AURORA-Bench introduced
658 |
659 | + [GIM: A Million-scale Benchmark for Generative Image Manipulation Detection and Localization](https://arxiv.org/abs/2406.16531) (2024-06-24)
660 | [](https://github.com/chenyirui/GIM)
661 |
662 |
663 | + [MultiEdits: Simultaneous Multi-Aspect Editing with Text-to-Image Diffusion Models](https://arxiv.org/abs/2406.00985) (2024-06-03)
664 | [](https://mingzhenhuang.com/projects/MultiEdits.html) [](https://huggingface.co/datasets/UB-CVML-Group/PIE_Bench_pp)
665 | > Note: PIE-Bench++, evaluating image-editing tasks involving multiple objects and attributes
666 |
667 | + [DiffUHaul: A Training-Free Method for Object Dragging in Images](https://arxiv.org/abs/2406.01594) (2024-06-03)
668 | >Note: foreground similarity, object traces and realism metric introduced
669 |
670 | + [HQ-Edit: A High-Quality Dataset for Instruction-based Image Editing](https://arxiv.org/abs/2404.09990) (2024-04-15)
671 | [](https://github.com/UCSC-VLAA/HQ-Edit)
672 | [](https://thefllood.github.io/HQEdit_web/)
673 | [](https://huggingface.co/datasets/UCSC-VLAA/HQ-Edit)
674 |
675 | + [FlexEdit: Flexible and Controllable Diffusion-based Object-centric Image Editing](https://arxiv.org/abs/2403.18605) (2024-03-27)
676 | [](https://flex-edit.github.io/)
677 | >Note: novel automatic mask-based evaluation metric tailored to various object-centric editing scenarios
678 |
679 | + TransformationOriented Paired Benchmark from [InstructBrush: Learning Attention-based Instruction Optimization for Image Editing](https://arxiv.org/abs/2403.18660) (2024-03-27)
680 | [](https://github.com/RoyZhao926/InstructBrush)
681 | [](https://royzhao926.github.io/InstructBrush/)
682 |
683 | + ImageNet Concept Editing Benchmark from [Editing Massive Concepts in Text-to-Image Diffusion Models](https://arxiv.org/abs/2403.13807) (2024-03-20)
684 | [](https://github.com/SilentView/EMCID)
685 | [](https://silentview.github.io/EMCID/)
686 |
687 | + [Editing Massive Concepts in Text-to-Image Diffusion Models](https://arxiv.org/abs/2403.13807) (2024-03-20)
688 | [](https://github.com/SilentView/EMCID) [](https://silentview.github.io/EMCID/)
689 | >Note: ImageNet Concept Editing Benchmark (ICEB), for evaluating massive concept editing for T2I models
690 |
691 | + [Make Me Happier: Evoking Emotions Through Image Diffusion Models](https://arxiv.org/abs/2403.08255) (2024-03-13)
692 | >Note: EMR, ESR, ENRD, ESS metric introduced
693 |
694 |
695 | + [Diffusion Model-Based Image Editing: A Survey](https://arxiv.org/abs/2402.17525) (2024-02-27)
696 | [](https://github.com/SiatMMLab/Awesome-Diffusion-Model-Based-Image-Editing-Methods)
697 | > Note: EditEval, benchmark for text-guided image editing and LLM Score
698 |
699 | + [Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks](https://arxiv.org/abs/2401.07709) (2024-01-15, AAAI 2024)
700 | [](https://github.com/xiaotianqing/InstDiffEdit)
701 | >Note: Editing-Mask, new benchmark to examine the mask accuracy and local editing ability
702 |
703 | + [RotationDrag: Point-based Image Editing with Rotated Diffusion Features](https://arxiv.org/abs/2401.06442) (2024-01-12)
704 | [](https://github.com/Tony-Lowe/RotationDrag)
705 | >Note: RotationBench introduced
706 |
707 | + [LEDITS++: Limitless Image Editing using Text-to-Image Models](https://arxiv.org/abs/2311.16711) (2023-11-28)
708 | [](https://huggingface.co/spaces/editing-images/leditsplusplus/tree/main) [](https://leditsplusplus-project.static.hf.space/index.html) [](https://huggingface.co/spaces/editing-images/leditsplusplus)
709 | > Note: TEdBench++, revised benchmark of TEdBench
710 |
711 | + [Emu Edit: Precise Image Editing via Recognition and Generation Tasks](https://arxiv.org/abs/2311.10089) (2023-11-16)
712 | [](https://huggingface.co/datasets/facebook/emu_edit_test_set)
713 | [](https://emu-edit.metademolab.com/)
714 |
715 |
716 | + [EditVal: Benchmarking Diffusion Based Text-Guided Image Editing Methods](https://arxiv.org/abs/2310.02426) (2023-10-03)
717 | [](https://github.com/deep-ml-research/editval_code)
718 | [](https://deep-ml-research.github.io/editval/)
719 |
720 | + PIE-Bench from [Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Code](https://arxiv.org/abs/2310.01506) (2023-10-02)
721 | [](https://github.com/cure-lab/PnPInversion)
722 | [](https://cure-lab.github.io/PnPInversion/)
723 |
724 | + [Iterative Multi-granular Image Editing using Diffusion Models](https://arxiv.org/abs/2309.00613) (2023-09-01)
725 |
726 | + [DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing](https://arxiv.org/abs/2306.14435) (2023-06-26)
727 | [](https://yujun-shi.github.io/projects/dragdiffusion.html)
728 | [](https://github.com/Yujun-Shi/DragDiffusion)
729 | > Note: drawbench benchmark introduced
730 |
731 | + [DreamEdit: Subject-driven Image Editing](https://arxiv.org/abs/2306.12624) (2023-06-22)
732 | > Note: DreamEditBench benchmark introduced
733 | [](https://dreameditbenchteam.github.io/)
734 | [](https://github.com/DreamEditBenchTeam/DreamEdit)
735 |
736 | + [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://arxiv.org/abs/2306.10012) (2023-06-16)
737 | [](https://github.com/OSU-NLP-Group/MagicBrush)
738 | [](https://osu-nlp-group.github.io/MagicBrush/)
739 | [](https://huggingface.co/datasets/osunlp/MagicBrush)
740 | > Note: dataset only
741 |
742 |
743 | + [Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting](https://arxiv.org/abs/2212.06909) (2022-12-13, CVPR 2023)
744 | [](https://research.google/blog/imagen-editor-and-editbench-advancing-and-evaluating-text-guided-image-inpainting/)
745 |
746 | + [Imagic: Text-Based Real Image Editing with Diffusion Models](https://arxiv.org/abs/2210.09276) (2022-10-17)
747 | [](https://github.com/imagic-editing/imagic-editing.github.io/tree/main/tedbench) [](https://imagic-editing.github.io/) [](https://huggingface.co/datasets/bahjat-kawar/tedbench)
748 | > Note: TEdBench, image editing benchmark
749 |
750 | + [Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Model](https://arxiv.org/abs/2111.13333) (2021-11-26)
751 | [](https://github.com/zipengxuc/PPE)
752 |
753 | + [Knowledge-Driven Generative Adversarial Network for Text-to-Image Synthesis](https://ieeexplore.ieee.org/abstract/document/9552559) (2021-09-29)
754 | [](https://github.com/pengjunn/KD-GAN)
755 | > Note: New evaluation system, Pseudo Turing Test (PTT)
756 |
757 | + [ManiGAN: Text-Guided Image Manipulation](https://arxiv.org/abs/1912.06203) (2019-12-12)
758 | [](https://github.com/mrlibw/ManiGAN)
759 | >Note: manipulative precision metric introduced
760 |
761 | + [Text Guided Person Image Synthesis](https://arxiv.org/abs/1904.05118) (2019-04-10)
762 | >Note: VQA perceptual score introduced
763 |
764 |
765 | ### 3.4. Evaluation of Neural Style Transfer
766 |
767 | + [ArtFID: Quantitative Evaluation of Neural Style Transfer](https://arxiv.org/abs/2207.12280) (2022-07-25)
768 | [](https://github.com/matthias-wright/art-fid)
769 |
770 |
771 | ### 3.5. Evaluation of Video Generation
772 |
773 | #### 3.5.1. Evaluation of Text-to-Video Generation
774 |
775 | + [Are Synthetic Videos Useful? A Benchmark for Retrieval-Centric Evaluation of Synthetic Videos](https://arxiv.org/abs/2507.02316) (2025-07-03)
776 |
777 | + [AIGVE-MACS: Unified Multi-Aspect Commenting and Scoring Model for AI-Generated Video Evaluation](https://arxiv.org/abs/2507.01255) (2025-07-02)
778 |
779 | + [BrokenVideos: A Benchmark Dataset for Fine-Grained Artifact Localization in AI-Generated Videos](https://arxiv.org/abs/2506.20103) (2025-06-25)
780 |
781 | + [OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation](https://arxiv.org/abs/2505.20292) (2025-06-03)
782 | [](https://github.com/PKU-YuanGroup/OpenS2V-Nexus)
783 | [](https://pku-yuangroup.github.io/OpenS2V-Nexus/)
784 | >Note: The first open-sourced infrastructure (OpenS2V-Eval & OpenS2V-5M) for Subject-to-Video generation
785 |
786 | + [LOVE: Benchmarking and Evaluating Text-to-Video Generation and Video-to-Text Interpretation](https://arxiv.org/abs/2505.12098) (2025-05-17)
787 |
788 | + [On the Consistency of Video Large Language Models in Temporal Comprehension](https://arxiv.org/abs/2411.12951) (2025-05-17)
789 |
790 | + [AIGVE-Tool: AI-Generated Video Evaluation Toolkit with Multifaceted Benchmark](https://arxiv.org/abs/2503.14064) (2025-04-18)
791 |
792 | + [VideoGen-Eval: Agent-based System for Video Generation Evaluation](https://arxiv.org/abs/2503.23452) (2025-03-30)
793 | [](https://github.com/AILab-CVC/VideoGen-Eval)
794 |
795 | + [Video-Bench: Human Preference Aligned Video Generation Benchmark](https://arxiv.org/abs/2504.04907) (2025-04-07)
796 | [](https://github.com/Video-Bench/Video-Bench)
797 |
798 | + [Morpheus: Benchmarking Physical Reasoning of Video Generative Models with Real Physical Experiments](https://arxiv.org/abs/2504.02918) (2025-04-03)
799 |
800 | + [Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing](https://arxiv.org/abs/2504.02826) (2025-04-03)
801 | [](https://github.com/PhoenixZ810/RISEBench)
802 |
803 | + [VinaBench: Benchmark for Faithful and Consistent Visual Narratives](https://arxiv.org/abs/2503.20871) (2025-03-26)
804 | [](https://github.com/Silin159/VinaBench)
805 |
806 | + [ETVA: Evaluation of Text-to-Video Alignment via Fine-grained Question Generation and Answering](https://arxiv.org/abs/2503.16867) (2025-03-21)
807 | + [Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generation](https://arxiv.org/abs/2412.16211) (2024-12-17)
808 | >Note: focus on storytelling.
809 |
810 | + [Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models](https://arxiv.org/abs/2412.09645) (2024-12-16)
811 | [](https://github.com/Vchitect/Evaluation-Agent)
812 | [](https://vchitect.github.io/Evaluation-Agent-project/)
813 | >Note: focus on efficient and dynamic evaluation.
814 |
815 | + [Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification](https://arxiv.org/abs/2411.16718) (2024-12-03)
816 | >Note: focus on temporally text-video alignment (event order, accuracy)
817 |
818 | + [AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Generation with LMM](https://arxiv.org/abs/2411.17221) (2024-11-26)
819 | [](https://github.com/wangjiarui153/AIGV-Assessor)
820 | >Note: fuild motion, light change, motion speed, event order.
821 |
822 | + [What You See Is What Matters: A Novel Visual and Physics-Based Metric for Evaluating Video Generation Quality](https://arxiv.org/abs/2411.13609) (2024-11-24)
823 | >Note: texture evaluation scheme introduced
824 |
825 | + [A Survey of AI-Generated Video Evaluation](https://arxiv.org/abs/2410.19884) (2024-10-24)
826 |
827 | + [The Dawn of Video Generation: Preliminary Explorations with SORA-like Models](https://arxiv.org/abs/2410.05227) (2024-10-10)
828 |
829 |
830 | + [Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation](https://arxiv.org/abs/2410.05363) (2024-10-07)
831 | [](https://github.com/OpenGVLab/PhyGenBench)
832 | [](https://phygenbench123.github.io/)
833 | >Note: Comprehensive physical (optical, mechanic, thermal, material) benchmark introduced
834 |
835 | + [Benchmarking AIGC Video Quality Assessment: A Dataset and Unified Model](https://arxiv.org/abs/2407.21408) (2024-07-31)
836 |
837 |
838 | + [T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation](https://arxiv.org/abs/2407.14505) (2024-07-19)
839 | [](https://github.com/KaiyueSun98/T2V-CompBench)
840 | [](https://t2v-compbench.github.io/)
841 |
842 |
843 | + [T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models](https://arxiv.org/abs/2407.05965) (2024-07-08)
844 | >Note: T2VSafetyBench introduced
845 |
846 |
847 |
848 | + [Evaluation of Text-to-Video Generation Models: A Dynamics Perspective](https://arxiv.org/abs/2407.01094) (2024-07-01)
849 |
850 |
851 | + [T2VBench: Benchmarking Temporal Dynamics for Text-to-Video Generation](https://openaccess.thecvf.com/content/CVPR2024W/EvGenFM/html/Ji_T2VBench_Benchmarking_Temporal_Dynamics_for_Text-to-Video_Generation_CVPRW_2024_paper.html) (2024-06)
852 |
853 |
854 | + [Evaluating and Improving Compositional Text-to-Visual Generation](https://openaccess.thecvf.com/content/CVPR2024W/EvGenFM/html/Li_Evaluating_and_Improving_Compositional_Text-to-Visual_Generation_CVPRW_2024_paper.html) (2024-06)
855 |
856 |
857 | + [TlTScore: Towards Long-Tail Effects in Text-to-Visual Evaluation with Generative Foundation Models](https://openaccess.thecvf.com/content/CVPR2024W/EvGenFM/html/Ji_TlTScore_Towards_Long-Tail_Effects_in_Text-to-Visual_Evaluation_with_Generative_Foundation_CVPRW_2024_paper.html) (2024-06)
858 |
859 |
860 | + [ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation](https://arxiv.org/abs/2406.18522) (2024-06-26)
861 | [](https://github.com/PKU-YuanGroup/ChronoMagic-Bench)
862 | [](https://pku-yuangroup.github.io/ChronoMagic-Bench/)
863 | [](https://huggingface.co/spaces/BestWishYsh/ChronoMagic-Bench)
864 | >Note: Comprehensive time-lapse (biological, human created, meteorological, physical) benchmark introduced
865 |
866 | + [VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation](https://arxiv.org/abs/2406.15252) (2024-06-21)
867 | [](https://github.com/TIGER-AI-Lab/VideoScore)
868 | [](https://huggingface.co/datasets/TIGER-Lab/VideoFeedback)
869 |
870 | + [TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generation](https://arxiv.org/abs/2406.08656) (2024-06-12)
871 | >Note: TC-Bench, TCR and TC-Score introduced
872 |
873 | + [VideoPhy: Evaluating Physical Commonsense for Video Generation](https://arxiv.org/abs/2406.03520v1) (2024-06-05)
874 | [](https://videophy.github.io)
875 | [](https://github.com/Hritikbansal/videophy)
876 |
877 | + [Illumination Histogram Consistency Metric for Quantitative Assessment of Video Sequences](https://arxiv.org/abs/2405.09716) (2024-05-15)
878 | [](https://github.com/LongChenCV/IHC-Metric)
879 |
880 | + [The Lost Melody: Empirical Observations on Text-to-Video Generation From A Storytelling Perspective](https://arxiv.org/abs/2405.08720) (2024-05-13)
881 | > Note: New evaluation framework T2Vid2T, Evaluation for storytelling aspects of videos
882 |
883 | + [Exposing AI-generated Videos: A Benchmark Dataset and a Local-and-Global Temporal Defect Based Detection Method](https://arxiv.org/abs/2405.04133) (2024-05-07)
884 |
885 | + [Sora Detector: A Unified Hallucination Detection for Large Text-to-Video Models](https://arxiv.org/abs/2405.04180) (2024-05-07)
886 | [](https://bytez.com/docs/arxiv/2405.04180/llm)
887 | > Note: hallucination detection
888 |
889 | + [Exploring AIGC Video Quality: A Focus on Visual Harmony, Video-Text Consistency and Domain Distribution Gap](https://arxiv.org/abs/2404.13573) (2024-04-21)
890 | [](https://github.com/Coobiw/TriVQA)
891 |
892 | + [Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessment](https://arxiv.org/abs/2403.11956) (2024-03-18)
893 | [](https://github.com/QMME/T2VQA)
894 |
895 | + [A dataset of text prompts, videos and video quality metrics from generative text-to-video AI models](https://www.sciencedirect.com/science/article/pii/S2352340924004839) (2024-02-22)
896 | [](https://github.com/Chiviya01/Evaluating-Text-to-Video-Models)
897 |
898 | + [Sora Generates Videos with Stunning Geometrical Consistency](https://arxiv.org/abs/2402.17403) (2024-02-27)
899 | [](https://github.com/meteorshowers/Sora-Generates-Videos-with-Stunning-Geometrical-Consistency)
900 | [](https://sora-geometrical-consistency.github.io)
901 |
902 |
903 | + [STREAM: Spatio-TempoRal Evaluation and Analysis Metric for Video Generative Models](https://arxiv.org/abs/2403.09669) (2024-01-30)
904 | [](https://github.com/pro2nit/STREAM)
905 |
906 |
907 | + [Towards A Better Metric for Text-to-Video Generation](https://arxiv.org/abs/2401.07781) (2024-01-15)
908 | [](https://github.com/showlab/T2VScore) [](https://showlab.github.io/T2VScore/) [](https://huggingface.co/datasets/jayw/t2v-gen-eval)
909 |
910 | + [PEEKABOO: Interactive Video Generation via Masked-Diffusion](https://arxiv.org/abs/2312.07509) (2023-12-12)
911 | [](https://github.com/microsoft/Peekaboo)
912 | > Note: Benchmark for interactive video generation
913 |
914 | + [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://arxiv.org/abs/2311.17982) (2023-11-29)
915 | [](https://github.com/Vchitect/VBench) [](https://vchitect.github.io/VBench-project/) [](https://huggingface.co/spaces/Vchitect/VBench_Leaderboard)
916 |
917 | + [SmoothVideo: Smooth Video Synthesis with Noise Constraints on Diffusion Models for One-shot Video Tuning](https://arxiv.org/abs/2311.17536) (2023-11-29)
918 | [](https://github.com/SPengLiang/SmoothVideo)
919 |
920 |
921 | + [FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation](https://arxiv.org/abs/2311.01813) (2023-11-03)
922 | [](https://github.com/llyx97/FETV)
923 |
924 | + [EvalCrafter: Benchmarking and Evaluating Large Video Generation Models](https://arxiv.org/abs/2310.11440) (2023-10-17)
925 | [](https://github.com/EvalCrafter/EvalCrafter)
926 | [](https://evalcrafter.github.io) [](https://huggingface.co/datasets/RaphaelLiu/EvalCrafter_T2V_Dataset) [](https://huggingface.co/spaces/AILab-CVC/EvalCrafter)
927 |
928 | + [Measuring the Quality of Text-to-Video Model Outputs: Metrics and Dataset](https://arxiv.org/abs/2309.08009) (2023-09-14)
929 |
930 | + [StoryBench: A Multifaceted Benchmark for Continuous Story Visualization](https://arxiv.org/abs/2308.11606) (2023-08-22, NeurIPS 2023)
931 | [](https://github.com/google/storybench)
932 |
933 | + [Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectives](https://arxiv.org/abs/2211.04894) (2023-03-07, ICCV 2023)
934 | [](https://github.com/VQAssessment/DOVER)
935 | > Note: Aesthetic View & Technical View
936 |
937 | + [CelebV-Text: A Large-Scale Facial Text-Video Dataset](https://arxiv.org/abs/2303.14717) (2023-03-26, CVPR 2023)
938 | [](https://github.com/CelebV-Text/CelebV-Text) [](https://celebv-text.github.io/)
939 | > Note: Benchmark on Facial Text-to-Video Generation
940 |
941 | + [Make It Move: Controllable Image-to-Video Generation with Text Descriptions](https://arxiv.org/abs/2112.02815) (2021-12-06, CVPR 2022)
942 | [](https://github.com/Youncy-Hu/MAGE)
943 |
944 |
945 | #### 3.5.2. Evaluation of Image-to-Video Generation
946 |
947 | + [VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models](https://arxiv.org/abs/2411.13503) (2024-11-20)
948 | [](https://github.com/Vchitect/VBench/tree/master/vbench2_beta_i2v)
949 | [](https://vchitect.github.io/VBench-project/)
950 |
951 | + I2V-Bench from [ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation](https://arxiv.org/abs/2402.04324) (2024-02-06)
952 | [](https://github.com/TIGER-AI-Lab/ConsistI2V) [](https://tiger-ai-lab.github.io/ConsistI2V/) [](https://huggingface.co/spaces/TIGER-Lab/ConsistI2V)
953 |
954 | + [AIGCBench: Comprehensive Evaluation of Image-to-Video Content Generated by AI](https://arxiv.org/abs/2401.01651) (2024-01-03)
955 | [](https://github.com/BenchCouncil/AIGCBench)
956 | [](https://www.benchcouncil.org/AIGCBench/) [](https://huggingface.co/datasets/stevenfan/AIGCBench_v1.0)
957 |
958 |
959 | + [A Benchmark for Controllable Text-Image-to-Video Generation](https://ieeexplore.ieee.org/abstract/document/10148799) (2023-06-12)
960 |
961 | + [Temporal Shift GAN for Large Scale Video Generation](https://arxiv.org/abs/2004.01823) (2020-04-04)
962 | [](https://github.com/amunozgarza/tsb-gan)
963 | >Note: Symmetric-Similarity-Score introduced
964 |
965 | + [Video Imagination from a Single Image with Transformation Generation](https://arxiv.org/abs/1706.04124) (2017-06-13)
966 | >Note: RIQA metric introduced
967 |
968 |
969 | #### 3.5.3. Evaluation of Talking Face Generation
970 |
971 | + [OpFlowTalker: Realistic and Natural Talking Face Generation via Optical Flow Guidance](https://arxiv.org/abs/2405.14709) (2024-05-23)
972 | > Note: VTCS to measures lip-readability in synthesized videos
973 |
974 | + [Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation](https://arxiv.org/abs/2405.04327) (2024-05-07)
975 |
976 | + [VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time](https://arxiv.org/abs/2404.10667) (2024-04-16)
977 | [](https://www.microsoft.com/en-us/research/project/vasa-1/)
978 | >Note: Contrastive Audio and Pose Pretraining (CAPP) score introduced
979 |
980 | + [THQA: A Perceptual Quality Assessment Database for Talking Heads](https://arxiv.org/abs/2404.09003) (2024-04-13)
981 | [](https://github.com/zyj-2000/THQA)
982 |
983 | + [A Comparative Study of Perceptual Quality Metrics for Audio-driven Talking Head Videos](https://arxiv.org/abs/2403.06421) (2024-03-11)
984 | [](https://github.com/zwx8981/ADTH-QA)
985 |
986 | + [Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert](https://arxiv.org/abs/2303.17480) (2023-03-29, CVPR 2023)
987 | [](https://github.com/Sxjdwang/TalkLip)
988 | > Note: Measuring intelligibility of the generated videos
989 |
990 | + [Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectors](https://arxiv.org/abs/2210.07055) (2022-10-13)
991 | [](https://github.com/v-iashin/SparseSync)
992 |
993 | + [Responsive Listening Head Generation: A Benchmark Dataset and Baseline](https://arxiv.org/abs/2112.13548) (2021-12-27, ECCV 2022)
994 | [](https://github.com/dc3ea9f/vico_challenge_baseline) [](https://project.mhzhou.com/vico/)
995 |
996 | + [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://arxiv.org/abs/2008.10010) (2020-08-23)
997 | >Note: new metric LSE-D and LSE-C introduced
998 |
999 | + [What comprises a good talking-head video generation?: A Survey and Benchmark](https://arxiv.org/abs/2005.03201) (2020-05-07)
1000 | [](https://github.com/lelechen63/talking-head-generation-survey)
1001 |
1002 | #### 3.5.4. Evaluation of World Generation
1003 |
1004 | + [WorldScore: A Unified Evaluation Benchmark for World Generation](https://arxiv.org/abs/2504.00983) (2025-04-01)
1005 | [](https://github.com/haoyi-duan/WorldScore) [](https://haoyi-duan.github.io/WorldScore/)
1006 |
1007 |
1008 | ### 3.6. Evaluation of Text-to-Motion Generation
1009 |
1010 | + [VMBench: A Benchmark for Perception-Aligned Video Motion Generation](https://arxiv.org/abs/2503.10076) (2024-03-13)
1011 |
1012 | + [MoDiPO: text-to-motion alignment via AI-feedback-driven Direct Preference Optimization](https://arxiv.org/abs/2405.03803) (2024-05-06)
1013 |
1014 | + [What is the Best Automated Metric for Text to Motion Generation?](https://arxiv.org/abs/2309.10248) (2023-09-19)
1015 |
1016 | + [Text-to-Motion Retrieval: Towards Joint Understanding of Human Motion Data and Natural Language](https://arxiv.org/abs/2305.15842) (2023-05-25)
1017 | [](https://github.com/mesnico/text-to-motion-retrieval)
1018 | > Note: Evaluation protocol for assessing the quality of the retrieved motions
1019 |
1020 | + [Establishing a Unified Evaluation Framework for Human Motion Generation: A Comparative Analysis of Metrics](https://arxiv.org/abs/2405.07680) (2024-05-13)
1021 | [](https://github.com/MSD-IRIMAS/Evaluating-HMG)
1022 |
1023 | + [Evaluation of text-to-gesture generation model using convolutional neural network](https://www.sciencedirect.com/science/article/pii/S0893608022001198) (2021-10-11)
1024 | [](https://github.com/GestureGeneration/text2gesture_cnn)
1025 |
1026 |
1027 |
1028 | ### 3.7. Evaluation of Model Trustworthiness
1029 |
1030 | #### 3.7.1. Evaluation of Visual-Generation-Model Trustworthiness
1031 |
1032 | + [Bias in Gender Bias Benchmarks: How Spurious Features Distort Evaluation](https://arxiv.org/abs/2509.07596) (2025-09-09)
1033 |
1034 | + [MLLM-as-a-Judge for Image Safety without Human Labeling](https://arxiv.org/abs/2501.00192) (2024-12-31)
1035 |
1036 | + [VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models](https://arxiv.org/abs/2411.13503) (2024-11-20)
1037 | [](https://github.com/Vchitect/VBench/tree/master/vbench2_beta_trustworthiness)
1038 | [](https://vchitect.github.io/VBench-project/)
1039 |
1040 | + [BIGbench: A Unified Benchmark for Social Bias in Text-to-Image Generative Models Based on Multi-modal LLM](https://arxiv.org/abs/2407.15240) (2024-07-21)
1041 | [](https://github.com/BIGbench2024/BIGbench2024/)
1042 |
1043 |
1044 | + [Towards Understanding Unsafe Video Generation](https://arxiv.org/abs/2407.12581) (2024-07-17)
1045 | >Note: Proposes Latent Variable Defense (LVD) which works within the model's internal sampling process
1046 |
1047 |
1048 | + [The Factuality Tax of Diversity-Intervened Text-to-Image Generation: Benchmark and Fact-Augmented Intervention](https://arxiv.org/abs/2407.00377) (2024-06-29)
1049 |
1050 |
1051 | + [FairCoT: Enhancing Fairness in Text-to-Image Generation via Chain of Thought Reasoning with Multimodal Large Language Models](https://arxiv.org/abs/2406.09070) (2024-06-13)
1052 | >Note: Normalized Entropy metric introduced
1053 |
1054 | + [Latent Directions: A Simple Pathway to Bias Mitigation in Generative AI](https://arxiv.org/abs/2406.06352) (2024-06-10)
1055 | [](https://github.com/blclo/latent-debiasing-directions) [](https://latent-debiasing-directions.compute.dtu.dk/)
1056 |
1057 | + [Evaluating and Mitigating IP Infringement in Visual Generative AI](https://arxiv.org/abs/2406.04662) (2024-06-07)
1058 | [](https://github.com/ZhentingWang/GAI_IP_Infringement)
1059 |
1060 | + [Improving Geo-diversity of Generated Images with Contextualized Vendi Score Guidance](https://arxiv.org/abs/2406.04551) (2024-06-06)
1061 |
1062 | + [AI-Face: A Million-Scale Demographically Annotated AI-Generated Face Dataset and Fairness Benchmark](https://arxiv.org/abs/2406.00783) (2024-06-02)
1063 | [](https://github.com/Purdue-M2/AI-Face-FairnessBench)
1064 |
1065 | + [FAIntbench: A Holistic and Precise Benchmark for Bias Evaluation in Text-to-Image Models](https://arxiv.org/abs/2405.17814) (2024-05-28)
1066 |
1067 | + [ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Users](https://arxiv.org/abs/2405.19360) (2024-05-24)
1068 |
1069 | + Condition Likelihood Discrepancy from [Membership Inference on Text-to-Image Diffusion Models via Conditional Likelihood Discrepancy](https://arxiv.org/abs/2405.14800) (2024-05-23)
1070 |
1071 | + [Could It Be Generated? Towards Practical Analysis of Memorization in Text-To-Image Diffusion Models](https://arxiv.org/abs/2405.05846) (2024-05-09)
1072 |
1073 |
1074 | + [Towards Geographic Inclusion in the Evaluation of Text-to-Image Models](https://arxiv.org/abs/2405.04457) (2024-05-07)
1075 |
1076 | + [UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Images](https://arxiv.org/abs/2405.03486) (2024-05-06)
1077 |
1078 | + [Espresso: Robust Concept Filtering in Text-to-Image Models](https://arxiv.org/abs/2404.19227) (2024-04-30)
1079 | > Note: Paper is about filtering unacceptable concepts, not evaluation.
1080 |
1081 | + [Ethical-Lens: Curbing Malicious Usages of Open-Source Text-to-Image Models](https://arxiv.org/abs/2404.12104) (2024-04-18)
1082 | [](https://github.com/yuzhu-cai/Ethical-Lens)
1083 |
1084 | + [OpenBias: Open-set Bias Detection in Text-to-Image Generative Models](https://arxiv.org/abs/2404.07990) (2024-04-11)
1085 | [](https://github.com/Picsart-AI-Research/OpenBias)
1086 |
1087 | + [Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigation](https://arxiv.org/abs/2404.01030) (2024-04-01)
1088 |
1089 | + [Lost in Translation? Translation Errors and Challenges for Fair Assessment of Text-to-Image Models on Multilingual Concepts](https://arxiv.org/abs/2403.11092) (2024-03-17, NAACL 2024)
1090 |
1091 | + [Evaluating Text-to-Image Generative Models: An Empirical Study on Human Image Synthesis](https://arxiv.org/abs/2403.05125) (2024-03-08)
1092 |
1093 | + [Position: Towards Implicit Prompt For Text-To-Image Models](https://arxiv.org/abs/2403.02118) (2024-03-04)
1094 | >Note: ImplicitBench, new benchmark
1095 |
1096 |
1097 | + [The Male CEO and the Female Assistant: Probing Gender Biases in Text-To-Image Models Through Paired Stereotype Test](https://arxiv.org/abs/2402.11089) (2024-02-16)
1098 |
1099 | + [Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help You](https://arxiv.org/abs/2401.16092) (2024-01-29)
1100 | [](https://github.com/felifri/magbig)
1101 |
1102 | + [Benchmarking the Fairness of Image Upsampling Methods](https://arxiv.org/abs/2401.13555) (2024-01-24)
1103 |
1104 | + [ViSAGe: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generation](https://arxiv.org/abs/2401.06310) (2024-01-02)
1105 |
1106 | + [New Job, New Gender? Measuring the Social Bias in Image Generation Models](https://arxiv.org/abs/2401.00763) (2024-01-01)
1107 |
1108 | + Distribution Bias, Jaccard Hallucination, Generative Miss Rate from [Quantifying Bias in Text-to-Image Generative Models](https://arxiv.org/abs/2312.13053) (2023-12-20)
1109 | [](https://huggingface.co/spaces/JVice/try-before-you-bias)
1110 | [](https://github.com/JJ-Vice/TryBeforeYouBias)
1111 |
1112 | + [TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Models](https://arxiv.org/abs/2312.01261) (2023-12-03)
1113 | >Note: CAS and BAV novel metric introduced
1114 |
1115 | + [Holistic Evaluation of Text-To-Image Models](https://arxiv.org/abs/2311.04287) (2023-11-07)
1116 | [](https://github.com/stanford-crfm/helm)
1117 | [](https://crfm.stanford.edu/helm/heim/v1.1.0/)
1118 |
1119 | + [Sociotechnical Safety Evaluation of Generative AI Systems](https://arxiv.org/abs/2310.11986) (2023-10-18)
1120 | [](https://deepmind.google/discover/blog/evaluating-social-and-ethical-risks-from-generative-ai/)
1121 |
1122 | + [Navigating Cultural Chasms: Exploring and Unlocking the Cultural POV of Text-To-Image Models](https://arxiv.org/abs/2310.01929) (2023-10-03)
1123 | > Note: Evaluate the cultural content of TTI-generated images
1124 |
1125 | + [ITI-GEN: Inclusive Text-to-Image Generation](https://arxiv.org/abs/2309.05569) (2023-09-11, ICCV 2023)
1126 | [](https://czhang0528.github.io/iti-gen)
1127 | [](https://github.com/humansensinglab/ITI-GEN)
1128 |
1129 | + [DIG In: Evaluating Disparities in Image Generations with Indicators for Geographic Diversity](https://arxiv.org/abs/2308.06198) (2023-08-11)
1130 | [](https://github.com/facebookresearch/DIG-In)
1131 |
1132 | + [On the Cultural Gap in Text-to-Image Generation](https://arxiv.org/abs/2307.02971) (2023-07-06)
1133 | [](https://github.com/longyuewangdcu/C3-Bench)
1134 |
1135 | + [Evaluating the Robustness of Text-to-image Diffusion Models against Real-world Attacks](https://arxiv.org/abs/2306.13103) (2023-06-16)
1136 |
1137 | + [Disparities in Text-to-Image Model Concept Possession Across Languages](https://dl.acm.org/doi/abs/10.1145/3593013.3594123) (2023-06-12)
1138 | > Note: Benchmark of multilingual parity in conceptual possession
1139 |
1140 | + [Evaluating the Social Impact of Generative AI Systems in Systems and Society](https://arxiv.org/abs/2306.05949) (2023-06-09)
1141 |
1142 | + [Word-Level Explanations for Analyzing Bias in Text-to-Image Models](https://arxiv.org/abs/2306.05500) (2023-06-03)
1143 |
1144 | + [Multilingual Conceptual Coverage in Text-to-Image Models](https://arxiv.org/abs/2306.01735) (2023-06-02, ACL 2023)
1145 | [](https://github.com/michaelsaxon/CoCoCroLa)
1146 | [](https://saxon.me/coco-crola/)
1147 | > Note: CoCo-CroLa, benchmark for multilingual parity of text-to-image models
1148 |
1149 | + [T2IAT: Measuring Valence and Stereotypical Biases in Text-to-Image Generation](https://arxiv.org/abs/2306.00905) (2023-06-01)
1150 | [](https://github.com/eric-ai-lab/T2IAT)
1151 |
1152 | + [SneakyPrompt: Jailbreaking Text-to-image Generative Models](https://arxiv.org/abs/2305.12082) (2023-05-20)
1153 | [](https://github.com/Yuchen413/text2image_safety)
1154 |
1155 | + [Inspecting the Geographical Representativeness of Images from Text-to-Image Models](https://arxiv.org/abs/2305.11080) (2023-05-18)
1156 |
1157 | + [Multimodal Composite Association Score: Measuring Gender Bias in Generative Multimodal Models](https://arxiv.org/abs/2304.13855) (2023-04-26)
1158 |
1159 | + [Uncurated Image-Text Datasets: Shedding Light on Demographic Bias](https://arxiv.org/abs/2304.02828) (2023-04-06, CVPR 2023)
1160 | [](https://github.com/noagarcia/phase)
1161 |
1162 | + [Social Biases through the Text-to-Image Generation Lens](https://arxiv.org/abs/2304.06034) (2023-03-30)
1163 |
1164 |
1165 | + [Stable Bias: Analyzing Societal Representations in Diffusion Models](https://arxiv.org/abs/2303.11408) (2023-03-20)
1166 |
1167 |
1168 |
1169 | + [Auditing Gender Presentation Differences in Text-to-Image Models](https://arxiv.org/abs/2302.03675) (2023-02-07)
1170 | [](https://github.com/SALT-NLP/GEP_data) [](https://salt-nlp.github.io/GEP/)
1171 |
1172 | + [Towards Equitable Representation in Text-to-Image Synthesis Models with the Cross-Cultural Understanding Benchmark (CCUB) Dataset](https://arxiv.org/abs/2301.12073) (2023-01-28)
1173 |
1174 | + [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://arxiv.org/abs/2211.05105) (2022-11-09, CVPR 2023)
1175 | [](https://github.com/ml-research/safe-latent-diffusion?tab=readme-ov-file)
1176 | > Note: SLD removes and suppresses inappropriate image parts during the diffusion process
1177 |
1178 | + [How well can Text-to-Image Generative Models understand Ethical Natural Language Interventions?](https://arxiv.org/abs/2210.15230) (2022-10-27)
1179 | [](https://github.com/j-min/DallEval)
1180 |
1181 | + [Exploiting Cultural Biases via Homoglyphs in Text-to-Image Synthesis](https://arxiv.org/abs/2209.08891) (2022-09-19)
1182 | [](https://github.com/LukasStruppek/Exploiting-Cultural-Biases-via-Homoglyphs)
1183 |
1184 |
1185 | + [DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models](https://arxiv.org/abs/2202.04053) (2022-02-08, ICCV 2023)
1186 | [](https://github.com/Hritikbansal/entigen_emnlp)
1187 | > Note: PaintSkills, evaluation for visual reasoning capabilities and social biases
1188 |
1189 | #### 3.7.2. Evaluation of Non-Visual-Generation-Model Trustworthiness
1190 | Not for visual generation, but related evaluations of other models like LLMs
1191 |
1192 | + [The African Woman is Rhythmic and Soulful: Evaluation of Open-ended Generation for Implicit Biases](https://arxiv.org/abs/2407.01270) (2024-07-01)
1193 |
1194 | + [Extrinsic Evaluation of Cultural Competence in Large Language Models](https://arxiv.org/abs/2406.11565) (2024-06-17)
1195 |
1196 | + [Benchmarking Trustworthiness of Multimodal Large Language Models: A Comprehensive Study](https://arxiv.org/abs/2406.07057) (2024-06-11)
1197 |
1198 | + [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://arxiv.org/abs/2402.04249) (2024-02-06)
1199 | [](https://github.com/centerforaisafety/HarmBench)
1200 | [](https://www.harmbench.org)
1201 |
1202 | + [FACET: Fairness in Computer Vision Evaluation Benchmark](https://arxiv.org/abs/2309.00035) (2023-08-31)
1203 | [](https://ai.meta.com/research/publications/facet-fairness-in-computer-vision-evaluation-benchmark/)
1204 | [](https://facet.metademolab.com/)
1205 |
1206 |
1207 | + [Gender Biases in Automatic Evaluation Metrics for Image Captioning](https://arxiv.org/abs/2305.14711) (2023-05-24)
1208 |
1209 | + [Fairness Indicators for Systematic Assessments of Visual Feature Extractors](https://arxiv.org/abs/2202.07603) (2022-02-15)
1210 | [](https://github.com/facebookresearch/vissl/tree/main/projects/fairness_indicators)
1211 | [](https://ai.meta.com/blog/meta-ai-research-explores-new-public-fairness-benchmarks-for-computer-vision-models/)
1212 |
1213 |
1214 |
1215 | ### 3.8. Evaluation of Entity Relation
1216 |
1217 | + Scene Graph(SG)-IoU, Relation-IoU, and Entity-IoU (using GPT-4v) from [SG-Adapter: Enhancing Text-to-Image Generation with Scene Graph Guidance](https://arxiv.org/abs/2405.15321) (2024-05-24)
1218 |
1219 | + Relation Accuracy & Entity Accuracy from [ReVersion: Diffusion-Based Relation Inversion from Images](https://arxiv.org/abs/2303.13495) (2023-03-23)
1220 | [](https://github.com/ziqihuangg/ReVersion)
1221 | [](https://ziqihuangg.github.io/projects/reversion.html)
1222 | [](https://huggingface.co/spaces/Ziqi/ReVersion)
1223 |
1224 | + [Testing Relational Understanding in Text-Guided Image Generation](https://arxiv.org/abs/2208.00005) (2022-07-29)
1225 |
1226 |
1227 | ### 3.9. Agentic Evaluation
1228 |
1229 | + [A Unified Agentic Framework for Evaluating Conditional Image Generation](https://arxiv.org/abs/2504.07046) (2025-04-09)
1230 | [](https://github.com/HITsz-TMG/Agentic-CIGEval)
1231 |
1232 | + [Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models](https://arxiv.org/abs/2412.09645) (2024-12-10)
1233 | [](https://github.com/Vchitect/Evaluation-Agent)
1234 | [](https://vchitect.github.io/Evaluation-Agent-project/)
1235 |
1236 | + [VideoGen-Eval: Agent-based System for Video Generation Evaluation](https://arxiv.org/abs/2503.23452) (2025-03-30)
1237 | [](https://github.com/AILab-CVC/VideoGen-Eval)
1238 | [](https://ailab-cvc.github.io/VideoGen-Eval/)
1239 |
1240 | + [Evaluating Hallucination in Text-to-Image Diffusion Models with Scene-Graph based Question-Answering Agent](https://arxiv.org/abs/2412.05722) (2024-12-07)
1241 |
1242 |
1243 |
1244 | ## 4. Improving Visual Generation with Evaluation / Feedback / Reward
1245 |
1246 | + [OneReward: Unified Mask-Guided Image Generation via Multi-Task Human Preference Learning](https://arxiv.org/abs/2508.21066) (2025-08-28)[](https://github.com/bytedance/OneReward) [](https://one-reward.github.io)
1247 |
1248 | + [Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLM](https://arxiv.org/abs/2412.15156) (2024-12-19) [](https://github.com/jiyt17/Prompt-A-Video)
1249 |
1250 | + [Improved video generation with human feedback](https://arxiv.org/pdf/2501.13918) (2025-01-23) [](https://gongyeliu.github.io/videoalign/)
1251 |
1252 | + [LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment](https://arxiv.org/pdf/2412.04814) (2024-12-24) []() [](https://codegoat24.github.io/LiFT/)
1253 |
1254 | + [VideoDPO: Omni-Preference Alignment for Video Diffusion Generation](https://arxiv.org/abs/2412.14167) (2024-12-18) []() [](https://github.com/CIntellifusion/VideoDPO)
1255 |
1256 | + [Boosting Text-to-Video Generative Model with MLLMs Feedback](https://openreview.net/pdf/4c9eebaad669788792e0a010be4031be5bdc426e.pdf) (2024-09-26,NeurIPS 2024)
1257 |
1258 | + [Direct Unlearning Optimization for Robust and Safe Text-to-Image Models](https://arxiv.org/abs/2407.21035) (2024-07-17)
1259 |
1260 | + [Safeguard Text-to-Image Diffusion Models with Human Feedback Inversion](https://arxiv.org/abs/2407.21032) (2024-07-17, ECCV 2024)
1261 |
1262 | + [Subject-driven Text-to-Image Generation via Preference-based Reinforcement Learning](https://arxiv.org/abs/2407.12164) (2024-07-16)
1263 |
1264 | + [Video Diffusion Alignment via Reward Gradients](https://arxiv.org/abs/2407.08737) (2024-07-11)
1265 | [](https://github.com/mihirp1998/VADER) [](https://vader-vid.github.io/)
1266 |
1267 |
1268 |
1269 | + [Powerful and Flexible: Personalized Text-to-Image Generation via Reinforcement Learning](https://arxiv.org/abs/2407.06642) (2024-07-09)
1270 | [](https://github.com/wfanyue/DPG-T2I-Personalization)
1271 |
1272 | + [Aligning Human Motion Generation with Human Perceptions](https://arxiv.org/abs/2407.02272) (2024-07-02)
1273 | [](https://github.com/ou524u/AlignHP)
1274 |
1275 |
1276 | + [PopAlign: Population-Level Alignment for Fair Text-to-Image Generation](https://arxiv.org/abs/2406.19668) (2024-06-28)
1277 | [](https://github.com/jacklishufan/PopAlignSDXL)
1278 |
1279 |
1280 | + [Prompt Refinement with Image Pivot for Text-to-Image Generation](https://arxiv.org/abs/2407.00247) (2024-06-28, ACL 2024)
1281 |
1282 | + [Diminishing Stereotype Bias in Image Generation Model using Reinforcemenlent Learning Feedback](https://arxiv.org/abs/2407.09551) (2024-06-27)
1283 |
1284 | + [Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation](https://arxiv.org/abs/2406.16807) (2024-06-24)
1285 |
1286 |
1287 |
1288 | + [Batch-Instructed Gradient for Prompt Evolution: Systematic Prompt Optimization for Enhanced Text-to-Image Synthesis](https://arxiv.org/abs/2406.08713) (2024-06-13)
1289 |
1290 |
1291 | + [InstructRL4Pix: Training Diffusion for Image Editing by Reinforcement Learning](https://arxiv.org/abs/2406.09973) (2024-06-14)
1292 | [](https://bair.berkeley.edu/blog/2023/07/14/ddpo/)
1293 |
1294 | + [Diffusion-RPO: Aligning Diffusion Models through Relative Preference Optimization](https://arxiv.org/abs/2406.06382) (2024-06-10)
1295 | > Note: new evaluation metric: style alignment
1296 |
1297 | + [Margin-aware Preference Optimization for Aligning Diffusion Models without Reference](https://arxiv.org/abs/2406.06424) (2024-06-10)
1298 |
1299 | + [ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimization](https://arxiv.org/abs/2406.04312) (2024-06-06)
1300 | [](https://github.com/ExplainableML/ReNO)
1301 |
1302 | + [Step-aware Preference Optimization: Aligning Preference with Denoising Performance at Each Step](https://arxiv.org/abs/2406.04314) (2024-06-06)
1303 |
1304 | + [Improving GFlowNets for Text-to-Image Diffusion Alignment](https://arxiv.org/abs/2406.00633) (2024-06-02)
1305 | > Note: Improves text-to-image alignment with reward function
1306 |
1307 | + [Enhancing Reinforcement Learning Finetuned Text-to-Image Generative Model Using Reward Ensemble](https://link.springer.com/chapter/10.1007/978-3-031-63031-6_19) (2024-06-01)
1308 |
1309 | + [Boost Your Own Human Image Generation Model via Direct Preference Optimization with AI Feedback](https://arxiv.org/abs/2405.20216) (2024-05-30)
1310 |
1311 | + [T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback](https://arxiv.org/abs/2405.18750) (2024-05-29)
1312 | [](https://github.com/Ji4chenLi/t2v-turbo) [](https://t2v-turbo.github.io/)
1313 |
1314 | + [Curriculum Direct Preference Optimization for Diffusion and Consistency Models](https://arxiv.org/abs/2405.13637) (2024-05-22)
1315 |
1316 | + [Class-Conditional self-reward mechanism for improved Text-to-Image models](https://arxiv.org/abs/2405.13473) (2024-05-22)
1317 | [](https://github.com/safouaneelg/SRT2I)
1318 |
1319 | + [Understanding and Evaluating Human Preferences for AI Generated Images with Instruction Tuning](https://arxiv.org/abs/2405.07346) (2024-05-12)
1320 |
1321 | + [Deep Reward Supervisions for Tuning Text-to-Image Diffusion Models](https://arxiv.org/abs/2405.00760) (2024-05-01)
1322 |
1323 | + [ID-Aligner: Enhancing Identity-Preserving Text-to-Image Generation with Reward Feedback Learning](https://arxiv.org/abs/2404.15449) (2024-04-23)
1324 | [](https://github.com/Weifeng-Chen/ID-Aligner) [](https://idaligner.github.io)
1325 |
1326 | + [Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis](https://arxiv.org/abs/2404.13686) (2024-04-21)
1327 | [](https://huggingface.co/ByteDance/Hyper-SD) [](https://hyper-sd.github.io/) [](https://huggingface.co/spaces/ByteDance/Hyper-SDXL-1Step-T2I)
1328 | >Note: Human feedback learning to enhance model performance in low-steps regime
1329 |
1330 |
1331 | + [Prompt Optimizer of Text-to-Image Diffusion Models for Abstract Concept Understanding](https://arxiv.org/abs/2404.11589) (2024-04-17)
1332 |
1333 | + [ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback](https://arxiv.org/abs/2404.07987) (2024-04-11)
1334 |
1335 | + [UniFL: Improve Stable Diffusion via Unified Feedback Learning](https://arxiv.org/abs/2404.05595) (2024-04-08)
1336 |
1337 | + [YaART: Yet Another ART Rendering Technology](https://arxiv.org/abs/2404.05666) (2024-04-08)
1338 |
1339 | + [ByteEdit: Boost, Comply and Accelerate Generative Image Editing](https://arxiv.org/abs/2404.04860) (2024-04-07)
1340 | [](https://byte-edit.github.io/)
1341 | > Note: ByteEdit, feedback learning framework for Generative Image Editing tasks
1342 |
1343 | + [Aligning Diffusion Models by Optimizing Human Utility](https://arxiv.org/abs/2404.04465) (2024-04-06)
1344 |
1345 | + [Dynamic Prompt Optimizing for Text-to-Image Generation](https://arxiv.org/abs/2404.04095) (2024-04-05)
1346 | [](https://github.com/Mowenyii/PAE)
1347 |
1348 | + [Pixel-wise RL on Diffusion Models: Reinforcement Learning from Rich Feedback](https://arxiv.org/abs/2404.04356) (2024-04-05)
1349 |
1350 | + [CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching](https://arxiv.org/abs/2404.03653) (2024-04-04)
1351 | [](https://github.com/CaraJ7/CoMat) [](https://caraj7.github.io/comat/)
1352 |
1353 | + [VersaT2I: Improving Text-to-Image Models with Versatile Reward](https://arxiv.org/abs/2403.18493) (2024-03-27)
1354 |
1355 |
1356 | + [Improving Text-to-Image Consistency via Automatic Prompt Optimization](https://arxiv.org/abs/2403.17804) (2024-03-26)
1357 |
1358 | + [RL for Consistency Models: Faster Reward Guided Text-to-Image Generation](https://arxiv.org/abs/2404.03673) (2024-03-25)
1359 | [](https://rlcm.owenoertell.com)
1360 | [](https://github.com/Owen-Oertell/rlcm)
1361 |
1362 | + [AGFSync: Leveraging AI-Generated Feedback for Preference Optimization in Text-to-Image Generation](https://arxiv.org/abs/2403.13352) (2024-03-20)
1363 |
1364 | + [Reward Guided Latent Consistency Distillation](https://arxiv.org/abs/2403.11027) (2024-03-16)
1365 | [](https://github.com/Ji4chenLi/rg-lcd) [](https://rg-lcd.github.io/)
1366 |
1367 | + [Optimizing Negative Prompts for Enhanced Aesthetics and Fidelity in Text-To-Image Generation](https://arxiv.org/abs/2403.07605) (2024-03-12)
1368 |
1369 |
1370 | + [Debiasing Text-to-Image Diffusion Models](https://arxiv.org/abs/2402.14577) (2024-02-22)
1371 |
1372 |
1373 |
1374 | + [Universal Prompt Optimizer for Safe Text-to-Image Generation](https://arxiv.org/abs/2402.10882) (2024-02-16, NAACL 2024)
1375 | [](https://github.com/wzongyu/POSI)
1376 |
1377 | + [Social Reward: Evaluating and Enhancing Generative AI through Million-User Feedback from an Online Creative Community](https://arxiv.org/abs/2402.09872) (2024-02-15, ICLR 2024)
1378 | [](https://github.com/Picsart-AI-Research/Social-Reward)
1379 |
1380 | + [A Dense Reward View on Aligning Text-to-Image Diffusion with Preference](https://arxiv.org/abs/2402.08265) (2024-02-13, ICML 2024)
1381 | [](https://github.com/Shentao-YANG/Dense_Reward_T2I)
1382 |
1383 | + [Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases](https://arxiv.org/abs/2402.08552) (2024-02-13, ICML 2024)
1384 | [](https://github.com/ZiyiZhang27/tdpo)
1385 |
1386 | + [PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Models](https://arxiv.org/abs/2402.08714) (2024-02-13)
1387 |
1388 | + [Human Aesthetic Preference-Based Large Text-to-Image Model Personalization: Kandinsky Generation as an Example](https://arxiv.org/abs/2402.06389) (2024-02-09)
1389 |
1390 | + [Divide and Conquer: Language Models can Plan and Self-Correct for Compositional Text-to-Image Generation](https://arxiv.org/abs/2401.15688) (2024-01-28)
1391 | [](https://github.com/zhenyuw16/CompAgent_code)
1392 |
1393 | + [Large-scale Reinforcement Learning for Diffusion Models](https://arxiv.org/abs/2401.12244) (2024-01-20)
1394 |
1395 | + [Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to-Image Generation](https://arxiv.org/abs/2401.05675) (2024-01-11)
1396 |
1397 | + [InstructVideo: Instructing Video Diffusion Models with Human Feedback](https://arxiv.org/abs/2312.12490) (2023-12-19)
1398 | [](https://instructvideo.github.io)
1399 |
1400 | + [Rich Human Feedback for Text-to-Image Generation](https://arxiv.org/abs/2312.10240) (2023-12-15, CVPR 2024)
1401 |
1402 | + [iDesigner: A High-Resolution and Complex-Prompt Following Text-to-Image Diffusion Model for Interior Design](https://arxiv.org/abs/2312.04326) (2023-12-07)
1403 |
1404 | + [InstructBooth: Instruction-following Personalized Text-to-Image Generation](https://arxiv.org/abs/2312.03011) (2023-12-04) [](https://sites.google.com/view/instructbooth)
1405 |
1406 | + [DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedback](https://arxiv.org/abs/2311.17946) (2023-11-29)
1407 |
1408 | + [Enhancing Diffusion Models with Text-Encoder Reinforcement Learning](https://arxiv.org/abs/2311.15657) (2023-11-27)
1409 | [](https://github.com/chaofengc/TexForce)
1410 |
1411 | + [AdaDiff: Adaptive Step Selection for Fast Diffusion](https://arxiv.org/abs/2311.14768) (2023-11-24)
1412 |
1413 | + [Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model](https://arxiv.org/abs/2311.13231) (2023-11-22)
1414 | [](https://github.com/yk7333/d3po)
1415 |
1416 | + [Diffusion Model Alignment Using Direct Preference Optimization](https://arxiv.org/abs/2311.12908) (2023-11-21)
1417 | [](https://github.com/SalesforceAIResearch/DiffusionDPO)[](https://github.com/huggingface/diffusers/tree/main/examples/research_projects/diffusion_dpo) [](https://blog.salesforceairesearch.com/diffusion-dpo/)
1418 |
1419 | + [BeautifulPrompt: Towards Automatic Prompt Engineering for Text-to-Image Synthesis](https://arxiv.org/abs/2311.06752) (2023-11-12)
1420 |
1421 |
1422 | + [Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization](https://arxiv.org/abs/2310.12103) (2023-10-18, ICML 2024)
1423 | [](https://github.com/ld-ing/qdhf) [](https://liding.info/qdhf/)
1424 |
1425 |
1426 | + [Aligning Text-to-Image Diffusion Models with Reward Backpropagation](https://arxiv.org/abs/2310.03739) (2023-10-05)
1427 | [](https://github.com/mihirp1998/AlignProp/)
1428 | [](https://align-prop.github.io/)
1429 |
1430 | + [Directly Fine-Tuning Diffusion Models on Differentiable Rewards](https://arxiv.org/abs/2309.17400) (2023-09-29)
1431 |
1432 | + [LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation](https://arxiv.org/abs/2308.05095) (2023-08-09, ACM MM 2023)
1433 | [](https://github.com/LayoutLLM-T2I/LayoutLLM-T2I) [](https://layoutllm-t2i.github.io/) [](https://huggingface.co/leigangqu/LayoutLLM-T2I/tree/main)
1434 |
1435 |
1436 | + [FABRIC: Personalizing Diffusion Models with Iterative Feedback](https://arxiv.org/abs/2307.10159) (2023-07-19)
1437 | [](https://github.com/sd-fabric/fabric) [
1438 |
1439 |
1440 | + [Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image Alignment with Iterative VQA Feedback](https://arxiv.org/abs/2307.04749) (2023-07-10, NeurIPS 2023)
1441 | [](https://github.com/1jsingh/Divide-Evaluate-and-Refine) [](https://1jsingh.github.io/divide-evaluate-and-refine)
1442 |
1443 | + [Censored Sampling of Diffusion Models Using 3 Minutes of Human Feedback](https://arxiv.org/abs/2307.02770) (2023-07-06, NeurIPS 2023)
1444 | [](https://github.com/tetrzim/diffusion-human-feedback)
1445 | > Note: Censored generation using a reward model
1446 |
1447 | + [StyleDrop: Text-to-Image Generation in Any Style](https://arxiv.org/abs/2306.00983) (2023-06-01)
1448 | [](https://styledrop.github.io/)
1449 | > Note: Iterative Training with Feedback
1450 |
1451 |
1452 | + [RealignDiff: Boosting Text-to-Image Diffusion Model with Coarse-to-fine Semantic Re-alignment](https://arxiv.org/abs/2305.19599) (2023-05-31)
1453 |
1454 |
1455 | + [DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models](https://arxiv.org/abs/2305.16381) (2023-05-25, NeurIPS 2023)
1456 | [](https://github.com/google-research/google-research/tree/master/dpok) [](https://sites.google.com/view/dpok-t2i-diffusion/home)
1457 |
1458 | + [Training Diffusion Models with Reinforcement Learning](https://arxiv.org/abs/2305.13301) (2023-05-22)
1459 | [](https://github.com/jannerm/ddpo)
1460 | [Website](https://img.shields.io/badge/Website-9cf)](https://rl-diffusion.github.io/)
1461 |
1462 | + [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://arxiv.org/abs/2304.05977) (2023-04-12)
1463 | [](https://github.com/THUDM/ImageReward)
1464 |
1465 | + [Confidence-aware Reward Optimization for Fine-tuning Text-to-Image Models](https://arxiv.org/abs/2404.01863) (2023-04-02, ICLR 2024)
1466 |
1467 |
1468 | + [Human Preference Score: Better Aligning Text-to-Image Models with Human Preference](https://arxiv.org/abs/2303.14420) (2023-03-25)
1469 | [](https://github.com/tgxs002/align_sd)
1470 | [](https://tgxs002.github.io/align_sd_web/)
1471 |
1472 | + [HIVE: Harnessing Human Feedback for Instructional Visual Editing](https://arxiv.org/abs/2303.09618) (2023-03-16)
1473 | ](https://github.com/salesforce/HIVE)
1474 |
1475 | + [Aligning Text-to-Image Models using Human Feedback](https://arxiv.org/abs/2302.12192) (2023-02-23)
1476 |
1477 | + [Optimizing Prompts for Text-to-Image Generation](https://arxiv.org/abs/2212.09611) (2022-12-19, NeurIPS 2023)
1478 | [](https://github.com/microsoft/LMOps/tree/main/promptist)
1479 | [](https://huggingface.co/spaces/microsoft/Promptist)
1480 |
1481 |
1485 |
1486 |
1487 | ## 5. Quality Assessment for AIGC
1488 |
1489 | ### 5.1. Image Quality Assessment for AIGC
1490 |
1491 | + [Descriptive Image Quality Assessment in the Wild](https://arxiv.org/abs/2405.18842) (2024-05-29)
1492 | [](https://depictqa.github.io/depictqa-wild/)
1493 |
1494 | + [PKU-AIGIQA-4K: A Perceptual Quality Assessment Database for Both Text-to-Image and Image-to-Image AI-Generated Images](https://arxiv.org/abs/2404.18409) (2024-04-29)
1495 | [](https://github.com/jiquan123/I2IQA)
1496 |
1497 | + [Large Multi-modality Model Assisted AI-Generated Image Quality Assessment](https://arxiv.org/abs/2404.17762) (2024-04-27)
1498 | [](https://github.com/wangpuyi/MA-AGIQA)
1499 |
1500 | + [Adaptive Mixed-Scale Feature Fusion Network for Blind AI-Generated Image Quality Assessment](https://arxiv.org/abs/2404.15163) (2024-04-23)
1501 |
1502 | + [PCQA: A Strong Baseline for AIGC Quality Assessment Based on Prompt Condition](https://arxiv.org/abs/2404.13299) (2024-04-20)
1503 |
1504 | + [AIGIQA-20K: A Large Database for AI-Generated Image Quality Assessment](https://arxiv.org/abs/2404.03407) (2024-04-04)
1505 | [](https://www.modelscope.cn/datasets/lcysyzxdxc/AIGCQA-30K-Image/summary)
1506 |
1507 | + [AIGCOIQA2024: Perceptual Quality Assessment of AI Generated Omnidirectional Images](https://arxiv.org/abs/2404.01024) (2024-04-01)
1508 |
1509 | + [Bringing Textual Prompt to AI-Generated Image Quality Assessment](https://arxiv.org/abs/2403.18714) (2024-03-27, ICME 2024)
1510 | [](https://github.com/Coobiw/IP-IQA)
1511 |
1512 | + [TIER: Text-Image Encoder-based Regression for AIGC Image Quality Assessment](https://arxiv.org/abs/2401.03854) (2024-01-08)
1513 | [](https://github.com/jiquan123/TIER)
1514 |
1515 | + [PSCR: Patches Sampling-based Contrastive Regression for AIGC Image Quality Assessment](https://arxiv.org/abs/2312.05897) (2023-12-10)
1516 | [](https://github.com/jiquan123/PSCR)
1517 |
1518 | + [Exploring the Naturalness of AI-Generated Images](https://arxiv.org/abs/2312.05476) (2023-12-09)
1519 | [](https://github.com/zijianchen98/AGIN)
1520 |
1521 | + [PKU-I2IQA: An Image-to-Image Quality Assessment Database for AI Generated Images](https://arxiv.org/abs/2311.15556) (2023-11-27)
1522 | [](https://github.com/jiquan123/I2IQA)
1523 |
1524 | + [Appeal and quality assessment for AI-generated images](https://ieeexplore.ieee.org/document/10178486) (2023-07-18)
1525 |
1526 | + [AIGCIQA2023: A Large-scale Image Quality Assessment Database for AI Generated Images: from the Perspectives of Quality, Authenticity and Correspondence](https://arxiv.org/abs/2307.00211) (2023-07-01)
1527 |
1528 | + [AGIQA-3K: An Open Database for AI-Generated Image Quality Assessment](https://arxiv.org/abs/2306.04717) (2023-06-07)
1529 | [](https://github.com/lcysyzxdxc/AGIQA-3k-Database)
1530 |
1531 | + [A Perceptual Quality Assessment Exploration for AIGC Images](https://arxiv.org/abs/2303.12618) (2023-03-22)
1532 |
1533 | + [SPS: A Subjective Perception Score for Text-to-Image Synthesis](https://ieeexplore.ieee.org/abstract/document/9401705) (2021-04-27)
1534 |
1535 |
1536 | + [GIQA: Generated Image Quality Assessment](https://arxiv.org/abs/2003.08932) (2020-03-19)
1537 | [](https://github.com/cientgu/GIQA)
1538 |
1539 | ### 5.2. Aesthetic Predictors for Generated Images
1540 |
1541 | + [Multi-modal Learnable Queries for Image Aesthetics Assessment](https://arxiv.org/abs/2405.01326) (2024-05-02, ICME 2024)
1542 |
1543 | + Aesthetic Scorer extension for SD Automatic WebUI (2023-01-15)
1544 | [](https://github.com/vladmandic/sd-extension-aesthetic-scorer)
1545 |
1546 |
1547 | + Simulacra Aesthetic-Models (2022-07-09)
1548 | [](https://github.com/crowsonkb/simulacra-aesthetic-models)
1549 |
1550 | + [Rethinking Image Aesthetics Assessment: Models, Datasets and Benchmarks](https://www.researchgate.net/publication/362037160_Rethinking_Image_Aesthetics_Assessment_Models_Datasets_and_Benchmarks) (2022-07-01)
1551 | [](https://github.com/woshidandan/TANet-image-aesthetics-and-quality-assessment)
1552 |
1553 |
1554 | + LAION-Aesthetics_Predictor V2: CLIP+MLP Aesthetic Score Predictor (2022-06-26)
1555 | [](https://github.com/christophschuhmann/improved-aesthetic-predictor)
1556 | [](http://captions.christoph-schuhmann.de/aesthetic_viz_laion_sac+logos+ava1-l14-linearMSE-en-2.37B.html)
1557 | [](https://laion.ai/blog/laion-aesthetics/#laion-aesthetics-v2)
1558 |
1559 |
1560 | + LAION-Aesthetics_Predictor V1 (2022-05-21)
1561 | [](https://github.com/LAION-AI/aesthetic-predictor)
1562 | [](https://laion.ai/blog/laion-aesthetics/#laion-aesthetics-v1)
1563 |
1564 |
1565 |
1567 |
1568 |
1569 | ## 6. Study and Rethinking
1570 |
1571 | ### 6.1. Evaluation of Evaluations
1572 | + [GAIA: Rethinking Action Quality Assessment for AI-Generated Videos](https://arxiv.org/abs/2406.06087) (2024-06-10)
1573 |
1574 | + [Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2)](https://arxiv.org/abs/2404.04251) (2024-04-05)
1575 | [](https://github.com/michaelsaxon/T2IScoreScore)
1576 | [](https://t2iscorescore.github.io) [](https://huggingface.co/datasets/saxon/T2IScoreScore)
1577 |
1578 |
1579 |
1580 | ### 6.2. Survey
1581 |
1582 | + [Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey](https://arxiv.org/abs/2503.12605) (2025-03-23)
1583 |
1584 | + [Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigation](https://arxiv.org/abs/2404.01030) (2024-05-01)
1585 |
1586 | + [Motion Generation: A Survey of Generative Approaches and Benchmarks](https://arxiv.org/abs/2507.05419) (2025-07-07)
1587 |
1588 | + [Advancing Talking Head Generation: A Comprehensive Survey of Multi-Modal Methodologies, Datasets, Evaluation Metrics, and Loss Functions](https://arxiv.org/abs/2507.02900) (2025-06-23)
1589 |
1590 | + [A Survey of Automatic Evaluation Methods on Text, Visual and Speech Generations](https://arxiv.org/abs/2506.10019) (2025-06-06)
1591 |
1592 | + [Survey of Video Diffusion Models: Foundations, Implementations, and Applications](https://arxiv.org/abs/2504.16081) (2025-04-22)
1593 |
1594 | + [A Survey on Quality Metrics for Text-to-Image Generation](https://arxiv.org/abs/2403.11821) (2024-03-18)
1595 |
1596 | + [A Survey of AI-Generated Video Evaluation](https://arxiv.org/abs/2410.19884) (2024-10-24)
1597 |
1598 | + [A Survey of Multimodal-Guided Image Editing with Text-to-Image Diffusion Models](https://arxiv.org/abs/2406.14555) (2024-06-20)
1599 |
1600 | + [From Sora What We Can See: A Survey of Text-to-Video Generation](https://arxiv.org/abs/2405.10674) (2024-05-17)
1601 | [](https://github.com/soraw-ai/Awesome-Text-to-Video-Generation)
1602 | > Note: Refer to Section 3.4 for Evaluation Datasets and Metrics
1603 |
1604 | + [A Survey on Personalized Content Synthesis with Diffusion Models](https://arxiv.org/abs/2405.05538) (2024-05-09)
1605 | > Note: Refere to Section 6 for Evaluation Datasets and Metrics
1606 |
1607 | + [A Survey on Long Video Generation: Challenges, Methods, and Prospects](https://arxiv.org/abs/2403.16407) (2024-03-25)
1608 |
1609 | > Note: Refer to table 2 for evaluation metrics for long video generation
1610 |
1611 | + [Sora as an AGI World Model? A Complete Survey on Text-to-Video Generation](https://arxiv.org/abs/2403.05131) (2024-03-08)
1612 |
1613 | + [State of the Art on Diffusion Models for Visual Computing](https://arxiv.org/abs/2310.07204) (2023-10-11)
1614 | > Note: Refer to Section 9 for Metrics
1615 |
1616 |
1617 | + [AI-Generated Images as Data Source: The Dawn of Synthetic Era](https://arxiv.org/abs/2310.01830) (2023-10-03)
1618 | [](https://github.com/mwxely/AIGS)
1619 | >Note: Refer to Section 4.2 for Evaluation Metrics
1620 |
1621 | + [A Survey on Video Diffusion Models](https://arxiv.org/abs/2310.10647) (2023-10-06)
1622 | [](https://github.com/ChenHsing/Awesome-Video-Diffusion-Models)
1623 | > Note: Refer to Section 2.3 for Evaluation Datasets and Metrics
1624 |
1625 | + [Text-to-image Diffusion Models in Generative AI: A Survey](https://arxiv.org/abs/2303.07909) (2023-03-14)
1626 | > Note: Refer to Section 5 for Evaulation from Techincal and Ethical Perspective
1627 |
1628 | + [Image synthesis: a review of methods, datasets, evaluation metrics, and future outlook](https://link-springer-com.remotexs.ntu.edu.sg/article/10.1007/s10462-023-10434-2#Sec20) (2023-02-28)
1629 | > Note: Refer to section 4 for evaluation metrics
1630 |
1631 | + [Adversarial Text-to-Image Synthesis: A Review](https://arxiv.org/abs/2101.09983) (2021-01-25)
1632 | > Note: Refer to Section 5 for Evaluation of T2I Models
1633 |
1634 | + [Generative Adversarial Networks (GANs): An Overview of Theoretical Model, Evaluation Metrics, and Recent Developments](https://arxiv.org/abs/2005.13178) (2020-05-27)
1635 | > Note: Refer to section 2.2 for Evaluation Metrics
1636 |
1637 | + [What comprises a good talking-head video generation?: A Survey and Benchmark](https://arxiv.org/abs/2005.03201) (2020-05-07)
1638 | [](https://github.com/lelechen63/talking-head-generation-survey)
1639 |
1640 | + [A Survey and Taxonomy of Adversarial Neural Networks for Text-to-Image Synthesis](https://arxiv.org/abs/1910.09399) (2019-10-21)
1641 | > Note: Refer to Section 5 for Benchmark and Evaluation
1642 |
1643 | + [Recent Progress on Generative Adversarial Networks (GANs): A Survey](https://ieeexplore.ieee.org/document/8667290) (2019-03-14)
1644 | > Note: Refer to section 5 for Evaluation Metrics
1645 |
1646 | + [Video Description: A Survey of Methods, Datasets and Evaluation Metrics](https://arxiv.org/abs/1806.00186) (2018-06-01)
1647 | > Note: Refer to section 5 for Evaluation Metrics
1648 |
1649 | ### 6.3. Study
1650 |
1651 | + [A-Bench: Are LMMs Masters at Evaluating AI-generated Images?](https://arxiv.org/abs/2406.03070) (2024-06-05)
1652 | [](https://github.com/Q-Future/A-Bench)
1653 |
1654 | + [On the Content Bias in Fréchet Video Distance](https://arxiv.org/abs/2404.12391) (2024-04-18, CVPR 2024)
1655 | [](https://github.com/songweige/content-debiased-fvd)
1656 | [](https://content-debiased-fvd.github.io)
1657 |
1658 | + [Text-to-Image Synthesis With Generative Models: Methods, Datasets, Performance Metrics, Challenges, and Future Direction](https://ieeexplore.ieee.org/abstract/document/10431766) (2024-02-09)
1659 |
1660 | + [On the Evaluation of Generative Models in Distributed Learning Tasks](https://arxiv.org/abs/2310.11714) (2023-10-18)
1661 |
1662 | + [Recent Advances in Text-to-Image Synthesis: Approaches, Datasets and Future Research Prospects](https://ieeexplore.ieee.org/abstract/document/10224242) (2023-08-18)
1663 |
1664 | + [Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models](https://arxiv.org/abs/2306.04675) (2023-06-07, NeurIPS 2023)
1665 | [](https://github.com/layer6ai-labs/dgm-eval)
1666 |
1667 | + [Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation](https://arxiv.org/abs/2304.01816) (2023-04-04, CVPR 2023)
1668 |
1669 | + [Revisiting the Evaluation of Image Synthesis with GANs](https://arxiv.org/abs/2304.01999) (2023-04-04)
1670 |
1671 |
1672 | + [A Study on the Evaluation of Generative Models](https://arxiv.org/abs/2206.10935) (2022-06-22)
1673 |
1674 | + [REALY: Rethinking the Evaluation of 3D Face Reconstruction](https://arxiv.org/abs/2203.09729) (2022-03-18)
1675 | [](https://github.com/czh-98/REALY)
1676 | [](https://realy3dface.com/)
1677 |
1678 | + [On Aliased Resizing and Surprising Subtleties in GAN Evaluation](https://arxiv.org/abs/2104.11222) (2021-04-22)
1679 | [](https://github.com/GaParmar/clean-fid)
1680 | [](https://www.cs.cmu.edu/~clean-fid/)
1681 |
1682 | + [Pros and Cons of GAN Evaluation Measures: New Developments](https://arxiv.org/abs/2103.09396) (2021-03-17)
1683 |
1684 | + [On the Robustness of Quality Measures for GANs](https://arxiv.org/abs/2201.13019) (2022-01-31, ECCV 2022)
1685 | [](https://github.com/MotasemAlfarra/R-FID-Robustness-of-Quality-Measures-for-GANs)
1686 |
1687 | + [Multimodal Image Synthesis and Editing: The Generative AI Era](https://arxiv.org/abs/2112.13592) (2021-12-27)
1688 | [](https://github.com/fnzhan/Generative-AI)
1689 |
1690 | + [An Analysis of Text-to-Image Synthesis](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3852950) (2021-05-25)
1691 |
1692 | + [Pros and Cons of GAN Evaluation Measures](https://arxiv.org/abs/1802.03446) (2018-02-09)
1693 |
1694 | + [A Note on the Inception Score](https://arxiv.org/abs/1801.01973) (2018-01-06)
1695 |
1696 | + [An empirical study on evaluation metrics of generative adversarial networks](https://arxiv.org/abs/1806.07755) (2018-06-19)
1697 | [](https://github.com/xuqiantong/GAN-Metrics)
1698 |
1699 | + [Are GANs Created Equal? A Large-Scale Study](https://arxiv.org/abs/1711.10337) (2017-11-28, NeurIPS 2018)
1700 |
1701 | + [A note on the evaluation of generative models](https://arxiv.org/abs/1511.01844) (2015-11-05)
1702 |
1703 | + [Appeal prediction for AI up-scaled Images](https://arxiv.org/abs/2502.14013) (2024-12-12) [](https://github.com/Telecommunication-Telemedia-Assessment/ai_upscaling)
1704 |
1705 | ### 6.4. Competition
1706 | + [NTIRE 2024 Quality Assessment of AI-Generated Content Challenge](https://arxiv.org/abs/2404.16687) (2024-04-25)
1707 |
1708 | + [CVPR 2023 Text Guided Video Editing Competition](https://arxiv.org/abs/2310.16003) (2023-10-24)
1709 | [](https://github.com/showlab/loveu-tgve-2023)
1710 | [](https://sites.google.com/view/loveucvpr23/track4)
1711 |
1712 |
1713 | ## 7. Other Useful Resources
1714 | + Stanford Course: CS236 "Deep Generative Models" - Lecture 15 "Evaluation of Generative Models" [[slides]](https://deepgenerativemodels.github.io/assets/slides/lecture15.pdf)
1715 |
1716 | + [Use of Neural Signals to Evaluate the Quality of Generative Adversarial Network Performance in Facial Image Generation](https://arxiv.org/abs/1811.04172) (2018-11-10)
1717 |
1718 |
1719 |
1724 |
--------------------------------------------------------------------------------