└── README.md


/README.md:
--------------------------------------------------------------------------------
  1 | <div align='center'>
  2 | <img src="https://cdn.jsdelivr.net/gh/MYJOKERML/imgbed/matebook14/image-20241111160012489.png" alt="image-20241111160012489" style="zoom: 30%;" />
  3 | </div>
  4 | 
  5 | # 🚀Quick Start
  6 | 
  7 | 1. [Introduction](#introduction)
  8 | 2. [Overall](#overall)
  9 |    - [1. The organization of this survey](#1-the-organization-of-this-survey)
 10 |    - [2. General classification of spoken dialogue systems](#2-general-classification-of-spoken-dialogue-systems)
 11 |    - [3. Key capabilities of speech dialogue systems](#3-key-capabilities-of-speech-dialogue-systems)
 12 |    - [4. Publicly Available Speech Dialogue Models](#4-publicly-available-speech-dialogue-models)
 13 | 3. [Representations of Spoken Dialogue Models](#representations-of-spoken-dialogue-models)
 14 | 4. [Training Paradigm of Spoken Dialogue Model](#training-paradigm-of-spoken-dialogue-model)
 15 | 5. [Streaming, Duplex, and Interaction](#streaming-duplex-and-interaction)
 16 | 6. [Training Resources and Evaluation](#training-resources-and-evaluation)
 17 |    - [1. Training resources](#1-training-resources)
 18 |    - [2. Evaluation](#2-evaluation)
 19 | 7. [Cite](#cite)
 20 | 
 21 | # 🔥What's new
 22 | 
 23 | - 2024.11.22: We release WavChat (A survey of spoken dialogue models about 60 pages) on arxiv! 🎉
 24 | - 2024.08.31: We release [WavTokenizer](https://github.com/jishengpeng/WavTokenizer) on arxiv.
 25 | 
 26 | ## Introduction
 27 | 
 28 | This repository is the official repository of the **WavChat: A Survey of Spoken Dialogue Models** [![Paper page](https://huggingface.co/datasets/huggingface/badges/raw/main/paper-page-sm-dark.svg)](https://arxiv.org/abs/2411.13577).
 29 | 
 30 | <div align='center'>
 31 | <img src="https://cdn.jsdelivr.net/gh/MYJOKERML/imgbed/matebook14/image-20241112151419833.png" alt="img1-paper-list" style="zoom: 20%;" />
 32 | 
 33 | Figure 1: The timeline of existing spoken dialogue models in recent years.
 34 | </div>
 35 | 
 36 | > Abstract
 37 | >
 38 | > Recent advancements in spoken dialogue models, exemplified by systems like GPT-4o, have captured significant attention in the speech domain. In the broader context of multimodal models, the speech modality offers a direct interface for human-computer interaction, enabling direct communication between AI and users. Compared to traditional three-tier cascaded spoken dialogue models that comprise speech recognition (ASR), large language models (LLMs), and text-to-speech (TTS), modern spoken dialogue models exhibit greater intelligence. These advanced spoken dialogue models not only comprehend audio, music, and other speech-related features, but also capture stylistic and timbral characteristics in speech. Moreover, they generate high-quality, multi-turn speech responses with low latency, enabling real-time interaction through simultaneous listening and speaking capability. Despite the progress in spoken dialogue systems, there is a lack of comprehensive surveys that systematically organize and analyze these systems and the underlying technologies. To address this, **we have first compiled existing spoken dialogue systems in the chronological order and categorized them into the cascaded and end-to-end paradigms.** We then provide an in-depth overview of the core technologies in spoken dialogue models, covering aspects such as **speech representation, training paradigm, streaming, duplex, and interaction capabilities.** Each section discusses the limitations of these technologies and outlines considerations for future research. Additionally, we present a thorough review of **relevant datasets, evaluation metrics, and benchmarks** from the perspectives of training and evaluating spoken dialogue systems. We hope this survey will contribute to advancing both academic research and industrial applications in the field of spoken dialogue systems.
 39 | 
 40 | ## Overall
 41 | 
 42 | #### 1. The organization of this survey
 43 | 
 44 | <div align='center'>
 45 | <img src="https://cdn.jsdelivr.net/gh/MYJOKERML/imgbed/matebook14/img4-framework-v7_00.png" alt="WavChat - 副本" style="zoom:16%;" />
 46 | 
 47 | 
 48 | 
 49 | Figure 2: Orgnization of this survey.
 50 | </div>
 51 | 
 52 | #### 2. General classification of spoken dialogue systems
 53 | 
 54 | <div align='center'>
 55 | <img src="https://cdn.jsdelivr.net/gh/MYJOKERML/imgbed/matebook14/image-20241112151732341.png" alt="img2-method" style="zoom: 30%;" />
 56 | 
 57 | Figure 3: A general overview of current spoken dialogue systems.
 58 | </div>
 59 | 
 60 | #### 3. Key capabilities of speech dialogue systems
 61 | 
 62 | <div align='center'>
 63 | <img src="https://cdn.jsdelivr.net/gh/MYJOKERML/imgbed/matebook14/image-20241111165006367.png" alt="image-20241111165006367" style="zoom: 25%;" />
 64 | 
 65 | Figure 4: An overview of the spoken dialogue systems' nine ideal capabilities.
 66 | </div>
 67 | 
 68 | #### 4. Publicly Available Speech Dialogue Models
 69 | 
 70 | <div align='center'>
 71 | <table>
 72 |     <thead>
 73 |         <tr>
 74 |             <th>Model</th>
 75 |             <th>URL</th>
 76 |         </tr>
 77 |     </thead>
 78 |     <tbody>
 79 |         <tr>
 80 |             <td>AudioGPT</td>
 81 |             <td><a href="https://github.com/AIGC-Audio/AudioGPT">https://github.com/AIGC-Audio/AudioGPT</a></td>
 82 |         </tr>
 83 |         <tr>
 84 |             <td>SpeechGPT</td>
 85 |             <td><a href="https://github.com/0nutation/SpeechGPT">https://github.com/0nutation/SpeechGPT</a></td>
 86 |         </tr>
 87 |         <tr>
 88 |             <td>Freeze-Omni</td>
 89 |             <td><a href="https://github.com/VITA-MLLM/Freeze-Omni">https://github.com/VITA-MLLM/Freeze-Omni</a></td>
 90 |         </tr>
 91 |         <tr>
 92 |             <td>Baichuan-Omni</td>
 93 |             <td><a href="https://github.com/westlake-baichuan-mllm/bc-omni">https://github.com/westlake-baichuan-mllm/bc-omni</a></td>
 94 |         </tr>
 95 |         <tr>
 96 |             <td>GLM-4-Voice</td>
 97 |             <td><a href="https://github.com/THUDM/GLM-4-Voice">https://github.com/THUDM/GLM-4-Voice</a></td>
 98 |         </tr>
 99 |         <tr>
100 |             <td>Mini-Omni</td>
101 |             <td><a href="https://github.com/gpt-omni/mini-omni">https://github.com/gpt-omni/mini-omni</a></td>
102 |         </tr>
103 |         <tr>
104 |             <td>Mini-Omni2</td>
105 |             <td><a href="https://github.com/gpt-omni/mini-omni2">https://github.com/gpt-omni/mini-omni2</a></td>
106 |         </tr>
107 |         <tr>
108 |             <td>FunAudioLLM</td>
109 |             <td><a href="https://github.com/FunAudioLLM">https://github.com/FunAudioLLM</a></td>
110 |         </tr>
111 |        <tr>
112 |             <td>Qwen-Audio</td>
113 |             <td><a href="https://github.com/QwenLM/Qwen-Audio">https://github.com/QwenLM/Qwen-Audio</a></td>
114 |         </tr>
115 |         <tr>
116 |             <td>Qwen2-Audio</td>
117 |             <td><a href="https://github.com/QwenLM/Qwen2-Audio">https://github.com/QwenLM/Qwen2-Audio</a></td>
118 |         </tr>
119 |         <tr>
120 |             <td>LLaMA3.1</td>
121 |             <td><a href="https://www.llama.com">https://www.llama.com</a></td>
122 |         </tr>
123 |         <tr>
124 |             <td>Audio Flamingo</td>
125 |             <td><a href="https://github.com/NVIDIA/audio-flamingo">https://github.com/NVIDIA/audio-flamingo</a></td>
126 |         </tr>
127 |        <tr>
128 |             <td>Ultravox</td>
129 |             <td><a href="https://github.com/fixie-ai/ultravox">https://github.com/fixie-ai/ultravox</a></td>
130 |         </tr>
131 |         <tr>
132 |             <td>Spirit LM</td>
133 |             <td><a href="https://github.com/facebookresearch/spiritlm">https://github.com/facebookresearch/spiritlm</a></td>
134 |         </tr>
135 |         <tr>
136 |             <td>dGSLM</td>
137 |             <td><a href="https://github.com/facebookresearch/fairseq/tree/main/examples/textless_nlp/dgslm">https://github.com/facebookresearch/fairseq/tree/main/examples/textless_nlp/dgslm</a></td>
138 |         </tr>
139 |         <tr>
140 |             <td>Spoken-LLM</td>
141 |             <td><a href="https://arxiv.org/abs/2305.11000">https://arxiv.org/abs/2305.11000</a></td>
142 |         </tr>
143 |         <tr>
144 |             <td>LLaMA-Omni</td>
145 |             <td><a href="https://github.com/ictnlp/LLaMA-Omni">https://github.com/ictnlp/LLaMA-Omni</a></td>
146 |         </tr>
147 |         <tr>
148 |             <td>Moshi</td>
149 |             <td><a href="https://github.com/kyutai-labs/moshi">https://github.com/kyutai-labs/moshi</a></td>
150 |         </tr>
151 |         <tr>
152 |             <td>SALMONN</td>
153 |             <td><a href="https://github.com/bytedance/SALMONN">https://github.com/bytedance/SALMONN</a></td>
154 |         </tr>
155 |         <tr>
156 |             <td>LTU-AS</td>
157 |             <td><a href="https://github.com/YuanGongND/ltu">https://github.com/YuanGongND/ltu</a></td>
158 |         </tr>
159 |         <tr>
160 |             <td>VITA</td>
161 |             <td><a href="https://github.com/VITA-MLLM/VITA">https://github.com/VITA-MLLM/VITA</a></td>
162 |         </tr>
163 |         <tr>
164 |             <td>SpeechGPT-Gen</td>
165 |             <td><a href="https://github.com/0nutation/SpeechGPT">https://github.com/0nutation/SpeechGPT</a></td>
166 |         </tr>
167 |        <tr>
168 |             <td>WavLLM</td>
169 |             <td><a href="https://github.com/microsoft/SpeechT5/tree/main/WavLLM">https://github.com/microsoft/SpeechT5/tree/main/WavLLM</a></td>
170 |         </tr>
171 |         <tr>
172 |             <td>Westlake-Omni</td>
173 |             <td><a href="https://github.com/xinchen-ai/Westlake-Omni">https://github.com/xinchen-ai/Westlake-Omni</a></td>
174 |         </tr>
175 |         <tr>
176 |             <td>MooER-Omni</td>
177 |             <td><a href="https://github.com/MooreThreads/MooER">https://github.com/MooreThreads/MooER</a></td>
178 |         </tr>
179 |         <tr>
180 |             <td>Hertz-dev</td>
181 |             <td><a href="https://github.com/Standard-Intelligence/hertz-dev">https://github.com/Standard-Intelligence/hertz-dev</a></td>
182 |         </tr>
183 |         <tr>
184 |             <td>Fish-Agent</td>
185 |             <td><a href="https://github.com/fishaudio/fish-speech">https://github.com/fishaudio/fish-speech</a></td>
186 |         </tr>
187 |         <tr>
188 |             <td>SpeechGPT2</td>
189 |             <td><a href="https://0nutation.github.io/SpeechGPT2.github.io/">https://0nutation.github.io/SpeechGPT2.github.io/</a></td>
190 |         </tr>
191 |     </tbody>
192 | </table>
193 | 
194 | Table 1: The list of publicly available speech dialogue models and their URL
195 | </div>
196 | 
197 | ## Representations of Spoken Dialogue Models
198 | 
199 | In the section Representations of Spoken Dialogue Models, we provide insights into how to represent the data in a speech dialogue model for better understanding and generation of speech. The choice of representation method directly affects the model's effectiveness in processing speech signals, system performance, and range of applications. The section covers two main types of representations: **semantic representations** and **acoustic representations**.
200 | 
201 | |              | Advantages of the comprehension side | Performance of unify music and audio | Compression rate of speech | Convert to historical context | Emotional and acoustic information | Pipeline for post-processing |
202 | | ------------ | ------------------------------------ | ------------------------------------ | -------------------------- | ----------------------------- | ---------------------------------- | ---------------------------- |
203 | | **Semantic** | Strong                               | Weak                                 | High                       | Easy                          | Less                               | Cascade                      |
204 | | **Acoustic** | Weak                                 | Strong                               | Low                        | Difficult                     | More                               | End-to-end                   |
205 | 
206 | <div align='center'>
207 | Table 2: The comparison of semantic and acoustic representations
208 | </div>
209 | 
210 | <br/>
211 | And we provide a comprehensive list of publicly available codec models and their URLs.
212 | 
213 | <br/>
214 | 
215 | <div align='center'>
216 | <table border="1" cellspacing="0" cellpadding="8" style="width: 100%; text-align: left;">
217 |   <table border="1" cellspacing="0" cellpadding="8" style="width: 100%; text-align: left;">
218 | 
219 |   <thead>
220 |     <tr>
221 |       <th><strong>Model</strong></th>
222 |       <th><strong>URL</strong></th>
223 |     </tr>
224 |   </thead>
225 |   <tbody>
226 |     <tr>
227 |       <td>Encodec</td>
228 |       <td><a href="https://github.com/facebookresearch/encodec">https://github.com/facebookresearch/encodec</a></td>
229 |     </tr>
230 |     <tr>
231 |       <td>SoundStream</td>
232 |       <td><a href="https://github.com/wesbz/SoundStream">https://github.com/wesbz/SoundStream</a></td>
233 |     </tr>
234 |     <tr>
235 |       <td>DAC</td>
236 |       <td><a href="https://github.com/descriptinc/descript-audio-codec">https://github.com/descriptinc/descript-audio-codec</a></td>
237 |     </tr>
238 |     <tr>
239 |       <td>WavTokenizer</td>
240 |       <td><a href="https://github.com/jishengpeng/WavTokenizer">https://github.com/jishengpeng/WavTokenizer</a></td>
241 |     </tr>
242 |     <tr>
243 |       <td>SpeechTokenizer</td>
244 |       <td><a href="https://github.com/ZhangXInFD/SpeechTokenizer">https://github.com/ZhangXInFD/SpeechTokenizer</a></td>
245 |     </tr>
246 |     <tr>
247 |       <td>SNAC</td>
248 |       <td><a href="https://github.com/hubertsiuzdak/snac">https://github.com/hubertsiuzdak/snac</a></td>
249 |     </tr>
250 |     <tr>
251 |       <td>SemantiCodec</td>
252 |       <td><a href="https://github.com/haoheliu/SemantiCodec-inference">https://github.com/haoheliu/SemantiCodec-inference</a></td>
253 |     </tr>
254 |     <tr>
255 |       <td>Mimi</td>
256 |       <td><a href="https://github.com/kyutai-labs/moshi">https://github.com/kyutai-labs/moshi</a></td>
257 |     </tr>
258 |     <tr>
259 |       <td>HiFi-Codec</td>
260 |       <td><a href="https://github.com/yangdongchao/AcademiCodec">https://github.com/yangdongchao/AcademiCodec</a></td>
261 |     </tr>
262 |     <tr>
263 |       <td>FunCodec</td>
264 |       <td><a href="https://github.com/modelscope/FunCodec">https://github.com/modelscope/FunCodec</a></td>
265 |     </tr>
266 |     <tr>
267 |       <td>APCodec</td>
268 |       <td><a href="https://github.com/YangAi520/APCodec/tree/main">https://github.com/YangAi520/APCodec/tree/main</a></td>
269 |     </tr>
270 |     <tr>
271 |       <td>AudioDec</td>
272 |       <td><a href="https://github.com/facebookresearch/AudioDec">https://github.com/facebookresearch/AudioDec</a></td>
273 |     </tr>
274 |     <tr>
275 |       <td>FACodec</td>
276 |       <td><a href="https://github.com/lifeiteng/naturalspeech3_facodec">https://github.com/lifeiteng/naturalspeech3_facodec</a></td>
277 |     </tr>
278 |     <tr>
279 |       <td>Language-Codec</td>
280 |       <td><a href="https://github.com/jishengpeng/Languagecodec">https://github.com/jishengpeng/Languagecodec</a></td>
281 |     </tr>
282 |     <tr>
283 |       <td>XCodec</td>
284 |       <td><a href="https://github.com/zhenye234/xcodec">https://github.com/zhenye234/xcodec</a></td>
285 |     </tr>
286 |     <tr>
287 |       <td>TiCodec</td>
288 |       <td><a href="https://github.com/y-ren16/TiCodec">https://github.com/y-ren16/TiCodec</a></td>
289 |     </tr>
290 |     <tr>
291 |       <td>SoCodec</td>
292 |       <td><a href="https://github.com/hhguo/SoCodec">https://github.com/hhguo/SoCodec</a></td>
293 |     </tr>
294 |     <tr>
295 |       <td>FUVC</td>
296 |       <td><a href="https://github.com/z21110008/FUVC">https://github.com/z21110008/FUVC</a></td>
297 |     </tr>
298 |     <tr>
299 |       <td>HILCodec</td>
300 |       <td><a href="https://github.com/aask1357/hilcodec">https://github.com/aask1357/hilcodec</a></td>
301 |     </tr>
302 |     <tr>
303 |       <td>LaDiffCodec</td>
304 |       <td><a href="https://github.com/haiciyang/LaDiffCodec">https://github.com/haiciyang/LaDiffCodec</a></td>
305 |     </tr>
306 |     <tr>
307 |       <td>LLM-Codec</td>
308 |       <td><a href="https://github.com/yangdongchao/LLM-Codec">https://github.com/yangdongchao/LLM-Codec</a></td>
309 |     </tr>
310 |     <tr>
311 |       <td>SpatialCodec</td>
312 |       <td><a href="https://github.com/XZWY/SpatialCodec">https://github.com/XZWY/SpatialCodec</a></td>
313 |     </tr>
314 |     <tr>
315 |       <td>BigCodec</td>
316 |       <td><a href="https://github.com/Aria-K-Alethia/BigCodec">https://github.com/Aria-K-Alethia/BigCodec</a></td>
317 |     </tr>
318 |     <tr>
319 |       <td>SuperCodec</td>
320 |       <td><a href="https://github.com/exercise-book-yq/Supercodec">https://github.com/exercise-book-yq/Supercodec</a></td>
321 |     </tr>
322 |     <tr>
323 |       <td>RepCodec</td>
324 |       <td><a href="https://github.com/mct10/RepCodec">https://github.com/mct10/RepCodec</a></td>
325 |     </tr>
326 |     <tr>
327 |       <td>EnCodecMAE</td>
328 |       <td><a href="https://github.com/habla-liaa/encodecmae">https://github.com/habla-liaa/encodecmae</a></td>
329 |     </tr>
330 |     <tr>
331 |       <td>MuCodec</td>
332 |       <td><a href="https://github.com/xuyaoxun/MuCodec">https://github.com/xuyaoxun/MuCodec</a></td>
333 |     </tr>
334 |     <tr>
335 |       <td>SPARC</td>
336 |       <td><a href="https://github.com/Berkeley-Speech-Group/Speech-Articulatory-Coding">https://github.com/Berkeley-Speech-Group/Speech-Articulatory-Coding</a></td>
337 |     </tr>
338 |     <tr>
339 |       <td>BANC</td>
340 |       <td><a href="https://github.com/anton-jeran/MULTI-AUDIODEC">https://github.com/anton-jeran/MULTI-AUDIODEC</a></td>
341 |     </tr>
342 |     <tr>
343 |       <td>SpeechRVQ</td>
344 |       <td><a href="https://huggingface.co/ibm/DAC.speech.v1.0">https://huggingface.co/ibm/DAC.speech.v1.0</a></td>
345 |     </tr>
346 |     <tr>
347 |       <td>QINCo</td>
348 |       <td><a href="https://github.com/facebookresearch/Qinco">https://github.com/facebookresearch/Qinco</a></td>
349 |     </tr>
350 |     <tr>
351 |       <td>SimVQ</td>
352 |       <td><a href="https://github.com/youngsheen/SimVQ">https://github.com/youngsheen/SimVQ</a></td>
353 |     </tr>
354 |   </tbody>
355 | </table>
356 | 
357 | </table>
358 | </div>
359 | 
360 | <div align='center'>
361 | Table 3: A comprehensive list of publicly available codec models and their URLs.
362 | </div>
363 | 
364 | ## **Training Paradigm of Spoken Dialogue Model**
365 | 
366 | In the Training Paradigm of Spoken Dialogue Model section, we focuse on how to adapt text-based large language models (LLMs) into dialogue systems with speech processing capabilities. The **selection and design of training paradigms** have a direct impact on the **performance, real-time performance, and multimodal alignment** of the model.
367 | 
368 | <div align='center'>
369 | <img src='https://cdn.jsdelivr.net/gh/MYJOKERML/imgbed/matebook14/image-20241113172433940.png' />
370 | </div>
371 | 
372 | <div align='center'>
373 | Figure 5: Categorization Diagram of Spoken Dialogue Model Architectural Paradigms (left) and Diagram of Multi-stage Training Steps (right)
374 | </div>
375 | 
376 | <br/>
377 | And we also comprehensively summarize an overview of <b>the Alignment Post-training Methods</b>.
378 | 
379 | <br/>
380 | 
381 | <div align='center'><img src="https://cdn.jsdelivr.net/gh/MYJOKERML/imgbed/matebook14/image-20241114142618698.png" alt="image-20241114142618698" style="zoom:30%;" /> </div>
382 | 
383 | <div align='center'>Figure 6: Alignment Post-training Methods</div>
384 | 
385 | ## Streaming, Duplex, and Interaction
386 | 
387 | The Streaming, Duplex, and Interaction section mainly discusses the implementation of **streaming processing, duplex communication, and interaction capabilities** inspeech dialogue models. These features are crucial for improving the response speed, naturalness, and interactivity of the model in real-time conversations.
388 | 
389 | <div align='center'>
390 | <img src="https://cdn.jsdelivr.net/gh/MYJOKERML/imgbed/matebook14/tu_03.png" alt="tu_03" style="zoom: 33%;" />
391 | 
392 | Figure 7: The Example Diagram of Duplex Interaction
393 | </div>
394 | 
395 | ## Training Resources and Evaluation
396 | 
397 | #### 1. Training resources
398 | 
399 | <div align='center'>
400 | <table>
401 |     <caption>Datasets used in the various training stages</caption>
402 |     <thead>
403 |         <tr>
404 |             <th>Stage</th>
405 |             <th>Task</th>
406 |             <th>Dataset</th>
407 |             <th>Size</th>
408 |             <th>URL</th>
409 |             <th>Modality</th>
410 |         </tr>
411 |     </thead>
412 |     <tbody>
413 |         <tr>
414 |             <td rowspan="14"><b>Modal Alignment</b></td>
415 |             <td>Multilingual TTS</td>
416 |             <td>Emilia</td>
417 |             <td>101k hrs</td>
418 |             <td><a href="https://huggingface.co/datasets/amphion/Emilia-Dataset">Link</a></td>
419 |             <td>Text, Speech</td>
420 |         </tr>
421 |         <tr>     
422 |             <td>Mandarin ASR</td>
423 |             <td>AISHELL-1</td>
424 |             <td>170 hrs</td>
425 |             <td><a href="https://www.openslr.org/33/">Link</a></td>
426 |             <td>Text, Speech</td>
427 |         </tr>
428 |         <tr>
429 |             <td>Mandarin ASR</td>
430 |             <td>AISHELL-2</td>
431 |             <td>1k hrs</td>
432 |             <td><a href="https://github.com/kaldi-asr/kaldi/tree/master/egs/aishell2">Link</a></td>
433 |             <td>Text, Speech</td>
434 |         </tr>
435 |         <tr>
436 |             <td>Mandarin TTS</td>
437 |             <td>AISHELL-3</td>
438 |             <td>85 hrs, 88,035 utt., 218 spk.</td>
439 |             <td><a href="https://www.aishelltech.com/aishell_3">Link</a></td>
440 |             <td>Text, Speech</td>
441 |         </tr>
442 |         <tr>
443 |             <td>TTS</td>
444 |             <td>LibriTTS</td>
445 |             <td>585 hrs</td>
446 |             <td><a href="https://www.openslr.org/60/">Link</a></td>
447 |             <td>Text, Speech</td>
448 |         </tr>
449 |         <tr>
450 |             <td>ASR</td>
451 |             <td>TED-LIUM</td>
452 |             <td>452 hrs</td>
453 |             <td><a href="https://lium.univ-lemans.fr/ted-lium3/">Link</a></td>
454 |             <td>Text, Speech</td>
455 |         </tr>
456 |         <tr>
457 |             <td>ASR</td>
458 |             <td>VoxPopuli</td>
459 |             <td>1.8k hrs</td>
460 |             <td><a href="https://github.com/facebookresearch/voxpopuli">Link</a></td>
461 |             <td>Text, Speech</td>
462 |         </tr>
463 |         <tr>
464 |             <td>ASR</td>
465 |             <td>Librispeech</td>
466 |             <td>1,000 hrs</td>
467 |             <td><a href="https://www.openslr.org/12">Link</a></td>
468 |             <td>Text, Speech</td>
469 |         </tr>
470 |         <tr>
471 |             <td>ASR</td>
472 |             <td>MLS</td>
473 |             <td>44.5k hrs</td>
474 |             <td><a href="https://www.openslr.org/">Link</a></td>
475 |             <td>Text, Speech</td>
476 |         </tr>
477 |         <tr>
478 |             <td>TTS</td>
479 |             <td>Wenetspeech</td>
480 |             <td>22.4k hrs</td>
481 |             <td><a href="https://wenet.org.cn/WenetSpeech/">Link</a></td>
482 |             <td>Text, Speech</td>
483 |         </tr>
484 |         <tr>
485 |             <td>ASR</td>
486 |             <td>Gigaspeech</td>
487 |             <td>40k hrs</td>
488 |             <td><a href="https://github.com/SpeechColab/GigaSpeech">Link</a></td>
489 |             <td>Text, Speech</td>
490 |         </tr>
491 |         <tr>
492 |             <td>ASR</td>
493 |             <td>VCTK</td>
494 |             <td>300 hrs</td>
495 |             <td><a href="https://paperswithcode.com/dataset/voice-bank-demand">Link</a></td>
496 |             <td>Text, Speech</td>
497 |         </tr>
498 |         <tr>
499 |             <td>TTS</td>
500 |             <td>LJSpeech</td>
501 |             <td>24 hrs</td>
502 |             <td><a href="https://keithito.com/LJ-Speech-Dataset/">Link</a></td>
503 |             <td>Text, Speech</td>
504 |         </tr>
505 |         <tr>
506 |             <td>ASR</td>
507 |             <td>Common Voice</td>
508 |             <td>2,500 hrs</td>
509 |             <td><a href="https://commonvoice.mozilla.org/zh-CN">Link</a></td>
510 |             <td>Text, Speech</td>
511 |         </tr>
512 |         <tr>
513 |             <td rowspan="7"><b>Dual-Stream Processing</b></td>
514 |             <td>Instruction</td>
515 |             <td>Alpaca</td>
516 |             <td>52,000 items</td>
517 |             <td><a href="https://huggingface.co/datasets/tatsu-lab/alpaca">Link</a></td>
518 |             <td>Text + TTS</td>
519 |         </tr>
520 |         <tr>
521 |             <td>Instruction</td>
522 |             <td>Moss</td>
523 |             <td>-</td>
524 |             <td><a href="https://huggingface.co/fnlp/moss-moon-003-sft">Link</a></td>
525 |             <td>Text + TTS</td>
526 |         </tr>
527 |         <tr>
528 |             <td>Instruction</td>
529 |             <td>BelleCN</td>
530 |             <td>-</td>
531 |             <td><a href="https://github.com/LianjiaTech/BELLE/tree/main">Link</a></td>
532 |             <td>Text + TTS</td>
533 |         </tr>
534 |         <tr>
535 |             <td>Dialogue</td>
536 |             <td>UltraChat</td>
537 |             <td>1.5 million</td>
538 |             <td><a href="https://github.com/thunlp/UltraChat">Link</a></td>
539 |             <td>Text + TTS</td>
540 |         </tr>
541 |         <tr>
542 |             <td>Instruction</td>
543 |             <td>Open-Orca</td>
544 |             <td>-</td>
545 |             <td><a href="https://huggingface.co/datasets/Open-Orca/OpenOrca">Link</a></td>
546 |             <td>Text + TTS</td>
547 |         </tr>
548 |         <tr>
549 |             <td>Noise</td>
550 |             <td>DNS</td>
551 |             <td>2425 hrs</td>
552 |             <td><a href="https://github.com/microsoft/DNS-Challenge">Link</a></td>
553 |             <td>Noise data</td>
554 |         </tr>
555 |         <tr>
556 |             <td>Noise</td>
557 |             <td>MUSAN</td>
558 |             <td>-</td>
559 |             <td><a href="https://www.openslr.org/17/">Link</a></td>
560 |             <td>Noise data</td>
561 |         </tr>
562 |         <tr>
563 |             <td rowspan="4"><b>Conversation Fine-Tune</b></td>
564 |             <td>Dialogue</td>
565 |             <td>Fisher</td>
566 |             <td>964 hrs</td>
567 |             <td><a href="https://catalog.ldc.upenn.edu/LDC2004T19">Link</a></td>
568 |             <td>Text, Speech</td>
569 |         </tr>
570 |         <tr>
571 |             <td>Dialogue</td>
572 |             <td>GPT-Talker</td>
573 |             <td>-</td>
574 |             <td><a href="https://github.com/AI-S2-Lab/GPT-Talker">Link</a></td>
575 |             <td>Text, Speech</td>
576 |         </tr>
577 |         <tr>
578 |             <td>Instruction</td>
579 |             <td>INSTRUCTS2S-200K</td>
580 |             <td>200k items</td>
581 |             <td><a href="https://github.com/ictnlp/LLaMA-Omni">Link</a></td>
582 |             <td>Text + TTS</td>
583 |         </tr>
584 |         <tr>
585 |             <td>Instruction</td>
586 |             <td>Open Hermes</td>
587 |             <td>900k items</td>
588 |             <td><a href="https://ollama.com/library/openhermes">Link</a></td>
589 |             <td>Text + TTS</td>
590 |         </tr>
591 |     </tbody>
592 | </table>
593 | 
594 | Table 4: Datasets used in the various training stages
595 | </div>
596 | 
597 | <div align='center'>
598 | <table>
599 | <caption>Music and Non-Speech Sound Datasets</caption>
600 | <thead>
601 |     <tr>
602 |     <th>Dataset</th>
603 |     <th>Size</th>
604 |     <th>URL</th>
605 |     <th>Modality</th>
606 |     </tr>
607 | </thead>
608 | <tbody>
609 |     <tr>
610 |     <td>ESC-50</td>
611 |     <td>2,000 clips (5s each)</td>
612 |     <td><a href="https://github.com/karoldvl/ESC-50">Link</a></td>
613 |     <td>Sound</td>
614 |     </tr>
615 |     <tr>
616 |     <td>UrbanSound8K</td>
617 |     <td>8,732 clips (<=4s each)</td>
618 |     <td><a href="https://urbansounddataset.weebly.com/urbansound8k.html">Link</a></td>
619 |     <td>Sound</td>
620 |     </tr>
621 |     <tr>
622 |     <td>AudioSet</td>
623 |     <td>2000k+ clips (10s each)</td>
624 |     <td><a href="https://research.google.com/audioset/">Link</a></td>
625 |     <td>Sound</td>
626 |     </tr>
627 |     <tr>
628 |     <td>TUT Acoustic Scenes 2017</td>
629 |     <td>52,630 segments</td>
630 |     <td><a href="https://zenodo.org/record/400515">Link</a></td>
631 |     <td>Sound</td>
632 |     </tr>
633 |     <tr>
634 |     <td>Warblr</td>
635 |     <td>10,000 clips</td>
636 |     <td><a href="https://warblr.net/">Link</a></td>
637 |     <td>Sound</td>
638 |     </tr>
639 |     <tr>
640 |     <td>FSD50K</td>
641 |     <td>51,197 clips (total 108.3 hours)</td>
642 |     <td><a href="https://zenodo.org/record/4060432">Link</a></td>
643 |     <td>Sound</td>
644 |     </tr>
645 |     <tr>
646 |     <td>DCASE Challenge</td>
647 |     <td>varies annually</td>
648 |     <td><a href="http://dcase.community/">Link</a></td>
649 |     <td>Sound</td>
650 |     </tr>
651 |     <tr>
652 |     <td>IRMAS</td>
653 |     <td>6,705 audio files (3s each)</td>
654 |     <td><a href="https://www.upf.edu/web/mtg/irmas">Link</a></td>
655 |     <td>Music</td>
656 |     </tr>
657 |     <tr>
658 |     <td>FMA</td>
659 |     <td>106,574 tracks</td>
660 |     <td><a href="https://github.com/mdeff/fma">Link</a></td>
661 |     <td>Music</td>
662 |     </tr>
663 |     <tr>
664 |     <td>NSynth</td>
665 |     <td>305,979 notes</td>
666 |     <td><a href="https://magenta.tensorflow.org/datasets/nsynth">Link</a></td>
667 |     <td>Music</td>
668 |     </tr>
669 |     <tr>
670 |     <td>EMOMusic</td>
671 |     <td>744 songs</td>
672 |     <td><a href="https://cvml.unige.ch/databases/emoMusic/">Link</a></td>
673 |     <td>Music</td>
674 |     </tr>
675 |     <tr>
676 |     <td>MedleyDB</td>
677 |     <td>122 multitrack recordings</td>
678 |     <td><a href="https://medleydb.weebly.com/">Link</a></td>
679 |     <td>Music</td>
680 |     </tr>
681 |     <tr>
682 |     <td>MagnaTagATune</td>
683 |     <td>25,863 clips (30s each)</td>
684 |     <td><a href="https://mirg.city.ac.uk/codeapps/the-magnatagatune-dataset">Link</a></td>
685 |     <td>Music</td>
686 |     </tr>
687 |     <tr>
688 |     <td>MUSDB</td>
689 |     <td>150 songs</td>
690 |     <td><a href="https://paperswithcode.com/dataset/musdb18">Link</a></td>
691 |     <td>Music</td>
692 |     </tr>
693 |     <tr>
694 |     <td>M4Singer</td>
695 |     <td>700 songs</td>
696 |     <td><a href="https://github.com/M4Singer/M4Singer">Link</a></td>
697 |     <td>Music</td>
698 |     </tr>
699 |     <tr>
700 |     <td>Jamendo</td>
701 |     <td>600k songs</td>
702 |     <td><a href="https://www.jamendo.com/?language=en">Link</a></td>
703 |     <td>Music</td>
704 |     </tr>
705 | </tbody>
706 | </table>
707 | 
708 | Table 5: Music and Non-Speech Sound Datasets
709 | </div>
710 | 
711 | #### 2. Evaluation
712 | 
713 | Evaluation is a crucial aspect of training and testing spoken dialogue models. In this section, we provide a comprehensive overview of the evaluation from **11 aspects**. The evaluation metrics are categorized into **two main types**: **Basic Evaluation**, and **Advanced Evaluation**.
714 | <div align='center'><img src="https://cdn.jsdelivr.net/gh/MYJOKERML/imgbed/matebook14/image-20241114143903640.png" alt="image-20241114143903640" style="zoom: 25%;" /></div>
715 | 
716 | <div align='center'>
717 | Table 6: This table evaluates model performance across various abilities, common tasks, representative benchmarks, and corresponding metrics.
718 | </div>
719 | 
720 | ## Cite
721 | 
722 | ```bibtex
723 | @article{ji2024wavchat,
724 |   title={WavChat: A Survey of Spoken Dialogue Models},
725 |   author={Ji, Shengpeng and Chen, Yifu and Fang, Minghui and Zuo, Jialong and Lu, Jingyu and Wang, Hanting and Jiang, Ziyue and Zhou, Long and Liu, Shujie and Cheng, Xize and others},
726 |   journal={arXiv preprint arXiv:2411.13577},
727 |   year={2024}
728 | }
729 | ```
730 | 


--------------------------------------------------------------------------------