├── README.md
├── assets
├── demo
│ ├── ff
│ │ ├── 6_444o0314.wav
│ │ ├── 6_None1_clean.png
│ │ ├── 6_None1_clean.wav
│ │ ├── 6_None1_noisy.png
│ │ ├── 6_None1_noisy.wav
│ │ ├── 6_None1_pre.png
│ │ ├── 6_None1_pre.wav
│ │ ├── 6_ref_444c020v.wav
│ │ ├── 7_053o020e.wav
│ │ ├── 7_None1_clean.png
│ │ ├── 7_None1_clean.wav
│ │ ├── 7_None1_noisy.png
│ │ ├── 7_None1_noisy.wav
│ │ ├── 7_None1_pre.png
│ │ ├── 7_None1_pre.wav
│ │ └── 7_ref_053o020r.wav
│ ├── fm_mf
│ │ ├── 0_445c0206.wav
│ │ ├── 0_None1_clean.png
│ │ ├── 0_None1_clean.wav
│ │ ├── 0_None1_noisy.png
│ │ ├── 0_None1_noisy.wav
│ │ ├── 0_None1_pre.png
│ │ ├── 0_None1_pre.wav
│ │ ├── 0_ref_445o030h.wav
│ │ ├── 1_22gc0105.wav
│ │ ├── 1_None1_clean.png
│ │ ├── 1_None1_clean.wav
│ │ ├── 1_None1_noisy.png
│ │ ├── 1_None1_noisy.wav
│ │ ├── 1_None1_pre.png
│ │ ├── 1_None1_pre.wav
│ │ └── 1_ref_22gc010r.wav
│ └── mm
│ │ ├── 4_22go0107.wav
│ │ ├── 4_None1_clean.png
│ │ ├── 4_None1_clean.wav
│ │ ├── 4_None1_noisy.png
│ │ ├── 4_None1_noisy.wav
│ │ ├── 4_None1_pre.png
│ │ ├── 4_None1_pre.wav
│ │ ├── 4_ref_22ga010b.wav
│ │ ├── 5_051c010u.wav
│ │ ├── 5_None1_clean.png
│ │ ├── 5_None1_clean.wav
│ │ ├── 5_None1_noisy.png
│ │ ├── 5_None1_noisy.wav
│ │ ├── 5_None1_pre.png
│ │ ├── 5_None1_pre.wav
│ │ └── 5_ref_051a050n.wav
├── framework.png
├── model.png
└── onset_offset_demo.png
└── data
└── wsj
├── mix_2_spk_voiceP_tt_WSJ.txt
├── spk_test_WSJ.lst
└── spk_train_WSJ.lst
/README.md:
--------------------------------------------------------------------------------
1 | # WASE
2 |
3 | ## Overview
4 |
5 | Demo samples of our paper *Wase: Learning When to Attend for Speaker Extraction in Cocktail Party Environments*. WASE first explicitly models **start/end time of speech** (**onset/offset cues**) in speaker extraction problem.
6 |
7 | ~~We will release code soon.~~ The source code is available in [WASE_202112](https://github.com/aispeech-lab/WASE_202112). If you have any question about implementation details, feel free to ask me (haoyunzhe2017@ia.ac.cn).
8 |
9 | 
10 |
11 | ## Model
12 |
13 | WASE is adapted based on our previous proposed [framework](http://www.interspeech2020.org/uploadfile/pdf/Mon-3-11-6.pdf), which includes five modules: voiceprint encoder, onset/offset detector, speech encoder, speech decoder, and speaker extraction module.
14 |
15 | In this work, we focus on the onset/offset cues of speech and verify their effectiveness in speaker extraction task. We also combine the onset/offset cues and voiceprint cue. Onset/offset
16 | cues model start/end time of speech and voiceprint cue models the
17 | voice characteristics. The combination of two perceptual cues bring a significant performance improvement, while extra needed parameters are negligible. Please see the figure below for detailed model structure.
18 |
19 | 
20 | ***************************************************************
21 |
22 | ## Datasets
23 | * [WSJ0](https://catalog.ldc.upenn.edu/LDC93S6A)
24 |
25 | The training samples are generated by randomly selecting speeches of different speakers from *si_tr_s* of WSJ0, and mixing them at various signal-to-noise ratios (SNR). The evaluating samples are generated by fixed list *./data/wsj/mix_2_spk_voiceP_tt_WSJ.txt*.
26 | You may need the command below to modify the evaluating data path.
27 | ```bash
28 | sed -i 's/home\/aa/YOUR PATH/g' data/wsj/mix_2_spk_voiceP_tt_WSJ.txt
29 | ```
30 |
31 | ## Result
32 |
33 | ### Audio Sample
34 |
35 |
36 | - Listen to audio samples at [*./assets/demo*](./assets/demo).
37 | - Spectrogram samples (clean/mixture/prediction).
38 |
39 |
40 |
41 |
42 |
43 |
44 |
45 |
50 |
51 |