6 | Official PyTorch code for Deep Audio-Signal Holistic Embeddings
7 | Scaling up masked audio encoder learning for general audio classification
8 |
11 | 
12 | 
13 | 
14 | 
15 | 
35 | 
36 | 
37 | 
60 |
61 | ## K-Nearest Neighbor results
62 |
63 | Performance of features without parameterized training.
64 |
65 | | | ESC50 | FSDKaggle18 | NSynth Instrument | Speech Commands 1 | Speech Commands 2 | US8k | VoxCeleb1 | RAVDESS-Speech | FluentSpeechCommands |
66 | |--------------------------|-------|--------|-------------|-------|-------|-------|-----------|---------|-------|
67 | | [MSM-MAE](https://github.com/nttcslab/msm-mae) | 2 | 2.18 | 20.58 | 3.7 | 1.5 | 11.5 | 0.12 | 6.77 | 1.85 |
68 | | MelSpec | 18.4 | 38.5 | 35.5 | 3.7 | 1.5 | 40.39 | 5.26 | 29.65 | 9.97 |
69 | | [CED-Base](https://github.com/RicherMans/CED) | 95.35 | 85.06 | 74.41 | 79.78 | 62.66 | 87.06 | 7.02 | 52.78 | 16.61 |
70 | | [AudioMAE](https://github.com/facebookresearch/AudioMAE) | 53.05 | 43.38 | 67.21 | 56.87 | 5.9 | 58.18 | 2.9 | 28.68 | 7.59 |
71 | | [WavLM-Large](https://github.com/microsoft/unilm/tree/master/wavlm) | 51.3 | 60.87 | | 96.97 | 92.69 | 58.67 | 28.54 | 51.39 | 83.28 |
72 | | [Wav2vec-large-100k-voxpopuli](https://huggingface.co/facebook/wav2vec2-large-100k-voxpopuli) | 44 | 59.5 | 60.42 | 80.86 | 66.61 | 59.84 | 18.22 | 45.76 | 30.48 |
73 | | Dasheng-Base | 61.9 | 70.31 | 70.02 | 93.55 | 86 | 73.87 | 34.21 | 58.12 | 52.33 |
74 | | Dasheng-0.6B | 66.55 | 72.06 | 70.87 | 93.36 | 87.27 | 75.92 | 37.78 | 61.81 | 57.63 |
75 | | Dasheng-1.2B | 68.55 | 72.06 | 71.19 | 95.9 | 90.9 | 77.71 | 39.39 | 61.94 | 62.38 |
76 |
77 | ## 1. Installation (Recommended for inference)
78 |
79 | Install the package.
80 |
81 | ```bash
82 | python3 -m pip install dasheng
83 | ```
84 |
85 | ### 1.2 Installation for Training
86 |
87 | ```bash
88 | python3 -m pip install dasheng[train]
89 | ```
90 |
91 | ## 2. Usage
92 |
93 | ```python
94 | # The three models of the paper
95 | from dasheng import dasheng_base, dasheng_06B, dasheng_12B
96 |
97 | model = dasheng_base()
98 | ```
99 |
100 | Forward some audio data (note should be 16khz)
101 |
102 | ```python
103 | import torch
104 | model = model.eval()
105 | features = model(torch.randn(1, 16000))
106 | print(features.shape)
107 | ```
108 |
109 | ## 3. Training
110 |
111 | Install dependencies:
112 |
113 | ```bash
114 | python3 -m pip install dasheng[train]
115 | ```
116 |
117 | ### 3.1 Prepare data
118 |
119 | We rely on the excellent [webdataset](https://github.com/webdataset) library for I/O.
120 | Thus one simply needs to pack their data into a bunch of `.tar` files.
121 |
122 | A simple example of such a file would be:
123 |
124 | ```bash
125 | find DIR -type f -name '*flac' | tar -rvf data.tgz -T -
126 | ```
127 |
128 | We also provide a simple script [wavlist_to_tar] that automates this process, which is installed with the package.
129 |
130 | ```bash
131 | wavlist_to_tar your_data.tsv shards/
132 | ```
133 |
134 | Creating `your_data.tsv` is simple:
135 |
136 | ```bash
137 | find data -type f | awk 'BEGIN{print "filename"} {print}' > your_data.tsv
138 | ```
139 |
140 | ### 3.2 Training from source
141 |
142 | To train one should first adjust the config in `dasheng/train/config/*yaml` accordingly, by adding their training data.
143 |
144 | ```bash
145 | python3 dasheng/train/train.py dasheng/train/config/dasheng_base.yaml
146 | ```
147 |
148 | MultiGPU support is realized using [Accelerate](https://huggingface.co/docs/accelerate/index)
149 |
150 | ```bash
151 | accelerate launch --mixed_precision='bf16' dasheng/train/train.py dasheng/train/config/dasheng_base.yaml
152 | ```
153 |
154 | ## FAQ
155 |
156 | ### Is there an Audioset-finetuned Dasheng?
157 |
158 | Yes, the performance for the base model is 49.7 mAP. One can use it as follows:
159 |
160 | ```python
161 | from typing import Any, Mapping
162 | import dasheng
163 | import torch
164 |
165 | class DashengAudiosetClassifier(torch.nn.Module):
166 |
167 | def __init__(self) -> None:
168 | super().__init__()
169 | self.dashengmodel = dasheng.dasheng_base()
170 | self.classifier = torch.nn.Sequential(torch.nn.LayerNorm(self.dashengmodel.embed_dim), torch.nn.Linear(self.dashengmodel.embed_dim, 527))
171 |
172 | def load_state_dict(self, state_dict: Mapping[str, Any], strict: bool = True, assign: bool = False):
173 | self.dashengmodel.load_state_dict(state_dict, strict=False)
174 | for_classifier_dict = {}
175 | for k,v in state_dict.items():
176 | if 'outputlayer' in k:
177 | for_classifier_dict[k.replace('outputlayer.','')] = v
178 | self.classifier.load_state_dict(for_classifier_dict)
179 | return self
180 |
181 | def forward(self, x):
182 | x = self.dashengmodel(x).mean(1)
183 | return self.classifier(x).sigmoid()
184 |
185 |
186 | mdl = DashengAudiosetClassifier()
187 | check = torch.hub.load_state_dict_from_url('https://zenodo.org/records/13315686/files/dasheng_audioset_mAP497.pt?download=1',map_location='cpu')
188 | mdl.load_state_dict(check)
189 |
190 | prediction = mdl(torch.randn(1,16000))
191 | ```
192 |
193 |
194 | ## Citation
195 |
196 | ```bibtex
197 | @inproceedings{dinkel2024dasheng,
198 | title={Scaling up masked audio encoder learning for general audio classification},
199 | author={Dinkel, Heinrich and Yan, Zhiyong and Wang, Yongqing and Zhang, Junbo and Wang, Yujun and Wang, Bin},
200 | booktitle={Interspeech 2024},
201 | year={2024}
202 | }
203 | ```
204 |
--------------------------------------------------------------------------------
/dasheng/__init__.py:
--------------------------------------------------------------------------------
1 | import importlib.metadata
2 | __version__ = importlib.metadata.version("dasheng")
3 |
4 | from .pretrained.pretrained import dasheng_base, dasheng_06B, dasheng_12B
5 |
--------------------------------------------------------------------------------
/dasheng/__pycache__/__init__.cpython-39.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/RicherMans/Dasheng/db7309358edbeea1b1cca37739f442c4139ac8e9/dasheng/__pycache__/__init__.cpython-39.pyc
--------------------------------------------------------------------------------
/dasheng/prepare/wavlist_to_tar.py:
--------------------------------------------------------------------------------
1 | from functools import partial
2 | from typing import Any, Dict, Iterable
3 | import json
4 | from pathlib import Path
5 | import pandas as pd
6 | import argparse
7 | import multiprocessing
8 | from webdataset import TarWriter
9 | from tqdm import tqdm
10 |
11 |
12 | def proxy_read(data: Dict, filename_column: str):
13 | filename = data.pop(filename_column)
14 | with open(filename, 'rb') as buf:
15 | raw_data = buf.read()
16 | fpath = Path(filename)
17 | stem_name = str(fpath.stem).replace('.', '_')
18 | suffix = fpath.suffix.replace('.', '')
19 | ret_data = {
20 | suffix: raw_data,
21 | '__key__': f"{stem_name}", # Just cast to str
22 | }
23 | # If we have some labels, also dump a .json file
24 | if len(data) > 0:
25 | ret_data['json'] = json.dumps(data).encode('utf-8')
26 | return ret_data
27 |
28 |
29 | def main():
30 | parser = argparse.ArgumentParser()
31 | parser.add_argument(
32 | 'input_filelist',
33 | type=Path,
34 | help=
35 | "Some input filelist. We will expect a column named