├── LICENSE ├── README.md ├── __init__.py ├── feature.py └── ivector.py /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # KaldiFeat 2 | 3 | KaldiFeat is a light-weight Python library for computing Kaldi-style acoustic features based on NumPy. It might be helpful if you want to: 4 | 5 | - Test a pre-trained model on new data without writing shell commands and creating a bunch of files. 6 | - Run a pre-trained model in a new environment without installing Kaldi. 7 | 8 | ## Example 9 | 10 | The following codes calculate MFCCs with the same settings in `kaldi/egs/voxceleb/v2` 11 | 12 | ``` 13 | import librosa 14 | 15 | from kaldifeat import compute_mfcc_feats, compute_vad, apply_cmvn_sliding 16 | 17 | # Assume we have a wav file called example.wav whose sample rate is 16000 Hz 18 | data, _ = librosa.load('example.wav', 16000) 19 | 20 | # We adopt 16 bits data, thus we need to transform dtype from float to int16 for librosa 21 | data = (data * 32768).astype(np.int16) 22 | 23 | raw_mfcc = compute_mfcc_feats(data, sample_frequency=16000, frame_length=25, frame_shift=10, low_freq=20, high_freq=-400, num_mel_bins=30, num_ceps=30, snip_edges=False) 24 | log_energy = raw_mfcc[:, 0] 25 | vad = compute_vad(log_energy, energy_threshold=5.5, energy_mean_scale=0.5, frames_context=2, proportion_threshold=0.12) 26 | mfcc = apply_cmvn_sliding(raw_mfcc, window=300, center=True)[vad] 27 | ``` 28 | 29 | ## Supported Functions 30 | 31 | ### compute_fbank_feats 32 | 33 | Compute (log) Mel filter bank energies (FBanks) in the same way as `kaldi/src/featbin/compute_fbank_feats` 34 | 35 | | Parameters | Description | 36 | | :--------- | :---------- | 37 | |blackman_coeff| Constant coefficient for generalized Blackman window. (float, default = 0.42)| 38 | |dither| Dithering constant (0.0 means no dither). If you turn this off, you should set the --energy-floor option, e.g. to 1.0 or 0.1 (float, default = 1)| 39 | |energy_floor| Floor on energy (absolute, not relative) in FBANK computation. Only makes a difference if --use-energy=true; only necessary if --dither=0.0. Suggested values: 0.1 or 1.0 (float, default = 0)| 40 | |frame_length| Frame length in milliseconds (float, default = 25)| 41 | |frame_shift| Frame shift in milliseconds (float, default = 10)| 42 | |high_freq| High cutoff frequency for mel bins (if <= 0, offset from Nyquist) (float, default = 0)| 43 | |low_freq| Low cutoff frequency for mel bins (float, default = 20)| 44 | |num_mel_bins| Number of triangular mel-frequency bins (int, default = 23)| 45 | |preemphasis_coefficient| Coefficient for use in signal preemphasis (float, default = 0.97)| 46 | |raw_energy| If true, compute energy before preemphasis and windowing (bool, default = true)| 47 | |remove_dc_offset| Subtract mean from waveform on each frame (bool, default = true)| 48 | |round_to_power_of_two| If true, round window size to power of two by zero-padding input to FFT. (bool, default = true)| 49 | |sample_frequency| Waveform data sample frequency (must match the waveform file, if specified there) (float, default = 16000)| 50 | |snip_edges| If true, end effects will be handled by outputting only frames that completely fit in the file, and the number of frames depends on the frame-length. If false, the number of frames depends only on the frame-shift, and we reflect the data at the ends. (bool, default = true)| 51 | |use_energy| Add an extra energy output. (bool, default = false)| 52 | |use_log_fbank| If true, produce log-filterbank, else produce linear. (bool, default = true)| 53 | |use_power| If true, use power, else use magnitude. (bool, default = true)| 54 | |window_type| Type of window ("hamming"\|"hanning"\|"povey"\|"rectangular"\|"sine"\|"blackmann") (string, default = "povey")| 55 | |dtype| Type of array (np.float32\|np.float64) (dtype or string, default=np.float32)| 56 | 57 | ### compute_mfcc_feats 58 | 59 | Compute Mel-frequency cepstral coefficients (MFCCs) in the same way as `kaldi/src/featbin/compute_mfcc_feats` 60 | 61 | | Parameters | Description | 62 | | :--------- | :---------- | 63 | |blackman_coeff| Constant coefficient for generalized Blackman window. (float, default = 0.42)| 64 | |cepstral_lifter| Constant that controls scaling of MFCCs (float, default = 22)| 65 | |dither| Dithering constant (0.0 means no dither). If you turn this off, you should set the --energy-floor option, e.g. to 1.0 or 0.1 (float, default = 1)| 66 | |energy_floor| Floor on energy (absolute, not relative) in MFCC computation. Only makes a difference if --use-energy=true; only necessary if --dither=0.0. Suggested values: 0.1 or 1.0 (float, default = 0)| 67 | |frame_length| Frame length in milliseconds (float, default = 25)| 68 | |frame_shift| Frame shift in milliseconds (float, default = 10)| 69 | |high_freq| High cutoff frequency for mel bins (if <= 0, offset from Nyquist) (float, default = 0)| 70 | |low_freq| Low cutoff frequency for mel bins (float, default = 20)| 71 | |num_ceps| Number of cepstra in MFCC computation (including C0) (int, default = 13)| 72 | |num_mel_bins| Number of triangular mel-frequency bins (int, default = 23)| 73 | |preemphasis_coefficient| Coefficient for use in signal preemphasis (float, default = 0.97)| 74 | |raw_energy| If true, compute energy before preemphasis and windowing (bool, default = true)| 75 | |remove_dc_offset| Subtract mean from waveform on each frame (bool, default = true)| 76 | |round_to_power_of_two| If true, round window size to power of two by zero-padding input to FFT. (bool, default = true)| 77 | |sample_frequency| Waveform data sample frequency (must match the waveform file, if specified there) (float, default = 16000)| 78 | |snip_edges| If true, end effects will be handled by outputting only frames that completely fit in the file, and the number of frames depends on the frame-length. If false, the number of frames depends only on the frame-shift, and we reflect the data at the ends. (bool, default = true)| 79 | |use_energy| Use energy (not C0) in MFCC computation (bool, default = true)| 80 | |window_type| Type of window ("hamming"\|"hanning"\|"povey"\|"rectangular"\|"sine"\|"blackmann") (string, default = "povey")| 81 | |dtype| Type of array (np.float32\|np.float64) (dtype or string, default=np.float32)| 82 | 83 | ### apply_cmvn_sliding 84 | 85 | Apply sliding-window cepstral mean (and optionally variance) normalization in the same way as `kaldi/src/featbin/apply_cmvn_sliding` 86 | 87 | | Parameters | Description | 88 | | :--------- | :---------- | 89 | |center| If true, use a window centered on the current frame (to the extent possible, modulo end effects). If false, window is to the left. (bool, default = false)| 90 | |window| Window in frames for running average CMN computation (int, default = 600)| 91 | |min_window| Minimum CMN window used at start of decoding (adds latency only at start). Only applicable if center == false, ignored if center==true (int, default = 100)| 92 | |norm_vars| If true, normalize variance to one. (bool, default = false)| 93 | 94 | ### compute_vad 95 | 96 | Apply energy-based voice activity detection in the same way as `kaldi/src/ivectorbin/compute_vad` 97 | 98 | | Parameters | Description | 99 | | :--------- | :---------- | 100 | |energy_mean_scale| If this is set to s, to get the actual threshold we let m be the mean log-energy of the file, and use s\*m + vad-energy-threshold (float, default = 0.5)| 101 | |energy_threshold| Constant term in energy threshold for VAD (also see energy_mean_scale) (float, default = 5)| 102 | |frames_context| Number of frames of context on each side of central frame, in window for which energy is monitored (int, default = 0)| 103 | |proportion_threshold| Parameter controlling the proportion of frames within the window that need to have more energy than the threshold (float, default = 0.6)| 104 | 105 | ### Related Projects 106 | 107 | - [python_speech_features](https://github.com/jameslyons/python_speech_features) 108 | - [python_kaldi_features](https://github.com/ZitengWang/python_kaldi_features) 109 | -------------------------------------------------------------------------------- /__init__.py: -------------------------------------------------------------------------------- 1 | from .feature import compute_fbank_feats, compute_mfcc_feats, apply_cmvn_sliding 2 | from .ivector import compute_vad 3 | -------------------------------------------------------------------------------- /feature.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from scipy.fftpack import dct 3 | 4 | 5 | # ---------- feature-window ---------- 6 | 7 | def sliding_window(x, window_size, window_shift): 8 | shape = x.shape[:-1] + (x.shape[-1] - window_size + 1, window_size) 9 | strides = x.strides + (x.strides[-1],) 10 | return np.lib.stride_tricks.as_strided(x, shape=shape, strides=strides)[::window_shift] 11 | 12 | 13 | def func_num_frames(num_samples, window_size, window_shift, snip_edges): 14 | if snip_edges: 15 | if num_samples < window_size: 16 | return 0 17 | else: 18 | return 1 + ((num_samples - window_size) // window_shift) 19 | else: 20 | return (num_samples + (window_shift // 2)) // window_shift 21 | 22 | 23 | def func_dither(waveform, dither_value): 24 | if dither_value == 0.0: 25 | return waveform 26 | waveform += np.random.normal(size=waveform.shape).astype(waveform.dtype) * dither_value 27 | return waveform 28 | 29 | 30 | def func_remove_dc_offset(waveform): 31 | return waveform - np.mean(waveform) 32 | 33 | 34 | def func_log_energy(waveform): 35 | return np.log(np.dot(waveform, waveform).clip(min=np.finfo(waveform.dtype).eps)) 36 | 37 | 38 | def func_preemphasis(waveform, preemph_coeff): 39 | if preemph_coeff == 0.0: 40 | return waveform 41 | assert 0 < preemph_coeff <= 1 42 | waveform[1:] -= preemph_coeff * waveform[:-1] 43 | waveform[0] -= preemph_coeff * waveform[0] 44 | return waveform 45 | 46 | 47 | def sine(M): 48 | if M < 1: 49 | return np.array([]) 50 | if M == 1: 51 | return np.ones(1, float) 52 | n = np.arange(0, M) 53 | return np.sin(np.pi*n/(M-1)) 54 | 55 | 56 | def povey(M): 57 | if M < 1: 58 | return np.array([]) 59 | if M == 1: 60 | return np.ones(1, float) 61 | n = np.arange(0, M) 62 | return (0.5 - 0.5*np.cos(2.0*np.pi*n/(M-1)))**0.85 63 | 64 | 65 | def feature_window_function(window_type, window_size, blackman_coeff): 66 | assert window_size > 0 67 | if window_type == 'hanning': 68 | return np.hanning(window_size) 69 | elif window_type == 'sine': 70 | return sine(window_size) 71 | elif window_type == 'hamming': 72 | return np.hamming(window_size) 73 | elif window_type == 'povey': 74 | return povey(window_size) 75 | elif window_type == 'rectangular': 76 | return np.ones(window_size) 77 | elif window_type == 'blackman': 78 | window_func = np.blackman(window_size) 79 | if blackman_coeff == 0.42: 80 | return window_func 81 | else: 82 | return window_func - 0.42 + blackman_coeff 83 | else: 84 | raise ValueError('Invalid window type {}'.format(window_type)) 85 | 86 | 87 | def process_window(window, dither, remove_dc_offset, preemphasis_coefficient, window_function, raw_energy): 88 | if dither != 0.0: 89 | window = func_dither(window, dither) 90 | if remove_dc_offset: 91 | window = func_remove_dc_offset(window) 92 | if raw_energy: 93 | log_energy = func_log_energy(window) 94 | if preemphasis_coefficient != 0.0: 95 | window = func_preemphasis(window, preemphasis_coefficient) 96 | window *= window_function 97 | if not raw_energy: 98 | log_energy = func_log_energy(window) 99 | return window, log_energy 100 | 101 | 102 | def extract_window(waveform, blackman_coeff, dither, window_size, window_shift, 103 | preemphasis_coefficient, raw_energy, remove_dc_offset, 104 | snip_edges, window_type, dtype): 105 | num_samples = len(waveform) 106 | num_frames = func_num_frames(num_samples, window_size, window_shift, snip_edges) 107 | num_samples_ = (num_frames - 1) * window_shift + window_size 108 | if snip_edges: 109 | waveform = waveform[:num_samples_] 110 | else: 111 | offset = window_shift // 2 - window_size // 2 112 | waveform = np.concatenate([ 113 | waveform[-offset - 1::-1], 114 | waveform, 115 | waveform[:-(offset + num_samples_ - num_samples + 1):-1] 116 | ]) 117 | frames = sliding_window(waveform, window_size=window_size, window_shift=window_shift) 118 | frames = frames.astype(dtype) 119 | log_enery = np.empty(frames.shape[0], dtype=dtype) 120 | for i in range(frames.shape[0]): 121 | frames[i], log_enery[i] = process_window( 122 | window=frames[i], 123 | dither=dither, 124 | remove_dc_offset=remove_dc_offset, 125 | preemphasis_coefficient=preemphasis_coefficient, 126 | window_function=feature_window_function( 127 | window_type=window_type, 128 | window_size=window_size, 129 | blackman_coeff=blackman_coeff 130 | ).astype(dtype), 131 | raw_energy=raw_energy 132 | ) 133 | return frames, log_enery 134 | 135 | # ---------- feature-window ---------- 136 | 137 | 138 | # ---------- feature-functions ---------- 139 | 140 | def compute_spectrum(frames, n): 141 | complex_spec = np.fft.rfft(frames, n) 142 | return np.absolute(complex_spec) 143 | 144 | 145 | def compute_power_spectrum(frames, n): 146 | return np.square(compute_spectrum(frames, n)) 147 | 148 | 149 | def apply_cmvn_sliding_internal(feat, center=False, window=600, min_window=100, norm_vars=False): 150 | num_frames, feat_dim = feat.shape 151 | std = 1 152 | if center: 153 | if num_frames <= window: 154 | mean = feat.mean(axis=0, keepdims=True).repeat(num_frames, axis=0) 155 | if norm_vars: 156 | std = feat.std(axis=0, keepdims=True).repeat(num_frames, axis=0) 157 | else: 158 | feat1 = feat[:window] 159 | feat2 = sliding_window(feat.T, window, 1) 160 | feat3 = feat[-window:] 161 | mean1 = feat1.mean(axis=0, keepdims=True).repeat(window // 2, axis=0) 162 | mean2 = feat2.mean(axis=2).T 163 | mean3 = feat3.mean(axis=0, keepdims=True).repeat((window - 1) // 2, axis=0) 164 | mean = np.concatenate([mean1, mean2, mean3]) 165 | if norm_vars: 166 | std1 = feat1.std(axis=0, keepdims=True).repeat(window // 2, axis=0) 167 | std2 = feat2.std(axis=2).T 168 | std3 = feat3.mean(axis=0, keepdims=True).repeat((window - 1) // 2, axis=0) 169 | std = np.concatenate([std1, std2, std3]) 170 | else: 171 | if num_frames <= min_window: 172 | mean = feat.mean(axis=0, keepdims=True).repeat(num_frames, axis=0) 173 | if norm_vars: 174 | std = feat.std(axis=0, keepdims=True).repeat(num_frames, axis=0) 175 | else: 176 | feat1 = feat[:min_window] 177 | mean1 = feat1.mean(axis=0, keepdims=True).repeat(min_window, axis=0) 178 | feat2_cumsum = np.cumsum(feat[:window], axis=0)[min_window:] 179 | cumcnt = np.arange(min_window + 1, min(window, num_frames) + 1, dtype=feat.dtype)[:, np.newaxis] 180 | mean2 = feat2_cumsum / cumcnt 181 | mean = np.concatenate([mean1, mean2]) 182 | if norm_vars: 183 | std1 = feat1.std(axis=0, keepdims=True).repeat(min_window, axis=0) 184 | feat2_power_cumsum = np.cumsum(np.square(feat[:window]), axis=0)[min_window:] 185 | std2 = np.sqrt(feat2_power_cumsum / cumcnt - np.square(mean2)) 186 | std = np.concatenate([std1, std2]) 187 | if num_frames > window: 188 | feat3 = sliding_window(feat.T, window, 1) 189 | mean3 = feat3.mean(axis=2).T 190 | mean = np.concatenate([mean, mean3[1:]]) 191 | if norm_vars: 192 | std3 = feat3.std(axis=2).T 193 | std = np.concatenate([std, std3[1:]]) 194 | feat = (feat - mean) / std 195 | return feat 196 | 197 | # ---------- feature-functions ---------- 198 | 199 | 200 | # ---------- mel-computations ---------- 201 | 202 | def inverse_mel_scale(mel_freq): 203 | return 700.0 * (np.exp(mel_freq / 1127.0) - 1.0) 204 | 205 | 206 | def mel_scale(freq): 207 | return 1127.0 * np.log(1.0 + freq / 700.0) 208 | 209 | 210 | def compute_mel_banks(num_bins, sample_frequency, low_freq, high_freq, n): 211 | """ Compute Mel banks. 212 | 213 | :param num_bins: Number of triangular mel-frequency bins 214 | :param sample_frequency: Waveform data sample frequency 215 | :param low_freq: Low cutoff frequency for mel bins 216 | :param high_freq: High cutoff frequency for mel bins (if <= 0, offset from Nyquist) 217 | :param n: Window size 218 | :return: Mel banks. 219 | """ 220 | assert num_bins >= 3, 'Must have at least 3 mel bins' 221 | num_fft_bins = n // 2 222 | 223 | nyquist = 0.5 * sample_frequency 224 | if high_freq <= 0: 225 | high_freq = nyquist + high_freq 226 | assert 0 <= low_freq < high_freq <= nyquist 227 | 228 | fft_bin_width = sample_frequency / n 229 | 230 | mel_low_freq = mel_scale(low_freq) 231 | mel_high_freq = mel_scale(high_freq) 232 | mel_freq_delta = (mel_high_freq - mel_low_freq) / (num_bins + 1) 233 | 234 | mel_banks = np.zeros([num_bins, num_fft_bins + 1]) 235 | for i in range(num_bins): 236 | left_mel = mel_low_freq + mel_freq_delta * i 237 | center_mel = left_mel + mel_freq_delta 238 | right_mel = center_mel + mel_freq_delta 239 | for j in range(num_fft_bins): 240 | mel = mel_scale(fft_bin_width * j) 241 | if left_mel < mel < right_mel: 242 | if mel <= center_mel: 243 | mel_banks[i, j] = (mel - left_mel) / (center_mel - left_mel) 244 | else: 245 | mel_banks[i, j] = (right_mel - mel) / (right_mel - center_mel) 246 | return mel_banks 247 | 248 | 249 | def compute_lifter_coeffs(q, M): 250 | """ Compute liftering coefficients (scaling on cepstral coeffs) 251 | the zeroth index is C0, which is not affected. 252 | 253 | :param q: Number of lifters 254 | :param M: Number of coefficients 255 | :return: Lifters. 256 | """ 257 | if M < 1: 258 | return np.array([]) 259 | if M == 1: 260 | return np.ones(1, float) 261 | n = np.arange(0, M) 262 | return 1 + 0.5*np.sin(np.pi*n/q)*q 263 | 264 | # ---------- mel-computations ---------- 265 | 266 | 267 | # ---------- compute-fbank-feats ---------- 268 | 269 | def compute_fbank_feats( 270 | waveform, 271 | blackman_coeff=0.42, 272 | dither=1.0, 273 | energy_floor=0.0, 274 | frame_length=25, 275 | frame_shift=10, 276 | high_freq=0, 277 | low_freq=20, 278 | num_mel_bins=23, 279 | preemphasis_coefficient=0.97, 280 | raw_energy=True, 281 | remove_dc_offset=True, 282 | round_to_power_of_two=True, 283 | sample_frequency=16000, 284 | snip_edges=True, 285 | use_energy=False, 286 | use_log_fbank=True, 287 | use_power=True, 288 | window_type='povey', 289 | dtype=np.float32): 290 | """ Compute (log) Mel filter bank energies 291 | 292 | :param waveform: Input waveform. 293 | :param blackman_coeff: Constant coefficient for generalized Blackman window. (float, default = 0.42) 294 | :param dither: Dithering constant (0.0 means no dither). If you turn this off, you should set the --energy-floor option, e.g. to 1.0 or 0.1 (float, default = 1) 295 | :param energy_floor: Floor on energy (absolute, not relative) in FBANK computation. Only makes a difference if --use-energy=true; only necessary if --dither=0.0. Suggested values: 0.1 or 1.0 (float, default = 0) 296 | :param frame_length: Frame length in milliseconds (float, default = 25) 297 | :param frame_shift: Frame shift in milliseconds (float, default = 10) 298 | :param high_freq: High cutoff frequency for mel bins (if <= 0, offset from Nyquist) (float, default = 0) 299 | :param low_freq: Low cutoff frequency for mel bins (float, default = 20) 300 | :param num_mel_bins: Number of triangular mel-frequency bins (int, default = 23) 301 | :param preemphasis_coefficient: Coefficient for use in signal preemphasis (float, default = 0.97) 302 | :param raw_energy: If true, compute energy before preemphasis and windowing (bool, default = true) 303 | :param remove_dc_offset: Subtract mean from waveform on each frame (bool, default = true) 304 | :param round_to_power_of_two: If true, round window size to power of two by zero-padding input to FFT. (bool, default = true) 305 | :param sample_frequency: Waveform data sample frequency (must match the waveform file, if specified there) (float, default = 16000) 306 | :param snip_edges: If true, end effects will be handled by outputting only frames that completely fit in the file, and the number of frames depends on the frame-length. If false, the number of frames depends only on the frame-shift, and we reflect the data at the ends. (bool, default = true) 307 | :param use_energy: Add an extra energy output. (bool, default = false) 308 | :param use_log_fbank: If true, produce log-filterbank, else produce linear. (bool, default = true) 309 | :param use_power: If true, use power, else use magnitude. (bool, default = true) 310 | :param window_type: Type of window ("hamming"|"hanning"|"povey"|"rectangular"|"sine"|"blackmann") (string, default = "povey") 311 | :param dtype: Type of array (np.float32|np.float64) (dtype or string, default=np.float32) 312 | :return: (Log) Mel filter bank energies. 313 | """ 314 | window_size = int(frame_length * sample_frequency * 0.001) 315 | window_shift = int(frame_shift * sample_frequency * 0.001) 316 | frames, log_energy = extract_window( 317 | waveform=waveform, 318 | blackman_coeff=blackman_coeff, 319 | dither=dither, 320 | window_size=window_size, 321 | window_shift=window_shift, 322 | preemphasis_coefficient=preemphasis_coefficient, 323 | raw_energy=raw_energy, 324 | remove_dc_offset=remove_dc_offset, 325 | snip_edges=snip_edges, 326 | window_type=window_type, 327 | dtype=dtype 328 | ) 329 | if round_to_power_of_two: 330 | n = 1 331 | while n < window_size: 332 | n *= 2 333 | else: 334 | n = window_size 335 | if use_power: 336 | spectrum = compute_power_spectrum(frames, n) 337 | else: 338 | spectrum = compute_spectrum(frames, n) 339 | mel_banks = compute_mel_banks( 340 | num_bins=num_mel_bins, 341 | sample_frequency=sample_frequency, 342 | low_freq=low_freq, 343 | high_freq=high_freq, 344 | n=n 345 | ).astype(dtype) 346 | feat = np.dot(spectrum, mel_banks.T) 347 | if use_log_fbank: 348 | feat = np.log(feat.clip(min=np.finfo(dtype).eps)) 349 | if use_energy: 350 | if energy_floor > 0.0: 351 | log_energy.clip(min=np.math.log(energy_floor)) 352 | return feat, log_energy 353 | return feat 354 | 355 | # ---------- compute-fbank-feats ---------- 356 | 357 | 358 | # ---------- compute-mfcc-feats ---------- 359 | 360 | def compute_mfcc_feats( 361 | waveform, 362 | blackman_coeff=0.42, 363 | cepstral_lifter=22, 364 | dither=1.0, 365 | energy_floor=0.0, 366 | frame_length=25, 367 | frame_shift=10, 368 | high_freq=0, 369 | low_freq=20, 370 | num_ceps=13, 371 | num_mel_bins=23, 372 | preemphasis_coefficient=0.97, 373 | raw_energy=True, 374 | remove_dc_offset=True, 375 | round_to_power_of_two=True, 376 | sample_frequency=16000, 377 | snip_edges=True, 378 | use_energy=True, 379 | window_type='povey', 380 | dtype=np.float32): 381 | """ Compute mel-frequency cepstral coefficients 382 | 383 | :param waveform: Input waveform. 384 | :param blackman_coeff: Constant coefficient for generalized Blackman window. (float, default = 0.42) 385 | :param cepstral_lifter: Constant that controls scaling of MFCCs (float, default = 22) 386 | :param dither: Dithering constant (0.0 means no dither). If you turn this off, you should set the --energy-floor option, e.g. to 1.0 or 0.1 (float, default = 1) 387 | :param energy_floor: Floor on energy (absolute, not relative) in MFCC computation. Only makes a difference if --use-energy=true; only necessary if --dither=0.0. Suggested values: 0.1 or 1.0 (float, default = 0) 388 | :param frame_length: Frame length in milliseconds (float, default = 25) 389 | :param frame_shift: Frame shift in milliseconds (float, default = 10) 390 | :param high_freq: High cutoff frequency for mel bins (if <= 0, offset from Nyquist) (float, default = 0) 391 | :param low_freq: Low cutoff frequency for mel bins (float, default = 20) 392 | :param num_ceps: Number of cepstra in MFCC computation (including C0) (int, default = 13) 393 | :param num_mel_bins: Number of triangular mel-frequency bins (int, default = 23) 394 | :param preemphasis_coefficient: Coefficient for use in signal preemphasis (float, default = 0.97) 395 | :param raw_energy: If true, compute energy before preemphasis and windowing (bool, default = true) 396 | :param remove_dc_offset: Subtract mean from waveform on each frame (bool, default = true) 397 | :param round_to_power_of_two: If true, round window size to power of two by zero-padding input to FFT. (bool, default = true) 398 | :param sample_frequency: Waveform data sample frequency (must match the waveform file, if specified there) (float, default = 16000) 399 | :param snip_edges: If true, end effects will be handled by outputting only frames that completely fit in the file, and the number of frames depends on the frame-length. If false, the number of frames depends only on the frame-shift, and we reflect the data at the ends. (bool, default = true) 400 | :param use_energy: Use energy (not C0) in MFCC computation (bool, default = true) 401 | :param window_type: Type of window ("hamming"|"hanning"|"povey"|"rectangular"|"sine"|"blackmann") (string, default = "povey") 402 | :param dtype: Type of array (np.float32|np.float64) (dtype or string, default=np.float32) 403 | :return: Mel-frequency cespstral coefficients. 404 | """ 405 | feat, log_energy = compute_fbank_feats( 406 | waveform=waveform, 407 | blackman_coeff=blackman_coeff, 408 | dither=dither, 409 | energy_floor=energy_floor, 410 | frame_length=frame_length, 411 | frame_shift=frame_shift, 412 | high_freq=high_freq, 413 | low_freq=low_freq, 414 | num_mel_bins=num_mel_bins, 415 | preemphasis_coefficient=preemphasis_coefficient, 416 | raw_energy=raw_energy, 417 | remove_dc_offset=remove_dc_offset, 418 | round_to_power_of_two=round_to_power_of_two, 419 | sample_frequency=sample_frequency, 420 | snip_edges=snip_edges, 421 | use_energy=use_energy, 422 | use_log_fbank=True, 423 | use_power=True, 424 | window_type=window_type, 425 | dtype=dtype 426 | ) 427 | feat = dct(feat, type=2, axis=1, norm='ortho')[:, :num_ceps] 428 | lifter_coeffs = compute_lifter_coeffs(cepstral_lifter, num_ceps).astype(dtype) 429 | feat = feat * lifter_coeffs 430 | if use_energy: 431 | feat[:, 0] = log_energy 432 | return feat 433 | 434 | # ---------- compute-mfcc-feats ---------- 435 | 436 | 437 | # ---------- apply-cmvn-sliding ---------- 438 | 439 | def apply_cmvn_sliding(feat, center=False, window=600, min_window=100, norm_vars=False): 440 | """ Apply sliding-window cepstral mean (and optionally variance) normalization 441 | 442 | :param feat: Cepstrum. 443 | :param center: If true, use a window centered on the current frame (to the extent possible, modulo end effects). If false, window is to the left. (bool, default = false) 444 | :param window: Window in frames for running average CMN computation (int, default = 600) 445 | :param min_window: Minimum CMN window used at start of decoding (adds latency only at start). Only applicable if center == false, ignored if center==true (int, default = 100) 446 | :param norm_vars: If true, normalize variance to one. (bool, default = false) 447 | :return: Normalized cepstrum. 448 | """ 449 | # double-precision 450 | feat = apply_cmvn_sliding_internal( 451 | feat=feat.astype(np.float64), 452 | center=center, 453 | window=window, 454 | min_window=min_window, 455 | norm_vars=norm_vars 456 | ).astype(feat.dtype) 457 | return feat 458 | 459 | # ---------- apply-cmvn-sliding ---------- 460 | -------------------------------------------------------------------------------- /ivector.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | from .feature import sliding_window 4 | 5 | 6 | # ---------- compute-vad ---------- 7 | 8 | def compute_vad(log_energy, energy_mean_scale=0.5, energy_threshold=0.5, frames_context=0, proportion_threshold=0.6): 9 | """ Apply voice activity detection 10 | 11 | :param log_energy: Log mel energy. 12 | :param energy_mean_scale: If this is set to s, to get the actual threshold we let m be the mean log-energy of the file, and use s*m + vad-energy-threshold (float, default = 0.5) 13 | :param energy_threshold: Constant term in energy threshold for VAD (also see energy_mean_scale) (float, default = 5) 14 | :param frames_context: Number of frames of context on each side of central frame, in window for which energy is monitored (int, default = 0) 15 | :param proportion_threshold: Parameter controlling the proportion of frames within the window that need to have more energy than the threshold (float, default = 0.6) 16 | :return: A vector of boolean that are True if we judge the frame voiced and False otherwise. 17 | """ 18 | assert len(log_energy.shape) == 1 19 | assert energy_mean_scale >= 0 20 | assert frames_context >= 0 21 | assert 0 < proportion_threshold < 1 22 | dtype = log_energy.dtype 23 | energy_threshold += energy_mean_scale * log_energy.mean() 24 | if frames_context > 0: 25 | num_frames = len(log_energy) 26 | window_size = frames_context * 2 + 1 27 | log_energy_pad = np.concatenate([ 28 | np.zeros(frames_context, dtype=dtype), 29 | log_energy, 30 | np.zeros(frames_context, dtype=dtype) 31 | ]) 32 | log_energy_window = sliding_window(log_energy_pad, window_size, 1) 33 | num_count = np.count_nonzero(log_energy_window > energy_threshold, axis=1) 34 | den_count = np.ones(num_frames, dtype=dtype) * window_size 35 | max_den_count = np.arange(frames_context + 1, min(window_size, num_frames) + 1, dtype=dtype) 36 | den_count[:-(frames_context + 2):-1] = max_den_count 37 | den_count[:frames_context + 1] = np.min([den_count[:frames_context + 1], max_den_count], axis=0) 38 | vad = num_count / den_count >= proportion_threshold 39 | else: 40 | vad = log_energy > energy_threshold 41 | return vad 42 | 43 | # ---------- compute-vad ---------- 44 | --------------------------------------------------------------------------------