45 |
56 | );
57 | };
58 |
59 | export default SliderComponent;
--------------------------------------------------------------------------------
/public/index.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
12 |
13 |
17 |
18 |
27 |
98 | {/* banner image centered */}
99 | 
100 | {!file ? (
101 |
102 | {({ getRootProps, getInputProps }) => (
103 |
109 | ) : (
110 |
132 | );
133 | };
134 |
135 | export default App
136 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | 
2 | ## Realtime Audio Segmentation
3 |
4 | The real-time audio segmentation algorithm described here is specifically developed to address the need for dynamic and coherent visual effects in audio reactive LED lighting systems. The goal is to create visually engaging displays that change and flow with the music. This algorithm segments the real-time audio into coherent sections and provides a signal when a change in the song occurs.
5 |
6 | ## Features
7 |
8 | - **Real-time**: The algorithm processes the audio in real-time, allowing it to respond to changes in the music as they occur.
9 | - **Coherent**: The algorithm segments the audio into coherent sections that are consistent with the music. Sometimes it might miss a section or detect a section that is not present, but overall, the sections are consistent enough with the music to create a pleasing visual effect.
10 | - **Lightweight**: The algorithm is lightweight and can be implemented on microcontrollers such as the ESP32. It is also highly optimized to minimize computational load.
11 |
12 |
13 |
104 |
105 |
107 | )}
108 | Drag and drop an audio file here, or click to browse
106 |
111 |
122 | )}
123 | {isLoading && (
124 |
112 |
115 |
121 |
125 |
128 | )}
129 |
130 |
131 | Loading...
127 |
14 |
15 |
16 | 
17 |
18 |
19 | [](https://not-matt.github.io/AudioSync/)
20 |
21 |
22 |
23 | ## Algorithmic Outline
24 |
25 | 1. **Frame-based analysis**: The audio is processed frame by frame in real-time. Each frame represents a short segment of the audio signal. The frame size is typically 512 samples, which corresponds to 11.6 ms at a sampling rate of 44.1 kHz. The frame size can be adjusted to fit the existing system requirements with little detriment to the algorithm's performance.
26 | 2. **Feature extraction**: Calculate audio features for each frame that capture general characteristics of the sound. Effective features include Low-Frequency Content (LFC), Energy, and Zero-Crossing Rate (ZCR). LFC is computed from the frequency spectrum obtained through FFT, while Energy and ZCR are calculated directly from the raw audio frame. These features are combined into a three-dimensional feature vector.
27 | 3. **Averaging**: Rolling averages are used to compare the current sounds to the previous sounds heard in the last few seconds. A rolling window is maintained, and for each frame, the feature vector is added to the start of the rolling window while discarding the oldest vector. Short and long averages are computed from the rolling window. The short average represents the current sound, and the long average represents the past sounds. To optimize the averaging process, the new data is added to the average and the old data is subtracted, avoiding recomputing the average by summing and dividing the entire window. See image for a visual representation of the window averaging process.
28 | 
29 | 4. **Change detection**: The Euclidean distance between the short and long averages is calculated to measure the change in the audio characteristics. A large distance indicates a significant change in the sound. The distance is compared to a threshold to determine if a change has occurred.
30 | 4. **Thresholding (WIP)** The threshold is dynamically adjusted based on the average distance between the short and long averages. This allows the algorithm to adapt to different songs and environments. A cooldown period is also implemented to prevent the threshold from firing too often and causing the lights to flicker if the music is very dynamic.
31 |
32 | ## Usage
33 |
34 | To use this algorithm in your project, follow these steps:
35 |
36 | 1. **Integration**: Integrate the algorithm into your system or firmware, considering the constraints of the target platform (e.g., ESP32 microcontroller).
37 |
38 | 2. **Audio Input**: Ensure that the algorithm has access to a real-time audio feed. This can be achieved by connecting a microphone or using the system audio output as input.
39 |
40 | 3. **Configuration**: Adjust the parameters of the algorithm based on your application's requirements. This includes frame size, feature selection, averaging window size, threshold value, and event handling mechanism. The most important values to configure are related to the rolling averages. You want the short average to capture the current sound and the long average to capture the past sounds. The short average should be around 50ms, and the long average should be around 2-5 seconds. You can experiment with different values in the live demo. Note that all music has different characteristics, so the optimal values will vary depending on the song. Test with different songs to find the best values for your application.
41 |
42 | 4. **Event Handling**: Integrate an event handling mechanism to respond to the detected audio sections or changes. For example, use the events triggered by the algorithm to change visual effects on an LED strip or perform other desired actions.
43 |
44 | ## License
45 |
46 | AudioSync is provided under the MIT License, allowing you to use, modify, and distribute it freely. However, please refer to the license file for the full terms and conditions.
47 |
48 | ## Contributing
49 |
50 | Contributions to enhance and improve AudioSync are welcome. If you encounter any issues, have suggestions for improvements, or would like to contribute new features, please feel free to submit an issue or a pull request to the project repository. Your contributions will help make the algorithm better and more robust.
51 |
52 | ## Contact
53 |
54 | For any questions, comments, or inquiries regarding the AudioSync algorithm, please open an issue or contact @not_matt on WLED or LedFx's Discord.
--------------------------------------------------------------------------------
/src/fft.js:
--------------------------------------------------------------------------------
1 | /* eslint-disable complexity, no-redeclare, no-var, one-var */
2 |
3 | /**
4 | * Calculate FFT - Based on https://github.com/corbanbrook/dsp.js
5 | *
6 | * @param {Number} bufferSize Buffer size
7 | * @param {Number} sampleRate Sample rate
8 | * @param {Function} windowFunc Window function
9 | * @param {Number} alpha Alpha channel
10 | */
11 |
12 | export default function FFT(bufferSize, sampleRate, windowFunc, alpha) {
13 | this.bufferSize = bufferSize;
14 | this.sampleRate = sampleRate;
15 | this.bandwidth = (2 / bufferSize) * (sampleRate / 2);
16 |
17 | this.sinTable = new Float32Array(bufferSize);
18 | this.cosTable = new Float32Array(bufferSize);
19 | this.windowValues = new Float32Array(bufferSize);
20 | this.reverseTable = new Uint32Array(bufferSize);
21 |
22 | this.peakBand = 0;
23 | this.peak = 0;
24 |
25 | var i;
26 | switch (windowFunc) {
27 | case 'bartlett':
28 | for (i = 0; i < bufferSize; i++) {
29 | this.windowValues[i] =
30 | (2 / (bufferSize - 1)) *
31 | ((bufferSize - 1) / 2 - Math.abs(i - (bufferSize - 1) / 2));
32 | }
33 | break;
34 | case 'bartlettHann':
35 | for (i = 0; i < bufferSize; i++) {
36 | this.windowValues[i] =
37 | 0.62 -
38 | 0.48 * Math.abs(i / (bufferSize - 1) - 0.5) -
39 | 0.38 * Math.cos((Math.PI * 2 * i) / (bufferSize - 1));
40 | }
41 | break;
42 | case 'blackman':
43 | alpha = alpha || 0.16;
44 | for (i = 0; i < bufferSize; i++) {
45 | this.windowValues[i] =
46 | (1 - alpha) / 2 -
47 | 0.5 * Math.cos((Math.PI * 2 * i) / (bufferSize - 1)) +
48 | (alpha / 2) *
49 | Math.cos((4 * Math.PI * i) / (bufferSize - 1));
50 | }
51 | break;
52 | case 'cosine':
53 | for (i = 0; i < bufferSize; i++) {
54 | this.windowValues[i] = Math.cos(
55 | (Math.PI * i) / (bufferSize - 1) - Math.PI / 2
56 | );
57 | }
58 | break;
59 | case 'gauss':
60 | alpha = alpha || 0.25;
61 | for (i = 0; i < bufferSize; i++) {
62 | this.windowValues[i] = Math.pow(
63 | Math.E,
64 | -0.5 *
65 | Math.pow(
66 | (i - (bufferSize - 1) / 2) /
67 | ((alpha * (bufferSize - 1)) / 2),
68 | 2
69 | )
70 | );
71 | }
72 | break;
73 | case 'hamming':
74 | for (i = 0; i < bufferSize; i++) {
75 | this.windowValues[i] =
76 | 0.54 -
77 | 0.46 * Math.cos((Math.PI * 2 * i) / (bufferSize - 1));
78 | }
79 | break;
80 | case 'hann':
81 | case undefined:
82 | for (i = 0; i < bufferSize; i++) {
83 | this.windowValues[i] =
84 | 0.5 * (1 - Math.cos((Math.PI * 2 * i) / (bufferSize - 1)));
85 | }
86 | break;
87 | case 'lanczoz':
88 | for (i = 0; i < bufferSize; i++) {
89 | this.windowValues[i] =
90 | Math.sin(Math.PI * ((2 * i) / (bufferSize - 1) - 1)) /
91 | (Math.PI * ((2 * i) / (bufferSize - 1) - 1));
92 | }
93 | break;
94 | case 'rectangular':
95 | for (i = 0; i < bufferSize; i++) {
96 | this.windowValues[i] = 1;
97 | }
98 | break;
99 | case 'triangular':
100 | for (i = 0; i < bufferSize; i++) {
101 | this.windowValues[i] =
102 | (2 / bufferSize) *
103 | (bufferSize / 2 - Math.abs(i - (bufferSize - 1) / 2));
104 | }
105 | break;
106 | default:
107 | throw Error("No such window function '" + windowFunc + "'");
108 | }
109 |
110 | var limit = 1;
111 | var bit = bufferSize >> 1;
112 | var i;
113 |
114 | while (limit < bufferSize) {
115 | for (i = 0; i < limit; i++) {
116 | this.reverseTable[i + limit] = this.reverseTable[i] + bit;
117 | }
118 |
119 | limit = limit << 1;
120 | bit = bit >> 1;
121 | }
122 |
123 | for (i = 0; i < bufferSize; i++) {
124 | this.sinTable[i] = Math.sin(-Math.PI / i);
125 | this.cosTable[i] = Math.cos(-Math.PI / i);
126 | }
127 |
128 | this.calculateSpectrum = function (buffer) {
129 | // Locally scope variables for speed up
130 | var bufferSize = this.bufferSize,
131 | cosTable = this.cosTable,
132 | sinTable = this.sinTable,
133 | reverseTable = this.reverseTable,
134 | real = new Float32Array(bufferSize),
135 | imag = new Float32Array(bufferSize),
136 | bSi = 2 / this.bufferSize,
137 | sqrt = Math.sqrt,
138 | rval,
139 | ival,
140 | mag,
141 | spectrum = new Float32Array(bufferSize / 2);
142 |
143 | var k = Math.floor(Math.log(bufferSize) / Math.LN2);
144 |
145 | if (Math.pow(2, k) !== bufferSize) {
146 | throw new Error('Invalid buffer size, must be a power of 2.');
147 | }
148 | if (bufferSize !== buffer.length) {
149 | throw new Error('Supplied buffer is not the same size as defined FFT. FFT Size: ' +
150 | bufferSize +
151 | ' Buffer Size: ' +
152 | buffer.length);
153 | }
154 |
155 | var halfSize = 1,
156 | phaseShiftStepReal,
157 | phaseShiftStepImag,
158 | currentPhaseShiftReal,
159 | currentPhaseShiftImag,
160 | off,
161 | tr,
162 | ti,
163 | tmpReal;
164 |
165 | for (var i = 0; i < bufferSize; i++) {
166 | real[i] =
167 | buffer[reverseTable[i]] * this.windowValues[reverseTable[i]];
168 | imag[i] = 0;
169 | }
170 |
171 | while (halfSize < bufferSize) {
172 | phaseShiftStepReal = cosTable[halfSize];
173 | phaseShiftStepImag = sinTable[halfSize];
174 |
175 | currentPhaseShiftReal = 1;
176 | currentPhaseShiftImag = 0;
177 |
178 | for (var fftStep = 0; fftStep < halfSize; fftStep++) {
179 | var i = fftStep;
180 |
181 | while (i < bufferSize) {
182 | off = i + halfSize;
183 | tr =
184 | currentPhaseShiftReal * real[off] -
185 | currentPhaseShiftImag * imag[off];
186 | ti =
187 | currentPhaseShiftReal * imag[off] +
188 | currentPhaseShiftImag * real[off];
189 |
190 | real[off] = real[i] - tr;
191 | imag[off] = imag[i] - ti;
192 | real[i] += tr;
193 | imag[i] += ti;
194 |
195 | i += halfSize << 1;
196 | }
197 |
198 | tmpReal = currentPhaseShiftReal;
199 | currentPhaseShiftReal =
200 | tmpReal * phaseShiftStepReal -
201 | currentPhaseShiftImag * phaseShiftStepImag;
202 | currentPhaseShiftImag =
203 | tmpReal * phaseShiftStepImag +
204 | currentPhaseShiftImag * phaseShiftStepReal;
205 | }
206 |
207 | halfSize = halfSize << 1;
208 | }
209 |
210 | for (var i = 0, N = bufferSize / 2; i < N; i++) {
211 | rval = real[i];
212 | ival = imag[i];
213 | mag = bSi * sqrt(rval * rval + ival * ival);
214 |
215 | if (mag > this.peak) {
216 | this.peakBand = i;
217 | this.peak = mag;
218 | }
219 | spectrum[i] = mag;
220 | }
221 | return spectrum;
222 | };
223 | }
224 |
--------------------------------------------------------------------------------
/src/AudioSync.js:
--------------------------------------------------------------------------------
1 | import FFT from './fft.js';
2 |
3 | /*
4 | Based on Wavesurfer's spectrogram plugin
5 | https://github.com/wavesurfer-js/wavesurfer.js/tree/master/src/plugin/spectrogram
6 | */
7 |
8 | export default class AudioSyncPlugin {
9 | /**
10 | * AudioSync plugin definition factory
11 | *
12 | * This function must be used to create a plugin definition which can be
13 | * used by wavesurfer to correctly instantiate the plugin.
14 | *
15 | * @param {AudioSyncPluginParams} params Parameters used to initialise the plugin
16 | * @return {PluginDefinition} An object representing the plugin.
17 | */
18 | static create(params) {
19 | return {
20 | name: 'audiosync',
21 | deferInit: params && params.deferInit ? params.deferInit : false,
22 | params: params,
23 | staticProps: {
24 | FFT: FFT
25 | },
26 | instance: AudioSyncPlugin
27 | };
28 | }
29 |
30 | constructor(params, ws) {
31 | this.params = params;
32 | this.wavesurfer = ws;
33 | this.util = ws.util;
34 |
35 | this.frequenciesDataUrl = params.frequenciesDataUrl;
36 | this._onScroll = e => {
37 | this.updateScroll(e);
38 | };
39 | this._onRender = () => {
40 | this.render();
41 | };
42 | this._onWrapperClick = e => {
43 | this._wrapperClickHandler(e);
44 | };
45 | this._onReady = () => {
46 | const drawer = (this.drawer = ws.drawer);
47 |
48 | this.container =
49 | 'string' == typeof params.container
50 | ? document.querySelector(params.container)
51 | : params.container;
52 |
53 | if (!this.container) {
54 | throw Error('No container for WaveSurfer spectrogram');
55 | }
56 |
57 | this.width = drawer.width;
58 | this.pixelRatio = this.params.pixelRatio || ws.params.pixelRatio;
59 | this.fftSamples = this.params.fftSamples || ws.params.fftSamples || 512;
60 | this.height = this.params.height || 450;
61 | this.noverlap = params.noverlap;
62 | this.windowFunc = params.windowFunc;
63 | this.alpha = params.alpha;
64 | this.splitChannels = params.splitChannels;
65 | this.channels = this.splitChannels ? ws.backend.buffer.numberOfChannels : 1;
66 |
67 | // Define variables and parameters
68 | this.settings = this.params.settings;
69 | console.log(this.settings)
70 | this.featureHistory = [];
71 | this.cooldownCounter = 0;
72 | this.shortAverageVector = null;
73 | this.longAverageVector = null;
74 | this.featuresData = [];
75 |
76 | this.createWrapper();
77 | this.createCanvas();
78 | this.render();
79 |
80 | drawer.wrapper.addEventListener('scroll', this._onScroll);
81 | ws.on('redraw', this._onRender);
82 | };
83 | }
84 |
85 | updateSettings(settings) {
86 | this.settings = settings;
87 | console.log(this.settings)
88 | if (this.featuresData) {
89 | this.calculateSpectralFeatures();
90 | }
91 | this.drawSpectralFeatures();
92 | }
93 |
94 | init() {
95 | // Check if wavesurfer is ready
96 | if (this.wavesurfer.isReady) {
97 | this._onReady();
98 | } else {
99 | this.wavesurfer.once('ready', this._onReady);
100 | }
101 | }
102 |
103 | destroy() {
104 | this.unAll();
105 | this.wavesurfer.un('ready', this._onReady);
106 | this.wavesurfer.un('redraw', this._onRender);
107 | this.drawer && this.drawer.wrapper.removeEventListener('scroll', this._onScroll);
108 | this.wavesurfer = null;
109 | this.util = null;
110 | this.params = null;
111 | if (this.wrapper) {
112 | this.wrapper.removeEventListener('click', this._onWrapperClick);
113 | this.wrapper.parentNode.removeChild(this.wrapper);
114 | this.wrapper = null;
115 | }
116 | }
117 |
118 | createWrapper() {
119 | const oldWrapper = this.container.querySelector('AudioSync');
120 | if (oldWrapper) {
121 | this.container.removeChild(oldWrapper);
122 | }
123 | const wsParams = this.wavesurfer.params;
124 | this.wrapper = document.createElement('AudioSync');
125 |
126 | this.drawer.style(this.wrapper, {
127 | display: 'block',
128 | position: 'relative',
129 | userSelect: 'none',
130 | webkitUserSelect: 'none',
131 | height: `${this.height}px`
132 | });
133 |
134 | if (wsParams.fillParent || wsParams.scrollParent) {
135 | this.drawer.style(this.wrapper, {
136 | width: '100%',
137 | overflowX: 'hidden',
138 | overflowY: 'hidden'
139 | });
140 | }
141 | this.container.appendChild(this.wrapper);
142 |
143 | this.wrapper.addEventListener('click', this._onWrapperClick);
144 | }
145 |
146 | _wrapperClickHandler(event) {
147 | event.preventDefault();
148 | const relX = 'offsetX' in event ? event.offsetX : event.layerX;
149 | this.fireEvent('click', relX / this.width || 0);
150 | }
151 |
152 | createCanvas() {
153 | const specCanvas = (this.specCanvas = this.wrapper.appendChild(
154 | document.createElement('canvas')
155 | ));
156 | const overlayCanvas = (this.overlayCanvas = this.wrapper.appendChild(
157 | document.createElement('canvas')
158 | ));
159 |
160 | this.spectrCc = specCanvas.getContext('2d');
161 | this.overlayCc = overlayCanvas.getContext('2d');
162 |
163 | this.util.style(specCanvas, {
164 | zIndex: 4,
165 | position: 'absolute',
166 | top: '0px',
167 | left: '0px',
168 | });
169 | this.util.style(overlayCanvas, {
170 | zIndex: 4,
171 | position: 'absolute',
172 | top: '0px',
173 | left: '0px',
174 | });
175 | }
176 |
177 | render() {
178 | this.updateCanvasStyle();
179 |
180 | if (this.frequenciesDataUrl) {
181 | this.loadFrequenciesData(this.frequenciesDataUrl);
182 | } else {
183 | this.calculateSpectralFeatures();
184 | }
185 | }
186 |
187 | calculateSpectralFeatures() {
188 | const fftSamples = this.fftSamples;
189 | const buffer = this.wavesurfer.backend.buffer;
190 | const channels = this.channels;
191 |
192 | if (!buffer) {
193 | this.fireEvent('error', 'Web Audio buffer is not available');
194 | return;
195 | }
196 |
197 | const sampleRate = buffer.sampleRate;
198 | this.featuresData = [];
199 |
200 | let noverlap = this.noverlap;
201 | if (!noverlap) {
202 | const uniqueSamplesPerPx = buffer.length / this.specCanvas.width;
203 | noverlap = Math.max(0, Math.round(fftSamples - uniqueSamplesPerPx));
204 | }
205 |
206 | const fft = new FFT(fftSamples, sampleRate, this.windowFunc, this.alpha);
207 |
208 | for (let c = 0; c < channels; c++) {
209 |
210 | const channelData = buffer.getChannelData(c);
211 | const channelFeatures = [];
212 |
213 | let currentOffset = 0;
214 |
215 | while (currentOffset + fftSamples < channelData.length) {
216 | const segment = channelData.slice(
217 | currentOffset,
218 | currentOffset + fftSamples
219 | );
220 | const spectrum = fft.calculateSpectrum(segment);
221 |
222 | const features = {
223 | spectrum: spectrum,
224 | lfc: null,
225 | zcr: null,
226 | energy: null,
227 | };
228 |
229 | features.lfc = this.calculateLFC(spectrum);
230 | features.zcr = this.calculateZeroCrossingRate(segment);
231 | features.energy = this.calculateEnergy(segment);
232 |
233 | channelFeatures.push(features);
234 |
235 | currentOffset += fftSamples - noverlap;
236 | }
237 |
238 | this.featuresData.push(channelFeatures);
239 | }
240 |
241 | this.drawSpectralFeatures();
242 | }
243 |
244 | calculateZeroCrossingRate(frame) {
245 | let zcr = 0;
246 | for (let i = 0; i < frame.length - 1; i++) {
247 | if (frame[i] * frame[i + 1] < 0) {
248 | zcr++;
249 | }
250 | }
251 | return zcr;
252 | }
253 |
254 | calculateEnergy(frame) {
255 | let energy = 0;
256 | for (let i = 0; i < frame.length; i++) {
257 | energy += frame[i] * frame[i];
258 | }
259 | return energy;
260 | }
261 |
262 | calculateLFC(spectrum) {
263 | const startIndex = 0;
264 | const endIndex = Math.ceil(spectrum.length * 0.05);
265 |
266 | let sum = 0;
267 | let total = 0;
268 |
269 | for (let i = startIndex; i <= endIndex; i++) {
270 | sum += spectrum[i];
271 | total += 1;
272 | }
273 |
274 | return total > 0 ? sum / total : 0;
275 | }
276 |
277 | // Function to update the vector feature rolling averages
278 | updateFeatureHistory(currentVector) {
279 | this.featureHistory.unshift(currentVector);
280 |
281 | if (this.featureHistory.length > this.settings.windowSize) {
282 | const removedVector = this.featureHistory.pop();
283 | const shortVector = this.featureHistory[this.settings.shortWindowSize];
284 |
285 | // Update the rolling average vectors
286 | for (let i = 0; i < this.longAverageVector.length; i++) {
287 | this.shortAverageVector[i] = this.shortAverageVector[i] - shortVector[i] + currentVector[i];
288 | this.longAverageVector[i] = this.longAverageVector[i] - removedVector[i] + shortVector[i];
289 | }
290 | } else {
291 | this.longAverageVector = Array.from(currentVector);
292 | this.shortAverageVector = Array.from(currentVector);
293 | }
294 | }
295 |
296 | // Function to calculate the squared distance between two vectors
297 | // This is like euclidian distance, but by skipping the sqrt and comparing squared values, we can save some processing power
298 | calculateSquaredDistance(vector1, vector2) {
299 | let sum = 0;
300 | for (let i = 0; i < vector1.length; i++) {
301 | sum += Math.pow(vector1[i] - vector2[i], 2);
302 | }
303 | return Math.pow(sum, 2);
304 | }
305 |
306 | drawSpectralFeatures() {
307 | const featuresData = this.featuresData;
308 | const spectrCc = this.spectrCc;
309 | const height = this.height;
310 | const width = this.width;
311 | const ratio = this.settings.shortWindowSize / (this.settings.windowSize - this.settings.shortWindowSize);
312 | const shortScale = 255 * 0.57735 / this.settings.shortWindowSize // 1 / sqrt(3), the diagonal of a unit cube
313 | const longScale = 255 * 0.57735 / (this.settings.windowSize - this.settings.shortWindowSize) // 1 / sqrt(3), the diagonal of a unit cube
314 |
315 | console.log(ratio, shortScale, longScale)
316 | let squaredDistances = [];
317 | this.featureHistory = []; // reset the feature history
318 |
319 | console.log(shortScale, longScale)
320 |
321 | const maxLFC = Math.max(
322 | ...featuresData.flatMap(channel => channel.flatMap(frame => frame.lfc))
323 | );
324 | const maxZCR = Math.max(
325 | ...featuresData.flatMap(channel => channel.flatMap(frame => frame.zcr))
326 | );
327 | const maxEnergy = Math.max(
328 | ...featuresData.flatMap(channel => channel.flatMap(frame => frame.energy))
329 | );
330 |
331 | if (!spectrCc) {
332 | throw new Error('No canvas context to draw spectrogram or overlay');
333 | }
334 |
335 | spectrCc.clearRect(0, 0, width, height);
336 |
337 | for (let c = 0; c < featuresData.length; c++) {
338 | const channelData = featuresData[c];
339 |
340 | for (let i = 0; i < channelData.length; i++) {
341 | const frame = channelData[i];
342 | const x = i * (width / channelData.length);
343 |
344 | // Scale the features to between 0 and 1
345 | const lfc = frame.lfc / maxLFC;
346 | const zcr = frame.zcr / maxZCR;
347 | const energy = frame.energy / maxEnergy;
348 |
349 | // Combine the features into a three-dimensional vector
350 | const currentVector = [lfc, energy, zcr];
351 |
352 | // Update the rolling window of feature vectors
353 | this.updateFeatureHistory(currentVector);
354 |
355 | // Change detection algorithm
356 | // Update the cooldown counter
357 | if (this.cooldownCounter > 0) {
358 | this.cooldownCounter *= 0.998;
359 | }
360 |
361 | // Calculate the squared distance between the short and long rolling average vectors
362 | // multiple this.longAverageVector by ratio to account for the difference in window sizes
363 | const squaredDistance = this.calculateSquaredDistance(this.shortAverageVector, this.longAverageVector.map(x => x * ratio));
364 | squaredDistances.push(squaredDistance)
365 |
366 | // Calculate the threshold for a significant change based on the cooldown duration
367 | // const threshold = this.cooldownCounter; // Adjust the threshold as needed
368 | // const triggered = squaredDistance > threshold
369 |
370 | // Draw the three spectral features in the top quarter of the canvas
371 | const lfcColor = "255, 0, 0"
372 | const zcrColor = "0, 255, 0"
373 | const energyColor = "0, 0, 255"
374 |
375 | const stackHeight = height / 4 / 3
376 | const lfcBarHeight = lfc * stackHeight;
377 | const lfcY = stackHeight - lfcBarHeight
378 | const zcrBarHeight = zcr * stackHeight;
379 | const zcrY = stackHeight * 2 - zcrBarHeight
380 | const energyBarHeight = energy * stackHeight;
381 | const energyY = stackHeight * 3 - energyBarHeight
382 |
383 | spectrCc.fillStyle = `rgba(${lfcColor}, ${lfcBarHeight / stackHeight})`;
384 | spectrCc.fillRect(x, lfcY, 1, lfcBarHeight);
385 | spectrCc.fillStyle = `rgba(${zcrColor}, ${zcrBarHeight / stackHeight})`;
386 | spectrCc.fillRect(x, zcrY, 1, zcrBarHeight);
387 | spectrCc.fillStyle = `rgba(${energyColor}, ${energyBarHeight / stackHeight})`;
388 | spectrCc.fillRect(x, energyY, 1, energyBarHeight);
389 |
390 | // Draw the spectral feature vector as a colour
391 | spectrCc.fillStyle = `rgba(${this.shortAverageVector[0] * shortScale}, ${this.shortAverageVector[1] * shortScale}, ${this.shortAverageVector[2] * shortScale}, 1)`;
392 | spectrCc.fillRect(x, height * 1/4, 1, height * 1/4);
393 | spectrCc.fillStyle = `rgba(${this.longAverageVector[0] * longScale}, ${this.longAverageVector[1] * longScale}, ${this.longAverageVector[2] * longScale}, 1)`;
394 | spectrCc.fillRect(x, height * 2/4, 1, height * 1/4);
395 |
396 | // if (triggered) {
397 | // // reset the cooldown counter
398 | // this.cooldownCounter = 65535
399 | // // Draw a white line if a change was detected
400 | // spectrCc.fillStyle = 'white';
401 | // spectrCc.fillRect(x, 0, 1, height);
402 | // }
403 |
404 | // // Draw the cooldown counter as a transparent white line
405 | // const cooldownBarHeight = this.cooldownCounter / 65535 * height * 0.5;
406 | // const cooldownBarY = height - cooldownBarHeight;
407 | // spectrCc.fillStyle = `rgba(255, 255, 255, ${this.cooldownCounter / 65535})`;
408 | // spectrCc.fillRect(x, cooldownBarY, 1, cooldownBarHeight);
409 |
410 | }
411 | }
412 |
413 | // set the last 5% to zero to avoid the graph being dominated by the cooldown period
414 | squaredDistances = squaredDistances.map((x, i) => i > squaredDistances.length * 0.95 ? 0 : x)
415 |
416 | // Draw the squared distance as solid red bars in the lower quarter of the canvas
417 | console.log(squaredDistances)
418 | const maxSquaredDistance = Math.max(...squaredDistances)
419 | for (let i = 0; i < squaredDistances.length; i++) {
420 | const x = i * (width / squaredDistances.length);
421 | const y = height * 3 / 4;
422 | const barHeight = squaredDistances[i] / maxSquaredDistance * height / 4;
423 | const barY = y + height / 4 - barHeight;
424 | spectrCc.fillStyle = `rgba(255, 0, 0, 1)`;
425 | spectrCc.fillRect(x, barY, 1, barHeight);
426 | }
427 |
428 | // Draw the legend
429 | const legendX = 10;
430 | spectrCc.fillStyle = 'black';
431 | spectrCc.font = '20px Arial';
432 | spectrCc.fillText('LFC', legendX, 30);
433 | spectrCc.fillText('ZCR', legendX, height/12 + 30);
434 | spectrCc.fillText('Energy', legendX, height/12 * 2 + 30);
435 | spectrCc.fillStyle = 'white';
436 | spectrCc.fillText('Short Rolling Average', legendX, height/4 + 30);
437 | spectrCc.fillText('(Feature Vector visualised as RGB)', legendX, height/4 + 60);
438 | spectrCc.fillText('Long Rolling Average', legendX, height/2 + 60);
439 | spectrCc.fillStyle = 'black';
440 | spectrCc.fillText('Change Detection', legendX, height/4 * 3 + 30);
441 | spectrCc.fillText('(Euclidean Distance between Short and Long Rolling Averages)', legendX, height/4 * 3 + 60);
442 |
443 | // Update playback position overlay
444 | this.updatePlaybackLine();
445 | }
446 |
447 | updatePlaybackLine() {
448 | if (!this.wavesurfer) {
449 | return;
450 | }
451 |
452 | const overlayCc = this.overlayCc
453 | const height = this.height;
454 | const playbackPos = this.wavesurfer.backend.getCurrentTime() * this.width / this.wavesurfer.getDuration();
455 |
456 | // Clear previous overlay
457 | overlayCc.clearRect(0, 0, this.width, height);
458 | overlayCc.beginPath();
459 | overlayCc.moveTo(playbackPos, 0);
460 | overlayCc.lineTo(playbackPos, height);
461 | overlayCc.strokeStyle = 'rgba(0,0,0,1)';
462 | overlayCc.lineWidth = 2;
463 | overlayCc.stroke();
464 | // Request animation frame for continuous updating
465 | requestAnimationFrame(() => this.updatePlaybackLine());
466 | }
467 |
468 | updateCanvasStyle() {
469 | const width = Math.round(this.width / this.pixelRatio) + 'px';
470 | this.specCanvas.width = this.width;
471 | this.specCanvas.height = this.height;
472 | this.specCanvas.style.width = width;
473 | this.specCanvas.style.height = this.height + 'px';
474 | this.overlayCanvas.width = this.width;
475 | this.overlayCanvas.height = this.height;
476 | this.overlayCanvas.style.width = width;
477 | this.overlayCanvas.style.height = this.height + 'px';
478 | }
479 |
480 | loadFrequenciesData(url) {
481 | const request = this.util.fetchFile({ url: url });
482 |
483 | request.on('success', data =>
484 | this.calculateSpectralFeatures(JSON.parse(data), this)
485 | );
486 | request.on('error', e => this.fireEvent('error', e));
487 |
488 | return request;
489 | }
490 |
491 | updateScroll(e) {
492 | if (this.wrapper) {
493 | this.wrapper.scrollLeft = e.target.scrollLeft;
494 | }
495 | }
496 | }
--------------------------------------------------------------------------------