├── .gitignore ├── .gitmodules ├── LICENSE ├── README.md ├── main.py ├── organize_wavs.py ├── requirements.txt ├── static ├── scripts │ └── app.js └── styles │ └── app.css └── templates ├── record.html └── welcome.html /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | *~ 3 | .DS_Store 4 | lib/ 5 | app.yaml 6 | -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "extract_loudest_section"] 2 | path = extract_loudest_section 3 | url = https://github.com/petewarden/extract_loudest_section 4 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright 2015 The TensorFlow Authors. All rights reserved. 2 | 3 | Apache License 4 | Version 2.0, January 2004 5 | http://www.apache.org/licenses/ 6 | 7 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 8 | 9 | 1. Definitions. 10 | 11 | "License" shall mean the terms and conditions for use, reproduction, 12 | and distribution as defined by Sections 1 through 9 of this document. 13 | 14 | "Licensor" shall mean the copyright owner or entity authorized by 15 | the copyright owner that is granting the License. 16 | 17 | "Legal Entity" shall mean the union of the acting entity and all 18 | other entities that control, are controlled by, or are under common 19 | control with that entity. For the purposes of this definition, 20 | "control" means (i) the power, direct or indirect, to cause the 21 | direction or management of such entity, whether by contract or 22 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 23 | outstanding shares, or (iii) beneficial ownership of such entity. 24 | 25 | "You" (or "Your") shall mean an individual or Legal Entity 26 | exercising permissions granted by this License. 27 | 28 | "Source" form shall mean the preferred form for making modifications, 29 | including but not limited to software source code, documentation 30 | source, and configuration files. 31 | 32 | "Object" form shall mean any form resulting from mechanical 33 | transformation or translation of a Source form, including but 34 | not limited to compiled object code, generated documentation, 35 | and conversions to other media types. 36 | 37 | "Work" shall mean the work of authorship, whether in Source or 38 | Object form, made available under the License, as indicated by a 39 | copyright notice that is included in or attached to the work 40 | (an example is provided in the Appendix below). 41 | 42 | "Derivative Works" shall mean any work, whether in Source or Object 43 | form, that is based on (or derived from) the Work and for which the 44 | editorial revisions, annotations, elaborations, or other modifications 45 | represent, as a whole, an original work of authorship. For the purposes 46 | of this License, Derivative Works shall not include works that remain 47 | separable from, or merely link (or bind by name) to the interfaces of, 48 | the Work and Derivative Works thereof. 49 | 50 | "Contribution" shall mean any work of authorship, including 51 | the original version of the Work and any modifications or additions 52 | to that Work or Derivative Works thereof, that is intentionally 53 | submitted to Licensor for inclusion in the Work by the copyright owner 54 | or by an individual or Legal Entity authorized to submit on behalf of 55 | the copyright owner. For the purposes of this definition, "submitted" 56 | means any form of electronic, verbal, or written communication sent 57 | to the Licensor or its representatives, including but not limited to 58 | communication on electronic mailing lists, source code control systems, 59 | and issue tracking systems that are managed by, or on behalf of, the 60 | Licensor for the purpose of discussing and improving the Work, but 61 | excluding communication that is conspicuously marked or otherwise 62 | designated in writing by the copyright owner as "Not a Contribution." 63 | 64 | "Contributor" shall mean Licensor and any individual or Legal Entity 65 | on behalf of whom a Contribution has been received by Licensor and 66 | subsequently incorporated within the Work. 67 | 68 | 2. Grant of Copyright License. Subject to the terms and conditions of 69 | this License, each Contributor hereby grants to You a perpetual, 70 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 71 | copyright license to reproduce, prepare Derivative Works of, 72 | publicly display, publicly perform, sublicense, and distribute the 73 | Work and such Derivative Works in Source or Object form. 74 | 75 | 3. Grant of Patent License. Subject to the terms and conditions of 76 | this License, each Contributor hereby grants to You a perpetual, 77 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 78 | (except as stated in this section) patent license to make, have made, 79 | use, offer to sell, sell, import, and otherwise transfer the Work, 80 | where such license applies only to those patent claims licensable 81 | by such Contributor that are necessarily infringed by their 82 | Contribution(s) alone or by combination of their Contribution(s) 83 | with the Work to which such Contribution(s) was submitted. If You 84 | institute patent litigation against any entity (including a 85 | cross-claim or counterclaim in a lawsuit) alleging that the Work 86 | or a Contribution incorporated within the Work constitutes direct 87 | or contributory patent infringement, then any patent licenses 88 | granted to You under this License for that Work shall terminate 89 | as of the date such litigation is filed. 90 | 91 | 4. Redistribution. You may reproduce and distribute copies of the 92 | Work or Derivative Works thereof in any medium, with or without 93 | modifications, and in Source or Object form, provided that You 94 | meet the following conditions: 95 | 96 | (a) You must give any other recipients of the Work or 97 | Derivative Works a copy of this License; and 98 | 99 | (b) You must cause any modified files to carry prominent notices 100 | stating that You changed the files; and 101 | 102 | (c) You must retain, in the Source form of any Derivative Works 103 | that You distribute, all copyright, patent, trademark, and 104 | attribution notices from the Source form of the Work, 105 | excluding those notices that do not pertain to any part of 106 | the Derivative Works; and 107 | 108 | (d) If the Work includes a "NOTICE" text file as part of its 109 | distribution, then any Derivative Works that You distribute must 110 | include a readable copy of the attribution notices contained 111 | within such NOTICE file, excluding those notices that do not 112 | pertain to any part of the Derivative Works, in at least one 113 | of the following places: within a NOTICE text file distributed 114 | as part of the Derivative Works; within the Source form or 115 | documentation, if provided along with the Derivative Works; or, 116 | within a display generated by the Derivative Works, if and 117 | wherever such third-party notices normally appear. The contents 118 | of the NOTICE file are for informational purposes only and 119 | do not modify the License. You may add Your own attribution 120 | notices within Derivative Works that You distribute, alongside 121 | or as an addendum to the NOTICE text from the Work, provided 122 | that such additional attribution notices cannot be construed 123 | as modifying the License. 124 | 125 | You may add Your own copyright statement to Your modifications and 126 | may provide additional or different license terms and conditions 127 | for use, reproduction, or distribution of Your modifications, or 128 | for any such Derivative Works as a whole, provided Your use, 129 | reproduction, and distribution of the Work otherwise complies with 130 | the conditions stated in this License. 131 | 132 | 5. Submission of Contributions. Unless You explicitly state otherwise, 133 | any Contribution intentionally submitted for inclusion in the Work 134 | by You to the Licensor shall be under the terms and conditions of 135 | this License, without any additional terms or conditions. 136 | Notwithstanding the above, nothing herein shall supersede or modify 137 | the terms of any separate license agreement you may have executed 138 | with Licensor regarding such Contributions. 139 | 140 | 6. Trademarks. This License does not grant permission to use the trade 141 | names, trademarks, service marks, or product names of the Licensor, 142 | except as required for reasonable and customary use in describing the 143 | origin of the Work and reproducing the content of the NOTICE file. 144 | 145 | 7. Disclaimer of Warranty. Unless required by applicable law or 146 | agreed to in writing, Licensor provides the Work (and each 147 | Contributor provides its Contributions) on an "AS IS" BASIS, 148 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 149 | implied, including, without limitation, any warranties or conditions 150 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 151 | PARTICULAR PURPOSE. You are solely responsible for determining the 152 | appropriateness of using or redistributing the Work and assume any 153 | risks associated with Your exercise of permissions under this License. 154 | 155 | 8. Limitation of Liability. In no event and under no legal theory, 156 | whether in tort (including negligence), contract, or otherwise, 157 | unless required by applicable law (such as deliberate and grossly 158 | negligent acts) or agreed to in writing, shall any Contributor be 159 | liable to You for damages, including any direct, indirect, special, 160 | incidental, or consequential damages of any character arising as a 161 | result of this License or out of the use or inability to use the 162 | Work (including but not limited to damages for loss of goodwill, 163 | work stoppage, computer failure or malfunction, or any and all 164 | other commercial damages or losses), even if such Contributor 165 | has been advised of the possibility of such damages. 166 | 167 | 9. Accepting Warranty or Additional Liability. While redistributing 168 | the Work or Derivative Works thereof, You may choose to offer, 169 | and charge a fee for, acceptance of support, warranty, indemnity, 170 | or other liability obligations and/or rights consistent with this 171 | License. However, in accepting such obligations, You may act only 172 | on Your own behalf and on Your sole responsibility, not on behalf 173 | of any other Contributor, and only if You agree to indemnify, 174 | defend, and hold each Contributor harmless for any liability 175 | incurred by, or claims asserted against, such Contributor by reason 176 | of your accepting any such warranty or additional liability. 177 | 178 | END OF TERMS AND CONDITIONS 179 | 180 | APPENDIX: How to apply the Apache License to your work. 181 | 182 | To apply the Apache License to your work, attach the following 183 | boilerplate notice, with the fields enclosed by brackets "[]" 184 | replaced with your own identifying information. (Don't include 185 | the brackets!) The text should be enclosed in the appropriate 186 | comment syntax for the file format. We also recommend that a 187 | file or class name and description of purpose be included on the 188 | same "printed page" as the copyright notice for easier 189 | identification within third-party archives. 190 | 191 | Copyright 2015, The TensorFlow Authors. 192 | 193 | Licensed under the Apache License, Version 2.0 (the "License"); 194 | you may not use this file except in compliance with the License. 195 | You may obtain a copy of the License at 196 | 197 | http://www.apache.org/licenses/LICENSE-2.0 198 | 199 | Unless required by applicable law or agreed to in writing, software 200 | distributed under the License is distributed on an "AS IS" BASIS, 201 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 202 | See the License for the specific language governing permissions and 203 | limitations under the License. 204 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ### Open Speech Recording 2 | 3 | This is a stripped down version of the [**Open Speech Recording**](https://github.com/petewarden/open-speech-recording) written by Pete Warden. For the full version that integrates with the google cloud please see the original repository. This version is optimized to run directly on your local machine. This repository also contains scripts written by Pete to manipulate the audio files that come out of the app. 4 | 5 | ### You can use the app by taking the following steps: 6 | 7 | 1. Clone the repository, initialize the submodules, and install the only requirement, flask: 8 | ``` 9 | git clone https://github.com/tinyMLx/open-speech-recording.git 10 | cd open-speech-recording 11 | git submodule update --init --recursive 12 | pip install flask 13 | ``` 14 | 2. To update the words you are recording and how many recordings of each word you are collecting, change the counts and values at the top of the app file in the open-speech-recording repository: ```open-speech-recording/static/scripts/app.js```. The default is 5 copies of each wake word (in this case just "hello") and 1 copy of each other/unknown/filler word (in this case just "world"). You can then run the server locally (from within the open-speech-recording folder) by running: 15 | ``` 16 | export FLASK_APP=main.py 17 | python -m flask run 18 | ``` 19 | 20 | 3. Then open the link provided in the terminal in a web browser to run the application. Make sure to **run the application in a private or incognito window** which avoids any cacheing issues. Also we've found that the app works best when **using Chrome**. Once the app opens you'll need give access to your microphone, and then you can click ```Record```. Once you finish recording all of the words a popup will appear and ask if you'd like to download the data. Simply click ```OK``` and the files will be downloaded into the folder from which you are running the flask app (which should be the open-speech-recording folder). 21 | 22 | Note: if you want to change the words or counts make sure to kill and re-start the app and open the link in a new incognito window to avoid any cacheing issues at the server or browser level! 23 | 24 | ### You can use the scripts to manipulate the data as follows: 25 | 26 | 1. You can convert the ```.ogg``` files to ```.wav``` files using ```ffmpeg```: 27 | ``` 28 | sudo apt-get install ffmpeg 29 | mkdir wavs 30 | find *.ogg -print0 | xargs -0 basename -s .ogg | xargs -I {} ffmpeg -i {}.ogg -ar 16000 wavs/{}.wav 31 | ``` 32 | 33 | 2. You can then trim the ```.wav``` files with Pete's tool. 34 | ``` 35 | mkdir trimmed_wavs 36 | make -C extract_loudest_section/ 37 | /tmp/extract_loudest_section/gen/bin/extract_loudest_section 'wavs/*.wav' trimmed_wavs/ 38 | ``` 39 | 3. Finally you can create the directory structure expected by the Tensorflow training script by running another of Pete's scripts adn then compress it into a zip file so it can be easily uploaded to Colab. 40 | ``` 41 | python organize_wavs.py 42 | cd output_wavs 43 | zip -r my_dataset.zip * 44 | ``` 45 | 46 | -Adapted by the Harvard CS249r F2020 team 47 | -------------------------------------------------------------------------------- /main.py: -------------------------------------------------------------------------------- 1 | from flask import Flask 2 | from flask import abort 3 | from flask import make_response 4 | from flask import redirect 5 | from flask import render_template 6 | from flask import request 7 | from flask import session 8 | from werkzeug.utils import secure_filename 9 | 10 | import os 11 | import uuid 12 | 13 | app = Flask(__name__) 14 | 15 | @app.route("/") 16 | def welcome(): 17 | session_id = request.cookies.get('session_id') 18 | if session_id: 19 | return render_template("record.html") 20 | else: 21 | return render_template("welcome.html") 22 | 23 | @app.route("/start") 24 | def start(): 25 | response = make_response(redirect('/')) 26 | session_id = uuid.uuid4().hex 27 | response.set_cookie('session_id', session_id) 28 | return response 29 | 30 | @app.route('/upload', methods=['POST']) 31 | def upload(): 32 | session_id = request.cookies.get('session_id') 33 | if not session_id: 34 | make_response('No session', 400) 35 | word = request.args.get('word') 36 | audio_data = request.data 37 | filename = word + '_' + session_id + uuid.uuid4().hex + '.ogg' 38 | secure_name = secure_filename(filename) 39 | with open(secure_name, 'wb') as f: 40 | f.write(audio_data) 41 | return make_response('All good') 42 | 43 | # CSRF protection, see http://flask.pocoo.org/snippets/3/. 44 | @app.before_request 45 | def csrf_protect(): 46 | if request.method == "POST": 47 | token = session['_csrf_token'] 48 | if not token or token != request.args.get('_csrf_token'): 49 | abort(403) 50 | 51 | def generate_csrf_token(): 52 | if '_csrf_token' not in session: 53 | session['_csrf_token'] = uuid.uuid4().hex 54 | return session['_csrf_token'] 55 | 56 | app.jinja_env.globals['csrf_token'] = generate_csrf_token 57 | 58 | app.secret_key = "DEPLOYING_LOCALLY_ONLY_PLACEHOLDER" 59 | 60 | if __name__ == "__main__": 61 | app.run(debug=True) 62 | -------------------------------------------------------------------------------- /organize_wavs.py: -------------------------------------------------------------------------------- 1 | # https://gist.githubusercontent.com/petewarden/64709e08226ba426b8708b60e497f813/raw/1a82ee500168752798bdbf32944d862fda7201ad/organize_wavs.py 2 | # To convert from .ogg files to .wavs: 3 | # find . -iname "*.ogg" -print0 | xargs -0 basename -s .ogg | xargs -I {} ffmpeg -i {}.ogg -ar 16000 ../converted_early_wavs/{}.wav 4 | # Then run the extract_loudest xcode project to get one-second clips. 5 | 6 | import glob 7 | import os 8 | import re 9 | import shutil 10 | 11 | data_index = {} 12 | 13 | os.chdir('trimmed_wavs') 14 | search_path = os.path.join('*.wav') 15 | for wav_path in glob.glob(search_path): 16 | matches = re.search('([^/_]+)_([^/_]+)_([^/]+)\.wav', wav_path) 17 | if not matches: 18 | raise Exception('File name not in a recognized form:"%s"' % wav_path) 19 | word = matches.group(1).lower() 20 | speaker = matches.group(2).lower() 21 | instance = matches.group(3).lower() 22 | if not word in data_index: 23 | data_index[word] = {} 24 | if not speaker in data_index[word]: 25 | data_index[word][speaker] = {} 26 | if instance in data_index[word][speaker]: 27 | raise Exception('Audio instance already seen:"%s"' % wav_path) 28 | data_index[word][speaker][instance] = wav_path 29 | 30 | output_dir = os.path.join('..', 'output_wavs') 31 | try: 32 | os.mkdir(output_dir) 33 | except: 34 | pass 35 | for word in data_index: 36 | word_dir = os.path.join(output_dir, word) 37 | try: 38 | os.mkdir(word_dir) 39 | except: 40 | pass 41 | print('Created ' + word_dir) 42 | for speaker in data_index[word]: 43 | for instance in data_index[word][speaker]: 44 | wav_path = data_index[word][speaker][instance] 45 | output_path = os.path.join(word_dir, speaker + '_' + instance + '.wav') 46 | shutil.copyfile(wav_path, output_path) -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | Flask==0.12.1 -------------------------------------------------------------------------------- /static/scripts/app.js: -------------------------------------------------------------------------------- 1 | // --------------------------------------------// 2 | // CHANGE THESE VALUS AND LISTS FOR THE WANTED // 3 | // AND FILLER WORDS YOU WANT TO RECORD // 4 | // --------------------------------------------// 5 | var COUNTS_FOR_WANTED_WORDS = 5; 6 | var COUNTS_FOR_FILLER_WORDS = 0; 7 | var wantedWords = [ 8 | 'Hello', 9 | ]; 10 | var fillerWords = []; 11 | // --------------------------------------------// 12 | 13 | // fork getUserMedia for multiple browser versions, for the future 14 | // when more browsers support MediaRecorder 15 | navigator.getUserMedia = ( navigator.getUserMedia || 16 | navigator.webkitGetUserMedia || 17 | navigator.mozGetUserMedia || 18 | navigator.msGetUserMedia); 19 | 20 | // set up basic variables for app 21 | var record = document.querySelector('.record'); 22 | var stop = document.querySelector('.stop'); 23 | var upload = document.querySelector('.upload'); 24 | var soundClips = document.querySelector('.sound-clips'); 25 | var canvas = document.querySelector('.visualizer'); 26 | var mediaRecorder = null; 27 | var mediaStreamSource = null; 28 | var ignoreAutoPlay = false; 29 | 30 | // disable stop button while not recording 31 | stop.disabled = true; 32 | upload.disabled = false; 33 | 34 | // visualiser setup - create web audio api context and canvas 35 | var audioCtx = new (window.AudioContext || webkitAudioContext)(); 36 | var canvasCtx = canvas.getContext("2d"); 37 | 38 | //main block for doing the audio recording 39 | if (navigator.getUserMedia) { 40 | console.log('getUserMedia supported.'); 41 | 42 | var constraints = { audio: true }; 43 | var chunks = []; 44 | 45 | var onSuccess = function(stream) { 46 | mediaRecorder = new MediaRecorder(stream); 47 | mediaStreamSource = audioCtx.createMediaStreamSource(stream); 48 | record.onclick = function() { 49 | // Initialize the wanted words 50 | COUNTS_FOR_WANTED_WORDS = parseInt(document.getElementById("input_counts").value) 51 | wantedWords = document.getElementById("input_words").value.split(",") 52 | 53 | // Turn on the visualizer 54 | visualize(stream); 55 | 56 | // Display a countdown before recording starts. 57 | var progress = document.querySelector('.progress-display'); 58 | progress.innerText = "3"; 59 | document.querySelector('.info-display').innerText = ""; 60 | setTimeout(function() { 61 | progress.innerText = "2"; 62 | setTimeout(function() { 63 | progress.innerText = "1"; 64 | setTimeout(function() { 65 | progress.innerText = ""; 66 | startRecording(); 67 | }, 1000); 68 | }, 1000); 69 | }, 1000); 70 | stop.disabled = false; 71 | record.disabled = true; 72 | } 73 | 74 | stop.onclick = function() { 75 | if (mediaRecorder.state == 'inactive') { 76 | // The user has already pressed stop, so don't set up another word. 77 | ignoreAutoPlay = true; 78 | } else { 79 | mediaRecorder.stop(); 80 | } 81 | mediaStreamSource.disconnect(); 82 | console.log(mediaRecorder.state); 83 | record.style.background = ""; 84 | record.style.color = ""; 85 | stop.disabled = true; 86 | record.disabled = false; 87 | } 88 | 89 | upload.onclick = function() { 90 | saveRecordings(); 91 | } 92 | 93 | mediaRecorder.onstop = function(e) { 94 | console.log("data available after MediaRecorder.stop() called."); 95 | 96 | var clipName = document.querySelector('.info-display').innerText; 97 | var clipContainer = document.createElement('article'); 98 | var clipLabel = document.createElement('p'); 99 | var audio = document.createElement('audio'); 100 | var deleteButton = document.createElement('button'); 101 | 102 | clipContainer.classList.add('clip'); 103 | clipLabel.classList.add('clip-label'); 104 | audio.setAttribute('controls', ''); 105 | deleteButton.textContent = 'Delete'; 106 | deleteButton.className = 'delete'; 107 | clipLabel.textContent = clipName; 108 | 109 | clipContainer.appendChild(audio); 110 | clipContainer.appendChild(clipLabel); 111 | clipContainer.appendChild(deleteButton); 112 | soundClips.appendChild(clipContainer); 113 | 114 | audio.controls = true; 115 | var blob = new Blob(chunks, { 'type' : 'audio/ogg; codecs=opus' }); 116 | chunks = []; 117 | var audioURL = window.URL.createObjectURL(blob); 118 | audio.src = audioURL; 119 | console.log("recorder stopped"); 120 | 121 | deleteButton.onclick = function(e) { 122 | evtTgt = e.target; 123 | evtTgt.parentNode.parentNode.removeChild(evtTgt.parentNode); 124 | updateProgress(); 125 | } 126 | } 127 | 128 | mediaRecorder.ondataavailable = function(e) { 129 | chunks.push(e.data); 130 | } 131 | } 132 | 133 | var onError = function(err) { 134 | console.log('The following error occured: ' + err); 135 | } 136 | 137 | navigator.getUserMedia(constraints, onSuccess, onError); 138 | } else { 139 | console.log('getUserMedia not supported on your browser!'); 140 | document.querySelector('.info-display').innerText = 141 | 'Your device does not support the HTML5 API needed to record audio (this is a known problem on iOS)'; 142 | } 143 | 144 | function visualize(stream) { 145 | var analyser = audioCtx.createAnalyser(); 146 | analyser.fftSize = 2048; 147 | var bufferLength = analyser.frequencyBinCount; 148 | var dataArray = new Uint8Array(bufferLength); 149 | 150 | mediaStreamSource.connect(analyser); 151 | 152 | WIDTH = canvas.width 153 | HEIGHT = canvas.height; 154 | 155 | draw() 156 | 157 | function draw() { 158 | 159 | requestAnimationFrame(draw); 160 | 161 | analyser.getByteTimeDomainData(dataArray); 162 | 163 | canvasCtx.fillStyle = 'rgb(200, 200, 200)'; 164 | canvasCtx.fillRect(0, 0, WIDTH, HEIGHT); 165 | 166 | canvasCtx.lineWidth = 2; 167 | canvasCtx.strokeStyle = 'rgb(0, 0, 0)'; 168 | 169 | canvasCtx.beginPath(); 170 | 171 | var sliceWidth = WIDTH * 1.0 / bufferLength; 172 | var x = 0; 173 | 174 | for(var i = 0; i < bufferLength; i++) { 175 | 176 | var v = dataArray[i] / 128.0; 177 | var y = v * HEIGHT/2; 178 | 179 | if(i === 0) { 180 | canvasCtx.moveTo(x, y); 181 | } else { 182 | canvasCtx.lineTo(x, y); 183 | } 184 | 185 | x += sliceWidth; 186 | } 187 | 188 | canvasCtx.lineTo(canvas.width, canvas.height/2); 189 | canvasCtx.stroke(); 190 | } 191 | } 192 | 193 | function getRecordedWords() { 194 | var wordElements = document.querySelectorAll('.clip-label'); 195 | var wordCounts = {}; 196 | wordElements.forEach(function(wordElement) { 197 | var word = wordElement.innerText; 198 | if (!wordCounts.hasOwnProperty(word)) { 199 | wordCounts[word] = 0; 200 | } 201 | wordCounts[word] += 1; 202 | }); 203 | return wordCounts; 204 | } 205 | 206 | function getAllWantedWords() { 207 | var wordCounts = {}; 208 | wantedWords.forEach(function(word) { 209 | wordCounts[word] = COUNTS_FOR_WANTED_WORDS; 210 | }); 211 | fillerWords.forEach(function(word) { 212 | wordCounts[word] = COUNTS_FOR_FILLER_WORDS; 213 | }); 214 | return wordCounts; 215 | } 216 | 217 | function getRemainingWords() { 218 | var recordedCounts = getRecordedWords(); 219 | var wantedCounts = getAllWantedWords(); 220 | var remainingCounts = {}; 221 | for (var word in wantedCounts) { 222 | wantedCount = wantedCounts[word]; 223 | var recordedCount; 224 | if (recordedCounts.hasOwnProperty(word)) { 225 | recordedCount = recordedCounts[word]; 226 | } else { 227 | recordedCount = 0; 228 | } 229 | var remainingCount = wantedCount - recordedCount; 230 | if (remainingCount > 0) { 231 | remainingCounts[word] = remainingCount; 232 | } 233 | } 234 | return remainingCounts; 235 | } 236 | 237 | function unrollWordCounts(wordCounts) { 238 | var result = []; 239 | for (var word in wordCounts) { 240 | count = wordCounts[word]; 241 | for (var i = 0; i < count; ++i) { 242 | result.push(word); 243 | } 244 | } 245 | return result; 246 | } 247 | 248 | function shuffleArray(array) { 249 | for (var i = array.length - 1; i > 0; i--) { 250 | var j = Math.floor(Math.random() * (i + 1)); 251 | var temp = array[i]; 252 | array[i] = array[j]; 253 | array[j] = temp; 254 | } 255 | } 256 | 257 | function getNextWord() { 258 | var remainingWords = unrollWordCounts(getRemainingWords()); 259 | if (remainingWords.length == 0) { 260 | return null; 261 | } 262 | shuffleArray(remainingWords); 263 | return remainingWords[0]; 264 | } 265 | 266 | function getProgressDescription() { 267 | var allWords = unrollWordCounts(getAllWantedWords()); 268 | var remainingWords = unrollWordCounts(getRemainingWords()); 269 | return ((allWords.length + 1) - remainingWords.length) + "/" + allWords.length; 270 | } 271 | 272 | function updateProgress() { 273 | var progress = getProgressDescription(); 274 | document.querySelector('.progress-display').innerText = progress; 275 | } 276 | 277 | function startRecording() { 278 | if (ignoreAutoPlay) { 279 | ignoreAutoPlay = false; 280 | return; 281 | } 282 | var word = getNextWord(); 283 | if (word === null) { 284 | promptToSave(); 285 | return; 286 | } 287 | updateProgress(); 288 | document.querySelector('.info-display').innerText = word; 289 | mediaRecorder.start(); 290 | console.log(mediaRecorder.state); 291 | console.log("recorder started"); 292 | record.style.background = "red"; 293 | setTimeout(endRecording, 1500); 294 | } 295 | 296 | function endRecording() { 297 | if (mediaRecorder.state == 'inactive') { 298 | // The user has already pressed stop, so don't set up another word. 299 | return; 300 | } 301 | mediaRecorder.stop(); 302 | console.log(mediaRecorder.state); 303 | console.log("recorder stopped"); 304 | record.style.background = ""; 305 | record.style.color = ""; 306 | setTimeout(startRecording, 1000); 307 | } 308 | 309 | function promptToSave() { 310 | if (confirm('Are you ready to download your data?\nIf not, press cancel now,' + 311 | ' and then press Download once you are ready.')) { 312 | saveRecordings(); 313 | } 314 | upload.disabled = false; 315 | } 316 | 317 | var allClips; 318 | var clipIndex; 319 | 320 | function saveRecordings() { 321 | mediaStreamSource.disconnect(); 322 | allClips = document.querySelectorAll('.clip'); 323 | clipIndex = 0; 324 | uploadNextClip(); 325 | } 326 | 327 | function uploadNextClip() { 328 | document.querySelector('.progress-display').innerText = 'Downloading clip ' + 329 | clipIndex + '/' + unrollWordCounts(getAllWantedWords()).length; 330 | var clip = allClips[clipIndex]; 331 | clip.style.display = 'None'; 332 | var audioBlobUrl = clip.querySelector('audio').src; 333 | var word = clip.querySelector('p').innerText; 334 | var xhr = new XMLHttpRequest(); 335 | xhr.open('GET', audioBlobUrl, true); 336 | xhr.responseType = 'blob'; 337 | xhr.onload = function(e) { 338 | if (this.status == 200) { 339 | var blob = this.response; 340 | var ajaxRequest = new XMLHttpRequest(); 341 | var uploadUrl = '/upload?word=' + word + '&_csrf_token=' + csrf_token; 342 | ajaxRequest.open('POST', uploadUrl, true); 343 | ajaxRequest.setRequestHeader('Content-Type', 'application/json'); 344 | ajaxRequest.onreadystatechange = function() { 345 | if (ajaxRequest.readyState == 4) { 346 | if (ajaxRequest.status === 200) { 347 | clipIndex += 1; 348 | if (clipIndex < allClips.length) { 349 | uploadNextClip(); 350 | } else { 351 | allDone(); 352 | } 353 | } else { 354 | alert('Downloading failed with error code ' + ajaxRequest.status); 355 | } 356 | } 357 | }; 358 | ajaxRequest.send(blob); 359 | } 360 | }; 361 | xhr.send(); 362 | } 363 | 364 | function allDone() { 365 | document.cookie = 'all_done=true; path=/'; 366 | location.reload(true); 367 | } 368 | -------------------------------------------------------------------------------- /static/styles/app.css: -------------------------------------------------------------------------------- 1 | * { 2 | margin: 0; 3 | padding: 0; 4 | box-sizing: border-box; 5 | } 6 | 7 | html, body { 8 | height: 100%; 9 | } 10 | 11 | body { 12 | font-family: "Roboto Mono", monospace; 13 | font-size: 0.8rem; 14 | } 15 | 16 | .wrapper { 17 | height: 100%; 18 | display: flex; 19 | flex-direction: column; 20 | } 21 | 22 | h1, h2 { 23 | font-size: 2rem; 24 | text-align: center; 25 | font-weight: normal; 26 | padding: 0.5rem 0 0 0; 27 | } 28 | 29 | .main-controls { 30 | padding: 0.5rem 0; 31 | } 32 | 33 | .top-display { 34 | font-size: 1.5rem; 35 | text-align: center; 36 | padding: 3.0rem 0.5rem 2.0rem 0.5rem; 37 | } 38 | 39 | .directions { 40 | font-size: 1.0rem; 41 | } 42 | 43 | .welcome-display { 44 | font-size: 1rem; 45 | text-align: center; 46 | padding: 0.5rem 0.5rem 0.5rem 0.5rem; 47 | } 48 | 49 | .legal { 50 | font-size: 1.0rem; 51 | text-align: left; 52 | padding: 0.5rem 0.5rem 0.5rem 0.5rem; 53 | font-family: Arial; 54 | } 55 | 56 | canvas { 57 | width: 100%; 58 | height: 60px; 59 | display: block; 60 | margin-bottom: 0.5rem; 61 | } 62 | 63 | #buttons { 64 | display: flex; 65 | flex-direction: row; 66 | justify-content: space-between; 67 | } 68 | 69 | #buttons button { 70 | font-size: 1rem; 71 | padding: 1rem; 72 | width: calc(33% - 0.25rem); 73 | } 74 | 75 | button { 76 | font-size: 1rem; 77 | background: #0088cc; 78 | text-align: center; 79 | color: white; 80 | border: none; 81 | transition: all 0.2s; 82 | padding: 0.5rem; 83 | } 84 | 85 | button:hover:enabled, button:focus:enabled { 86 | box-shadow: inset 0px 0px 10px rgba(255, 255, 255, 1); 87 | background: #0ae; 88 | } 89 | 90 | button:active:enabled { 91 | box-shadow: inset 0px 0px 20px rgba(0,0,0,0.5); 92 | transform: translateY(2px); 93 | } 94 | 95 | button:disabled { 96 | background: #888888; 97 | } 98 | 99 | .continue-link { 100 | width:33%; 101 | margin-left: 33%; 102 | } 103 | 104 | #buttons .continue { 105 | width: 100%; 106 | } 107 | 108 | /* Make the clips use as much space as possible, and 109 | * also show a scrollbar when there are too many clips to show 110 | * in the available space */ 111 | .sound-clips { 112 | flex: 1; 113 | overflow: auto; 114 | } 115 | 116 | section, article { 117 | display: block; 118 | } 119 | 120 | .clip { 121 | padding-bottom: 1rem; 122 | } 123 | 124 | audio { 125 | width: 100%; 126 | display: block; 127 | margin: 1rem auto 0.5rem; 128 | } 129 | 130 | .clip p { 131 | display: inline-block; 132 | font-size: 1rem; 133 | } 134 | 135 | .clip button { 136 | font-size: 1rem; 137 | float: right; 138 | } 139 | 140 | button.delete { 141 | background: #f00; 142 | padding: 0.5rem 0.75rem; 143 | font-size: 0.8rem; 144 | } 145 | 146 | /* Checkbox hack to control information box display */ 147 | 148 | label { 149 | font-size: 3rem; 150 | position: absolute; 151 | top: 2px; 152 | right: 3px; 153 | z-index: 5; 154 | cursor: pointer; 155 | } 156 | 157 | input[type=checkbox] { 158 | position: absolute; 159 | top: -100px; 160 | } 161 | 162 | aside { 163 | position: fixed; 164 | top: 0; 165 | left: 0; 166 | width: 100%; 167 | height: 100%; 168 | transform: translateX(100%); 169 | transition: 0.3s all ease-out; 170 | background-color: #efefef; 171 | padding: 1rem; 172 | } 173 | 174 | aside p { 175 | font-size: 1.2rem; 176 | margin: 0.5rem 0; 177 | } 178 | 179 | aside a { 180 | color: #666; 181 | } 182 | 183 | /* Toggled State of information box */ 184 | input[type=checkbox]:checked ~ aside { 185 | transform: translateX(0); 186 | } 187 | 188 | /* Cursor when clip name is clicked over */ 189 | 190 | .clip p { 191 | cursor: pointer; 192 | } 193 | 194 | /* Adjustments for wider screens */ 195 | @media all and (min-width: 500px) { 196 | /* Don't take all the space as readability is lost when line length 197 | goes past a certain size */ 198 | .wrapper { 199 | width: 90%; 200 | max-width: 700px; 201 | margin: 0 auto; 202 | } 203 | } 204 | 205 | footer { 206 | font-family: Arial; 207 | color: #222; 208 | background-color: rgb(242, 242, 242); 209 | width: 100%; 210 | bottom: 0; 211 | position: fixed; 212 | padding-left: 30px; 213 | } 214 | 215 | footer a { 216 | margin-left: 20px; 217 | } 218 | 219 | .info-display { 220 | border-style: solid; 221 | } 222 | -------------------------------------------------------------------------------- /templates/record.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | Open Speech Recording 7 | 8 | 9 | 12 | 13 | 14 |
15 | 16 |
17 |

Open Speech Recording

18 |
19 | 20 |
21 |
22 |
    23 |
  1. 24 | Input the words you would like to record as a comma separated list: 25 |
    26 | 27 |
    2. 28 |
    29 |
  2. 30 | Input the number of samples of each word you would like to record: 31 |
    32 | 33 |
    3. 34 |
    35 |
  3. 36 | Press Record to start recording. And record only the word show in the box below when instructed (the record button will switch between red and grey to indicate each new recording and will randomly select words from your list to record). 37 |
    4. 38 |
    39 |
  4. 40 | Press Stop to stop recording, and when all the requested words are completed you'll be able to download the recordings. 41 |
    5. 42 |
    43 | 44 |
45 |
46 |
47 |
48 |
49 |
50 | 51 |
52 | 53 |
54 | 55 | 56 | 57 |
58 |
59 | 60 |
61 | 62 | 71 | 72 |
73 | 74 |
75 | 76 | 77 | Fork me on GitHub 78 | 79 | 80 | 81 | 82 | 83 | 84 | -------------------------------------------------------------------------------- /templates/welcome.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | Open Speech Recording 7 | 8 | 9 | 10 | 11 |
12 | 13 |
14 |

Open Speech Recording

15 |
16 | 17 |
18 |
19 | Welcome to the TinyML Open Speech Recording App. 20 |
21 | Please use this app in a Chrome Incognito Window. We find that sometimes there is odd cacheing behavior and forcibly reloading it in a new Incognito Window tends to fix any issues. 22 |
23 |
24 | 29 |
30 | 31 |
32 | 33 | 34 | Fork me on GitHub 35 | 36 | 37 | 38 | 39 | --------------------------------------------------------------------------------