├── .gitattributes ├── Dockerfile ├── requirements.txt ├── LICENSE ├── transcripts └── lSQ4vtkwW1Y ├── README.md ├── .gitignore └── YoutubeTranscriber.py /.gitattributes: -------------------------------------------------------------------------------- 1 | # Auto detect text files and perform LF normalization 2 | * text=auto 3 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.7.9-alpine3.12 2 | ADD . /app 3 | WORKDIR /app 4 | RUN apk add youtube-dl ffmpeg flac 5 | RUN pip install -r requirements.txt 6 | ENTRYPOINT ["python","./YoutubeTranscriber.py"] -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | google-api-python-client==1.6.4 2 | httplib2==0.10.3 3 | oauth2client==4.1.2 4 | pyasn1==0.4.2 5 | pyasn1-modules==0.2.1 6 | rsa==3.4.2 7 | six==1.11.0 8 | SpeechRecognition==3.8.1 9 | tqdm==4.19.5 10 | uritemplate==3.0.0 11 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Çalgan Aygün 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /transcripts/lSQ4vtkwW1Y: -------------------------------------------------------------------------------- 1 | 00:00:00 örgütün izin lideri kim kaç kişi öldürdüm vesaire bir gün dedim Itır gizli örgütün çok zenginiz Aslında Sürekli insan öldürüyoruz bahsettim sonra dedim bir gün dedim gel dedim Şahane ev yemeklerimiz var dedim ama neler neler Yani bütün paramı lafın korkuyorum onun gibi saçma sapan kuyruk bir arkadaşı var Ona söylemiş o gel 2 | 00:00:30 ama altıma sıçacağım yani korkudan kötü davrandım kovulduk azalıp kağıdın restaurantlar öyle kurtulduk yani bakın benim parolam sevgidir ama ben şimdi de Yerine göre bir enstrüman olarak kullanırım bu kız bunu görsün istedim kız için yani anlatabiliyor muyum şimdi açık olalım karşılıklı bu sorgularmı olgulardan varmazlar bilmem neler Bunların hepsi artistliği misalen söylüyorum Bakın ben eğer işbirlikçi olmak İstemezse 3 | 00:01:00 ne kullanırsa kullansın benim ağzımdan tek bir kelime laf alabilir mi aramaz işte burada karşılıklı bir ilişki var Bana demiyor ki serbest çift bankada 21 milyon dolarım var Teşekkür ederim sağ olun ya benim canım geçen akşam beni nasıl daha alamadım ya benim canım geçen akşam meyve suyu istedi alamadım Ya kusura bakmayın ama ben hep ben anasını sikeyim ya öyle çok gördün Ben anasını sikeyim Ne iş yapıyorum ben 4 | 00:01:30 iş yapıyorum Ben adamı öldürüyorum Adam cebinde 24 lira Ben ne yapıyorum adamı öldürüyorum yapılan Show nedir kardeşim Ne yapıyorsunuz siz benim babamın kutu kola gibidir fatura örgüt olarak beni karşınıza Allah gafletine düşmeyin bakın Yemin ediyorum Yemin ediyorum sakın sakın öyle bir hata yapmayın patlama hazırım 5 | 00:02:00 Bıçak Kemiğe Dayandı her gece yeminler ediyorum 6 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # YoutubeTranscriber 2 | A Python script to search strings in YouTube videos. 3 | 4 | Uses Google Cloud Speech-to-Text API to generate transcripts. You can use Google Cloud Free Tier credits. 5 | 6 | 7 | ## How to use? 8 | 9 | 1. Clone this repo. 10 | 2. Sign-in to [GCP](https://console.cloud.google.com/). 11 | 3. Go to [Speech-to-Text API](https://console.cloud.google.com/apis/api/speech.googleapis.com/overview) 12 | select project and enable this API. 13 | 1. Click "Credentials". 14 | 2. Click "Create Credentials". 15 | 3. Select "Service Account Key". 16 | 4. Under "Service Account" select "New service account". 17 | 5. Name service. 18 | 6. Select Role: "Project" -> "Owner". 19 | 7. Finish creating credential. 20 | 8. Select your "Service Account" from list. 21 | 9. Click "Add Key" button and select "Create New Key". 22 | 10. Leave "JSON" option selected. 23 | 11. Click "Create". 24 | 12. Save generated API key file to repo's main directory. 25 | 13. Rename file to "api-key.json" or, specify `GC_CREDENTIAL` env variable with your json file name 26 | while running docker image. 27 | 28 | > This project uses parallel processing. You can use `NUM_OF_THREADS` env variable to specify number of 29 | concurrent threads while running docker image. Unless you change program use *8* threads. 30 | 31 | ``` bash 32 | # Build Docker image 33 | docker build . -t calganaygun/youtube-transcriber:latest 34 | 35 | # Run program and search or generate text 36 | docker run -it calganaygun/youtube-transcriber:latest -v \ 37 | -w \ 38 | -l 39 | ```` 40 | 41 | ## Examples 42 | 43 | [![asciicast](https://asciinema.org/a/2JU2fJgAkEia8P6C9LYCyUcyO.svg)](https://asciinema.org/a/2JU2fJgAkEia8P6C9LYCyUcyO) 44 | 45 | [![asciicast](https://asciinema.org/a/jC43dK5YmvSRMlzv6imSrQ7ZT.svg)](https://asciinema.org/a/jC43dK5YmvSRMlzv6imSrQ7ZT) 46 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | pip-wheel-metadata/ 24 | share/python-wheels/ 25 | *.egg-info/ 26 | .installed.cfg 27 | *.egg 28 | MANIFEST 29 | 30 | # PyInstaller 31 | # Usually these files are written by a python script from a template 32 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 33 | *.manifest 34 | *.spec 35 | 36 | # Installer logs 37 | pip-log.txt 38 | pip-delete-this-directory.txt 39 | 40 | # Unit test / coverage reports 41 | htmlcov/ 42 | .tox/ 43 | .nox/ 44 | .coverage 45 | .coverage.* 46 | .cache 47 | nosetests.xml 48 | coverage.xml 49 | *.cover 50 | *.py,cover 51 | .hypothesis/ 52 | .pytest_cache/ 53 | 54 | # Translations 55 | *.mo 56 | *.pot 57 | 58 | # Django stuff: 59 | *.log 60 | local_settings.py 61 | db.sqlite3 62 | db.sqlite3-journal 63 | 64 | # Flask stuff: 65 | instance/ 66 | .webassets-cache 67 | 68 | # Scrapy stuff: 69 | .scrapy 70 | 71 | # Sphinx documentation 72 | docs/_build/ 73 | 74 | # PyBuilder 75 | target/ 76 | 77 | # Jupyter Notebook 78 | .ipynb_checkpoints 79 | 80 | # IPython 81 | profile_default/ 82 | ipython_config.py 83 | 84 | # pyenv 85 | .python-version 86 | 87 | # pipenv 88 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 89 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 90 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 91 | # install all needed dependencies. 92 | #Pipfile.lock 93 | 94 | # celery beat schedule file 95 | celerybeat-schedule 96 | 97 | # SageMath parsed files 98 | *.sage.py 99 | 100 | # Environments 101 | .env 102 | .venv 103 | env/ 104 | venv/ 105 | ENV/ 106 | env.bak/ 107 | venv.bak/ 108 | 109 | # Spyder project settings 110 | .spyderproject 111 | .spyproject 112 | 113 | # Rope project settings 114 | .ropeproject 115 | 116 | # mkdocs documentation 117 | /site 118 | 119 | # mypy 120 | .mypy_cache/ 121 | .dmypy.json 122 | dmypy.json 123 | 124 | # Pyre type checker 125 | .pyre/ 126 | 127 | #VSCode 128 | .vscode/ 129 | 130 | #Credentials 131 | api-key.json -------------------------------------------------------------------------------- /YoutubeTranscriber.py: -------------------------------------------------------------------------------- 1 | import getopt 2 | import os 3 | import shutil 4 | import subprocess 5 | import sys 6 | from multiprocessing.dummy import Pool 7 | 8 | import speech_recognition as sr 9 | from tqdm import tqdm 10 | 11 | numOfThreads = int(os.getenv('NUM_OF_THREADS', '8')) 12 | credentialsGC = os.getenv('GC_CREDENTIAL', 'api-key.json') 13 | 14 | with open(credentialsGC) as f: 15 | GOOGLE_CLOUD_SPEECH_CREDENTIALS = f.read() 16 | 17 | r = sr.Recognizer() 18 | 19 | 20 | def getWavFile(id): 21 | # Run youte-dl to download Youtube video and translate to WAV 22 | print("Video is downloading...") 23 | os.system('youtube-dl -f bestaudio --extract-audio --audio-format wav --audio-quality 3 -o ./temp/{id}.webm https://www.youtube.com/watch?v={id} > /dev/null'.format(id=id)) 24 | 25 | 26 | def splitWaw(id): 27 | try: 28 | os.mkdir('temp/parts-' + id) 29 | except: 30 | pass 31 | # Run ffmpeg to split WAV file 32 | # This command generates 30 second pieces of WAV file 33 | print("Sound file splitting...") 34 | os.system('ffmpeg -i temp/{id}.wav -f segment -segment_time 30 -c copy ./temp/parts-{id}/out%06d.wav 2> /dev/null'.format(id=id)) 35 | 36 | 37 | def prepareTranscript(id, language): 38 | global langOfVideo 39 | langOfVideo = language 40 | 41 | # If the transcript of the video is already created, directly returns saved transcript 42 | if id in os.listdir('./transcripts'): 43 | with open('./transcripts/' + id, 'r', encoding='utf-8') as transcriptFile: 44 | return transcriptFile.read() 45 | 46 | getWavFile(id) 47 | splitWaw(id) 48 | files = sorted(os.listdir('temp/parts-' + id)) 49 | global idofVideo 50 | idofVideo = id 51 | transcript = transcribeFiles(files) 52 | 53 | # Save generated transcript for next uses 54 | with open('transcripts/' + id, 'w', encoding='utf-8') as transcriptFile: 55 | transcriptFile.write(transcript) 56 | shutil.rmtree('./temp/parts-' + id) 57 | os.remove('./temp/'+id+'.wav') 58 | 59 | return transcript 60 | 61 | 62 | def transcribe(data): 63 | try: 64 | global idofVideo 65 | idx, file = data 66 | name = 'temp/parts-'+idofVideo+'/' + file 67 | # Load audio file 68 | with sr.AudioFile(name) as source: 69 | audio = r.record(source) 70 | # Transcribe audio file 71 | text = r.recognize_google_cloud( 72 | audio, credentials_json=GOOGLE_CLOUD_SPEECH_CREDENTIALS, language=langOfVideo) 73 | return { 74 | 'idx': idx, 75 | 'text': text 76 | } 77 | except sr.UnknownValueError: # Google Cloud STT API gives error if can not detect speech in the audio 78 | return { 79 | 'idx': idx, 80 | 'text': '' 81 | } 82 | 83 | 84 | def transcribeFiles(files): 85 | print("Sound files is processing on Google Cloud STT API...") 86 | pool = Pool(numOfThreads) # Number of concurrent threads 87 | all_text = pool.map(transcribe, enumerate(files)) 88 | pool.close() 89 | pool.join() 90 | 91 | transcript = '' 92 | for t in sorted(all_text, key=lambda x: x['idx']): 93 | total_seconds = t['idx'] * 30 94 | # Cool shortcut from: 95 | # https://stackoverflow.com/questions/775049/python-time-seconds-to-hms 96 | # to get hours, minutes and seconds 97 | m, s = divmod(total_seconds, 60) 98 | h, m = divmod(m, 60) 99 | 100 | # Format time as h:m:s - 30 seconds of text 101 | transcript = transcript + \ 102 | '{:0>2d}:{:0>2d}:{:0>2d} {}\n'.format(h, m, s, t['text']) 103 | return transcript 104 | 105 | 106 | def main(argv): 107 | videoID = 'lSQ4vtkwW1Y' 108 | word = 'getAll' 109 | language = 'tr-TR' 110 | try: 111 | opts, _ = getopt.getopt(argv, 'hv:w:l:', ['video=', 'word=']) 112 | except getopt.GetoptError: 113 | print( 114 | 'YoutubeTranscriber.py -v -w -l ') 115 | sys.exit(2) 116 | 117 | for opt, arg in opts: 118 | if opt == '-h': 119 | print( 120 | 'YoutubeTranscriber.py -v -w -l ') 121 | sys.exit() 122 | elif opt in ('-v', '--video'): 123 | videoID = arg 124 | elif opt in ('-w', '--word'): 125 | word = arg 126 | elif opt in ('-l', '--language'): 127 | language = arg 128 | 129 | if videoID != '': 130 | if word == 'getAll': 131 | print(prepareTranscript(videoID, language)) 132 | elif word != '': 133 | count = 0 134 | for i in prepareTranscript(videoID, language).split('\n'): 135 | if word.lower() in i.lower(): 136 | print(i) 137 | count += 1 138 | print('Transcript searched and {} result(s) found.'.format(count)) 139 | else: 140 | print( 141 | 'YoutubeTranscriber.py -v -w -l ') 142 | sys.exit(2) 143 | else: 144 | print( 145 | 'YoutubeTranscriber.py -v -w -l ') 146 | sys.exit(2) 147 | 148 | 149 | if __name__ == '__main__': 150 | main(sys.argv[1:]) --------------------------------------------------------------------------------