233 | 
6 |
7 | *Transcribe, translate, annotate and subtitle audio and video files with OpenAI's [Whisper](https://github.com/openai/whisper) ... fast!*
8 |
9 | `whisply` combines [faster-whisper](https://github.com/SYSTRAN/faster-whisper) and [insanely-fast-whisper](https://github.com/Vaibhavs10/insanely-fast-whisper) to offer an easy-to-use solution for batch processing files on Windows, Linux and Mac. It also enables word-level speaker annotation by integrating [whisperX](https://github.com/m-bain/whisperX) and [pyannote](https://github.com/pyannote/pyannote-audio).
10 |
11 | ## Table of contents
12 |
13 | - [Features](#features)
14 | - [Requirements](#requirements)
15 | - [Installation](#installation)
16 | - [Install `ffmpeg`](#install-ffmpeg)
17 | - [Installation with `pip`](#installation-with-pip)
18 | - [Installation from `source`](#installation-from-source)
19 | - [Nvidia GPU fix for Linux users (March 2025)](#nvidia-gpu-fix-for-linux-users-march-2025)
20 | - [Usage](#usage)
21 | - [CLI](#cli)
22 | - [App](#app)
23 | - [Speaker annotation and diarization](#speaker-annotation-and-diarization)
24 | - [Requirements](#requirements-1)
25 | - [How speaker annotation works](#how-speaker-annotation-works)
26 | - [Post correction](#post-correction)
27 | - [Batch processing](#batch-processing)
28 | - [Using config files for batch processing](#using-config-files-for-batch-processing)
29 | - [Citation](#citation)
30 |
31 | ## Features
32 |
33 | * 🚴♂️ **Performance**: `whisply` selects the fastest Whisper implementation based on your hardware:
34 | * CPU/GPU (Nvidia CUDA): `fast-whisper` or `whisperX`
35 | * MPS (Apple M1-M4): `insanely-fast-whisper`
36 |
37 | * ⏩ **large-v3-turbo Ready**: Support for [whisper-large-v3-turbo](https://huggingface.co/openai/whisper-large-v3-turbo) on all devices. **Note**: Subtitling and annotations on CPU/GPU use `whisperX` for accurate timestamps, but `whisper-large-v3-turbo` isn’t currently available for `whisperX`.
38 |
39 | * ✅ **Auto Device Selection**: `whisply` automatically chooses `faster-whisper` (CPU) or `insanely-fast-whisper` (MPS, Nvidia GPUs) for transcription and translation unless a specific `--device` option is passed.
40 |
41 | * 🗣️ **Word-level Annotations**: Enabling `--subtitle` or `--annotate` uses `whisperX` or `insanely-fast-whisper` for word segmentation and speaker annotations. `whisply` approximates missing timestamps for numeric words.
42 |
43 | * 💬 **Customizable Subtitles**: Specify words per subtitle block (e.g., "5") to generate `.srt` and `.webvtt` files with fixed word counts and timestamps.
44 |
45 | * 🧺 **Batch Processing**: Handle single files, folders, URLs, or lists via `.list` documents. See the [Batch processing](#batch-processing) section for details.
46 |
47 | * 👩💻 **CLI / App**: `whisply` can be run directly from CLI or as an app with a graphical user-interface (GUI).
48 |
49 | * ⚙️ **Export Formats**: Supports `.json`, `.txt`, `.txt (annotated)`, `.srt`, `.webvtt`, `.vtt`, `.rttm` and `.html` (compatible with [noScribe's editor](https://github.com/kaixxx/noScribe)).
50 |
51 | ## Requirements
52 |
53 | * [FFmpeg](https://ffmpeg.org/)
54 | * \>= Python3.10
55 | * GPU processing requires:
56 | * Nvidia GPU (CUDA: cuBLAS and cuDNN for CUDA 12)
57 | * Apple Metal Performance Shaders (MPS) (Mac M1-M4)
58 | * Speaker annotation requires a [HuggingFace Access Token](https://huggingface.co/docs/hub/security-tokens)
59 |
60 | ## Installation
61 |
62 | ### Install `ffmpeg`
63 |
64 | ```shell
65 | # --- macOS ---
66 | brew install ffmpeg
67 |
68 | # --- Linux ---
69 | sudo apt-get update
70 | sudo apt-get install ffmpeg
71 |
72 | # --- Windows ---
73 | winget install Gyan.FFmpeg
74 | ```
75 |
76 | For more information you can visit the [FFmpeg website](https://ffmpeg.org/download.html).
77 |
78 | ### Installation with `pip`
79 |
80 | 1. Create a Python virtual environment
81 |
82 | ```shell
83 | python3 -m venv venv
84 | ```
85 |
86 | 2. Activate the environment
87 |
88 | ```shell
89 | # --- Linux & macOS ---
90 | source venv/bin/activate
91 |
92 | # --- Windows ---
93 | venv\Scripts\activate
94 | ```
95 |
96 | 3. Install whisply
97 |
98 | ```shell
99 | pip install whisply
100 | ```
101 |
102 | ### Installation from `source`
103 |
104 | 1. Clone this repository
105 |
106 | ```shell
107 | git clone https://github.com/tsmdt/whisply.git
108 | ```
109 |
110 | 2. Change to project folder
111 |
112 | ```shell
113 | cd whisply
114 | ```
115 |
116 | 3. Create a Python virtual environment
117 |
118 | ```shell
119 | python3 -m venv venv
120 | ```
121 |
122 | 4. Activate the Python virtual environment
123 |
124 | ```shell
125 | # --- Linux & macOS ---
126 | source venv/bin/activate
127 |
128 | # --- Windows ---
129 | venv\Scripts\activate
130 | ```
131 |
132 | 5. Install whisply
133 |
134 | ```shell
135 | pip install .
136 | ```
137 |
138 | ### Nvidia GPU fix (May 2025)
139 |
232 |
233 |
{transcription}
17 |Transcribed with whisply for noScribe's Editor.
Audio source: {audio_filepath}
' 581 | f'[{segment_start}] {speaker}: {text}' 582 | f'
\n') 583 | else: 584 | segment_html = f'{text}
\n' 585 | 586 | body_content += segment_html 587 | 588 | html_output = html_template.replace('{body_content}', body_content) 589 | 590 | return html_output 591 | -------------------------------------------------------------------------------- /whisply/post_correction.py: -------------------------------------------------------------------------------- 1 | import yaml 2 | from pathlib import Path 3 | from dataclasses import dataclass, field 4 | from typing import Dict, List 5 | 6 | @dataclass 7 | class Corrections: 8 | """ 9 | A dataclass to encapsulate both simple and pattern-based corrections. 10 | """ 11 | simple: Dict[str, str] = field(default_factory=dict) 12 | patterns: List[Dict[str, str]] = field(default_factory=list) 13 | 14 | 15 | def load_correction_list(filepath: str | Path) -> Corrections: 16 | """ 17 | Load the correction dictionary and patterns from a YAML file. 18 | 19 | :param filepath: Path to the YAML correction file. 20 | :return: Corrections object containing simple and pattern-based 21 | corrections. 22 | """ 23 | try: 24 | with open(filepath, 'r') as file: 25 | data = yaml.safe_load(file) 26 | 27 | if not isinstance(data, dict): 28 | raise ValueError("→ Correction file must contain a YAML dictionary.") 29 | 30 | # Extract simple corrections 31 | simple_corrections = {k: v for k, v in data.items() if k != 'patterns'} 32 | 33 | # Extract pattern-based corrections 34 | pattern_corrections = data.get('patterns', []) 35 | 36 | # Validate patterns 37 | for entry in pattern_corrections: 38 | if 'pattern' not in entry or 'replacement' not in entry: 39 | raise ValueError("→ Each pattern entry must contain 'pattern' \ 40 | and 'replacement' keys.") 41 | 42 | return Corrections( 43 | simple=simple_corrections, 44 | patterns=pattern_corrections 45 | ) 46 | 47 | except FileNotFoundError: 48 | print(f"→ Correction file not found: {filepath}") 49 | return Corrections() 50 | except yaml.YAMLError as e: 51 | print(f"→ Error parsing YAML file: {e}") 52 | return Corrections() 53 | except Exception as e: 54 | print(f"→ Unexpected error loading correction list: {e}") 55 | return Corrections() 56 | -------------------------------------------------------------------------------- /whisply/transcription.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import time 3 | from pathlib import Path 4 | from datetime import datetime 5 | from functools import partial 6 | from rich import print 7 | import whisperx.diarize 8 | 9 | from whisply import little_helper, output_utils, models 10 | from whisply.little_helper import FilePathProcessor 11 | from whisply.post_correction import Corrections 12 | 13 | # Set logging configuration 14 | log_dir = little_helper.ensure_dir(Path('./logs')) 15 | log_filename = f"log_whisply_{datetime.now().strftime('%Y-%m-%d')}.log" 16 | log_file = f"{log_dir}/{log_filename}" 17 | 18 | logging.basicConfig( 19 | filename=log_file, 20 | level=logging.INFO, 21 | format="%(asctime)s %(levelname)s [%(funcName)s]: %(message)s", 22 | ) 23 | 24 | class TranscriptionHandler: 25 | """ 26 | Handles transcription and diarization of audio/video files using various 27 | Whisper-based models. 28 | 29 | This class leverages different implementations of OpenAI's Whisper models 30 | (whisperX, insanely-fast-whisper, faster-whisper) to transcribe audio and 31 | video files. It supports features like language detection, speaker 32 | diarization, translation, subtitle generation, and exporting transcriptions 33 | in multiple formats. It is capable of processing single files, directories, 34 | URLs, and lists of files, providing flexibility for diverse transcription 35 | needs. 36 | 37 | Args: 38 | base_dir (str, optional): Directory to store transcription outputs. 39 | Defaults to './transcriptions'. 40 | model (str, optional): Whisper model variant to use (e.g., 'large-v2'). 41 | Defaults to 'large-v3-turbo'. 42 | device (str, optional): Compute device ('cpu', 'cuda', etc.). 43 | Defaults to 'cpu'. 44 | file_language (str, optional): Language of the input audio. 45 | If not provided, language detection is performed. 46 | annotate (bool, optional): Enable speaker diarization. 47 | Defaults to False. 48 | hf_token (str, optional): Hugging Face token for accessing restricted 49 | models or features. 50 | subtitle (bool, optional): Generate subtitles with word-level timestamps. 51 | Defaults to False. 52 | sub_length (int, optional): Maximum number of words per subtitle chunk. 53 | Required if subtitle is True. 54 | translate (bool, optional): Translate transcription to English if the 55 | original language is different. Defaults to False. 56 | verbose (bool, optional): Enable detailed logging and output. 57 | Defaults to False. 58 | export_formats (str or list, optional): Formats to export transcriptions 59 | (e.g., 'json', 'srt'). Defaults to 'all'. 60 | 61 | Attributes: 62 | base_dir (Path): Directory for storing transcriptions. 63 | device (str): Compute device in use. 64 | file_language (str or None): Detected or specified language of the 65 | audio. 66 | annotate (bool): Indicates if speaker diarization is enabled. 67 | translate (bool): Indicates if translation is enabled. 68 | subtitle (bool): Indicates if subtitle generation is enabled. 69 | verbose (bool): Indicates if verbose mode is active. 70 | export_formats (str or list): Selected formats for exporting 71 | transcriptions. 72 | processed_files (list): List of processed file information and results. 73 | 74 | Methods: 75 | get_filepaths(filepath: str): 76 | Retrieves and validates file paths from various input types. 77 | 78 | detect_language(file: Path, audio_array) -> str: 79 | Detects the language of the given audio file. 80 | 81 | process_files(files: list): 82 | Processes a list of audio files for transcription and diarization. 83 | 84 | transcribe_with_whisperx(filepath: Path) -> dict: 85 | Transcribes an audio file using the whisperX implementation. 86 | 87 | transcribe_with_insane_whisper(filepath: Path) -> dict: 88 | Transcribes an audio file using the insanely-fast-whisper 89 | implementation. 90 | 91 | transcribe_with_faster_whisper(filepath: Path, num_workers: int = 1) 92 | -> dict: 93 | Transcribes an audio file using the faster-whisper implementation. 94 | 95 | adjust_word_chunk_length(result: dict) -> dict: 96 | Splits transcription text into chunks based on a maximum word 97 | count. 98 | 99 | to_transcription_dict(insanely_annotation: list[dict]) -> dict: 100 | Converts speaker-annotated results into a standardized dictionary. 101 | 102 | to_whisperx(transcription_result: dict) -> dict: 103 | Normalizes transcription results to the whisperX format. 104 | 105 | create_text_with_speakers(transcription_dict: dict, 106 | delimiter: str = '.') -> dict: 107 | Inserts speaker labels into the transcription text upon speaker 108 | changes. 109 | """ 110 | def __init__( 111 | self, 112 | base_dir='./transcriptions', 113 | model='large-v3-turbo', 114 | device='cpu', 115 | file_language=None, 116 | annotate=False, 117 | num_speakers=None, 118 | hf_token=None, 119 | subtitle=False, 120 | sub_length=None, 121 | translate=False, 122 | verbose=False, 123 | del_originals=False, 124 | corrections=Corrections, 125 | export_formats='all' 126 | ): 127 | self.base_dir = little_helper.ensure_dir(Path(base_dir)) 128 | self.file_formats = little_helper.return_valid_fileformats() 129 | self.device = device 130 | self.file_language = file_language 131 | self.file_language_provided = file_language is not None 132 | self.model = None 133 | self.model_provided = model 134 | self.annotate = annotate 135 | self.num_speakers = num_speakers 136 | self.translate = translate 137 | self.hf_token = hf_token 138 | self.subtitle = subtitle 139 | self.sub_length = sub_length 140 | self.verbose = verbose 141 | self.del_originals = del_originals 142 | self.corrections = corrections 143 | self.export_formats = export_formats 144 | self.metadata = self._collect_metadata() 145 | self.filepaths = [] 146 | self.output_dir = None 147 | self.processed_files = [] 148 | 149 | def _collect_metadata(self): 150 | return { 151 | 'output_dir': str(self.base_dir), 152 | 'file_language': self.file_language, 153 | 'model': self.model_provided, 154 | 'device': self.device, 155 | 'annotate': self.annotate, 156 | 'num_speakers': self.num_speakers, 157 | 'translate': self.translate, 158 | 'subtitle': self.subtitle, 159 | 'sub_length': self.sub_length 160 | } 161 | 162 | def adjust_word_chunk_length(self, result: dict) -> dict: 163 | """ 164 | Generates text chunks based on the maximum number of words. 165 | 166 | Parameters: 167 | result (dict): The nested dictionary containing segments 168 | and words. 169 | max_number (int): The maximum number of words per chunk. 170 | Default is 6. 171 | 172 | Returns: 173 | dict: A dictionary containing a list of chunks, each with 174 | 'text', 'timestamp', and 'words'. 175 | """ 176 | # Flatten all words from all segments 177 | words = [ 178 | word_info 179 | for segment in result.get('segments', []) 180 | for word_info in segment.get('words', []) 181 | ] 182 | 183 | # Split words into chunks of size max_number 184 | def split_into_chunks(lst, n): 185 | """Yield successive n-sized chunks from lst.""" 186 | for i in range(0, len(lst), n): 187 | yield lst[i:i + n] 188 | 189 | chunks = [] 190 | for word_chunk in split_into_chunks(words, self.sub_length): 191 | chunk_text = ' '.join(word_info['word'] for word_info in word_chunk) 192 | chunk_start = word_chunk[0]['start'] 193 | chunk_end = word_chunk[-1]['end'] 194 | chunk = { 195 | 'timestamp': [chunk_start, chunk_end], 196 | 'text': chunk_text, 197 | 'words': word_chunk 198 | } 199 | chunks.append(chunk) 200 | 201 | result_temp = { 202 | 'text': ' '.join(chunk['text'].strip() for chunk in chunks), 203 | 'chunks': chunks 204 | } 205 | 206 | return result_temp 207 | 208 | def to_transcription_dict(self, insanely_annotation: list[dict]) -> dict: 209 | """ 210 | Transform insanely-fast-whisper speaker annotation result to dict. 211 | """ 212 | chunks = [] 213 | for s in insanely_annotation: 214 | chunk = { 215 | 'text': s['text'], 216 | 'timestamp': (s['timestamp'][0], s['timestamp'][1]), 217 | 'speaker': s['speaker'] 218 | } 219 | chunks.append(chunk) 220 | 221 | result = { 222 | 'text': ''.join([s['text'] for s in insanely_annotation]), 223 | 'chunks': chunks 224 | } 225 | return result 226 | 227 | def to_whisperx(self, transcription_result: dict) -> dict: 228 | """ 229 | Normalize insanely-fast-whisper transcription result to whisperX dict. 230 | """ 231 | words = [] 232 | for c in transcription_result['chunks']: 233 | if 'speaker' in c: 234 | word = { 235 | 'word': c['text'].strip(), 236 | 'start': c['timestamp'][0], 237 | 'end': c['timestamp'][1], 238 | 'speaker': c['speaker'] 239 | } 240 | else: 241 | word = { 242 | 'word': c['text'].strip(), 243 | 'start': c['timestamp'][0], 244 | 'end': c['timestamp'][1] 245 | } 246 | words.append(word) 247 | 248 | result = { 249 | 'segments': [ 250 | { 251 | 'start': transcription_result['chunks'][0]['timestamp'][0], 252 | 'end': transcription_result['chunks'][-1]['timestamp'][1], 253 | 'text': transcription_result['text'].strip(), 254 | 'words': words 255 | } 256 | ] 257 | } 258 | return result 259 | 260 | def create_text_with_speakers( 261 | self, 262 | transcription_dict: dict, 263 | delimiter: str = '.' 264 | ) -> dict: 265 | """ 266 | Iterates through all chunks of each language and creates the complete 267 | text with speaker labels inserted when there is a speaker change. 268 | 269 | Args: 270 | transcription_dict (dict): The dictionary containing transcription 271 | data. 272 | 273 | Returns: 274 | dict: A dictionary mapping each language to its formatted text with 275 | speaker labels. 276 | """ 277 | transcriptions = transcription_dict.get('transcriptions', {}) 278 | 279 | for lang, lang_data in transcriptions.items(): 280 | text = "" 281 | current_speaker = None 282 | chunks = lang_data.get('chunks', []) 283 | 284 | for chunk in chunks: 285 | words = chunk.get('words', []) 286 | 287 | for word_info in words: 288 | speaker = word_info.get('speaker') 289 | word = word_info.get('word', '') 290 | start_timestamp = little_helper.format_time( 291 | word_info.get('start'), 292 | delimiter 293 | ) 294 | 295 | # Insert speaker label if a speaker change is detected 296 | if speaker != current_speaker: 297 | text += f"\n[{start_timestamp}] [{speaker}] " 298 | current_speaker = speaker 299 | 300 | # Append the word with a space 301 | text += word + " " 302 | 303 | transcription_dict['transcriptions'][lang]['text_with_speaker_annotation'] = text.strip() 304 | 305 | return transcription_dict 306 | 307 | def transcribe_with_whisperx(self, filepath: Path) -> dict: 308 | """ 309 | Transcribe a file with the whisperX implementation that returns word-level 310 | timestamps and speaker annotation:https://github.com/m-bain/whisperX 311 | 312 | This implementation is used when a specific subtitle length (e.g. 5 words 313 | per individual subtitle) is needed. 314 | """ 315 | import torch 316 | import whisperx 317 | import gc 318 | 319 | def empty_cuda_cache(model): 320 | gc.collect() 321 | torch.cuda.empty_cache() 322 | del model 323 | 324 | def fill_missing_timestamps(segments: list) -> list: 325 | """ 326 | whisperX does not provide timestamps for words containing only 327 | numbers (e.g. "1.5", "2024" etc.). 328 | 329 | The function fills these missing timestamps by padding the last 330 | known 'end' timestamp for a missing 'start' timestamp or by cutting 331 | the next known 'start' timestamp for a missing 'end' timestamp. 332 | """ 333 | padding = 0.05 # in seconds 334 | 335 | for segment in segments: 336 | words = segment['words'] 337 | num_words = len(words) 338 | 339 | for i, word in enumerate(words): 340 | # If the 'start' key is missing 341 | if 'start' not in word: 342 | if i > 0 and 'end' in words[i-1]: 343 | word['start'] = round(words[i-1]['end'] + padding, 2) 344 | else: 345 | word['start'] = segment['start'] 346 | 347 | # If the 'end' key is missing 348 | if 'end' not in word: 349 | if i < num_words - 1 and 'start' in words[i+1]: 350 | word['end'] = round(words[i+1]['start'] - padding, 2) 351 | elif i == num_words - 1: 352 | word['end'] = round(words[i]['start'] + padding, 2) 353 | segment['end'] = word['end'] 354 | else: 355 | word['end'] = round(words[i]['start'] + padding, 2) 356 | 357 | # If 'score' key is missing 358 | if 'score' not in word: 359 | word['score'] = 0.5 360 | 361 | # If 'speaker' key is missing 362 | if self.annotate and 'speaker' not in word: 363 | speaker_assigned = False 364 | 365 | # Case 1: If it's the first word, look forward for the 366 | # next speaker 367 | if i == 0: 368 | for j in range(i + 1, num_words): 369 | if 'speaker' in words[j]: 370 | word['speaker'] = words[j]['speaker'] 371 | speaker_assigned = True 372 | break 373 | 374 | # Case 2: If it's the last word, look backward for the 375 | # previous speaker 376 | elif i == num_words - 1: 377 | for j in range(i - 1, -1, -1): 378 | if 'speaker' in words[j]: 379 | word['speaker'] = words[j]['speaker'] 380 | speaker_assigned = True 381 | break 382 | 383 | # Case 3: For other words, prefer the previous speaker; 384 | # If not found, look forward 385 | if not speaker_assigned: 386 | # Look backward 387 | for j in range(i - 1, -1, -1): 388 | if 'speaker' in words[j]: 389 | word['speaker'] = words[j]['speaker'] 390 | speaker_assigned = True 391 | break 392 | 393 | if not speaker_assigned: 394 | # Look forward 395 | for j in range(i + 1, num_words): 396 | if 'speaker' in words[j]: 397 | word['speaker'] = words[j]['speaker'] 398 | speaker_assigned = True 399 | break 400 | 401 | if not speaker_assigned: 402 | # Default speaker if none found 403 | word['speaker'] = 'UNKNOWN' 404 | 405 | return segments 406 | 407 | def whisperx_task(task: str = 'transcribe', language = None): 408 | """ 409 | Define a transcription / translation task with whisperX 410 | """ 411 | # Set parameters 412 | device = 'cuda' if self.device == 'cuda:0' else 'cpu' 413 | 414 | # Transcribe or Translate 415 | model = whisperx.load_model( 416 | whisper_arch=self.model, 417 | device=device, 418 | compute_type='float16' if self.device == 'cuda:0' else 'int8', 419 | language=self.file_language or None, 420 | asr_options={ 421 | "hotwords": None, 422 | "multilingual": False 423 | }) 424 | audio = whisperx.load_audio(str(filepath), sr=16000) 425 | result = model.transcribe( 426 | audio, 427 | batch_size=16 if self.device == 'cuda:0' else 8, 428 | task=task 429 | ) 430 | model_a, metadata = whisperx.load_align_model( 431 | device=device, 432 | language_code=language 433 | ) 434 | result = whisperx.align( 435 | result["segments"], 436 | model_a, 437 | metadata, 438 | audio, 439 | device, 440 | return_char_alignments=False 441 | ) 442 | 443 | # Empty CUDA cache 444 | if self.device == 'cuda:0': 445 | empty_cuda_cache(model_a) 446 | 447 | return result 448 | 449 | def whisperx_annotation(transcription_result: dict) -> dict: 450 | # Set parameters 451 | device = 'cuda' if self.device == 'cuda:0' else 'cpu' 452 | 453 | diarize_model = whisperx.diarize.DiarizationPipeline( 454 | use_auth_token=self.hf_token, 455 | device=device 456 | ) 457 | diarize_segments = diarize_model( 458 | str(filepath), 459 | max_speakers=self.num_speakers 460 | ) 461 | result = whisperx.assign_word_speakers( 462 | diarize_segments, 463 | transcription_result 464 | ) 465 | 466 | # Empty CUDA cache 467 | if self.device == 'cuda:0': 468 | empty_cuda_cache(diarize_model) 469 | 470 | return result 471 | 472 | # Start and time transcription 473 | logging.info(f"👨💻 Transcription started with whisper🆇 for {filepath.name}") 474 | t_start = time.time() 475 | 476 | # Run the transcription 477 | transcription_task = partial( 478 | whisperx_task, 479 | task='transcribe', 480 | language=self.file_language 481 | ) 482 | transcription_result = little_helper.run_with_progress( 483 | description=f"[cyan]→ Transcribing ({'CUDA' if self.device == 'cuda:0' else 'CPU'}) [bold]{filepath.name}", 484 | task=transcription_task 485 | ) 486 | 487 | # Speaker annotation 488 | if self.annotate: 489 | annotation_task = partial( 490 | whisperx_annotation, 491 | transcription_result 492 | ) 493 | transcription_result = little_helper.run_with_progress( 494 | description=f"[purple]→ Annotating ({self.device.upper()}) [bold]{filepath.name}", 495 | task=annotation_task 496 | ) 497 | 498 | # Fill in missing timestamps and adjust word chunk length 499 | transcription_result['segments'] = fill_missing_timestamps( 500 | transcription_result['segments'] 501 | ) 502 | transcription_result = self.adjust_word_chunk_length( 503 | transcription_result 504 | ) 505 | 506 | # Create result dict and append transcription to it 507 | result = {'transcriptions': {}} 508 | result['transcriptions'][self.file_language] = transcription_result 509 | 510 | if self.verbose: 511 | print(f"{result['transcriptions'][self.file_language]['text']}") 512 | 513 | # Translation task (to English) 514 | if self.translate and self.file_language != 'en': 515 | translation_task = partial( 516 | whisperx_task, 517 | task='translate', 518 | language='en' 519 | ) 520 | translation_result = little_helper.run_with_progress( 521 | description=f"[dark_blue]→ Translating ({'CUDA' if self.device == 'cuda:0' else 'CPU'}) [bold]{filepath.name}", 522 | task=translation_task 523 | ) 524 | 525 | # Speaker annotation 526 | if self.annotate: 527 | annotation_task = partial( 528 | whisperx_annotation, 529 | translation_result 530 | ) 531 | translation_result = little_helper.run_with_progress( 532 | description=f"[purple]→ Annotating ({self.device.upper()}) [bold]{filepath.name}", 533 | task=annotation_task 534 | ) 535 | 536 | # Fill in missing timestamps and adjust word chunk length 537 | translation_result['segments'] = fill_missing_timestamps( 538 | translation_result['segments'] 539 | ) 540 | translation_result = self.adjust_word_chunk_length( 541 | translation_result 542 | ) 543 | result['transcriptions']['en'] = translation_result 544 | 545 | if self.verbose: 546 | print(f"{result['transcriptions']['en']['text']}") 547 | 548 | # Create full transcription with speaker annotation 549 | if self.annotate: 550 | result = self.create_text_with_speakers(result) 551 | 552 | logging.info(f"👨💻 Transcription completed in {time.time() - t_start:.2f} sec.") 553 | 554 | return {'transcription': result} 555 | 556 | def transcribe_with_insane_whisper(self, filepath: Path) -> dict: 557 | """ 558 | Transcribes a file using the 'insanely-fast-whisper' implementation: 559 | https://github.com/Vaibhavs10/insanely-fast-whisper 560 | 561 | This method utilizes the 'insanely-fast-whisper' implementation of 562 | OpenAI Whisper for automatic speech recognition on Mac M1-M4 devices. 563 | 564 | Parameters: 565 | - filepath (Path): The path to the audio file for transcription. 566 | 567 | Returns: 568 | - dict: A dictionary containing the transcription result and, if 569 | speaker detection is enabled, the speaker diarization result. 570 | The transcription result includes the recognized text and 571 | timestamps if available. 572 | """ 573 | import torch 574 | from transformers import pipeline 575 | from transformers import logging as hf_logger 576 | from whisply import diarize_utils 577 | 578 | hf_logger.set_verbosity_error() 579 | 580 | def insane_whisper_annotation(transcription_result: dict) -> dict: 581 | # Speaker annotation 582 | annotation_result = diarize_utils.diarize( 583 | transcription_result, 584 | diarization_model='pyannote/speaker-diarization-3.1', 585 | hf_token=self.hf_token, 586 | file_name=str(filepath), 587 | num_speakers=self.num_speakers, 588 | min_speakers=None, 589 | max_speakers=None, 590 | ) 591 | # Transform annotation_result to correct dict structure 592 | transcription_result = self.to_transcription_dict(annotation_result) 593 | return transcription_result 594 | 595 | # Start and time transcription 596 | logging.info( 597 | f"👨💻 Transcription started with 🚅 insane-whisper for {filepath.name}" 598 | ) 599 | t_start = time.time() 600 | 601 | try: 602 | pipe = pipeline( 603 | 'automatic-speech-recognition', 604 | model = self.model, 605 | torch_dtype = torch.float16, 606 | device = self.device, 607 | model_kwargs = { 608 | 'attn_implementation': 'eager' 609 | } 610 | ) 611 | 612 | # Define transcription function 613 | def transcription_task(): 614 | transcription_result = pipe( 615 | str(filepath), 616 | batch_size=1, 617 | return_timestamps='word', 618 | generate_kwargs={ 619 | 'use_cache': True, 620 | 'return_legacy_cache': False, 621 | 'language': self.file_language, 622 | 'task': "transcribe", 623 | 'forced_decoder_ids': None 624 | } 625 | ) 626 | return transcription_result 627 | 628 | # Transcription 629 | transcription_result = little_helper.run_with_progress( 630 | description=f"[cyan]→ Transcribing ({self.device.upper()}) [bold]{filepath.name}", 631 | task=transcription_task 632 | ) 633 | 634 | # Speaker annotation 635 | if self.annotate: 636 | transcription_result = little_helper.run_with_progress( 637 | description=f"[purple]→ Annotating ({self.device.upper()}) [bold]{filepath.name}", 638 | task=partial( 639 | insane_whisper_annotation, 640 | transcription_result 641 | ) 642 | ) 643 | 644 | # Adjust word chunk length 645 | transcription_result = self.to_whisperx(transcription_result) 646 | transcription_result = self.adjust_word_chunk_length(transcription_result) 647 | 648 | # Build result dict 649 | result = {'transcriptions': {}} 650 | result['transcriptions'] = { 651 | self.file_language: transcription_result 652 | } 653 | 654 | if self.verbose: 655 | print(result['transcriptions'][self.file_language]['text']) 656 | 657 | # Translation 658 | if self.translate and self.file_language != 'en': 659 | def translation_task(): 660 | translation_result = pipe( 661 | str(filepath), 662 | batch_size=1, 663 | return_timestamps='word', 664 | generate_kwargs={ 665 | 'use_cache': True, 666 | 'return_legacy_cache': False, 667 | 'task': 'translate', 668 | # 'forced_decoder_ids': None 669 | } 670 | ) 671 | return translation_result 672 | 673 | # Run the translation task 674 | translation_result = little_helper.run_with_progress( 675 | description=f"[dark_blue]→ Translating ({self.device.upper()}) [bold]{filepath.name}", 676 | task=translation_task 677 | ) 678 | 679 | # Speaker annotation 680 | if self.annotate: 681 | translation_result = little_helper.run_with_progress( 682 | description=f"[purple]→ Annotating ({self.device.upper()}) [bold]{filepath.name}", 683 | task=partial( 684 | insane_whisper_annotation, 685 | translation_result 686 | ) 687 | ) 688 | 689 | # Adjust word chunk length 690 | translation_result = self.to_whisperx(translation_result) 691 | translation_result = self.adjust_word_chunk_length(translation_result) 692 | 693 | result['transcriptions']['en'] = translation_result 694 | 695 | if self.verbose: 696 | print(result['transcriptions']['en']['text']) 697 | 698 | if self.annotate: 699 | # Create full transcription with speaker annotation 700 | result = self.create_text_with_speakers(result) 701 | 702 | except Exception as e: 703 | print(f'{e}') 704 | 705 | # Stop timing transcription 706 | logging.info(f"👨💻 Transcription completed in {time.time() - t_start:.2f} sec.") 707 | 708 | return {'transcription': result} 709 | 710 | def transcribe_with_faster_whisper( 711 | self, 712 | filepath: Path, 713 | num_workers: int = 1 714 | ) -> dict: 715 | """ 716 | Transcribes an audio file using the 'faster-whisper' implementation: 717 | https://github.com/SYSTRAN/faster-whisper 718 | 719 | This method utilizes the 'faster-whisper' implementation of OpenAI 720 | Whisper for automatic speech recognition. It loads the model and sets 721 | parameters for transcription. After transcription, it formats the 722 | result into segments with timestamps and combines them into a single 723 | text. If speaker detection is enabled, it also annotates speakers in 724 | the transcription result. 725 | 726 | Parameters: 727 | - filepath (Path): The path to the audio file for transcription. 728 | - num_workers (int): The number of workers to use for transcription. 729 | 730 | Returns: 731 | - dict: A dictionary containing the transcription result and, if 732 | speaker detection is enabled, the speaker diarization result. 733 | The transcription result includes the recognized text and 734 | segmented chunks with timestamps if available. 735 | """ 736 | from faster_whisper import WhisperModel, BatchedInferencePipeline 737 | 738 | # Start and time transcription 739 | logging.info(f"👨💻 Transcription started with 🏃♀️➡️ faster-whisper for {filepath.name}") 740 | t_start = time.time() 741 | 742 | # Load model and set parameters 743 | model = BatchedInferencePipeline( 744 | model = WhisperModel( 745 | self.model, 746 | device='cpu' if self.device in ['mps', 'cpu'] else 'cuda', 747 | num_workers=num_workers, 748 | compute_type='int8' if self.device in ['mps', 'cpu'] else 'float16' 749 | )) 750 | 751 | # Define the transcription task 752 | def transcription_task(): 753 | segments, _ = model.transcribe( 754 | str(filepath), 755 | beam_size=5, 756 | language=self.file_language, 757 | word_timestamps=True, 758 | batch_size=16 759 | ) 760 | 761 | chunks = [] 762 | for segment in segments: 763 | seg = { 764 | 'timestamp': (float(f"{segment.start:.2f}"), float(f"{segment.end:.2f}")), 765 | 'text': segment.text.strip(), 766 | 'words': [{ 767 | 'word': i.word.strip(), 768 | 'start': float(f"{i.start:.2f}"), 769 | 'end': float(f"{i.end:.2f}"), 770 | 'score': float(f"{i.probability:.2f}") 771 | } for i in segment.words] 772 | } 773 | chunks.append(seg) 774 | 775 | # If verbose Flag 776 | if self.verbose: 777 | print(seg['text']) 778 | 779 | return chunks 780 | 781 | # Add progress bar and run the transcription task 782 | chunks = little_helper.run_with_progress( 783 | description=f"[cyan]→ Transcribing ({self.device.upper()}) [bold]{filepath.name}", 784 | task=transcription_task 785 | ) 786 | 787 | # Create result dict and append transcriptions to it 788 | result = {'transcriptions': {}} 789 | result['transcriptions'][self.file_language] = { 790 | 'text': ' '.join([segment['text'].strip() for segment in chunks]), 791 | 'chunks': chunks 792 | } 793 | 794 | # Translation 795 | if self.translate and self.file_language != 'en': 796 | # Define the translation task 797 | def translation_task(): 798 | segments, _ = model.transcribe( 799 | str(filepath), 800 | beam_size=5, 801 | task='translate', 802 | language='en', 803 | word_timestamps=True 804 | ) 805 | 806 | translation_chunks = [] 807 | for segment in segments: 808 | seg = { 809 | 'timestamp': (float(f"{segment.start:.2f}"), float(f"{segment.end:.2f}")), 810 | 'text': segment.text.strip(), 811 | 'words': [{ 812 | 'word': i.word.strip(), 813 | 'start': float(f"{i.start:.2f}"), 814 | 'end': float(f"{i.end:.2f}"), 815 | 'score': float(f"{i.probability:.2f}") 816 | } for i in segment.words] 817 | } 818 | translation_chunks.append(seg) 819 | 820 | # If verbose Flag 821 | if self.verbose: 822 | print(seg['text']) 823 | 824 | return translation_chunks 825 | 826 | # Add progress bar and run the translation task 827 | translation_chunks = little_helper.run_with_progress( 828 | description=f"[dark_blue]→ Translating ({self.device.upper()}) [bold]{filepath.name}", 829 | task=translation_task 830 | ) 831 | 832 | # Add translation to result dict 833 | result['transcriptions']['en'] = { 834 | 'text': ' '.join([segment['text'].strip() for segment in translation_chunks]), 835 | 'chunks': translation_chunks 836 | } 837 | 838 | # Stop timing transcription 839 | logging.info(f"👨💻 Transcription completed in {time.time() - t_start:.2f} sec.") 840 | 841 | return {'transcription': result} 842 | 843 | def detect_language(self, filepath, audio_array) -> str: 844 | """ 845 | Detects the language of the input file. 846 | """ 847 | from faster_whisper import WhisperModel 848 | 849 | logging.info(f"Detecting language of file: {filepath.name}") 850 | 851 | def run_language_detection(): 852 | lang_detection_model = WhisperModel( 853 | models.set_supported_model( 854 | model=self.model_provided, 855 | implementation='faster-whisper', 856 | translation=self.translate 857 | ), 858 | device='cpu' if self.device in ['mps', 'cpu'] else 'cuda', 859 | compute_type='int8' if self.device in ['mps', 'cpu'] else 'float16' 860 | ) 861 | lang, score, _ = lang_detection_model.detect_language(audio_array) 862 | return lang, score 863 | 864 | lang, score = little_helper.run_with_progress( 865 | description=f"[dark_goldenrod]→ Detecting language for [bold]{filepath.name}", 866 | task=run_language_detection 867 | ) 868 | 869 | self.file_language = lang 870 | 871 | print(f'[blue1]→ Detected language "{lang}" with probability {score:.2f}') 872 | logging.info(f'Detected language → "{lang}" with probability {score:.2f}') 873 | 874 | def process_files(self, files) -> None: 875 | """ 876 | Processes a list of audio files for transcription and/or diarization. 877 | 878 | This method logs the processing parameters, extracts filepaths from the 879 | input list, and initializes an empty list for storing results. Each 880 | file is processed based on the compute device specified ('mps', 'cuda:0', 881 | or 'cpu'). Appropriate transcription method is chosen based on the 882 | device. Results, including file ids, paths, transcriptions, and 883 | diarizations, are stored in a dictionary and saved to a designated 884 | output directory. Each result is also appended to `self.processed_files`. 885 | 886 | Parameters: 887 | files (list of str): A list of file paths or file-like objects 888 | representing the audio files to be processed. 889 | """ 890 | logging.info(f"Provided parameters for processing: {self.metadata}") 891 | 892 | # Get filepaths 893 | filepath_handler = FilePathProcessor(self.file_formats) 894 | [filepath_handler.get_filepaths(f) for f in files] 895 | self.filepaths = filepath_handler.filepaths 896 | 897 | # Process filepaths 898 | logging.info(f"Processing files: {self.filepaths}") 899 | 900 | self.processed_files = [] 901 | for idx, filepath in enumerate(self.filepaths): 902 | 903 | # Create and set output_dir and output_filepath 904 | self.output_dir = little_helper.set_output_dir(filepath, self.base_dir) 905 | output_filepath = self.output_dir / Path(filepath).stem 906 | 907 | # Convert file format 908 | filepath, audio_array = little_helper.check_file_format( 909 | filepath=filepath, 910 | del_originals=self.del_originals 911 | ) 912 | 913 | # Detect file language 914 | if not self.file_language: 915 | self.detect_language(filepath, audio_array) 916 | 917 | logging.info(f"Transcribing file: {filepath.name}") 918 | 919 | # Transcription and speaker annotation 920 | if self.device == 'mps': 921 | self.model = models.set_supported_model( 922 | self.model_provided, 923 | implementation='insane-whisper', 924 | translation=self.translate 925 | ) 926 | print(f'[blue1]→ Using {self.device.upper()} and 🚅 Insanely-Fast-Whisper with model "{self.model}"') 927 | result_data = self.transcribe_with_insane_whisper(filepath) 928 | 929 | elif self.device in ['cpu', 'cuda:0']: 930 | if self.annotate or self.subtitle: 931 | # WhisperX for annotation / subtitling 932 | self.model = models.set_supported_model( 933 | self.model_provided, 934 | implementation='whisperx', 935 | translation=self.translate 936 | ) 937 | print(f'[blue1]→ Using {self.device.upper()} and whisper🆇 with model "{self.model}"') 938 | result_data = self.transcribe_with_whisperx(filepath) 939 | else: 940 | # Faster-Whisper for raw transcription 941 | self.model = models.set_supported_model( 942 | self.model_provided, 943 | implementation='faster-whisper', 944 | translation=self.translate 945 | ) 946 | print(f'[blue1]→ Using {self.device.upper()} and 🏃♀️➡️ Faster-Whisper with model "{self.model}"') 947 | result_data = self.transcribe_with_faster_whisper(filepath) 948 | 949 | result = { 950 | 'id': f'file_00{idx + 1}', 951 | 'created': datetime.now().strftime('%Y-%m-%d %H:%M:%S'), 952 | 'input_filepath': str(filepath.absolute()), 953 | 'output_filepath': str(Path(output_filepath).absolute()), 954 | 'written_files': None, 955 | 'device': self.device, 956 | 'model': self.model, 957 | 'transcription': result_data['transcription']['transcriptions'], 958 | } 959 | 960 | # Save results 961 | result['written_files'] = output_utils.OutputWriter( 962 | corrections=self.corrections 963 | ).save_results( 964 | result=result, 965 | export_formats=self.export_formats 966 | ) 967 | 968 | self.processed_files.append(result) 969 | 970 | if not self.file_language_provided: 971 | self.file_language = None 972 | --------------------------------------------------------------------------------