├── sample.wav ├── sample_tflite.wav ├── README.md ├── inference.py └── inference_tflite.py /sample.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/monatis/german-tts/HEAD/sample.wav -------------------------------------------------------------------------------- /sample_tflite.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/monatis/german-tts/HEAD/sample_tflite.wav -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # german-tts 2 | German Tacotron 2 and Multi-band MelGAN in TensorFlow with TF Lite inference support 3 | 4 | ## Acknowledgement 5 | Google supported this work by providing Google Cloud credit. Thank you Google for supporting the open source! 🎉 6 | 7 | ## Overview 8 | I am releasing pretrained German neural text-to-speech (TTS) models Tacotron 2 and Multi-band MelGAN. It supports inference with `saved_model` and `TF Lite` formats, and all the models can be found on [TensorFlow Hub](https://tfhub.dev/monatis). 9 | 10 | 💬 Say hello in [Discussions](https://github.com/monatis/german-tts/discussions/1) if you find it useful for anything. 11 | 12 | - See [`inference.py`](https://github.com/monatis/german-tts/blob/main/inference.py) to infer with `saved_model`. 13 | - See [`inference_tflite.py`](https://github.com/monatis/german-tts/blob/main/inference_tflite.py) to infer with `TF Lite`. 14 | - See [`e2e-notebook.ipynb`](https://github.com/monatis/german-tts/blob/main/e2e-notebook.ipynb) to check how I exported to these model formats. 15 | - see [releases](https://github.com/monatis/german-tts/releases) to download pretrained models. 16 | 17 | ## Dataset 18 | I trained these models on [Thorsten dataset](https://github.com/thorstenMueller/deep-learning-german-tts) by Thorsten Müller. It is licensed under the terms of Creative Commons Zero V1 Universal (CC0), which is used to opt out of copyright entirely and ensure that the work has the widest reach. Thanks [@thorstenMueller](https://github.com/thorstenMueller) for such a great contribution to the community. 19 | 20 | ## Notes 21 | Some good guys are doing a great job at [tensorspeech/TensorFlowTTS](https://github.com/tensorspeech/TensorFlowTTS), which was already supporting TTS in English, Chinese and Korean. I wanted to contribute with support for German and trained these models. Now it supports both training and inference with proper processors. A detailed blog post will follow up, but some quick notes for now: 22 | 23 | - I made use of [german_transliterate](https://github.com/repodiac/german_transliterate) For text preprocessing. Basically it normalizes numbers (e.g. converts digits to words), expands abbreviations and cares German umlauts and punctuations. For inference examples released in this repo, it is the only dependency apart from TensorFlow. 24 | - You need to convert input text to numerical IDs to feed into the model. I am sharing a reference implementation for this in inference examples, and you need to code this logic to use the models in non-Python environments (e.g., Android). 25 | - `Tacotron 2` produces some noise at the end, and you need to cut it off. Again, inference examples show how to do this. 26 | - I exported `Multi-band MelGAN` to `TF Lite` without optimizations because it produced some background noise when I exported with the default ones. I used default optimizations in `Tacotron 2`. 27 | - `saved_model` formats that I am releasing here are not suitable for finetuning. Architecture implementation uses `Subclassing API` in TensorFlow 2.x and gets multiple inputs in `call` method for teacher forcing during training. This caused some problems when exporting to `saved_model` and I had to remove this logic before exporting. If you want to finetune models, please see [my fork of TensorFlowTTS](https://github.com/monatis/TensorFlowTTS). 28 | 29 | ## License 30 | You can use these pretrained model artifacts and code examples under the terms of Apache 2.0 license. On the other hand, you may want to contact me for paid consultancies and/or collaborations in speech and/or NLP projects at the email address shown on [my profile](https://github.com/monatis). -------------------------------------------------------------------------------- /inference.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # Copyright 2020 TensorFlowTTS Team. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Perform preprocessing and raw feature extraction for LJSpeech dataset.""" 16 | 17 | import os 18 | import re 19 | import time 20 | from scipy.io import wavfile 21 | from german_transliterate.core import GermanTransliterate 22 | 23 | 24 | _pad = "pad" 25 | _eos = "eos" 26 | _punctuation = "!'(),.? " 27 | _special = "-" 28 | _letters = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" 29 | 30 | # Export all symbols: 31 | ALL_SYMBOLS = ( 32 | [_pad] + list(_special) + list(_punctuation) + list(_letters) + [_eos] 33 | ) 34 | 35 | # Regular expression matching text enclosed in curly braces: 36 | _curly_re = re.compile(r"(.*?)\{(.+?)\}(.*)") 37 | 38 | def german_cleaners(text): 39 | """Pipeline for German text, including number and abbreviation expansion.""" 40 | text = GermanTransliterate(replace={';': ',', ':': ' '}, sep_abbreviation=' -- ').transliterate(text) 41 | print(text) 42 | return text 43 | 44 | class Processor(): 45 | """German processor.""" 46 | 47 | def __init__(self): 48 | self.symbol_to_id = {symbol: id for id, symbol in enumerate(ALL_SYMBOLS)} 49 | self.eos_id = self.symbol_to_id["eos"] 50 | 51 | def text_to_sequence(self, text): 52 | sequence = [] 53 | # Check for curly braces and treat their contents as ARPAbet: 54 | while len(text): 55 | m = _curly_re.match(text) 56 | if not m: 57 | sequence += self._symbols_to_sequence( 58 | german_cleaners(text) 59 | ) 60 | break 61 | sequence += self._symbols_to_sequence( 62 | german_cleaners(m.group(1)) 63 | ) 64 | sequence += self._arpabet_to_sequence(m.group(2)) 65 | text = m.group(3) 66 | 67 | # add eos tokens 68 | sequence += [self.eos_id] 69 | return sequence 70 | 71 | def _symbols_to_sequence(self, symbols): 72 | return [self.symbol_to_id[s] for s in symbols if self._should_keep_symbol(s)] 73 | 74 | def _arpabet_to_sequence(self, text): 75 | return self._symbols_to_sequence(["@" + s for s in text.split()]) 76 | 77 | def _should_keep_symbol(self, s): 78 | return s in self.symbol_to_id and s != "_" and s != "~" 79 | 80 | if __name__ == "__main__": 81 | import tensorflow as tf 82 | path_to_mbmelgan = tf.keras.utils.get_file( 83 | 'german-tts-mbmelgan.tar.gz', 84 | 'https://storage.googleapis.com/mys-released-models/german-tts-mbmelgan.tar.gz', 85 | extract=True, 86 | cache_subdir='german-tts-mbmelgan' 87 | ) 88 | mbmelgan = tf.saved_model.load(os.path.dirname(path_to_mbmelgan)) 89 | 90 | 91 | path_to_tacotron2 = tf.keras.utils.get_file( 92 | 'german-tts-tacotron2.tar.gz', 93 | 'https://storage.googleapis.com/mys-released-models/german-tts-tacotron2.tar.gz', 94 | extract=True, 95 | cache_subdir='german-tts-tacotron2' 96 | ) 97 | tacotron2 = tf.saved_model.load(os.path.dirname(path_to_tacotron2)) 98 | 99 | # Infer 100 | proc = Processor() 101 | start = time.time() 102 | input_ids = proc.text_to_sequence("Möchtest du das meiner Frau erklären? Nein? Ich auch nicht.") 103 | 104 | _, mel_outputs, _, _ = tacotron2.inference( 105 | tf.expand_dims(tf.convert_to_tensor(input_ids, dtype=tf.int32), 0), 106 | tf.convert_to_tensor([len(input_ids)], dtype=tf.int32), 107 | tf.convert_to_tensor([0], dtype=tf.int32) 108 | ) 109 | audio = mbmelgan.inference(mel_outputs)[0, :-1024, 0] 110 | duration = time.time() - start 111 | print(f"it took {duration} secs") 112 | wavfile.write("sample.wav", 22050, audio.numpy()) -------------------------------------------------------------------------------- /inference_tflite.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # Copyright 2020 TensorFlowTTS Team. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | """Perform preprocessing and raw feature extraction for LJSpeech dataset.""" 16 | 17 | import os 18 | os.environ['CUDA_VISIBLE_DEVICES'] = '-1' 19 | import re 20 | import time 21 | from scipy.io import wavfile 22 | from german_transliterate.core import GermanTransliterate 23 | 24 | 25 | _pad = "pad" 26 | _eos = "eos" 27 | _punctuation = "!'(),.? " 28 | _special = "-" 29 | _letters = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" 30 | 31 | # Export all symbols: 32 | ALL_SYMBOLS = ( 33 | [_pad] + list(_special) + list(_punctuation) + list(_letters) + [_eos] 34 | ) 35 | 36 | # Regular expression matching text enclosed in curly braces: 37 | _curly_re = re.compile(r"(.*?)\{(.+?)\}(.*)") 38 | 39 | def german_cleaners(text): 40 | """Pipeline for German text, including number and abbreviation expansion.""" 41 | text = GermanTransliterate(replace={';': ',', ':': ' '}, sep_abbreviation=' -- ').transliterate(text) 42 | return text 43 | 44 | class Processor(): 45 | """German processor.""" 46 | 47 | def __init__(self): 48 | self.symbol_to_id = {symbol: id for id, symbol in enumerate(ALL_SYMBOLS)} 49 | self.eos_id = self.symbol_to_id["eos"] 50 | 51 | def text_to_sequence(self, text): 52 | sequence = [] 53 | # Check for curly braces and treat their contents as ARPAbet: 54 | while len(text): 55 | m = _curly_re.match(text) 56 | if not m: 57 | sequence += self._symbols_to_sequence( 58 | german_cleaners(text) 59 | ) 60 | break 61 | sequence += self._symbols_to_sequence( 62 | german_cleaners(m.group(1)) 63 | ) 64 | sequence += self._arpabet_to_sequence(m.group(2)) 65 | text = m.group(3) 66 | 67 | # add eos tokens 68 | sequence += [self.eos_id] 69 | return sequence 70 | 71 | def _symbols_to_sequence(self, symbols): 72 | return [self.symbol_to_id[s] for s in symbols if self._should_keep_symbol(s)] 73 | 74 | def _arpabet_to_sequence(self, text): 75 | return self._symbols_to_sequence(["@" + s for s in text.split()]) 76 | 77 | def _should_keep_symbol(self, s): 78 | return s in self.symbol_to_id and s != "_" and s != "~" 79 | 80 | 81 | 82 | def prepare_input(input_ids): 83 | return (tf.expand_dims(tf.convert_to_tensor(input_ids, dtype=tf.int32), 0), 84 | tf.convert_to_tensor([len(input_ids)], tf.int32), 85 | tf.convert_to_tensor([0], dtype=tf.int32)) 86 | 87 | processor = Processor() 88 | 89 | def infer_tflite(input_text, interpreter, mbmelgan_interpreter): 90 | input_ids = processor.text_to_sequence(input_text) 91 | interpreter.resize_tensor_input(input_details[0]['index'], [1, len(input_ids)]) 92 | interpreter.allocate_tensors() 93 | input_data = prepare_input(input_ids) 94 | for i, detail in enumerate(input_details): 95 | interpreter.set_tensor(detail['index'], input_data[i]) 96 | 97 | interpreter.invoke() 98 | 99 | # The function `get_tensor()` returns a copy of the tensor data. 100 | mel_outputs = interpreter.get_tensor(output_details[0]['index']) 101 | mbmelgan_interpreter.resize_tensor_input(mbmelgan_input_details[0]['index'], mel_outputs.shape) 102 | mbmelgan_interpreter.allocate_tensors() 103 | mbmelgan_interpreter.set_tensor(mbmelgan_input_details[0]['index'], mel_outputs) 104 | mbmelgan_interpreter.invoke() 105 | 106 | # Get audio and remove noise at the end 107 | audio = mbmelgan_interpreter.get_tensor(mbmelgan_output_details[0]['index'])[0, :-1024, 0] 108 | return audio 109 | 110 | 111 | 112 | if __name__ == "__main__": 113 | import tensorflow as tf 114 | path_to_melgan = tf.keras.utils.get_file( 115 | 'german-tts-mbmelgan-lite.tar.gz', 116 | 'https://storage.googleapis.com/mys-released-models/german-tts-mbmelgan-lite.tar.gz', 117 | extract=True, 118 | cache_subdir='german-tts-mbmelgan' 119 | ) 120 | 121 | path_to_tacotron2 = tf.keras.utils.get_file( 122 | 'german-tts-tacotron2-lite.tar.gz', 123 | 'https://storage.googleapis.com/mys-released-models/german-tts-tacotron2-lite.tar.gz', 124 | extract=True, 125 | cache_subdir='german-tts-tacotron2' 126 | ) 127 | 128 | # Load TFLite models and allocate tensors. 129 | interpreter = tf.lite.Interpreter(model_path=path_to_tacotron2[:-6] + "tflite") 130 | interpreter.allocate_tensors() 131 | 132 | # Get input and output tensors. 133 | input_details = interpreter.get_input_details() 134 | output_details = interpreter.get_output_details() 135 | 136 | mbmelgan_interpreter = tf.lite.Interpreter(model_path=path_to_melgan[:-6] + "tflite") 137 | mbmelgan_interpreter.allocate_tensors() 138 | 139 | # Get input and output tensors. 140 | mbmelgan_input_details = mbmelgan_interpreter.get_input_details() 141 | mbmelgan_output_details = mbmelgan_interpreter.get_output_details() 142 | start = time.time() 143 | audio = infer_tflite("Möchtest du das meiner Frau erklären? Nein? Ich auch nicht.", interpreter, mbmelgan_interpreter) 144 | duration = time.time() - start 145 | print(F"it took {duration} secs") 146 | wavfile.write("sample_tflite.wav", 22050, audio) --------------------------------------------------------------------------------