Around the Web
10 |You can find me on Twitter at https://twitter.com/kastnerkyle
12 |
13 | For more code, see my GitHub at https://github.com/kastnerkyle
Papers I've written or helped write can be found on Google Scholar
![](\"the_more_you_know.jpg\")
"
71 | ]
72 | },
73 | {
74 | "cell_type": "markdown",
75 | "metadata": {},
76 | "source": [
77 | "When All The Planets Align\n",
78 | "==========================\n",
79 | "\n",
80 | "To begin our adventure, we can imagine wanting a tool that takes normal, human understandable words and sentences, and \n",
81 | "converts them into a series of sounds (phonemes) that would make sense. This task is complicated, and has many possible approaches. However, one of the most straightforward approaches is to use something called a pronunciation dictionary to look up which phonemes you need to pronounce each word. \n",
82 | "\n",
83 | "There are a number of pronunciation dictionaries but one of the biggest/most complete dictionaries I have found is used by [CMU Sphinx](http://cmusphinx.sourceforge.net/), an open source speech recognition toolkit. Sidenote - if you are interested in recognition, I have made an \"all-in-one\" wrapper to a light version of Sphinx, called pocketsphinx. You can find that code here https://github.com/kastnerkyle/ez-phones.\n",
84 | "\n",
85 | "Continuing on our path for synthesis, we now have a human crafted lookup table to go from words to sounds. Unfortunately, we almost immediately run into a common problem in sequential modeling - the length of the text and the length of the pronunciation/phoneme sequence are not the same!\n",
86 | "\n",
87 | "This normally requires somewhat complicated machine learning techniques, in order to learn a valid alignment and predict it. However in the papers I listed above (such as [Letter to Sound Rules For Accented Lexicon Compression, Pagel, Lenzo, Black](https://www.cs.cmu.edu/~awb/papers/ICSLP98-lts.pdf)) the authors describe a fairly straightforward dynamic programming algorithm for inputting a new character (\"\\_\") in the predicted sequence which makes the mapping from text to extended phones 1:1. This makes the machine learning portion of this system much simpler.\n",
88 | "\n",
89 | "For example in the original CMU dict we have lines such as:\n",
90 | "```\n",
91 | "BEATNIK B IY T N IH K\n",
92 | "```\n",
93 | "\n",
94 | "After running the t2p script and generating alignments for these sequences, we see word and phoneme sequence pairs like this:\n",
95 | "```\n",
96 | "BEATNIK B IY _ T N IH K\n",
97 | "```\n",
98 | "\n",
99 | "This aligned dictionary file is commited in the repo as ```cmudict.0.7a_SPHINX_40.align```, which was generated from ```cmudict.0.7a_SPHINX_40```. You can also try your own custom dictionary - I included copies of the relevant t2p scripts as well. Processing the alignments for this dictionary took about 10 minutes on my machine (Macbook Air 2011)."
100 | ]
101 | },
102 | {
103 | "cell_type": "markdown",
104 | "metadata": {},
105 | "source": [
106 | "Creature Features\n",
107 | "=========\n",
108 | "\n",
109 | "After processing the pronunciation dictionary to get alignments, we have about 138,000 aligned sequence pairs. The original paper trained a decision tree for unit selection using 5 character windows (n-gram character windows, $$n=5$$).\n",
110 | "This results in a feature window like the following:\n",
111 | " \n",
112 | "Original string and phones:\n",
113 | "```\n",
114 | "BEATNIK ['B', 'IY', '_', 'T', 'N', 'IH', 'K']\n",
115 | "```\n",
116 | "\n",
117 | "After padding with 2 \"-\" characters (floor(n / 2))) to accomodate for beginning and end windows:\n",
118 | "```\n",
119 | "--BEATNIK--\n",
120 | "```\n",
121 | "\n",
122 | "After feature extraction, we get the resulting features. Target phone to predict is the last entry, readable version in commas (formatted as features > target) with center character in the $(n = 5$) character window bolded:\n",
123 | "\n",
124 | "```['-', '-', 'B', 'E', 'A', 'B'] (--```***B***```EA > B)```\n",
125 | "\n",
126 | "```['-', 'B', 'E', 'A', 'T', 'IY'] (-B```***E***```AT > IY)```\n",
127 | "\n",
128 | "```['B', 'E', 'A', 'T', 'N', '_'] (BE```***A***```TN > _)```\n",
129 | "\n",
130 | "```['E', 'A', 'T', 'N', 'I', 'T'] (EA```***T***```NI > T)```\n",
131 | "\n",
132 | "```['A', 'T', 'N', 'I', 'K', 'N'] (AT```***N***```IK > N)```\n",
133 | "\n",
134 | "```['T', 'N', 'I', 'K', '-', 'IH'] (TN```***I***```K- > IH)```\n",
135 | "\n",
136 | "```['N', 'I', 'K', '-', '-', 'K'] (NI```***K***```-- > K)```\n",
137 | "\n",
138 | "In general this gives us a nice simple feature set in order to train a classifier modeling $p(y_i\\>|\\>x_{i-n/2} ... x_{i+n/2})$.\n",
139 | "\n",
140 | "After reading the [Pixel RNN paper](http://arxiv.org/abs/1601.06759) and reviewing the [Deep Learning book (Figure 10.4 and section 10.2.1)](http://www.deeplearningbook.org/contents/rnn.html#pf9), I have become interested in trying to train sequential decision models in parallel, using teacher forced (ground truth) data as a proxy for the sequential predictions. This means you can train in parallel, but apply recursively at test time to still get a sequential decision. A fully bidirection approach to this can be seen in [Natural Language Processing From Scratch, Collobert, Weston, Bouttou, Karlen, Kavukcuoglu, Kuksa](http://arxiv.org/abs/1103.0398), but for now we will use a \"left-to-right\" approach since I find it easier to understand and work with.\n",
141 | "\n",
142 | "Given this new fascination I added one extra wrinkle using \"Markov features\" (for lack of a better name) to make features modeling $p(y_i\\>|\\>x_{i-n/2} ... x_{i+n/2}, y_{i-1})$. Due to adding a $y_{i-1}$ feature, the phone sequence to predict was padded in front with one \"SIL\" phoneme at the very start, allong the decision tree to \"learn to start\". After making all the ngram + Markov features, there end up being ~900,000 samples generated from the original 138,000 aligned sequences. After this processing, the final features looked like the below example (remember, last entry is the target to predict). This is our training data for the classification algorithm which will map text to phonemes.\n",
143 | "\n",
144 | "Padded string and padded phones:\n",
145 | "\n",
146 | "```\n",
147 | "--BEATNIK-- ['SIL', 'B', 'IY', '_', 'T', 'N', 'IH', 'K']\n",
148 | "```\n",
149 | "\n",
150 | "Resulting features (target variable is the last entry):\n",
151 | "\n",
152 | "```\n",
153 | "['SIL', '-', '-', 'B', 'E', 'A', 'B']\n",
154 | "['B', '-', 'B', 'E', 'A', 'T', 'IY']\n",
155 | "['IY', 'B', 'E', 'A', 'T', 'N', '_']\n",
156 | "['_', 'E', 'A', 'T', 'N', 'I', 'T']\n",
157 | "['T', 'A', 'T', 'N', 'I', 'K', 'N']\n",
158 | "['N', 'T', 'N', 'I', 'K', '-', 'IH']\n",
159 | "['IH', 'N', 'I', 'K', '-', '-', 'K']\n",
160 | "```\n",
161 | "\n",
162 | "These Markov features seemed to make a decent improvement for 1,000 and 10,000 samples in training, but it is unclear to me if that holds as more data is added, or if the Markov feature is simply acting as a bigger effective character window (since $y_{i-1}$ is based on one extra character backward in time). Either way, it is something I hope to explore further another time.\n",
163 | "\n",
164 | "Decisions, Decisions, Decisions\n",
165 | "===============================\n",
166 | "I will not review the decision tree learning algorithm in depth, but most of my implementation is derived from [Kevin Davenport](http://kldavenport.com/pure-python-decision-trees/) and [Patrick L. Le](http://www.patricklamle.com/Tutorials/Decision%20tree%20python/tuto_decision%20tree.html), who in turn derived from [Programming Collective Intelligence, Toby Segaran](http://shop.oreilly.com/product/9780596529321.do). The primary reason I chose a pure Python approach, rather than using the (much superior) implementation in [Scikit-Learn](http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier) is so that I could serialize the tree once and save it in this repo, not worrying about possible API changes in future versions of sklearn. I was also curious how to serialize Python objects in pure json, so this was a nice experiment for that although jsonpickle seems a more robust and general solution.\n",
167 | "\n",
168 | "At a high level, decision tree learning tries to split the data along every column (in my implementation, by trying each split). Testing the resulting 2 datasets for \"purity\", measured by the entropy of the labels in each half, our goal is to find splits where all the labels are the same in one of the halves. Choosing the candidate split which was the most pure, we recusively apply the same logic in each half until we reach splits with entropy 0, meaning all the labels in this split are the same - sometimes phrased as \"the leaf is pure\". This means that as long as there is valid 1:1 mapping in the feature set (the same set of features don't map to 2 different labels), a deep enough decision tree will be *perfect* on the training set.\n",
169 | "\n",
170 | "Generally this may seem like a bad thing, since it means decision trees are easy to overfit (leading to algorithms like random forests, boosted trees, and so on which try to regularize the power of the tree) but in this context it is actually great - it means that we can put words we want to pronounce correctly in the training set, and the decision tree should be able to perfectly pronounce them. This makes the system easy to modify and extend, not to mention the fact that decision tree rules are easy to visualize (see the previous links) and potentially modify by hand using expert analysis.\n",
171 | "\n",
172 | "Using my own implementation of the decision tree means the algorithm does not scale particularly well (though good implementations are quite scalable). We are only able to train on around 10,000 examples out of the potential ***900,000*** available, and I am sure adding more training data would greatly boost performance. On the other hand, if the results were too good I couldn't title this article \"Bad Speech Synthesis\"."
173 | ]
174 | },
175 | {
176 | "cell_type": "markdown",
177 | "metadata": {},
178 | "source": [
179 | "Diphone Fanatics\n",
180 | "=========\n",
181 | "\n",
182 | "Since the actual waveforms we want to look up in the provided database are diphone based, not phoneme based, we need to stitch the phoneme sequence into a valid set of diphones. This is probably the most custom code in the whole system, so I will walk through it in a bit of detail.\n",
183 | "\n",
184 | "Starting at [line 13](https://github.com/kastnerkyle/diphone_synthesizer/blob/master/diphone_synthesis.py#L13) we read in the aligned dictionary and split into text sequences and phone sequences. Note that we also skip lines for alternatives, denoted blah(2), blah(3) and so on.\n",
185 | "\n",
186 | "Next in [lines 23 to 40](https://github.com/kastnerkyle/diphone_synthesizer/blob/master/diphone_synthesis.py#L23) we read in the two phone lists for Sphinx and Festival, stripping them down to the common subset between each list, placing the results in sorted order with \"SIL\"(Sphinx) and \"pau\" (Festival) at the end.\n",
187 | "\n",
188 | "After this we read in the diphone lookup list from ```kaldiph.est``` - this is the mapping from Festival dphones to actual wavfiles, along with start, middle, and end times for the diphone in question. This will be important when finally stitching the sound together.\n",
189 | "\n",
190 | "The next step is to construct lookups for cmu2radio (since we are using CMU Sphinx and Festival's radio phoneset) as well as the reverse (radio2cmu). Once this is done we construct real pairs (pau-pau, t-t, and so on) as well as \"fake pairs\" - these are described in more detail in the papers, but basically some sequences of \"phone1, \\_, phone2\" can also be looked up directly and have a unique wavfile. I will describe how this is handled later - just know that \"fake\" is referring to things with \"\\_\" in them.\n",
191 | "\n",
192 | "Once we have all these helper lookups and mappings, it is time to start constructing diphones. Broadly, there are a few steps in the process, which can be seen in the [synthesize function](https://github.com/kastnerkyle/diphone_synthesizer/blob/master/diphone_synthesis.py#L122). We step through the list one element at a time, looking ahead to the next element for some rules to try. Note that this logic is handcrafted by me, and there could be one or many bugs in the logic - or even a misunderstanding of how the \"\\_\" character should really function. One of the biggest questions I have is if this \"backward continuation\" in 1a makes sense. To track which real_match phones are not in the database, I also added a print statement of \"No match found? p1 p2\" which can be ignored in general.\n",
193 | "\n",
194 | "1. Is there a \"\\_\" character in the first position?\n",
195 | "\n",
196 | " a. If so, try to continue the second phoneme backwards (real_match(p2, p2))\n",
197 | " \n",
198 | " b. Take a step forward and continue the logic\n",
199 | " \n",
200 | "2. Is there a \"\\_\" character in the second position?\n",
201 | "\n",
202 | " a. Try to make a fake phone with the third phone (if possible), and check for a match\n",
203 | " \n",
204 | " b. If the fake phone didn't work, try to treat the second phone as a continuation (real_match(p1, p1))\n",
205 | " \n",
206 | " c. Take a step forward and continue the logic\n",
207 | " \n",
208 | "3. If there are no \"\\_\" characters at all\n",
209 | "\n",
210 | " a. Try a real match (real_match(p1, p2))\n",
211 | " \n",
212 | " b. Take a step forward and continue the logic\n",
213 | " \n",
214 | " \n",
215 | "For each of these steps, if we find a match we add it to a list of wavfile chunks that will be stitched together. Once all the wav chunks have been gathered, we perform \"overlap-add\" processing in the time domain [line 225](https://github.com/kastnerkyle/diphone_synthesizer/blob/master/diphone_synthesis.py#L225) by adding up the waveforms in the time domain (taken from Wikipedia article on [overlap add](https://en.wikipedia.org/wiki/Overlap%E2%80%93add_method)).\n",
216 | "\n",
217 | "![](\"overlap_add.png\")
\n",
218 | "\n",
219 | "This is the simplest way to combine all the chunks into one continuous sound, but much better results can be obtained by performing overlap add in the frequency domain, performing frequency smoothing/interpolation, WSOLA (matching the overlap of pieces in time based on autocorrelation), using fancier representations such as vocoder parameters (LPC, STRAIGHT, WORLD), or tons of other tricks. This is just the minimal way to get something working - whole books can be written just on the signal processing used to combine individual sounds into continuous, smooth sounds. But once again if we did it too well, this post couldn't be \"Bad Speech Synthesis\"."
220 | ]
221 | },
222 | {
223 | "cell_type": "markdown",
224 | "metadata": {},
225 | "source": [
226 | "Putting The Pieces Together Again\n",
227 | "==================\n",
228 | "Finally we have all the parts to make bad speech. Because of the limited training, the model pronounces some words wildly incorrectly, and it has a lot of trouble with common short words (ME, THE, YOU). I hard coded a few quality of life replacements into the input processing [such as line 301](https://github.com/kastnerkyle/diphone_synthesizer/blob/master/diphone_synthesis.py#L301), but generally you can rewrite words in a more \"pronouncible\" way to get different sounds. I also added an additional emphasis on the very first and last syllables in [line 124](https://github.com/kastnerkyle/diphone_synthesizer/blob/master/diphone_synthesis.py#L124) by doubling the first and last phones, this generally improved sound quality to my ears but is not really justifiable otherwise.\n",
229 | "\n",
230 | "The final result will look something like this:\n",
231 | "\n",
232 | "python diphone_synthesis.py congratulations this is the final message\n",
233 | "\n",
234 | "```\n",
235 | "Text: CONGRATULATIONS\n",
236 | "Predicted phones: ['K', 'OW', '_', 'NG', 'R', 'AE', 'CH', 'UW', 'L', 'EY', '_', 'SH', 'AH', 'N', 'Z']\n",
237 | "No match found?\n",
238 | "ng ng\n",
239 | "Text: THIS\n",
240 | "Predicted phones: ['_', 'TH', 'IH', 'S']\n",
241 | "Text: IS\n",
242 | "Predicted phones: ['IH', 'S']\n",
243 | "Replacing THE -> THEEE\n",
244 | "Text: THEEE\n",
245 | "Predicted phones: ['_', 'TH', 'IY', 'IY', '_']\n",
246 | "Text: FINAL\n",
247 | "Predicted phones: ['F', 'IH', 'N', 'AH', 'L']\n",
248 | "Text: MESSAGE\n",
249 | "Predicted phones: ['M', 'EH', '_', 'S', 'IH', 'JH', '_']\n",
250 | "```"
251 | ]
252 | },
253 | {
254 | "cell_type": "code",
255 | "execution_count": 2,
256 | "metadata": {
257 | "collapsed": false
258 | },
259 | "outputs": [
260 | {
261 | "data": {
262 | "text/html": [
263 | ""
264 | ],
265 | "text/plain": [
266 | "Bad Speech Synthesis Made Simple
5 |Introduction To Gaussian Processes
5 |Introduction To Page Rank
5 |Linear Regression
5 |Polyphase Signal Processing
5 |Robust Matrix Decomposition
5 |Single Speaker Word Recognition With Hidden Markov Models
5 |Using Pytables For Larger Than Ram Data Processing
5 |Wavelets
5 |Around the Web
12 |You can find me on Twitter at https://twitter.com/kastnerkyle
14 |
15 | For more code, see my GitHub at https://github.com/kastnerkyle
Papers I've written or helped write can be found on Google Scholar
21 | 22 |