71 |
72 | I've found that linear interpolation gives comparable results to proper
73 | sinc interpolation, so that's what I went with. To avoid glitches, I
74 | added some hysteresis to the threshold detection. The interpolation
75 | ratio is 20.
76 |
77 | # Clock recovery
78 |
79 | The first step in decoding any kind of signal without an explicit clock
80 | is recovering the clock from the signal itself. This usually aided by
81 | some kind of line coding that ensures that there are only so many
82 | consecutive bits without a transition. In our case the line coding is
83 | [Eight-to-fourteen
84 | modulation](https://en.wikipedia.org/wiki/Eight-to-fourteen_modulation)
85 | that maps one byte to 14 channel bits. It is then
86 | pressed onto the disc using NRZ-I (Non-return to zero inverted)
87 | encoding, that is a '1' is encoded as a transition
88 | and a '0' as no transition. The combination of these two
89 | encodings guarantees that there are no more than 11 and no less
90 | than 3 unchanging consecutive bits.
91 |
92 | This means that when looking at the signal we can't assume that the
93 | shortest time between two transitions is one unit interval, instead
94 | it's three.
95 |
96 | The usual way of recovering the clock from a signal with an embedded
97 | clock is by means of a [phase-locked
98 | loop](https://en.wikipedia.org/wiki/Phase-locked_loop). While these are
99 | usually implemented in a mixed-signal circuit, implementing one in
100 | software can be surprisingly easy. In this instance, it can be seen as
101 | a discrete-time simulation that's clocked at 400 MHz, i.e. on every
102 | interpolated bit.
103 |
104 | Same as with a hardware PLL we need three main components. VCO, phase
105 | detector and loop filter.
106 |
107 | ## VCO
108 |
109 | A simple way of implementing a VCO in software is by means of an
110 | [Numerically-controlled
111 | oscillator](https://en.wikipedia.org/wiki/Numerically-controlled_oscillator).
112 | Since all we need is know when to sample the input signal, the
113 | phase-to-amplitude converter part of the NCO can be reduced to
114 | detecting if the accumulator has wrapped around.
115 |
116 | The phase accumulator is as simple as adding the frequency tuning word
117 | to the accumulator modulo the accumulator size on every clock cycle :
118 |
119 | ```python
120 | acc = 0
121 | last_acc = 0
122 | ftw = 42
123 | acc_size = 1000
124 | for bit in all_bits :
125 | if acc < last_acc :
126 | # integrator has wrapped around, sample the input
127 | last_acc = acc
128 | acc = (acc+ftw)%acc_size # that's the actual NCO
129 | ```
130 |
131 | ## Phase detector
132 |
133 | The job of the phase detector is to convert the phase difference
134 | between the output of the VCO and the incoming data stream into a
135 | proportional voltage. With the VCO being an NCO, implementing the phase detector is
136 | as simple as sampling the value of the phase accumulator whenever the
137 | input signal changes. That way, the phase detector also keeps its output
138 | constant in the absence of transitions at the input. To keep the sampling point
139 | as far away from the input transitions as possible, we want the phase of the
140 | VCO to be 180° at the transitions.
141 |
142 | ```python
143 | last_bit = False
144 | phase_delta = 0
145 | for bit in all_bits :
146 | if last_bit != bit :
147 | phase_delta = (acc_size/2 - acc)
148 | last_bit = bit
149 |
150 | ```
151 |
152 | ## Loop filter
153 |
154 | To smooth the output of the phase detector before feeding it into the
155 | VCO, we need some kind of low-pass filter. I went with the equivalent
156 | of a first order low pass since that's trivial to implement and turned
157 | out be good enough.
158 |
159 | ```python
160 | last_bit = False
161 | phase_delta = 0
162 | delta_filtered = 0
163 | for bit in all_bits :
164 | ...
165 | alpha = .005
166 | delta_filtered = delta*alpha + delta_filtered*(1-alpha)
167 | ...
168 | ```
169 |
170 | ## Putting it all together
171 |
172 | Here's how it all looks connected:
173 |
174 | 
176 |
177 | I found it really instructive to
178 | discover that a
179 | PLL-based clock recovery can be implemented in about a dozen lines of
180 | Python or any other imperative language without the use of any
181 | high-level tools such as Simulink. Apart from that it's quite
182 | fascinating how little it takes to implement a system that exhibits
183 | complex dynamic behaviour. Contrast that to other code, where the same
184 | number of lines just adds a couple of buttons to a window or so and
185 | requires calling into thousands of lines of library code.
186 |
187 | ## Making it lock
188 |
189 | Anyone who has ever dealt with closed-loop feedback systems will tell you that
190 | debugging them can be really difficult since it's hard to separate
191 | cause from effect.
192 |
193 | To get around this, we first operate our PLL open-loop by setting the loop
194 | gain to zero. It's also worth noting that a PLL with this kind of phase
195 | detector that's not sensitive to frequency will have a fairly tight
196 | lock range, which means that we need to get the VCO center frequency
197 | fairly close to its nominal frequency.
198 |
199 | After playing with the center frequency of the VCO and the loop filter
200 | corner frequency this is what we get:
201 |
202 | 
203 |
204 | We can see that the phase detector outputs a sawtooth waveform which
205 | indicates that there's a frequency offset between the VCO and input
206 | frequency. Closing the loop by increasing the loop gain, the PLL locks
207 | and the phase error becomes constant. Why constant and not zero, you
208 | may ask? To bring the VCO to the correct frequency, its tuning voltage,
209 | i.e. the output of the phase detector and loop filter must be non-zero,
210 | resulting in a residual phase offset. We can eliminate that offset by
211 | introducing and integrator the loop so that we can get a zero phase
212 | detector output and non-zero tuning voltage. Tweaking the integrator
213 | gain, we get this:
214 |
215 | 
216 |
217 |
218 | The output of the phase detector being relatively close to zero indicates that our
219 | PLL is working as intended so we can move on to the next step, that is
220 | sampling the input signal with the recovered clock. As mentioned in the
221 | VCO section, this is as simple as capturing the value of the input
222 | signal every time the phase accumulator overflows.
223 |
224 | ```python
225 | sampled_bits = []
226 | for bit in all_bits :
227 | if acc < last_acc :
228 | sampled_bits.append(bit)
229 | ...
230 | ```
231 |
232 | This leaves us with a stream of bits that we need to make sense of.
233 |
234 | Here's another plot to verify that the clock recovery is working as it
235 | should:
236 |
237 | 
238 |
239 | We see that the VCO's phase is close to 180° when the input
240 | transitions and thus the phase wraparounds are as far from the edges as
241 | they can be.
242 |
243 | ## Lock range
244 |
245 | Now that the clock recovery was working, I was curious to see how close
246 | to the actual frequency I need to get the VCO's initial frequency, i.e.
247 | what the PLL's lock range is. The correct frequency tuning word the
248 | locked PLL settles on is 42.6. The minimum initial frequency tuning
249 | word to achieve this is 41.6, the maximum is 43.9. This results in a
250 | lock range of about ±2.3%. I don't know if this is particularly good or
251 | bad for a PLL-based clock recovery.
252 |
253 | # NRZI decoding
254 |
255 | As mentioned before, ones and zeros aren't encoded as-is on a CD,
256 | instead a '1' is econded as
257 | a transition and a '0' as no transition. This means it's insignificant
258 | whether a land on the disc is a high level or a low level. Decoding it is as simple as
259 | comparing each value to its predecessor:
260 |
261 | ```python
262 | nrz_bits = [a != b for a,b in zip(sampled_bits[1:], sampled_bits)]
263 | ```
264 |
265 | # Framing
266 |
267 | Data on a compact disc is structured as frames of 588 channel bits:
268 |
269 | 
270 |
271 | First, we have a 24 bit long sync pattern. The sync pattern has been
272 | chosen in such a way that it can't appear in valid data, so we can be
273 | absolutely sure to have found the start of a frame if we've seen the
274 | sync pattern.
275 |
276 | To extract the frames from the bitstream, we do exactly this and write
277 | the sync pattern as well as the following 588-24 = 564 channel bits to
278 | a file for further decoding. The file contains each frame encoded as 588
279 | `1`s and `0`s per line.
280 |
281 | The sync word is followed by the EFM-encoded control byte and 32
282 | payload bytes, 24 of which are actual PCM samples and 8 of which are
283 | parity bytes. Each EFM-encoded byte is separated from its neighboring
284 | bytes by three merging bits to guarantee DC balance and meet the
285 | specified run length requirements. The actual value of the merging bits
286 | is irrelevant and isn't used in the decoding process.
287 |
288 | See [analyze.py](/analyze.py) for the implementation of all of this.
289 |
290 | # Frame decoding
291 |
292 | With the frames neatly put into a file we're now safely in the digital
293 | realm and can start to make sense of
294 | the data stored in them. Since the python code for this is much less
295 | interesting and magic than the clock recovery, there'll be fewer code snippets throughout this section.
296 |
297 | ## Subcode
298 |
299 | Apart from the actual PCM samples, there's metadata in the form of
300 | subcode on the disc. We'll deal with this first since it's simpler to
301 | decode than the PCM samples and it's much easier to tell if we got it right.
302 |
303 | This is the first time where we actually have to decode the
304 | eight-to-fourteen modulation. Decoding it is as simple as going through
305 | a lookup table that maps 14 bit words to 8 bit words. The contents of
306 | the lookup table are given as images in the standard. Too lazy to type
307 | thousands of ones and zeros, I OCR'd them using tesseract and
308 | massaged the resulting text into a lookup table that's read from the
309 | python script[^3].
310 |
311 | The subcode is stored in a way that I found confusing at first: A CD
312 | contains 8 channels of subcode, one of which is actually interesting.
313 | Rather than storing the content of each subcode channel contiguously,
314 | The subcode byte in the frame carries 1 bit for each of the 8 subcode
315 | channels at once, so we get one bit for each subcode channel per frame.
316 |
317 | 
318 |
319 |
320 | The subcode itself also has some kind of framing structure, called
321 | blocks. Each block contains 96 payload bits. And is delimited by two
322 | 14-bit synchronization words S₀ and S₁ that don't map to anything the EFM LUT, so
323 | there's no ambiguity in finding the start of a block.
324 |
325 | The first (P) subcode channel encodes the start of a track and is '0'
326 | for the remainder of the track. The second (Q) channel is the one
327 | that contains data we can make sense of.
328 |
329 | All Q-channel blocks I decoded have the ADR bits set to `0001`, meaning that the
330 | DATA-Q bits are to be interpreted as Mode 1:
331 |
332 | 
333 |
334 | It's a bit odd that all numbers are BCD-encoded, I guess it was done
335 | that way so displaying them on a 7-segment display requires the least
336 | amount of logic.
337 |
338 | - Track number: Current track on the Disc
339 | - Index: Tracks can be subdivided by indices, but very few discs use this
340 | - Min/Sec: Current run time of the track
341 | - Frame: Each second is subdivided into 75 frames (This frame is
342 | different from the 588-bit channel frame)
343 |
344 | Given that one frame of 588 channel bits contains 24 bytes of audio
345 | that make up 6 samples at a sample rate of 44.1 kHz, we get
346 | frames at a rate of 44.1kHz / 6 = 7350 frames/sec. Since it takes 98
347 | frames to form a block, this results in a block rate of 44.1kHz / 6 /
348 | 98 = 75 Hz. This explains why each second is subdivided the way it is.
349 |
350 | Decoding the data I captured, we get this:
351 |
352 | ```
353 | Track 1.1 R=01:08:70 A=01:10:70
354 | Track 1.1 R=01:08:71 A=01:10:71
355 | Track 1.1 R=01:08:72 A=01:10:72
356 | Track 1.1 R=01:08:73 A=01:10:73
357 | Track 1.1 R=01:08:74 A=01:10:74
358 | Track 1.1 R=01:09:00 A=01:11:00
359 | Track 1.1 R=01:09:01 A=01:11:01
360 | Track 1.1 R=01:09:02 A=01:11:02
361 | Track 1.1 R=01:09:03 A=01:11:03
362 | ```
363 |
364 | We can see that track number matches what was on the DVD player when I
365 | captured the waveform and that the frame number is incrementing as it
366 | should and wraps around at 75.
367 |
368 | Looking good so far!
369 |
370 | The only thing left to do is to check that the CRC is correct. The
371 | standard specifies the polynomial to be x¹⁶ + x¹² + x⁵ + 1 which is
372 | identical to the 16-bit CRC-CCITT and that the
373 | parity bits are stored inverted. Feeding the CONTROL, ADR, DATA-Q and
374 | inverted parity bits into the CRC should yield a zero CRC.
375 |
376 | ```python
377 | def calc_crc(bits) :
378 | poly = 0x1021
379 | crc = 0
380 | for bit in bits :
381 | if (crc>>15)&1 != bit :
382 | crc = ((crc<<1)&0xffff) ^ poly
383 | else :
384 | crc = ((crc<<1)&0xffff)
385 | return crc
386 | ```
387 |
388 | I was both surprised and relieved when the CRC did indeed turned out to
389 | be zero for all blocks, since I wouldn't have had much of an idea where to start
390 | debugging this other than staring at the code.
391 |
392 | ## Samples
393 |
394 | Successfully having decoded the Q-Channel subcode gave me the confidence that
395 | all of the previous steps, including OCR'ing the EFM LUT, were done correctly, so it's now time for the
396 | main event, the audio samples in glorious 16 bit linear PCM.
397 |
398 | Revisiting the frame structure, we're now looking at the data and
399 | parity bytes:
400 |
401 | 
402 |
403 | Each frame contains 32 payload bytes, 24 of which are data and 8 of
404 | which are parity byes.
405 |
406 | As every article on compact disc digital audio mentions, samples are
407 | encoded using [Cross-interleaved reed-solomon coding
408 | (CIRC)](https://en.wikipedia.org/wiki/Cross-interleaved_Reed%E2%80%93Solomon_coding). The
409 | specification helpfully provides block diagrams of the encoder and
410 | decoder, which I've redrawn and put side-by-side for clarity:
411 |
412 | 
413 |
414 | Tracing each byte from input to output, we see that each byte is
415 | subject to the same delay of 111 frames, so the data stays as-is.
416 |
417 | The important takeaway from the encoder block diagram is that the
418 | reed-solomon forward error correction (FEC) in the C₁ and C₂ encoders merely tacks on extra parity bytes and leaves the
419 | sample data as-is. That means we can focus on deinterleaving first
420 | and deal with the FEC later on.
421 |
422 | Ignoring the FEC, the decoder is exactly the inverse of the
423 | encoder.
424 |
425 | Based on that knowledge, we can substitute the C₁ and C₂ decoder boxes in
426 | the above block with pass-throughs as indicated by the dashed lines. This leaves us
427 | with having to implement the various delay and reordering elements.
428 |
429 | I implemented the delay elements as a shift register encapsulated in a
430 | class. Calling the `step` method returns the value from a prior
431 | invocation, depending on `n_delay`.
432 |
433 | ```python
434 | class Delay:
435 | def __init__(self, n_delay, fill = None) :
436 | self.register = [fill]*n_delay
437 |
438 | def step(self, v) :
439 | if len(self.register) == 0 :
440 | return v
441 | r = self.register[-1]
442 | self.register = [v] + self.register[:-1]
443 | return r
444 |
445 | d = Delay(3)
446 | print(d.step(1)) # prints None
447 | print(d.step(2)) # prints None
448 | print(d.step(3)) # prints None
449 | print(d.step(4)) # prints 1
450 | print(d.step(5)) # prints 2
451 | ....
452 |
453 | ```
454 |
455 | For each column of delay stages, we create a list that has the
456 | individual delay elements:
457 |
458 | ```python
459 | first_delays = [Delay(0 if i%2 == 0 else 1, 0) for i in range(32)]
460 | ```
461 |
462 | Applying the delay to all input symbols then is as simple as zipping
463 | them with the delay elements and calling `step` on each of them:
464 |
465 | ```python
466 | symbols_delayed1 = [d.step(x) for d,x in zip(first_delays, symbols_in)]
467 | ```
468 |
469 | The other delay columns work more or less identical apart from different
470 | delay values.
471 |
472 | Right before the last delay columns, we need to shuffle the samples as
473 | indicated in the decoder diagram. Thanks to list comprehensions, that's
474 | a one-liner as well:
475 |
476 | ```python
477 | # output to input sample
478 | deinterleave_tab = ( 0, 1, 6, 7, 16, 17, 22, 23, 2, 3, 8, 9, 18, 19, 24, 25, 4, 5, 10, 11, 20, 21, 26, 27)
479 | symbols_deinterleaved = [symbols_delayed2[i] for i in deinterleave_tab]
480 | ```
481 |
482 | After this deinterleaving step, we get 24 bytes that form 6 consecutive
483 | stereo samples. All that's left to do is combine the bytes into the
484 | appropriate samples taking into account two's complement.
485 |
486 | We then run this process for all frames and dump the samples into a
487 | wave file at 44.1 kHz sample rate, so we get something we can
488 | listen to. Even though we only got about 0.8 s of audio, it sounds
489 | plausible. To know if I got it all right, I ripped that particular
490 | track as an uncompressed wave file and imported it into Audacity.
491 | I aligned it to the decoded audio snippet with the help of the
492 | timestamps from the subcode and inverted it. Adding both tracks
493 | resulted in absolute silence, confirming that the decoded samples are
494 | indeed correct!
495 |
496 | See [decode.py](/decode.py) for the implementation of all of this.
497 |
498 |
499 | # Something's not quite right...
500 |
501 | Remember that we initially captured 4 MSa at 20 MSa/s on the
502 | oscilloscope? This
503 | corresponds to a record length of 0.2 s. However, we got 0.78
504 | seconds of audio. Since compact disc digital audio operates in real
505 | time, this is doesn't quite add up. The frequency tuning word from the
506 | PLL backs this up: The raw bit rate of a compact disc should be:
507 | 44.1 kHz / 6 samples per frame × 588 bits per frame =
508 | 4.3218 MBit/s. However, the bit rate as per the VCO frequency is
509 | 42.6 / 1000 × 20 MSa/s × 20 = 17.04 MBit/s. Multiplying this
510 | by the ratio between the oscilloscope record length and the length
511 | of the decoded audio, we get: 17.04 MBit/s × (0.2 s /
512 | 0.78 s) = 4.33 MBit/s, which is fairly close to the expected
513 | bit rate, so at least that that checks out.
514 |
515 | All of this didn't quite make sense at first. If the DVD player is
516 | reading the disc about 4× faster than it needs to, where's that data
517 | going? Then I remembered the dropouts I initially saw on the
518 | oscilloscope. Maybe these dropouts aren't actually dust or scratches,
519 | but the DVD player too fast to and then jumping
520 | back on the disc to maintain the
521 | correct data rate on average?
522 |
523 | To find out if that's the case, I captured another portion of the
524 | waveform that has such a dropout:
525 |
526 | 
528 |
529 | At first attempt, the clock recovery PLL didn't relock after the first
530 | dropout and stayed unlocked thereafter. Looking at the PLL's signals, I
531 | found out that the integrator went off the rails during the dropout to
532 | the point where it was so far off that the PLL had no chance of ever
533 | locking again. Clamping the magnitude of the integrator value to
534 | slightly above its nominal value did the trick to get the PLL to relock
535 | after the dropouts.
536 |
537 | With that obstacle out of the way we can decode the Q-Channel subcode
538 | as before:
539 |
540 | ```
541 | [...]
542 | Track 2.1 R=00:17:45 A=06:32:53
543 | Track 2.1 R=00:17:46 A=06:32:54
544 | Track 2.1 R=00:17:47 A=06:32:55
545 | CRC error
546 | Track 2.1 R=00:17:27 A=06:32:35
547 | Track 2.1 R=00:17:28 A=06:32:36
548 | Track 2.1 R=00:17:29 A=06:32:37
549 | [...]
550 | Track 2.1 R=00:17:45 A=06:32:53
551 | Track 2.1 R=00:17:46 A=06:32:54
552 | Track 2.1 R=00:17:47 A=06:32:55
553 | CRC error
554 | Track 2.1 R=00:17:27 A=06:32:35
555 | Track 2.1 R=00:17:28 A=06:32:36
556 | Track 2.1 R=00:17:29 A=06:32:37
557 | [...]
558 | Track 2.1 R=00:17:45 A=06:32:53
559 | Track 2.1 R=00:17:46 A=06:32:54
560 | Track 2.1 R=00:17:47 A=06:32:55
561 | CRC error
562 | Track 2.1 R=00:17:27 A=06:32:35
563 | Track 2.1 R=00:17:28 A=06:32:36
564 | Track 2.1 R=00:17:29 A=06:32:37
565 | ```
566 |
567 | Based on time codes, we can see that the DVD player is indeed reading
568 | the same part of the disc multiple times. Suspicion confirmed!
569 |
570 | My best guess for why the DVD player isn't reading the disc at the
571 | nominal rate is that the signal conditioning and clock recovery circuitry in the
572 | SoC can't handle the nominal rate since it also has to
573 | process the significantly faster signal from a DVD.
574 |
575 | # Closing thoughts
576 |
577 | Decoding compact disc digital audio from the pickup signal to PCM
578 | samples was a really interesting project and was surprisingly easy
579 | after getting the clock recovery right and fixing a couple of stupid
580 | bugs in the deinterleaver. It's also worth noting that this project
581 | only required a digital oscilloscope with half-decent memory depth, a CD
582 | player and some basic DSP and bit shuffling knowledge, so I'm wondering
583 | why I haven't found much evidence of people having done this before.
584 |
585 | One thing that I didn't cover is implementing the Reed-Solomon
586 | decoder. That'll has to wait until I've developed a better
587 | understanding of Reed-Solomon forward error correction.
588 |
589 |
590 | [^1]: While writing this article, a friend of mine pointed me to
591 | [this](http://www.pmonta.com/compact-disc-microscopy.html) post where
592 | someone did just that!
593 |
594 | [^2]: In the process of implementing the clock recovery, I was scratching my
595 | head why I was seeing lots of jitter on the acquired signal. Turns out
596 | that Keysight InfiniiVision oscilloscopes by default randomize the the
597 | time between samples at low sample rates to prevent aliasing. Turning
598 | off the antialiasing option in the display menu indeed fixed the
599 | problem. Another thing worth mentioning is that one needs to press the
600 | single shot button to get maximum memory depth on this series of
601 | oscilloscopes. Just pressing stop only yields half the memory depth.
602 |
603 | [^3]: Only later I discovered that the freely available [ECMA-130
604 | Standard](https://www.ecma-international.org/wp-content/uploads/ECMA-130_2nd_edition_june_1996.pdf)
605 | for CD-ROM includes a textual representation of the EFM LUT.
606 |
--------------------------------------------------------------------------------
/media/frame.svg:
--------------------------------------------------------------------------------
1 |
2 |
3 |
--------------------------------------------------------------------------------
/media/subcode2.svg:
--------------------------------------------------------------------------------
1 |
2 |
3 |
--------------------------------------------------------------------------------