27 |
28 |
29 | [ RIFFUSION ]
30 |
31 |
32 |
33 | (noun): riff + diffusion
34 |
35 |
62 |
63 |
64 | You've heard of{" "}
65 |
66 | Stable Diffusion
67 |
68 | , the open-source AI model that generates images from text?
69 |
70 |
74 |
75 | Well, we fine-tuned the model to generate images of spectrograms,
76 | like this:
77 |
78 |
82 |
83 | The magic is that this spectrogram can then be converted to an audio
84 | clip:
85 |
86 |
87 |
90 |
91 |
🔥🔥🔥😱
92 |
93 | Really? Yup.
94 |
95 |
96 | This is the v1.5 stable diffusion model with no modifications, just
97 | fine-tuned on images of spectrograms paired with text. Audio
98 | processing happens downstream of the model.
99 |
100 |
101 | It can generate infinite variations of a prompt by varying the seed.
102 | All the same web UIs and techniques like img2img, inpainting,
103 | negative prompts, and interpolation work out of the box.
104 |
105 |
106 |
107 | Code:{" "}
108 |
109 | https://github.com/riffusion/riffusion
110 |
111 |
112 | Discord:{" "}
113 |
114 | https://discord.gg/yu6SRwvX4v
115 |
116 |
117 |
118 |
Spectrograms
119 |
120 | An audio{" "}
121 | spectrogram{" "}
122 | is a visual way to represent the frequency content of a sound clip.
123 | The x-axis represents time, and the y-axis represents frequency. The
124 | color of each pixel gives the amplitude of the audio at the
125 | frequency and time given by its row and column.
126 |
127 |
132 |
133 | The spectogram can be computed from audio using the{" "}
134 |
135 | Short-time Fourier transform
136 | {" "}
137 | (STFT), which approximates the audio as a combination of sine waves
138 | of varying amplitudes and phases.
139 |
140 |
145 |
146 | The STFT is invertible, so the original audio can be reconstructed
147 | from a spectrogram. However, the spectrogram images from our model
148 | only contain the amplitude of the sine waves and not the phases,
149 | because the phases are chaotic and hard to learn. Instead, we use
150 | the{" "}
151 |
152 | Griffin-Lim
153 | {" "}
154 | algorithm to approximate the phase when reconstructing the audio
155 | clip.
156 |
157 |
158 | The frequency bins in our spectrogram use the{" "}
159 | Mel scale,
160 | which is a perceptual scale of pitches judged by listeners to be
161 | equal in distance from one another.
162 |
163 |
164 | Below is a hand-drawn image interpreted as a spectrogram and
165 | converted to audio. Play it back to get an intuitive sense of how
166 | they work. Note how you can hear the pitches of the two curves on
167 | the bottom half, and how the four vertical lines at the top make
168 | beats similar to a hi-hat sound.
169 |
170 |
175 |
176 |
183 |
184 |
185 | We use{" "}
186 |
187 | Torchaudio
188 |
189 | , which has excellent modules for efficient audio processing on the
190 | GPU. Check out our audio processing code{" "}
191 |
192 | here
193 |
194 | .
195 |
196 |
Image-to-Image
197 |
198 | With diffusion models, it is possible to condition their creations
199 | not only on a text prompt but also on other images. This is
200 | incredibly useful for modifying sounds while preserving the
201 | structure of the an original clip you like. You can control how much
202 | to deviate from the original clip and towards a new prompt using the
203 | denoising strength parameter.
204 |
205 |
206 | For example, here is that funky sax riff again, followed by a
207 | modification to crank up the piano:
208 |
209 |
210 |
211 |
216 |
217 |
218 |
221 |
222 |
223 |
224 |
229 |
230 |
231 |
234 |
235 |
236 |
237 |
238 | The next example adapts a rock and roll solo to an acoustic folk
239 | fiddle:
240 |
241 |
242 |
243 |
248 |
249 |
250 |
256 |
257 |
258 |
259 |
264 |
265 |
266 |
269 |
270 |
271 |
272 |
273 | Looping and Interpolation
274 |
275 |
276 | Generating short clips is a blast, but we really wanted infinite
277 | AI-generated jams.
278 |
279 |
280 | Let's say we put in a prompt and generate 100 clips with varying
281 | seeds. We can't concatenate the resulting clips because they differ
282 | in key, tempo, and downbeat.
283 |
284 |
285 | Our strategy is to pick one initial image and generate variations of
286 | it by running image-to-image generation with different seeds and
287 | prompts. This preserves the key properties of the clips. To make
288 | them loop-able, we also create initial images that are an exact
289 | number of measures.
290 |
291 |
296 |
297 | However, even with this approach it's still too abrupt to transition
298 | between clips. Multiple interpretations of the same prompt with the
299 | same overall structure can still vary greatly in their vibe and
300 | melodic motifs.
301 |
302 |
303 | To address this, we smoothly interpolate between prompts and seeds
304 | in the latent space of the model. In diffusion models, the{" "}
305 |
306 | latent space
307 | {" "}
308 | is a feature vector that embeds the entire possible space of what
309 | the model can generate. Items which resemble each other are close in
310 | the latent space, and every numerical value of the latent space
311 | decodes to a viable output.
312 |
313 |
314 | The key is that it's possible to sample the latent space between a
315 | prompt with two different seeds, or two different prompts with the
316 | same seed. Here is an example with the visual model:
317 |
318 |
322 |
323 | We can do the same thing with our model, which often produces
324 | buttery smooth transitions, even between starkly different prompts.
325 | This is much more interesting than interpolating the raw audio,
326 | because in the latent space all in-between points still sound like
327 | plausible clips. The figure below is colorized to show the latent
328 | space interpolation between two seeds of the same prompt. Playing
329 | this sequence is much smoother than just playing the two endpoints.
330 | The interpolated clips are often diverse and have their own riffs
331 | and motifs come and go.
332 |
333 |
338 | {/* TODO(hayk): Move one of these examples to the bottom. */}
339 |
340 | Here is one of our favorites, a beautiful 20-step interpolation from
341 | typing to jazz:
342 |
343 |
344 |
347 |
348 |
349 | And another one from church bells to electronic beats
350 | :
351 |
352 |
353 |
360 |
361 |
362 | Interpolation of arabic gospel, this time with the same
363 | prompt between two seeds:
364 |
365 |
366 |
369 |
370 |
371 | The Hugging Face{" "}
372 | diffusers{" "}
373 | library implements a wide range of pipelines including
374 | image-to-image and prompt interpolation, but we needed an
375 | implementation for interpolation combined with image-to-image
376 | conditioning. We implemented this pipeline, along with support for
377 | masking to limit generation to only parts of an image. Code{" "}
378 |
379 | here
380 |
381 | .
382 |
383 |
Interactive Web App
384 |
385 | To put it all together, we made an interactive web app to type in
386 | prompts and infinitely generate interpolated content in real time,
387 | while visualizing the spectrogram timeline in 3D.
388 |
389 |
390 | As the user types in new prompts, the audio smoothly transitions to
391 | the new prompt. If there is no new prompt, the app will interpolate
392 | between different seeds of the same prompt. Spectrograms are
393 | visualized as 3D height maps along a timeline with a translucent
394 | playhead.
395 |
396 |
401 |
402 | The app is built using Next.js,{" "}
403 | React,{" "}
404 | Typescript,{" "}
405 | three.js,{" "}
406 | Tailwind, and{" "}
407 | Vercel.
408 |
409 |
410 | The app communicates over an API to run the inference calls on a GPU
411 | server. We used Truss to
412 | package the model and test it locally before deploying it to Baseten
413 | which provided GPU-backed inference, auto-scaling, and
414 | observability. We used NVIDIA A10Gs in production.
415 |
416 |
417 | If you have a GPU powerful enough to generate stable diffusion
418 | results in under five seconds, you can run the experience locally
419 | using our test flask server.
420 |
421 |
Code
422 |
442 |
443 |
Prompt Guide
444 |
445 | Like other diffusion models, the quality of the results depends on
446 | the prompt and other settings. This section provides some tips for
447 | getting good results.
448 |
449 |
450 | Seed image - The app does image-to-image conditioning, and
451 | the seed image used for conditioning locks in the BPM and overall
452 | vibe of the prompt. There can still be a large amount of diversity
453 | with a given seed image, but the effect is present. In the app
454 | settings, you can change the seed image to explore this effect.
455 |
456 |
457 | Denoising - The higher the denoising, the more creative the
458 | results but the less they will resemble the seed image. The default
459 | denoising is 0.75, which does a good job of keeping on beat for most
460 | prompts. The settings allow raising the denoising, which is often
461 | fun but can quickly result in chaotic transitions and tempos.
462 |
463 |
464 | Prompt - When providing prompts, get creative! Try your
465 | favorite styles, instruments like saxophone or violin, modifiers
466 | like arabic or jamaican, genres like jazz or rock, sounds like
467 | church bells or rain, or any combination. Many words that are not
468 | present in the training data still work because the text encoder can
469 | associate words with similar semantics. The closer a prompt is in
470 | spirit to the seed image And BPM, the better the results. For
471 | example, a prompt for a genre that is much faster BPM than the seed
472 | image will result in poor, generic audio.
473 |
474 |
475 | Prompt Reweighting - We have support for providing weights
476 | for tokens in a prompt, to emphasize certain words more than others.
477 | An example syntax to boost a word is (vocals:1.2), which applies a
478 | 1.2x multiplier. The shorthand (vocals) is supported for a 1.1x
479 | boost or [vocals] for a 1.1x reduction.
480 |
481 |
482 | Parameters can also be specified via URL, for example:
483 |
484 |
485 | https://www.riffusion.com/?&prompt=rainy+day&
486 | denoising=0.85&seedImageId=og_beat
487 |
488 |
489 |
Examples
490 |
491 | The app suggests some of our favorite prompts, and the share panel
492 | allows grabbing the spectrogram, audio, or a shareable URL. We're
493 | also posting some favorites at{" "}
494 |
499 | /r/riffusion
500 |
501 | .
502 |
503 |
504 | Here are some longer-form interpolations we like:
505 |
506 |
507 | Sunrise DJ Set to hard synth solo:
508 |
509 |
510 |
517 |
518 |
519 | Detroit Rap to Jazz:
520 |
521 |
522 |
529 |
530 |
531 | Cinematic New York City in a Dust Storm to{" "}
532 | Golden hour vibes:
533 |
534 |
535 |
542 |
543 |
544 | Techno beat to Jamaican rap:
545 |
546 |
547 |
554 |
555 |
556 | Fantasy ballad, female voice to teen boy pop star:
557 |
558 |
559 |
566 |
567 |
568 |
Citation
569 |
570 | If you build on this work, please cite it as follows:
571 |
572 |
573 | {`@software{Forsgren_Martiros_2022,
574 | author = {Forsgren, Seth*
575 | and Martiros, Hayk*},
576 | title = {{Riffusion -
577 | Stable diffusion for
578 | real-time music generation}},
579 | url = {https://riffusion.com/about},
580 | year = {2022}
581 | }`}
582 |
583 |
584 |
585 | {`@software{Forsgren_Martiros_2022,
586 | author = {Forsgren, Seth* and Martiros, Hayk*},
587 | title = {{Riffusion - Stable diffusion for real-time music generation}},
588 | url = {https://riffusion.com/about},
589 | year = {2022}
590 | }`}
591 |
592 |
593 |