Samuel text-to-speech

Apart from mimicking your voice, Samuel can now do TTS. Finally, a text-to-speech for the 80s!

My original concept for Samuel was to create a text-to-speech that sounds like 1982's S.A.M. (try a JS remake here) but give it a natural prosody using machine learning from 2026. I decided to use Pink Trombone as a fun and visual way to make SAM-like speech.

Method

The way you usually train a TTS is by collecting a dataset of speech with corresponding transcripts. You feed in a transcript to the model and you ask it to predict the speech:

And this is the tricky part of Samuel TTS: how do you get the targets? The model doesn't predict waveforms, it predicts instead parameter trajectories for Pink Trombone. But what are the correct trajectories that we want it to output? There is no ground truth available.

Imagine we have a magic box that will take a speech waveform and tell us what parameter trajectories for Pink Trombone reproduce it. Let's call this magic box a codec. Afterwards, training the TTS becomes feasible: we just pass our speech-text dataset through the codec. That gives us ground truth trajectories that we can use to set up a loss function and train the model.

(Or, even simpler, we could just take an existing TTS and attach the codec on top, pink-trombonizing the waveform that comes out. That requires no extra training, but it's wasteful. Predicting Pink Trombone trajectories is easier than predicting waveforms, meaning we can get away with a smaller model.)

I use the term codec here in analogy with neural audio codecs: models that compress audio into a compact representation that makes training audio models easier. The Samuel codec is particular: we only train the encoder, the "latents" are the trajectories and the decoder is Pink Trombone.

So the harder part of this project is training the codec. You can read more about how I did that in this post. Here, let's look at how the text-to-speech itself works:

Building on Pocket TTS

In a fortunate coincidence, my colleagues from Kyutai had open-sourced the training code for Pocket TTS (code, blog, paper) around the time when I finished training the codec. It's a small text-to-speech architecture for CPU-friendly models.

Like all text-to-speech models nowadays, it uses a neural audio codec. to compress audio before passing it into the main model. That meant that the development of Samuel TTS consisted of pretty much just pointing Claude Code at the Pocket TTS training code and telling it to replace its codec by Samuel.

Specifically, Pocket TTS uses a continuous codec rather than the more classic discrete RVQ codec described in my blog post. That's convenient for us, because our parameter trajectories are continuous as well.

I also tuned the model size to obtain the smallest model that reaches a performance threshold (in terms of word error rate), and landed at 7M params for the TTS + 3M params for the codec = 10M for the whole thing!

In comparison, Pocket TTS, already considered a tiny model, has 100M parameters. Of course, mine is not a serious TTS and I'm not saying these two can be compared apples-to-apples, but it shows that the compressing the audio using Pink Trombone makes learning easier.

Training

Pocket TTS is a flow matching model, meaning it's trained to denoise its data - in this case, the parameter trajectories. Here's the high-level idea of how training works:

Take a sample and pass the audio through the codec to get the parameter trajectories. For clarity, we only show four out of the eight Pink Trombone parameters here.

Tokenize: Standard byte-pair encoding for the text. For the audio, we group the trajectories into groups of 8 frames, meaning 93ms per token.

Add noise: Take each trajectory token and create a noised version. Some tokens get more noise, others get less.

Ask the model to denoise: At each step $t$, the model can see all text tokens, the past trajectory tokens up to step $t-1$, and the noised trajectory token at time $t$. We also tell it how much noise was applied. Based on these inputs, it tries to predict the clean version of the trajectory token, and we apply a loss on the difference between its prediction and the ground truth. In practice, this is batched across the time axis for efficiency, rather than happening one-by-one as the animation shows it.

Inference

At inference time, we give the model complete noise and ask it to denoise it, knowing the text and the previously generated audio. In classic flow matching, you need to denoise gradually over multiple steps, but we use a technique called LSD (Lagrangian Self-Distillation) that allows us to denoise in a single step.

The generated frame is then added to the history and we denoise the next one. Unlike in training, each denoising depends on the previously-generated frames. Therefore this process cannot be batched and does actually happen sequentially like in the animation.

More info

Those were the important bits, but here are the details in case you're a speech AI researcher or want to reproduce this for some reason:

Samuel TTS follows Pocket TTS in many aspects. You can read more about the architecture in the blog post and paper.

Here are the ways in which Samuel TTS differs, apart from the codec being replaced:

Smaller model: Backbone hidden dim 1024 → 256, flow head hidden dim 512 → 256, flow head depth 6 → 3.

Smaller text tokenizer: since the model is tiny now, an embedding for 1024 tokens would be about 40% of the model parameters. We use a vocab of 256 tokens, but it does make the model worse at pronouncing rarer words.

Frame rate: Pocket TTS' codec Mimi uses 12.5 Hz, but the Samuel codec operates at ~86 Hz (44100 Hz / 512 samples per frame). To get to a similar rate, we group the codec frames by eight, yielding a weird ~10.8 Hz.

No distillation: The Pocket TTS training recipe is to first train a 24-layer model and then distill it into a 6-layer one. We just directly train a small model. Could probably get better results if we did distill (but who cares).

No voice cloning: Pocket TTS supports voice cloning, meaning you can put a short voice sample and it will speak using that voice. Here we skip that for simplicity, so the TTS just picks a voice at random at sampling time. If you sample multiple times for the same text, you'll hear that the voice is deep some times and shrill at other times.