dots.tts Technical Report

dots.tts is a fully open (Apache 2.0, code + checkpoints) 2B-parameter TTS system that generates speech autoregressively in a continuous latent space — no discrete acoustic tokens anywhere — and still hits the best average WER/speaker-similarity on Seed-TTS-Eval while streaming its first audio packet in 54 ms. The interesting part isn't any single trick; it's that the paper assembles a complete, reproducible answer to the question that has kept continuous-AR TTS out of production: how do you stop tiny per-step prediction errors from compounding into drift over a long rollout?

Why continuous autoregression is the hard road

Modern TTS has three camps. Non-autoregressive flow-matching models (F5-TTS, Voicebox) generate a whole utterance in parallel — great for offline dubbing, awkward for live dialogue. Discrete-token AR models (CosyVoice, Qwen3-TTS, Seed-TTS) quantize speech into a vocabulary and reuse the entire LLM stack: next-token training, preference tuning, vLLM serving. That's why they dominate deployment.

But the tokenizer is a ceiling. A low-bitrate codec flattens exactly the things users increasingly want — emotional paralinguistics, singing, ambient sound, subtle timbre. The obvious fix is to skip quantization and predict continuous latents directly (DiTAR, VibeVoice, VoxCPM, KALL-E all do variants of this). The catch is subtle and important:

Discrete tokens are self-correcting; continuous latents are not. When a discrete-token model samples an imperfect token, the codec snaps it back to a valid point on the acoustic manifold before it reaches the waveform or the model's own context. A continuous decoder faithfully reconstructs every error, and — worse — feeds the erroneous latent back as conditioning for the next step. Errors compound. This is the same exposure-bias problem that plagued early seq2seq models, but with no discrete grid to absorb the drift.

dots.tts attacks this compounding at three different layers, and each attack is worth understanding.

The architecture: separate "what to say" from "how it sounds"

The system has a frozen AudioVAE (48 kHz audio ↔ 128-dim latents at 25 Hz, a 1920× temporal compression) and a three-part backbone:

The feedback loop is deliberately lossy. Each generated 4-frame latent patch is compressed through the semantic encoder before returning to the LLM, so acoustic prediction errors are stripped out of the LLM's context. The DiT head, meanwhile, sees the full raw acoustic history.

The design principle: the LLM should reason about content and the DiT head should render acoustics, and neither should be able to corrupt the other. Two decisions enforce this.

First, the LLM never sees raw VAE latents. Each newly generated 25 Hz patch is compressed by a pretrained semantic encoder into a single 6.25 Hz embedding — a summary aligned with the LLM's text-semantic space, with high-variance acoustic detail stripped out. The authors say flatly that this was necessary for stable continuous-AR rollouts. Intuitively, this is a soft version of what discrete tokens do for free: it projects the noisy generated latent onto a low-dimensional "meaning" manifold before it can pollute the LLM's context.

Second — and this is the "full-history conditioning" from the abstract — the flow-matching head is not a local patch renderer conditioned only on the current LLM state (as in DiTAR-style designs). Its context at step n is the interleaved sequence [H_0, P_0, H_1, P_1, \ldots, H_n, Z_n], where H_i is the LLM hidden state at audio position i, P_i are the clean previously generated latent patches, and Z_n is the noisy patch currently being denoised. The DiT sees the entire raw acoustic history, which is what preserves long-range timbre and prosody consistency and reduces drift. A neat consequence: since the DiT's prefix already contains every LLM hidden state, the DiT is on its own a complete text-conditioned speech generator (the ARDiT observation) — which is exactly what pushes the LLM toward encoding semantics rather than acoustics.

Training this AR head efficiently requires a clever attention layout (Section 2.5.2, worth reading): the sequence is duplicated into a clean "cause" half and a noisy "generation" half with shared RoPE position IDs, and a four-block mask makes the parallel training pass numerically identical to a step-by-step inference rollout. All N patches train in one forward pass with zero train/inference mismatch in the attention pattern.

A latent space designed to be predicted, not just reconstructed

The AudioVAE gets its own two-stage treatment, and the second stage encodes a lesson the field has been converging on: reconstruction quality and predictability are different axes, and a heavily compressed latent can be excellent at one and terrible at the other.

Stage 1 is a standard BigVGAN-v2-style adversarial reconstruction recipe plus a KL + flow-regularized prior. Stage 2 keeps those losses and adds two supervisions directly on the latent: a frame-level cosine alignment against frozen WavLM features (layer 23), and a multitask block (small encoder + small LM head) trained on ASR, emotion, and speaker classification. The multitask LM is thrown away; the small encoder survives and becomes the backbone's semantic frontend. So the same module that made the latent space semantically structured during VAE training is the one that later summarizes patches for the LLM — a tidy piece of infrastructure reuse.

The reconstruction numbers confirm the latent isn't the bottleneck: on LibriSpeech test-other, the VAE reconstructs at WER 4.14% and SIM 0.969 versus ground truth's 4.59%/1.000 — essentially transparent — while operating at only 25 Hz. Discrete codecs in the same table sit at WER 5.8–13.4% and SIM 0.68–0.85. That gap is the "perceptual ceiling" argument for continuous latents in one table.

Reward-free self-correction: train on your own mistakes

The third attack on error accumulation is a post-training stage (adapted from SOAR) applied only to the DiT head. The insight: flow-matching training only ever shows the model points on ideal noise→data interpolation lines, but at inference the multi-step Euler solver wanders off those lines, and the model has never been taught what to do there.

The fix needs no reward model, no human preferences, no external teacher:

The model generates its own inference-time errors (one CFG-guided Euler step, gradients detached), scatters re-noised copies of the erroneous state, and is then supervised to steer each of them back to the ground-truth clean latent. The model learns a "return-to-manifold" vector field in exactly the regions its solver actually visits.

Concretely: sample a flow time \tau, take one detached Euler step with the model's own CFG-guided velocity to get an off-trajectory state \hat{x}, draw K=6 re-noised variants of it, and add an auxiliary regression loss whose target is the endpoint-consistent velocity (x_1 - x_{\text{aux}})/(1 - \tau_{\text{aux}}) — the straight line back to the true clean patch. It's a supervised, flow-matching-native cousin of scheduled sampling: expose the model to its own mistakes, but with an analytic correction target instead of a reward signal.

Does it work? The Pretrain→SOAR deltas are real but modest on most benchmarks (Seed-TTS-Eval average SIM 78.8→79.2), with one striking exception: hard-English WER on CV3-Eval drops from 5.99% to 4.49%, the largest single gain in the paper, and Syntactic Complexity on EmergentTTS-Eval jumps +7.3 points to 65.7% — above every closed-source system in the table. That pattern (biggest gains exactly on the hardest, most drift-prone inputs) is consistent with the mechanism doing what it claims.

Distillation and streaming: 54 ms to first audio

For latency, the paper distills the corrected DiT into a MeanFlow student that predicts the mean velocity over an interval [t_a, t_b] rather than the instantaneous velocity, letting the ODE be solved in 2–4 big steps instead of 10. The twist is "CFG-aware": teacher trajectories are integrated with classifier-free guidance already applied (16 Euler steps, CFG scale 1.2), so the student's single conditional forward pass bakes in the guidance. The pretrained model needs 10 steps × 2 passes (conditional + unconditional) = 20 function evaluations per patch; the student needs 4. A 5× reduction in the acoustic head's compute.

Streaming comes from the "1T1A" interleaved layout: one BPE text token alternates with one 6.25 Hz audio step, so an upstream dialogue LLM's token stream can drive synthesis directly — speech starts within a single text token of the dialogue model beginning its response, no utterance buffering. On one H800 under vLLM: first-packet latency of 85.4 ms (plain mode, RTF 0.231) or 54.4 ms (interleaved, RTF 0.245). For voice-agent stacks where TTS latency stacks on top of ASR and LLM latency, sub-60 ms first audio from an open checkpoint is a genuinely useful number.

The evidence

On Seed-TTS-Eval, dots.tts (SOAR) leads average SIM at 79.2 — 1.4 above closed-source Seed-TTS — and average WER at 2.95% vs. CosyVoice 3's 3.06%. On the 24-language MiniMax test set it takes the SIM lead on 19 of 24 languages (average 83.9 vs. VoxCPM 2's 82.3). On CV3-Eval's cross-lingual cloning subset — arguably the hardest test of timbre disentanglement — it leads SIM by 6–8 absolute points over CosyVoice 3 (75.0/72.8 vs. 66.9/66.4).

Crucially, the distilled 4-step model inherits nearly all of it: Seed-TTS-Eval WER identical to SOAR to within 0.01, SIM down about one point. The latency numbers come essentially for free in quality terms.

What to be skeptical about

The multilingual WER story is genuinely mixed, and the paper is honest about it. The average MiniMax WER (6.8%) is worse than MiniMax (2.8%) and Fish-Audio S2 (3.7%), dragged down by catastrophic outliers: Arabic at 36–40% WER (vs. MiniMax's 1.67%), plus double-digit gaps on Hindi, Thai, Turkish. The diagnosis — feeding the LLM raw BPE instead of phonemes buys prosody and text normalization but starves script-divergent, low-resource languages of coverage — is plausible, and SIM remains normal on those languages, but "leads average SIM" should not be read as "leads multilingually."

Expressiveness shows a real trade-off. On EmergentTTS-Eval, SOAR raises syntactic faithfulness (+7.3) but drops Emotions from 72.7% to 63.9% and Paralinguistics from 54.7% to 52.7%. Self-correction seems to tighten the distribution around text-faithful renderings at the cost of expressive variance — an interesting datapoint on what "robustness post-training" costs. Complex Pronunciation (16–18% win rate) and Foreign Words (36–40%) are weak across all dots.tts variants.

Also worth noting: all baseline numbers are taken from original papers or default open-source configs with no re-tuning; cross-lingual WER still trails CosyVoice 3; and the released model, despite the modality-agnostic VAE, is speech-only — no singing or general audio in this release, which was part of the original motivation for going continuous. And no ablation isolates full-history conditioning against a DiTAR-style local head at matched scale, so the "full history reduces drift" claim rests on the overall result rather than a controlled comparison.

Why it matters

If continuous-AR TTS was waiting for a proof that its instability problem is solvable with a documented, reproducible recipe, this is a credible candidate: a prediction-friendly VAE (Stage-2 supervision), a semantically filtered feedback loop, exact train/inference-matched parallel training, reward-free error-exposure post-training, and CFG-fused few-step distillation — each addressing a specific failure mode, all released under Apache 2.0 with pretrain/SOAR/MF checkpoints. The block-causal training layout (Section 2.5.2) and the self-corrective alignment objective (Section 2.6.3) are the two sections most worth your time; both are general recipes for AR-diffusion hybrids well beyond TTS — the same exposure-bias problem shows up in continuous-AR video and any "LLM + diffusion head" architecture. The 54 ms 1T1A streaming mode is the most immediately deployable piece: an open drop-in voice for streaming dialogue LLMs that starts speaking before the LLM finishes its first sentence.