studio-dots-ai/dots.tts

dots.tts is a 2B-parameter, Apache-2.0 text-to-speech system from studio-dots-ai that goes fully continuous: no discrete audio tokens anywhere — an LLM backbone drives an autoregressive flow-matching head directly in a 48 kHz VAE latent space. It's interesting because it posts the best average numbers on Seed-TTS-Eval among open models, ships everything (weights, training code, distillation code, streaming runtimes, an instruction-controlled speech editor), and includes 1–2 step distilled variants plus a token-by-token "double streaming" mode built for duplex LLM voice agents.

The idea: continuous end-to-end AR TTS

Most open TTS stacks (CosyVoice, FireRed, Qwen3-TTS) quantize audio into discrete tokens for the LM to predict, then decode with a separate acoustic model. dots.tts instead keeps everything continuous, in the DiTAR lineage but scaled up and productionized:

• A frozen AudioVAE encodes 48 kHz mono into continuous latent patches (~160 ms each), decoded by a BigVGAN-style causal decoder.
• A semantic encoder re-embeds each generated patch into a compact representation for the LM, stripping high-variance acoustic detail — this is the trick that makes AR over continuous latents stable.
• The LLM (initialized from Qwen2.5-1.5B-Base) consumes raw BPE text — no phonemizer — and emits one hidden state per audio step.
• An AR flow-matching DiT head conditions on the LLM state, the AR prefix, and a frozen CAM++ speaker x-vector to denoise the next latent patch.

Two sequence layouts matter: plain mode (full text prefix, standard TTS) and an interleaved double-streaming mode where you push BPE tokens one at a time from an upstream LLM and get audio chunks back — a genuinely useful contract for duplex voice agents that most open TTS releases lack.

What's released and how good it is

Seven checkpoints on HF: base, SOAR (post-trained, best similarity), MeanFlow-distilled students at NFE 4/2/1, a streaming-TTS variant, and dots.tts.edit for instruction-controlled editing (text edits, pitch, rate, pauses, emotion, background, speaker transfer via tags). Full inference, fine-tuning, and MeanFlow distillation code included — you can distill your own fine-tuned teacher.

The striking result is how little distillation costs: the 1-step student holds 78.3 avg SIM vs 79.2 for the 10-step teacher. On the 24-language MiniMax benchmark, dots.tts leads SIM on 19/24 languages (avg 83.9). The honest caveat, which the README states plainly: average multilingual WER (6.8) is worse than MiniMax (2.8) and Fish-Audio (3.7), with big gaps on Arabic (~36% WER), Hindi, Thai, Turkish — the price of a BPE text backbone with no phonemes on low-resource scripts. On EmergentTTS-Eval it sits below all the closed frontier systems overall (47.6% vs Gemini's 70.7%), though it beats them on syntactic complexity. Content fidelity in zh/en is where it genuinely leads.

Practicalities

Efficiency is well-documented: with --optimize (torch.compile, ~3 min warmup), the MeanFlow model hits RTF 0.15 p50 and 68 ms first-chunk latency (text-only) on one H800, ~5.7 GB peak. SGLang Omni serves it with continuous batching — 4.76 req/s / ~20 audio-seconds/s at concurrency 16 on one H100 at 1.35% WER. Note Omni serving doesn't yet do same-request token-by-token streaming; true double streaming is only in the Python runtime. Community ports already exist (audio.cpp, MLX Python and Swift, ComfyUI), which is a strong adoption signal three months in.

Numbers are all self-reported against published baselines; the multilingual WER weakness and speech-only training (no singing, no sound effects) are the main gaps. License is Apache-2.0 for both code and weights.

Try it

pip install dots.tts

dots.tts \
  --model-name-or-path dots-studio/dots.tts-soar \
  --text "Hello, this is a zero-shot voice cloning demonstration." \
  --prompt-audio /path/to/reference.wav \
  --prompt-text "The exact transcript of the reference audio." \
  --output clone.wav

Use dots-studio/dots.tts-mf for low latency (NFE 4), keep references ~10 s, and reseed for prosody variation. Repo: https://github.com/studio-dots-ai/dots.tts