Taming Long-form Text-to-Speech

TL;DR: State-of-the-art autoregressive TTS models like Qwen3-TTS fall apart on long text — worst-case word error rate goes from ~5% on short prompts to 35–48% past 1500 words. This paper shows the failures are visible in real time as breakdowns in emergent attention "alignment heads," and fixes them with a purely inference-time loop: watch the model's reading position, detect when it skips ahead or stalls, roll back to the failure onset, and regenerate under a temporary hard attention mask. Worst-of-10 WER on >1500-word prompts drops from 35.2% to 3.4% — better than the baseline's short-form reliability — with no retraining and negligible overhead. There's also a genuinely useful diagnostic contribution: a sliding-window speaker-similarity metric (wSIM) that exposes voice-cloning failures the standard global metric completely averages away.

The problem: long-form TTS is a tail-risk problem

Modern open-weights TTS systems (Qwen3-TTS, VoxCPM2) are autoregressive neural codec language models: they consume text tokens and generate a stream of discrete audio tokens, one step at a time, sampled with some temperature. On short prompts they're excellent. But the real use cases — audiobook narration, multi-turn voice agents that carry a whole conversation in context, voice cloning from minutes of reference audio — require long prompts, and there the models degrade badly.

The important framing move in this paper is that this is a reliability problem, not an average-quality problem. Because generation is sampled, they run every prompt across 10 RNG seeds and report worst-of-N WER alongside mean WER. For Qwen3-TTS-0.6B, mean WER on >1500-word prompts is 16.6% — already bad — but the worst seed hits 35.2%, versus 5.4% worst-case on short prompts. For the 1.7B model it's worse: 47.8%. If you're shipping a voice agent, the worst seed is what your unluckiest customer hears.

The authors decompose the gap into two failure modes with concrete definitions:

• Skip: a contiguous deletion of ≥10 words — the model jumps ahead in the text.
• Hallucination: a contiguous run of ≥20 substituted/inserted words — the model stops reading and free-generates.

Everything else (small substitutions, minor errors) occurs at the same rate in short and long prompts; skips and hallucinations account for essentially the entire long-form gap. That decomposition matters: it says you don't need to make the model better, you need to catch and repair two specific, localized events.

The obvious inference-time fix — best-of-N reranking with an ASR model — costs N× compute, can't stream, and doesn't help when a failure persists across seeds. The obvious training-time fix — duration models, forced alignment, monotonic attention losses à la Tacotron 2 — requires retraining and complicates training stability. What you want is something that detects a failure as it happens and repairs it locally.

The key insight: the model has a reading finger, and you can watch it slip

Here's the aha. When these codec LMs generate speech correctly, certain attention heads spontaneously develop a clean, near-diagonal text–audio alignment: at each audio generation step, the head's attention argmax sits on the text token currently being spoken, and it advances monotonically through the prompt at roughly speaking pace. This emergent alignment appears in all Qwen3-TTS variants and in VoxCPM2 — a different model family — without any alignment supervision.

Think of it as the model reading with a finger under the words. In every high-accuracy generation, the finger tracks left to right at a steady pace. In every low-accuracy generation, the finger either jumps forward (skip) or freezes in place while the mouth keeps moving (hallucination). The failure isn't hidden in the model's internals; it's written directly in the attention weights of a handful of heads, in real time.

The paper condenses this into a single scalar instability measure: the deviation of the alignment head's reading position from its expected pace. It sits at zero during healthy generation and departs from zero at the onset of a skip or hallucination — which means you can detect failures within a few seconds of generated audio, without an external ASR model, without waiting for the generation to finish.

The mechanism: detect, roll back, guard temporarily

The detector is simple enough to sketch. Split the text prompt into equal-sized token bins. During streaming generation, count how many steps the alignment head spends attending to each bin (via the argmax). Then:

• Skip: the reading position moves past a bin that received far fewer steps than the bins before it. Some words never got their airtime.
• Hallucination: the reading position stops advancing for much longer than a bin normally takes. Audio keeps coming, text doesn't move.
• Termination: once the last bin has received its share of attention, the text has been fully read — a bonus, since end-of-generation detection is itself flaky in these models.

On detection, LACI (Localized Attention-Constrained Inference) does three things: roll back the KV cache and audio stream to the failure onset, reseed the RNG, and regenerate under a hard attention mask that forces the alignment heads onto the correct text region — but only until generation moves past the point where the failure was detected. Then the mask lifts and the model runs free again. Persistent failures can trigger multiple retries, with a cap to bound overhead.

The mask itself is borrowed from prior work, ACI (Attention-Constrained Inference), and the comparison against ACI is where the design argument gets sharp. ACI applies the same hard mask always-on, at every step. The paper's results show why that's the wrong tradeoff: an always-on constraint fixes long-form failures but breaks generations the baseline got right — ACI raises worst-of-10 WER on 500–1000-word prompts from 6.5% to 9.3% for Qwen3-TTS-0.6B. It's the classic guardrail dilemma: a constraint strong enough to prevent derailment is strong enough to interfere with normal operation. LACI resolves it by making the constraint event-triggered and probationary — the model is only handcuffed while it's actively misbehaving, and gets a fresh RNG seed to escape the failure basin. And because ACI has no detection, it can't retry when a failure persists; LACI can.

A note on latency: detection takes a few seconds of generated audio, and these systems generate several times faster than real time into a playback buffer. So the buffer absorbs detection + rollback + regeneration without interrupting playback — the repair is invisible to a streaming listener.

The evidence

Table 2 is the headline. Qwen3-TTS-0.6B, worst-of-10 WER across prompt-length buckets on the AppTek Call Center dataset (144 prompts × 10 seeds):

Three things to notice. First, LACI's worst seed on 1500+-word prompts (3.4%) beats the baseline's worst seed on short prompts (5.4%) — the long-form gap isn't just closed, it's inverted. Second, LACI exactly matches the baseline on short prompts (5.4% vs 5.4%), which is the signature of an intervention that fires only when something is actually wrong. Third, ACI's short-prompt regression (5.4 → 6.0, 6.5 → 9.3) confirms that always-on constraints have real costs.

Figure 4 of the paper isolates the 146 (prompt, seed) pairs where the baseline had at least one skip or hallucination: LACI pulls every one into the same low WER band as short prompts, while ACI leaves a long upper tail. That's the difference between lowering a median and eliminating a failure mode.

Generalization (Table 3): on Qwen3-TTS-1.7B — which is, interestingly, less reliable on long prompts than the smaller model — baseline worst-of-10 WER is 45.7% and 47.8% in the two longest buckets; LACI brings both below 4.5% (ACI leaves 22.7% on the longest). On VoxCPM2, a different model family, LACI cuts the skip/hallucination rate in the 1000–1300-word range from 15.3% of (prompt, seed) pairs to 3.3%. Beyond 1300 words, though, VoxCPM2 resists repair: LACI only reduces the failure fraction from 70% to 57%, and its worst-of-10 WER stays at 32.3%. The authors read this as a coherence limit of the model itself — the failures are detected fine, but regeneration keeps failing. That honesty is worth flagging: LACI is a repair harness, not a capability injection, and it can't push a model past the point where its long-context generation is fundamentally unstable.

Voice cloning and the wSIM diagnostic

The voice-cloning result contains what may be the most broadly useful nugget in the paper. Intuitively, more reference audio should mean a better clone. Empirically, the opposite: the baseline's catastrophic-failure rate (WER > 30%) grows from 5% with <15-second references to 26% with ~120-second references. Long reference audio doesn't help these models — it actively destabilizes them.

Worse, the standard speaker-similarity metric hides this. SIM is a cosine similarity between speaker embeddings computed over the whole generated clip. A five-second span where the voice switches identity or badly distorts gets averaged into oblivion across a minute of otherwise-fine audio. The paper's fix, wSIM, computes SIM over sliding 8-second windows (4-second stride) and reports the worst window across all seeds. The authors manually verified that the lowest wSIM windows correspond to real identity switches or severe distortions — so it's measuring something perceptually real, not embedding noise.

With LACI, the relationship flips to what it should be: longer reference audio yields higher similarity and a catastrophic rate below 1% at 120 seconds. And independently of LACI, wSIM deserves adoption — any evaluation of long-form voice cloning that reports only global SIM is systematically blind to transient identity failures, which are arguably the failures that matter most in deployment.

What to make of it

What's genuinely new here isn't the attention mask (that's ACI) or the observation that alignment heads exist (monotonic attention has a long history in TTS). It's the combination: emergent alignment heads in modern codec LMs are reliable enough to serve as a real-time failure detector, and a detect → rollback → temporarily constrain → release loop dominates both always-on constraints and best-of-N resampling. That's a pattern worth stealing beyond TTS — it's speculative decoding's rollback machinery repurposed for semantic failure recovery, and one can imagine analogues wherever an autoregressive model has an interpretable internal progress signal. The proposed future direction — using LACI to generate paired good/bad rollouts as RL training data so the reliability gets distilled into the weights — is a natural next step.

Things to be skeptical about:

• The headline numbers are worst-of-10. That's the right metric for deployment reliability, but it amplifies the improvement. The mean-WER story is still strong (16.6% → 2.9% at ≥1500 words) but less dramatic; keep both in mind.
• One dataset, one domain. All content-accuracy experiments are on AppTek Call Center (customer-support speech). 144 prompts is modest. Audiobook-style prose, multilingual text, or code-switched prompts might stress the alignment heads differently.
• WER is ASR-judged (Parakeet-TDT-0.6B-v2), standard practice but capable of masking or inventing errors at the margins. No human listening evaluation or MOS is reported, so we don't know whether rollback seams are ever audible.
• Overhead is claimed "negligible" but not quantified — no numbers on retry frequency, tokens regenerated, or wall-clock overhead distribution. With a retry cap and few-second detection latency it's plausibly small, but "negligible" is asserted, not measured.
• It requires identifiable alignment heads. All three models tested have them, which is encouraging, but the method presupposes the phenomenon. Head selection and detector thresholds live in code "to be open-sourced upon publication," so per-model tuning burden is unclear. And it's inherently an AR-decoder technique — flow-matching and NAR systems are out of scope (though those tend to have explicit duration models and don't fail this way).
• The VoxCPM2 long-tail result (57% failure fraction remaining beyond 1300 words) is a clear boundary: when the base model can't coherently continue, no amount of rollback saves it.

None of these undercut the core claim, which is well-supported across two model sizes and two model families with a clean baseline comparison (always-on ACI) that isolates exactly why the localized design wins.

Where to spend your reading time: Section 3.2 with Figures 2 and 3 — the alignment-head analysis and the binned-coverage detector — is the intellectual core and the part you'd need to reimplement this on your own model. Then Table 3 for the generalization-and-limits picture, and the voice-cloning discussion around Figure 5 if you care about cloning evaluation: wSIM is a two-line metric change you can adopt today.