Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering

TL;DR: Full-duplex speech models — the ones that listen and talk at the same time, like Moshi — turn out to have an internal "mode switch" between a generative state (focused on producing their own speech) and a perceptive state (focused on tracking your speech). This paper shows, via a logit-lens probe, that the switch is sticky: when you barge in while the model is talking, it stays in generative mode for roughly half a second and literally misses the first word you say. The authors call this state inertia, build a benchmark that puts the critical word at the very start of the interruption to expose it, and then fix much of it with a single training-free steering vector added to one hidden layer at interruption onset — lifting PersonaPlex's interruption correctness from 28% to 45% and its first-word capture rate from 40% to 72%.

Why barge-in is the hard part of full-duplex

A half-duplex voice assistant is a walkie-talkie: you talk, it detects end-of-turn, it talks. A full-duplex spoken language model (FD-SLM) is a phone call: the model processes two synchronized token streams at every timestep — a user stream (audio tokens of whatever's coming in over the mic) and a model stream (its own generated text and audio tokens). Formally, at each timestep t the model emits its own tokens (m^{(t)}_{\text{audio}}, m^{(t)}_{\text{text}}) conditioned on the incoming user audio u^{(t)}_{\text{audio}}, its own previous outputs, and dialogue history. This architecture (Moshi, PersonaPlex, Raon-SpeechChat) is what enables backchannels, overlapping speech, and interruptions.

Interruptions are the stress test. Human conversation has overlap in over 40% of turns, and the whole point of full-duplex is that the model should be able to hear you while it's mid-sentence. Existing benchmarks check whether the model eventually yields the floor and says something plausible. What they don't check is whether the model heard the beginning of your interruption — and it turns out that's exactly where these models fail, for a reason that's visible in their activations.

Probing the mode switch with a logit lens

The analysis tool is the logit lens: take the hidden state h^{(t)} at some intermediate layer, project it through the unembedding matrix, and see what token distribution that layer "already believes." In a text LLM this shows how predictions refine with depth. Here it reveals something more interesting, because an FD-SLM has two streams it could be predicting.

Finding 1: during listening, intermediate layers preferentially predict the user's upcoming tokens; during speaking, they predict the model's own next tokens. Concretely: while the user says "...explain their pros and cons...", the model's middle layers — even though its output stream is emitting silence/padding — decode tokens like "pro," "and," "cons," anticipating the user's next words. The model is running a next-token predictor over your speech internally while saying nothing.

To make this quantitative, the authors define two scalar affinity scores per timestep from the logit-lens distribution:

\mathcal{S}_{\text{gen}}(t) = \tfrac{1}{2}\left(P(m^{(t)}_{\text{audio}} \mid h^{(t)}) + P(m^{(t)}_{\text{text}} \mid h^{(t)})\right), \qquad \mathcal{S}_{\text{perc}}(t) = P(u^{(t+1)}_{\text{audio}} \mid h^{(t)}),

i.e., how much probability the intermediate hidden state assigns to the model's own current output tokens versus the next incoming user audio token (using the first, semantically dominant codec codebook). Finding 2: averaged over 100 turn-by-turn dialogues aligned at the end of the user's utterance, \mathcal{S}_{\text{perc}} is high while the user speaks and decays right after; \mathcal{S}_{\text{gen}} rises as the model takes the floor. The model isn't doing both jobs uniformly at all times — it reconfigures its internal predictive focus depending on whose turn it is. Final layers are the exception (they must always emit output tokens, even if it's silence), which is why the probe targets middle layers.

State inertia: the switch is sticky

Now the key experiment. Compare two conditions with the same user utterance: one where it arrives during silence (no-interruption), and one where the model has first been prompted into a long-winded answer ("explain how antibiotics work...") and the utterance barges in mid-response.

In the no-interruption condition, \mathcal{S}_{\text{perc}} jumps up essentially immediately when the user starts speaking. In the interruption condition, it stays flat for 7–8 timesteps — about 0.6 seconds — before recovering. Appendix C shows the mirror image: \mathcal{S}_{\text{gen}} stays elevated for roughly 20 timesteps (~2 seconds) after the barge-in. The model's internal state has momentum: it keeps predicting its own continuation and is slow to reallocate to the incoming stream.

State inertia, schematically (after Figures 5–7 of the paper). Perception affinity — the probability the model's mid-layer hidden state assigns to the next incoming user audio token — stays suppressed for ~7–8 timesteps after a barge-in, exactly spanning the first word of the interruption. Injecting the perception vector at onset collapses the lag.

The authors note a lovely neuroscience parallel: speech-induced suppression, where speaking suppresses activity in your own auditory cortex and increases auditory response latency. Whether that's a deep computational analogy or a coincidence of two systems that must share machinery between production and perception, it's the same functional signature: producing output dampens sensitivity to input.

A benchmark designed to hit the blind spot

Half a second of deafness matters only if there's content in that half-second. Existing full-duplex benchmarks accidentally hide the problem: their interruption queries start with filler ("Hold on, what time is...?") that gives the model time to switch states before anything important is said.

The Zero-Buffer Benchmark (ZBB) removes the buffer. Each example: (1) a speech-inducing prompt to get the model talking, then (2) a barge-in query of the form <Subject>, <Description>, <Confirmation Request> — e.g., "Banana is a red fruit, right?" — where the subject is the first word. Miss the onset and you've lost the one word needed to answer. 50 subjects, each paired with one true and one false description (100 queries), so subject difficulty is controlled. Two LLM-judged metrics on ASR transcripts of the model's reply:

Correctness: did it answer the confirmation question right?
IWOR (Initial Word Occurrence Rate): did the response explicitly name the subject (or a synonym)? This is the direct probe of "did you hear the first word," with the judge instructed to penalize pronoun-dodging and naked yes/no answers.

Interruption devastates both metrics across all three models tested — e.g., PersonaPlex IWOR falls from 0.74 to 0.40 — confirming that state inertia isn't just a curiosity in an activation plot; it deletes real information.

The fix: one vector, three timesteps, no training

Since the two states are represented in activation space, the natural intervention is activation steering. Using the affinity scores (thresholded, averaged over layers 12–24), classify timesteps from held-out turn-by-turn conversations into generation-dominant (T_{\text{gen}}) and perception-dominant (T_{\text{perc}}) sets, then take the mean-difference direction:

\mu_{g\to p} = \frac{1}{|T_{\text{perc}}|}\sum_{t\in T_{\text{perc}}} h^{(t)} \;-\; \frac{1}{|T_{\text{gen}}|}\sum_{t\in T_{\text{gen}}} h^{(t)}.

This is the classic contrastive steering-vector recipe, but pointed at a behavioral mode of an interactive system rather than a semantic concept. A PCA check (Appendix D) shows the two timestep populations form cleanly separated clusters across most layers, so the vector is a real axis of the representation, not a noisy difference of overlapping blobs.

At inference: an energy-based onset detector flags when the user starts speaking during model generation; for the next \Delta T_{\text{steer}} = 3 timesteps, add the vector to one layer's hidden state with linearly decaying strength, \tilde{h}^{(t)} = h^{(t)} + \alpha(1 - \frac{t - t_{\text{int}}}{\Delta T_{\text{steer}}})\mu_{g\to p} (\alpha = 5.5, layer 23 for PersonaPlex). That's the whole mechanism — a vector add on three frames. No fine-tuning, negligible compute.

The evidence

The IWOR recovery is the striking number: on PersonaPlex and Moshi, steering brings first-word capture nearly back to the no-interruption ceiling (0.72 vs. 0.74; 0.64 vs. 0.67). Correctness recovers less fully, which makes sense — hearing the word is necessary but not sufficient for a right answer.

Three corroborating analyses strengthen the causal story. First, the affinity traces: with steering, \mathcal{S}_{\text{perc}} recovers immediately at onset instead of after 7–8 steps. Second, an attention analysis (Appendix G): without steering, subsequent timesteps attend weakly back to the first few interruption frames; steering restores that attention to near no-interruption levels. So the vector doesn't just cosmetically flip a probe — it changes what the model retrieves from context. Third, a sanity check on Full-Duplex Bench shows steering leaves general dialogue quality unchanged (e.g., PersonaPlex 3.34 → 3.41 on a 1–5 scale, within error), and a false-trigger study shows quality degrades only gradually if the onset detector fires spuriously.

What to be skeptical about

Hyperparameters were tuned on the evaluation. The vector itself is built from data disjoint from ZBB, but the steering layer, strength \alpha, and span were grid-searched on ZBB performance for PersonaPlex (Appendix F), and per-model thresholds and layers differ substantially (Raon uses layer 26, \alpha=1.2, thresholds several nats lower). There's no held-out test split of ZBB, so the headline numbers have some tuning optimism baked in. The FDB result mitigates the worry about overfitting to ZBB's format but doesn't test other buffer-free scenarios.

Raon-SpeechChat barely works to begin with (0.10 correctness even without interruption), so its "200% recovery" is arithmetic on tiny numbers. The real evidence is two models, both Moshi-lineage-adjacent dual-stream architectures. Whether state inertia and this fix apply to other full-duplex designs (e.g., time-multiplexed or non-dual-channel models) is open — and the authors are candid that open-source FD-SLM scarcity limits the survey.

Deployment questions remain. The energy-based onset detector will fire on coughs and background noise; the false-trigger study shows graceful degradation, but a production system needs semantic VAD. Also, ZBB queries are synthetic TTS with a rigid template; real interruptions are messier, though arguably that only makes zero-buffer onsets rarer, not the mechanism wrong.

A quieter question: why does inertia exist? The paper diagnoses and patches it but doesn't explain its origin. A plausible story is training-data statistics — real interruptions in training corpora mostly have filler onsets, so the model never learned to switch instantly — which would suggest the right long-term fix is data curation or an auxiliary objective, with steering as the stopgap. The paper doesn't test this.

Why it matters anyway

Three things make this paper worth your time despite the caveats. First, it's one of the cleanest demonstrations that interpretability tooling (logit lens → contrastive steering vector) can fix a concrete product-relevant failure in an interactive multimodal system, closing the loop from mechanism to metric. Second, the generative/perceptive state decomposition is a genuinely useful lens for thinking about any always-on streaming model that must arbitrate between producing and perceiving — the same tension will show up in real-time video agents and robot policies. Third, "state inertia" is now a named, measurable phenomenon with a benchmark, which tends to be how failure modes actually get fixed by the field.

If you read one part of the paper, read Section 3 (the affinity-score construction and Figures 2–5) — that's where the conceptual contribution lives — then skim Appendix G for the attention-recovery heatmaps, which are the most convincing evidence that the steering vector changes computation rather than just the probe.