Issue 27 · Pick 09 AI / ML ✓ read
Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents
TL;DR: ConvFill splits a voice agent into a tiny on-device "Talker" (135M–1.7B params) that starts speaking within ~500ms, and a slow frontier "Reasoner" that streams condensed knowledge chunks into the Talker's context mid-utterance. The Talker is fine-tuned to generate contextual filler when it has nothing yet, and to fold arriving knowledge into fluent speech when it does. The result: time-to-first-response stays at small-model levels (478–976ms vs. 2.9–8.1s for frontier models), while QA accuracy lands within 0–6.3% of the reasoner feeding it. The interesting contribution isn't a smarter model — it's an interaction protocol, a training format, and a data-validation pipeline that make "thinking while speaking" a learnable skill for very small models.
The tension: thinking is slow, talking must be instant
Human conversation has a brutal timing budget. Turn-taking gaps of a few hundred milliseconds feel natural; a few seconds of silence feels broken. Meanwhile, everything that makes modern LLMs capable — chain-of-thought, retrieval, tool calls over MCP — is iterative and takes seconds. In this paper's own measurements, a frontier model answering a direct question takes ~2.6s to a first full sentence; add retrieval and it's ~4s; add an email-inbox tool call and it's ~8s.
The two standard escapes both fail. You can run a small on-device model — but a 1B model can't answer hard factual questions (0.4–2.5% on SimpleQA), and it turns out it doesn't even solve latency for retrieval tasks: an unassisted small model handed a pile of RAG context must prefill all of it before speaking, and clocks 3.8s TTFR, nearly as slow as the frontier model. Or you can play canned filler ("Let me think about that...") while the big model works — but generic filler is hollow, and systems where a fast model answers and a slow model later corrects it are known to erode user trust with visible revisions.
The idea: one voice, two brains, streamed mid-sentence
The paper's task, conversational infill, is a specific division of labor. The Reasoner (any frontier model, unmodified except for prompting) does its slow thing and emits knowledge chunks — terse, non-conversational factual phrases — into a queue as they become available. The Talker is the only model the user ever hears. At each phrase within a turn, it looks at the queue:
- If a knowledge chunk is there, generate the next phrase conditioned on it — a conversational rendering of that fact.
- If the queue is empty (signaled by a special
<sil>token), generate a contextually grounded filler phrase — not "hmm," but something contingent on what the user just said ("Let me take a look at your inbox for that...").
Formally, within turn \tau the Talker builds its response as a sequence of phrases T_\tau = (t_1, \dots, t_p), where each t_i is conditioned on conversation history, the user utterance u_\tau, its own prior phrases in this turn, and either a Reasoner chunk r_i or a silence token s_i. The <sil> token is inserted dynamically by the runtime whenever the TTS queue is about to run dry — so the amount of filler automatically stretches to cover whatever the Reasoner's latency happens to be, with no per-task tuning.
Two design choices distinguish this from lookalikes. Unlike speculative decoding, there's no verify-and-revise step — the Talker never asserts facts it doesn't have, so nothing needs retraction. Unlike Qwen2.5-Omni's Thinker-Talker (where the Talker consumes latent representations from a co-trained backbone), here the coupling is plain text at inference time, so any Reasoner slots in with zero retraining — the authors run the same seven Talkers against three different frontier APIs.
Mechanism: a new role token and a lot of data hygiene
Training is refreshingly simple. Each Talker gets one new special token, <sil>, and one new chat role, [KNOWLEDGE], mapped through the model's existing control-token template. A training example looks like: history turn, current user utterance, a knowledge chunk (or <sil>) in the [KNOWLEDGE] role, the phrases already spoken this turn, and the target next phrase. Fine-tuning the seven models (SmolLM2 135M/360M/1.7B, Gemma 3 270M/1B, Qwen3 0.6B, Llama 3.2 1B) took 122 GPU-hours total — about $134. Inference runs INT8 via MLX on a MacBook M2.
The real engineering is in the data. There is no natural corpus of "person paraphrasing a streamed fact mid-sentence while filling time contextually," so they synthesize 8,443 conversations (290,571 training examples: 111,552 filler-conditioned, 179,019 knowledge-conditioned) from two tracks — freeform topic-seeded dialogues, and re-skinned DSTC8 task-oriented dialogues where the original system utterances become the Reasoner's "thoughts."
Here's the subtle failure mode that makes naive generation useless: the generating LLM sees the whole conversation, including future turns, so it leaks future facts into early filler — an infill phrase mentioning the restaurant name three turns before it's retrieved. At inference time the Talker can't see the future, so training on such data teaches hallucination. They fight this with a four-stage validation cascade: structural checks, a DeBERTa-MNLI contradiction gate on every (thought, response) pair, a BERTScore cross-alignment check (each response must match its own thought better than any sibling — catching transposed arrays), and a regex-based proper-noun visibility check: no entity may appear in a response before appearing in the user's or Reasoner's visible context. Conversations are regenerated until they pass. Total dataset cost: ~$2,400. Appendix B is worth reading as a template for causally-constrained synthetic data generation generally.
Evidence
The single-turn QA results tell a clean story. On SimpleQA (adversarially hard factual questions), base small models are useless; fed streamed Reasoner knowledge, they recover nearly all of the Reasoner's accuracy:
Note what's being measured: the Talker's job is to not corrupt knowledge in transit, and the "conditional accuracy" column (accuracy restricted to questions the Reasoner got right) shows it succeeds 87–99% of the time, scaling with Talker size. Even the 135M model relays ~90% faithfully. NLI-based entailment of knowledge-conditioned phrases runs 83–97%, and filler phrases avoid contradicting later knowledge 81–99% of the time.
The latency side is where the architecture earns its keep:
The buried gem is that RAG row: a small model alone is nearly as slow as the frontier model on retrieval tasks, because it must chew through the retrieved context itself. ConvFill's Talker stays under 1s because it only ever ingests the Reasoner's condensed phrases. The system also self-adapts: mean <sil> insertions per turn rise from 1.16 (Direct) to 2.35 (MCP) as Reasoner latency grows.
The live user study (n=18, counterbalanced, real voice interaction, IRB-approved) found ConvFill statistically equivalent to the frontier configuration on clarity, fluency, coherence, task completion, and satisfaction (TOST equivalence tests, \delta=0.5), better on perceived latency (4.24 vs 3.46), and worse on naturalness (3.83 vs 4.41) — some users disliked the filler style. In rankings, ConvFill tied frontier on Direct and MCP tasks but was significantly preferred for RAG (12 vs 5 first-place votes, p=.044). Notably, RAG — medium latency, ~1.3 fillers/turn — beat MCP — longest latency, ~2.4 fillers/turn — hinting that filler helps up to a point and then starts costing you. The authors' latency-adaptive proposal (predict Reasoner delay; stay quiet for short waits, generate explicit pauses for long ones) follows directly.
What to be skeptical about
The "6.3% gap" is measured against the Reasoner, not the frontier model. Reformatting a frontier model to emit phrase-style knowledge chunks itself costs accuracy — usually ~2%, but 7.4% for Gemini 3.1 Pro on SimpleQA. So the Talker-vs-standalone-frontier gap can approach 10% in the worst configuration. The authors are upfront about this (it's prompt-format sensitivity, not Talker failure), but the headline number is the friendlier comparison.
The intelligence transfer is shallow by design. On single-turn QA, the Talker is essentially doing constrained paraphrase of a fact it was just handed. That this works at 135M parameters is genuinely useful, but it's not knowledge distillation or reasoning transfer — the hard multi-hop composition all happens in the Reasoner. MultiWOZ multi-turn results are correspondingly weaker (entailment ~83%, faithfulness ~4.6/5), and most judge metrics elsewhere sit near ceiling, which limits what the benchmarks can discriminate.
Everything downstream of a $2,400 synthetic corpus. All training data is Claude-generated, English-only, in one stylistic register. The user study does show generalization to RAG and MCP tasks absent from training, which is the strongest evidence the learned skill is a general "integrate streamed knowledge" behavior rather than domain memorization — but the study is n=18, single institution, and the Reasoner-timeout regime (very long delays, filler fatigue) was explicitly not evaluated.
Naturalness is a real cost. Users noticed the filler. The trade was worth it in aggregate here, but "always fill silence" is clearly not the final answer, and the paper's own MCP-vs-RAG asymmetry says so.
Also a small oddity: the paper names models like GPT-5.5, Claude Opus 4.7, and Gemini 3.1 Pro with 2026 version tags; take the specific frontier-model identities as given by the text.
Why this matters
The lasting idea is the decoupling: conversational behavior lives in a tiny, cheaply fine-tunable, on-device model with a stable voice; capability lives in whatever frontier stack you point at it, swappable without retraining. That directly addresses two deployment pains — users hating personality drift across model upgrades, and the impossibility of making tool-calling frontier loops feel conversational. The mechanism is simple enough (one role token, one silence token, a queue) that it could plausibly become a standard pattern for voice agents, the way RAG became a pattern for grounding. And the demonstration that a 135M-parameter model can learn to fluently interleave filler and streamed facts — trained for three dollars — is a nice data point on how little capacity "conversational behavior" actually requires once you separate it from knowledge.
If you read two things: Section 8 (Discussion) for the latency/filler analysis and the honest treatment of the Gemini anomaly and naturalness trade-off, and Appendix B for the validation cascade — the proper-noun visibility check is a reusable trick for anyone generating synthetic data with causal-visibility constraints.