vysri/conversational-infill

TL;DR: ConvFill (UW, Patel/Iyer labs) turns the latency–accuracy tradeoff in voice agents into a trainable task called conversational infill: a 135M–1.7B "Talker" running locally starts speaking within a fraction of a second, while a cloud "Reasoner" (Claude/GPT/Gemini) does the slow reasoning, retrieval, and tool calls in the background and streams knowledge back mid-utterance for the Talker to weave into the ongoing response. The release is unusually complete — full training and inference code, seven fine-tuned Talkers on HuggingFace, a 290k-example dataset, and an end-to-end browser speech-to-speech demo with RAG and MCP tool use — all MIT-licensed.

The idea

Every architecture for voice agents has to pick a side: a small on-device model answers instantly but hallucinates or can't reason, while a frontier model with tools takes multiple seconds — conversational death. The usual dodges (canned filler audio, speculative decoding, streaming TTS over a slow model) don't change the underlying fact that the useful content arrives late.

ConvFill's move is to make "talking while waiting" a learned behavior. The user only ever converses with the small Talker. It's fine-tuned on paired sequences of thought/response phrases so it produces natural cover ("Let me look that up for you…") and then — this is the trained skill — incorporates the Reasoner's streamed facts into a sentence it has already started, without contradicting itself or breaking flow. The Reasoner is a stock API model with a prompt template; swapping it needs no retraining, so the Talker's infill skill is model-agnostic on the reasoning side.

What's actually in the release

• Seven Talker checkpoints across SmolLM2 (135M/360M/1.7B), Gemma 3 (270M/1B), Llama 3.2 1B, and Qwen3 0.6B, in bf16 (HF backend) and int8 (MLX for Apple Silicon). Evaluated on an M2 MacBook.
• The ConvFill dataset: 290k validated examples across six domains from a synthetic pipeline, plus the preprocessing script (dataset_gen/dataset_preprocess.py) that unrolls per-turn thought/response phrase pairs into streaming training format.
• Training pipeline: Lightning-based fine-tuning with DDP multi-GPU, WandB logging, one JSON config per model.
• Full inference stack + web demo: FastAPI backend, React/Vite frontend, ASR → Talker+Reasoner → TTS (macOS say or cross-platform Piper ONNX). Three task modes: normal, rag (committed FAISS index + MiniLM cross-encoder reranker; embeddings via OpenAI so an OpenAI key is required even with a Claude Reasoner), and mcp (default wiring for a mail server over stdio/Docker).

The README claims the system "approaches the accuracy of the cloud model used on its own" and cites a user study, but gives no numbers — latency, accuracy, or WER — deferring entirely to the paper (arXiv 2511.07397). Judge the quantitative claims there, not here.

What to expect

Maturity is "fresh research artifact": four days between creation and last update, 19 stars, leftover frontend/backend naming throughout the configs, and a committed demo RAG index about UW PhD programs (clearly the paper's eval setup). Dependency surface for the full demo is nontrivial: Python 3.11, Node 24, FFMPEG, provider API keys, and Piper voice files if you're not on macOS. Training a Talker is cheap by design — these are sub-2B models.

The interesting open questions are exactly the failure modes the architecture invites: what does the Talker do when the Reasoner's answer contradicts the filler it already committed to aloud? Can a 135M model reliably restate streamed facts without garbling them? And does infill quality survive outside the six synthetic training domains? The repo gives you everything needed to poke at these yourself, which is the right kind of release. Note the Talker is text-only — this is not an end-to-end speech model; ASR/TTS are conventional bolt-ons, so audio-native prosody during infill (hesitations, repairs) is out of scope.

This pattern — small model owns the interaction loop, big model streams knowledge asynchronously — is a plausible default for on-device assistants generally, and it's notable that the Reasoner-swapping works without retraining.

Try it

git clone https://github.com/vysri/conversational-infill && cd conversational-infill
python3.11 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt && pip install -e .

export ANTHROPIC_API_KEY="..."   # or OPENAI_API_KEY / GEMINI_API_KEY
cd web_demo/frontend && npm ci && cd ../..   # needs Node 24, FFMPEG

bash scripts/run_web_demo.sh     # open http://127.0.0.1:5173 in Chrome

Models are in the vysri HuggingFace collection (e.g. vysri/gemma3-270m-IT-ConvFill); paper at arXiv 2511.07397.