amywork777/lipflow

TL;DR: Lipflow is a push-to-talk silent-speech interface that runs entirely on your laptop: hold a key, silently mouth words at your webcam, and text appears at your cursor in any app. It's a camera-based cousin of the EMG/imagined-speech work you follow — a stock Auto-AVSR lip-reading model made usable in daily life through per-user fine-tuning on your own face, personalized language modeling from your dictation history, and LLM cleanup of homophene confusions. MIT-licensed code, but the LRS3-trained weights are non-commercial research only.

Why it's interesting

Visual speech recognition has been "a benchmark result" for years (Auto-AVSR's 19.1% WER on LRS3). Lipflow is a serious attempt to turn that into a working interface, and the engineering choices are where the value is: it treats lip reading as an inherently lossy channel and compensates with every personalization trick available rather than hoping for a better encoder. That's the right framing for any silent-speech modality — the same playbook (user adaptation + personal LM prior + top-k rescoring by an LLM) applies directly to sEMG, ear-EEG, or imagined-speech decoders.

How it works

The model is Auto-AVSR visual-only: 3D-conv ResNet front end → Conformer encoder → Transformer decoder + CTC, with a subword Transformer LM in the beam search. MediaPipe face landmarking runs during recording so there's no post-hoc detection pass, and the encoder runs on the Apple GPU (0.16 s for a 9 s utterance on M4 Pro vs 1.7 s on CPU). Total latency is ~1–2 s from key release to pasted text. Two vendored-ESPnet patches are noted (cross-attention KV projected once per utterance → 25% faster beam search).

The onboarding is the clever part: it imports your Wispr Flow dictation history locally, has you mouth 24 sentences (drawn from your own phrasing), then fine-tunes both the LM on your phrasing and the visual encoder on your face — with 6 sentences held out and the adapted face model kept only if it beats stock. It also learns continuously: fix a word within ~30 s and that clip + correction feeds the next training round (Mac-only for now).

Because p/b/m, f/v, t/d/n are indistinguishable on lips, raw output can read "WALLET OFFICER" for "while in office." The LLM cleanup stage gets the top-3 beam hypotheses plus your recent dictations and picks the intended sentence — via Claude API, Codex CLI, Ollama, or a fully-offline Qwen3-0.6B 4-bit on MLX (~0.2 s/sentence).

Evidence

The README shows near-perfect raw transcripts on public-domain White House footage (with one dropped clause), and is honest that silently mouthed speech is harder than filmed speech — smaller lip movements. The strongest number is whisper mode: adding a soft-whisper audio stream to the Auto-AVSR audio-visual model cuts errors dramatically on test sentences.

No independent evaluation exists; there are no numbers for silent mouthing on a real webcam across lighting/pose/vocabulary, which is the actual use case. Names are acknowledged as the weak point (handled via a custom-words list that overrides beam hypotheses).

What's there and caveats

• Full Python source, setup scripts for macOS (Apple Silicon, macOS 13+), Windows, and Linux; ~1.2 GB of models downloaded at setup, ~2–3 GB disk. CI runs the full Windows setup and decodes a real clip on every push.
• Mac is the first-class target (MLX local cleanup, correction learning, cursor-context names). Windows/Linux fall back to CPU inference unless you install CUDA PyTorch.
• License gotcha: code is MIT, but the LRS3-trained weights are non-commercial research use only. Don't ship this.
• Privacy posture is good: everything local, data in one deletable folder, mic only active in whisper mode while the key is held.

Try it

On an Apple Silicon Mac:

git clone https://github.com/amywork777/lipflow.git ~/code/lipflow
cd ~/code/lipflow && ./setup.sh
open /Applications/Lipflow.app
# then: hold Right Option, mouth words, release

Or sanity-check the model on a video file without any setup ceremony:

uv run lipflow file talk.mp4 --start 10 --end 20

Worth an hour: the per-user adaptation loop with held-out validation is a template any silent-speech or BCI decoding project should steal, and the lips+whisper 6.9% WER suggests the practical sweet spot is multimodal, not purely visual.