Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models

Note up front: I had the full paper text, but it's a short technical report — the "experiments" section is almost entirely latency accounting, with no quality benchmarks. Read the numbers below with that in mind.

TL;DR: Alibaba's Wan team built a single Transformer that ingests and emits interleaved text, audio, and video tokens in one continuous causal stream — no separate VAD, ASR, LLM, TTS, or avatar renderer anywhere in the loop. The agent watches you, listens to you, and simultaneously generates its own speech and a synchronized 25 fps video of itself, in 160 ms streaming units, with ~200 ms model-side latency (~550 ms end-to-end over a network). It's the most complete realization yet of the "one model, one causal timeline, all modalities in both directions" idea — but it's a v0.1 at 192p output, and the paper offers essentially zero evidence about response quality.

Why cascades are the wrong shape for conversation

Today's talking-avatar systems are pipelines: a voice activity detector decides when you've stopped speaking, ASR transcribes you, an LLM composes a reply, TTS speaks it, and an audio-driven animation model puppets a face. Each module boundary adds latency, and text becomes a lossy bottleneck between components — prosody, hesitation, and visual context get discarded and then have to be re-synthesized downstream.

But the deeper problem isn't latency; it's that human interaction is full-duplex. While you speak, a real interlocutor nods, shifts gaze, and prepares to jump in. While they speak, they're still watching your face for signs of confusion and will cut themselves off if you interrupt. A cascade fundamentally can't learn this, because its modules alternate: perceive, then reason, then render. Turn-taking becomes hand-coded VAD logic bolted on from outside. The paper's framing is that streamability and duplexity are modeling constraints, not serving optimizations — a system built on offline encoders, bidirectional video decoders, and round-based dialogue "cannot recover truly low-latency full-duplex behavior by engineering alone."

That's the genuinely interesting claim here. Moshi (Kyutai) made the analogous move for speech-only dialogue: put user and assistant audio on parallel streams of one model and let turn-taking be learned. Wan-Streamer extends that to the full audio-visual case — the model both perceives video of the user and generates video of itself, in the same stream.

The core idea: one causal stream, six token types

Time is chopped into 160 ms streaming units. At unit k, the user side contributes observations u_k = (u_k^{\mathrm{t}}, u_k^{\mathrm{a}}, u_k^{\mathrm{v}}) — text, audio, and video — and the agent produces a response y_k = (y_k^{\mathrm{t}}, y_k^{\mathrm{a}}, y_k^{\mathrm{v}}). Everything is concatenated into one interleaved sequence, and a single Transformer factorizes the interaction autoregressively over units:

Every generated unit is committed back into the history, so the model conditions on its own past outputs as well as everything it has observed — that's how identity, scene state, and speaking rhythm can persist over a long session. Attention is block-causal: tokens within a 160 ms unit see each other freely; across units, attention flows strictly forward. This is what makes incremental streaming possible with a KV cache.

The mechanism: hybrid discrete + continuous generation

The three output modalities are handled with two different objectives inside the same backbone:

• Text is discrete tokens, trained with standard next-token cross-entropy.
• Audio and video live in continuous latent spaces (from strictly causal VAEs — no future-frame lookahead, so encoding and decoding can stream) and are generated jointly with conditional flow matching. Given a clean target latent z_0^m for modality m \in \{\mathrm{a}, \mathrm{v}\} and noise \epsilon^m, a noisy interpolant z_\tau^m = (1-\tau)z_0^m + \tau\epsilon^m is constructed at flow time \tau, and the Transformer f_\theta predicts the velocity \epsilon^m - z_0^m conditioned on the entire clean causal history c_k.

The crucial detail: the same clean context conditions both velocity predictions, and audio and video latents are denoised jointly. Lip sync, prosody, and facial dynamics are coupled before decoding, in latent space, rather than repaired by post-hoc alignment (the standard trick in audio-driven avatars). After denoising, the clean latents are appended to history as context for future units.

This "autoregressive over blocks, diffusion within a block" pattern is the same family of ideas as recent causal video diffusion / self-forcing work, but pushed into a fully interactive, tri-modal, bidirectional setting.

Training runs in three stages: (1) independent-task pretraining — initialize from a language model, then mix understanding tasks (VQA, ASR, TTS, dialogue) and generation tasks (image/audio/video/joint AV generation) so that all latent interfaces are grounded in one sequence model; (2) end-to-end interaction training on duplex data where inputs and outputs from all modalities interleave on one timeline — this is where response timing, listening behavior, and interruption handling get learned; (3) distillation of a CFG-guided many-step teacher into a few-step student, plus rolling distillation: the student is rolled out over consecutive units on its own generated history (self-forcing with distribution matching) to close the train-test gap that otherwise makes long streams degrade.

The serving trick: thinker and performer

A subtle scheduling problem: perception + language reasoning is a cheap token-causal pass, but latent denoising (even distilled to few steps) is expensive, and decoding latents to pixels/waveform costs more still. Run all of this serially and you can't fit in 160 ms.

The deployment splits the one model across two GPUs by function. The thinker hosts the encoders, the language/state Transformer pass, and the causal decoders. The performer hosts only the flow-matching solver. They stay in sync by exchanging KV-cache slices — so semantically it remains a single model with a single full-history state — and the pipeline overlaps four things across adjacent units: encoding the current user unit, decoding the previous response latents for immediate emission, denoising the next latents, and the KV/latent communication itself.

Note the distinction the paper is careful about: throughput (performer time must fit in one 160 ms unit, or the stream falls behind) versus response latency (the full path from a user unit arriving to the corresponding response unit being decoded — encode + thinker pass + denoise + decode ≈ 200 ms). Pipelining fixes throughput; it can't shrink latency below the serial path.

The numbers, and how much to trust them

The comparison table is honest about a real problem: nobody in this space measures the same thing. OpenAI reports 232/320 ms "audio response" for GPT-4o but ~500 ms API TTFB; Moshi reports 160 ms theoretical / 200 ms practical model latency but is speech-only; Qwen3-Omni reports first-packet latency; avatar renderers report FPS and first-frame delay while assuming clean input audio already exists. The paper reports both boundaries for itself — ~200 ms model-side, ~550 ms total with a 350 ms network budget — and refuses to pretend the columns are commensurable.

The genuinely striking comparison: Wan-Streamer's ~200 ms model-side latency matches Moshi — a speech-only model — while also generating a synchronized 25 fps video of the agent, and it's competitive with or faster than speech-only products on the end-to-end path. Among visually embodied systems (Table 2), the closest full-loop competitors (Body of Her, X-Streamer, MIDAS) don't disclose absolute signal-to-signal latency at all, and the fast avatar renderers (VASA-1 at 40 fps, AvatarForcing at 34 ms/frame) sit downstream of an entire dialogue stack whose latency they don't count.

Now the skepticism, and it's substantial. This is a latency paper wearing a foundation-model title:

• No quality evaluation whatsoever. No understanding benchmarks, no speech quality (MOS/WER), no video quality (FVD, lip-sync scores), no dialogue quality, no human preference study against a cascaded baseline. The "Naturalness" and "Interruption" sections are purely descriptive claims with no measurements. The claim that end-to-end training beats cascades on behavior — the paper's central thesis — is asserted, not demonstrated.
• 192p output resolution. The conclusion calls scaling "straightforward," but flow-matching solver cost grows with latent size, and the whole real-time budget hinges on the performer fitting in 160 ms. Higher resolution directly attacks the latency claim; it is not obviously straightforward.
• Undisclosed scale. No parameter counts, no data quantities, no GPU specs beyond "two-GPU serving." The end-to-end duplex interaction data — arguably the hardest ingredient to obtain, since natural full-duplex audiovisual dialogue at scale barely exists — gets one paragraph.
• The unified-vs-cascade tradeoff is real. A cascade lets you drop in a frontier LLM; here, reasoning ability is capped by whatever fits inside this jointly trained model. If the "thinker" is a modest LM, you've bought 300 ms of latency at the cost of a much dumber conversational partner — and the paper gives us no way to check.

What changes if this holds

If the quality follows in later versions, this is the architecture template for interactive agents: interleaved multimodal blocks, discrete text + flow-matched continuous latents in one backbone, block-causal attention, and functional (rather than layer-wise) model sharding synchronized by KV exchange. Moshi made the case that turn-taking should be learned, not engineered; Wan-Streamer makes the same case for the entire audiovisual embodiment — listening face included. The thinker-performer split is independently useful: it's a general recipe for serving any hybrid AR+diffusion streaming model, and I'd expect it to be copied.

The version number is the tell. This is a proof-of-concept that the full-duplex tri-modal streaming plumbing works at real-time rates — a nontrivial systems result on its own — with the capability story deferred. Most worth your time: Section 2.4 (the thinker-performer schedule and the throughput/latency distinction) and Section 2.1 (the hybrid cross-entropy + flow-matching formulation). The demo videos at wan-streamer.com are, for now, the only quality evidence that exists.