MOSS-VL Technical Report

TL;DR: MOSS-VL is an open 11.3B vision–language model built around one capability nearly all current video LLMs lack: it keeps watching while it talks. The trick is architectural nostalgia with a new justification — it revives Flamingo-style gated cross-attention, which the field abandoned in favor of interleaving visual tokens into the decoder sequence, because cross-attention turns out to be exactly what real-time interaction needs: a new frame just appends to a KV cache without touching the decoding state. Combined with a synthetic corpus that supervises when to speak, when to stay silent, and when to revise (installed in a final stage using under 3% of training tokens), it sweeps the proactive-behavior subsets of streaming benchmarks — 66.0 vs. 37.5 on OmniMMI Proactive Alerting — and its time-to-first-token advantage over same-backbone Qwen3-VL-8B grows with visual context, from 2.8× to 5.1×. All five checkpoints, the curriculum, and real-time inference code are released.

The problem: every video LLM goes blind mid-sentence

The standard open VLM watches a finished clip, then answers questions. Recent "streaming" models improve on this: frames arrive continuously, the model can stay silent until evidence appears, and it can re-answer a resident question as the scene evolves. But there's a subtle ceiling the paper makes explicit with a five-level taxonomy (Table 1 of the paper, worth two minutes of your time): at levels L2–L4, the model is blind for the duration of each reply. Once it starts generating, incoming frames pile up unprocessed until the reply ends.

Why? Because of how mainstream VLMs integrate vision. Qwen-VL, LLaVA, and nearly every modern open model interleave visual tokens directly into the decoder's token sequence. That's elegant for offline understanding — one unified sequence, one attention mechanism — but it makes mid-generation perception awkward. If the model is halfway through decoding a sentence and a new frame arrives, where do you put its 500+ visual tokens? Splicing them into the sequence disrupts the autoregressive state; buffering them means the model is reasoning about a stale world. It also means prefill cost, and therefore time-to-first-token, grows with the full visual history sitting inside the decoded sequence.

L5 — perceiving while generating, revising or cutting short a reply the moment the scene overturns it — is what a live assistant, a sports commentator, or a robot narrating its own actions actually needs. MOSS-VL is designed for L5 from the ground up.

The aha: the "obsolete" architecture was the right one all along

DeepMind's Flamingo (2022) attached vision to a frozen language model through gated cross-attention layers: text hidden states are queries, visual features are keys and values, and visual tokens never enter the decoded sequence. The field largely moved on — interleaved token architectures won on offline benchmarks and simplicity.

MOSS-VL's core observation is that cross-attention has a property nobody valued at the time: the visual context is a side-channel cache, decoupled from the autoregressive state. When a new frame arrives, you encode only that frame, append its keys and values to the cross-attention cache, and the very next generated token already sees it through the gated layers. Nothing in the decoded sequence needs to move. The decoder's own sequence grows by only a few timestamp tokens and a single placeholder token per frame — the hundreds of patch tokens stay on the vision side.

MOSS-VL: gated cross-attention side channel Decoded sequence text t + ⟨ph⟩ text t + ⟨ph⟩ reply keeps going

Cross-attn KV cache frame 1 K,V frame 2 K,V frame 3 K,V queries attend down through 12 gated layers new frame: append only Decoding state untouched; next token already sees the new frame. Latency grows slowly with visual history.

The decoder is Qwen3-8B's 36 self-attention layers plus 12 tanh-gated cross-attention layers (one every fourth layer, gates zero-initialized so training starts from an intact language backbone). Total: 11.3B parameters — 8.2B backbone, 2.3B cross-attention, 0.8B vision encoder. Note the framing the authors use: more parameters, faster serving, because the extra 2.3B sit outside the decoded sequence.

Two supporting pieces make the side-channel coherent. XRoPE — the authors claim the first rotary position encoding for the cross-attention channel — puts text tokens and visual patches in one shared three-axis coordinate space (t, h, w): a text token advances all three axes together, while a frame anchored at position t tiles its patches from (t,t,t) outward, so height/width offsets ride on a shared temporal anchor. Without it, the cross-attention channel has no notion of where frames sit relative to text. Second, since positions are logical rather than temporal (frame rate is motion-adaptive, 1–16 fps), each frame is preceded by an explicit plain-text timestamp — the model reads wall-clock time rather than inferring it.

There's real systems work underneath: the streaming visibility pattern (each text query sees the visual prefix up to its frame) isn't served by stock attention kernels, so they extended FlashAttention-3 with a per-query-row prefix boundary (cross_kv_boundary) so kernel cost tracks visibility instead of computing-then-masking. This backend is released too.

Teaching the model when to speak: Realtime-SFT

Architecture makes perceiving-while-generating possible; it doesn't make the model do anything with it. That behavior is installed in a single light final stage — 34.8B tokens, under 3% of the training budget — atop a conventional four-stage pretraining curriculum (150B→203B→459B→450B tokens, sequence lengths 8K→256K) and standard SFT. No RL, no thinking mode.

The mechanism is minimal. Training samples interleave frames and text in arrival order, and every frame is followed by a decision slot. Each slot resolves to one of three forms: <|silence|> (keep watching), <|response|> plus a text chunk (speak, not yet finished), or a chunk ending in <|silence|> (reply complete). A long reply is spread across consecutive slots, one chunk per frame, emulating rate-limited real-time output — and a single user turn can contain several separate self-initiated emissions. The entire apparatus costs exactly two new vocabulary tokens and no decision head: speak-or-wait is plain next-token prediction. If <|silence|> is the argmax, wait for the next frame; otherwise, decode.

Corpus stats: 2.2M supervised emission decisions · 58.7% self-timed (unprompted) 5.1% of samples: event never occurs — correct answer is silence throughout streams up to 768 frames (~12.8 min) at 1 fps

The corpus is where the behavioral engineering lives. Synthesis follows a caption-driven pipeline: densely time-anchored captions are mined for state transitions of a focal object, each transition yields a question, an immediately-answerable reply, and a trajectory of updates; temporal anchoring is verified frame-by-frame against the actual footage, and a quality gate keeps only replies grounded in what's visible at emission time. Notably, a reply overtaken by a new event is rewritten as a plain-language self-correction — revision is taught in natural language, not through a special interrupt token.

Two training details are unusually well-motivated. First, silence slots vastly outnumber emissions, so uniform loss teaches permanent silence; the two state tokens get a focal-style reweighting w_i = \alpha_{y_i}(1-p_i)^\gamma with per-global-batch inverse-frequency coefficients \alpha_k, equalizing the aggregate weight of speak vs. wait regardless of how rare speaking is. Second — my favorite detail — they mask the assistant's turn-final end token from the loss, because in a stream a turn boundary usually means the user interjected, not that the conversation is over. In a controlled single-variable comparison, that one masking choice raised emission frequency by 39% and mean reply length by 68%. That's the kind of ablation that tells you the team actually debugged the behavior rather than just scaling data.

Mode control is one system prompt: the same weights run offline (no prompt), streaming, and real-time.

The evidence

Proactive behavior is where the wins concentrate. Across four streaming benchmarks against open streaming baselines (AURA, M4, ROMA, ViSpeak, MMDuet family, etc.), MOSS-VL-Realtime posts the best average on three — OVO-Bench 70.2 vs. 65.3, OmniMMI 32.7 vs. 25.4, ProactiveVideoQA 47.2 vs. 42.7 — and second on StreamingBench's visual average (69.7 vs. AURA's 71.1). The subset pattern is more informative than the averages:

Every subset that tests speaking unprompted at the right moment goes to MOSS-VL, with Proactive Alerting nearly doubled. Where a subset reduces to perception QA over the current scene, AURA keeps the edge — the authors say this plainly, which is to their credit.

Latency scales the right way. Against Qwen3-VL-8B — same Qwen3-8B backbone, so the comparison isolates vision integration — measured in SGLang on a single H200, the TTFT gap widens from 2.8× to 5.1× as visual context grows, end-to-end from 1.9× to 4.3×. The same-video comparison is stricter than it sounds: MOSS-VL forgoes temporal compression (so each arriving frame can be encoded immediately) and carries roughly twice the vision tokens on identical input, yet never falls behind. The widening gap is exactly the regime a live assistant occupies: visual history accumulates by the minute while replies must land on time.

Offline, it holds ground where you'd expect. MOSS-VL-Instruct (trained without the real-time corpus) leads temporal-reasoning video sets — Minerva 40.5, TOMATO 39.5, VideoMME-Logical 17.1, each by ≥4.9 points — plus perception benchmarks like BLINK (78.0, +8.9). It clearly trails Qwen3-VL-8B on MMMU (51.1 vs. 69.6), document/OCR (InfoVQA 68.9 vs. 83.4), and RefCOCO grounding. No thinking mode, and the training budget went to real-time video, not exam-style reasoning.

What to be skeptical about

The most important caveat is stated by the authors themselves: L5 — the headline capability — has no quantitative benchmark. The four streaming benchmarks cover L2–L4, where the model is still evaluated one reply at a time. Perceiving-during-generation is demonstrated only through live demos (a standing conditional instruction firing exactly at four cat-touches-carrot events; live soccer commentary tracking a free kick through whistle, strike, and scoreline). The demos are compelling but qualitative, and "revises its reply mid-sentence when the scene changes" is exactly the behavior no public benchmark measures. Their proposed L5 benchmark is future work.

Second, the proactive wins land precisely on the behaviors the Realtime-SFT corpus directly supervises — standing instructions, timed emissions, silence-until-evidence. That's the intended design ("the data injects the behavior"), but it means the gains partly reflect a well-matched training distribution rather than an emergent capability, and the benchmark suites and corpus were built by overlapping communities. Decontamination is claimed via held-out video lists; distributional similarity is harder to rule out. Third, most baseline numbers come from official reports under those authors' own inference settings, not re-runs. Fourth, there's no ablation isolating cross-attention vs. interleaving at matched training — the efficiency comparison is clean, but the capability comparison conflates architecture, data, and curriculum. The authors call this "co-design"; a skeptic calls it unattributed.

Why it matters

If the result holds, the interesting claim isn't any single number — it's that the field's consensus architecture for VLMs (interleaved visual tokens) is a poor fit for the interactive regime, and a design widely considered superseded is the right substrate for it. The recipe is also strikingly cheap to adopt: two vocabulary tokens, one system prompt, a reweighted next-token loss, and <3% of training tokens convert a strong offline model into one that decides at every frame whether to speak. With all five checkpoints, the FlashAttention-3 streaming kernel, and the real-time inference code released, this is positioned to become the reference open stack for live video assistants — and the natural substrate for a speech-output omni-model, where "perceive while you talk" stops being optional.

Where to spend your time: Section 4.2–4.3 (Realtime-SFT format and loss — the real contribution) and Appendix A.1, whose five-point walkthrough of the dialogue template, including the leading-slot dropout and the frame-release handshake at inference, is the most concrete account of how a real-time VLM actually runs that I've seen in print. Then Figure 4 for the latency curves and Section 7 for an unusually honest limitations discussion.