netease-youdao/Confucius4-R2T2

TL;DR: Confucius4-R2T2 is NetEase Youdao's streaming ASR model, a fine-tune of Qwen3-ASR that emits an append-only transcript — text is committed once and never revised — at 80 ms–2 s chunk granularity, claiming 200–600 ms average latency at near-offline accuracy. This attacks the single most annoying property of real-time ASR pipelines: unstable partial hypotheses that flicker on screen and force downstream LLM agents, translators, and captioning systems to constantly re-process retracted text. Weights, vLLM inference code, and a WebSocket server are all shipped.

Why it matters

Almost every "streaming" ASR you can actually deploy (Whisper-based real-time wrappers, most commercial APIs) is pseudo-streaming: it re-decodes a sliding window and revises earlier words. Truly append-only decoders exist (transducer-style), but LLM-based ASR — where the accuracy now is — decodes autoregressively over full context and doesn't naturally know when a prefix is safe to commit. R2T2's contribution is teaching an LLM-ASR model to make that call itself.

How it works

The base is Qwen3-ASR (audio-conditioned LLM decoder; parameter count not stated on the card). The training recipe adds three data-construction techniques — stable-prefix data, forced time-alignment data, and token-level audio segmentation — combined under a "Longest Stable Prefix" (LSP) learning paradigm. The model learns to dynamically decide, per chunk, how much of its current hypothesis is stable enough to emit and when it needs more audio. Emitted text becomes fixed context conditioning future predictions, with a guarantee it will never change. A small rollback window (unfixed_token_num, default 1) holds the trailing token(s) as tentative. The tech report is "coming soon," so the LSP training details are currently a promise, not a paper.

The LSP idea: the model itself decides how much of the hypothesis is safe to commit each chunk, so downstream consumers never see retractions.

Evidence

The headline result: at a 160 ms chunk, R2T2 stays close to offline Qwen3-ASR, while naively streaming the base model at the same chunk size collapses.

Chinese is similar: e.g. Wenet-meeting CER 7.27 (R2T2) vs 5.97 offline vs 20.38 for base-model streaming. Two honest caveats when reading these tables. First, several comparison systems (Qwen3-ASR 2s config, WhisperRT, AssemblyAI, the two "commercial" systems) are marked as pseudo-streaming — they get to revise text, R2T2 doesn't — which makes R2T2's parity impressive, but also means "Commercial B" beating R2T2 on most English sets is not an apples-to-apples loss. Second, latency is reported with a nonstandard metric ("retrospective chunk-wise mean fuzzy latency"), and the eval figures themselves are images not reproduced in the text, so treat the 200–600 ms claim as vendor-reported until the tech report lands.

What's actually there

Weights on HF (safetensors), under the NetEase Model Use License (not a standard open license — check before commercial use). Code is Apache-2.0.
Inference: vLLM backend (offline + streaming Python API, batched, hotword/context prompts) plus a transformers fallback.
A ready WebSocket server with VAD (FireRedVAD Stream-VAD), a reference client, and Docker deployment on the Qwen3-ASR image — this is unusually production-shaped for a research release.
Not there yet: tech report, RTF/throughput numbers, model size disclosure, non-CJK/English latency characterization. Needs an NVIDIA GPU and vLLM's usual CUDA version pinning.

Optimized for Chinese and English; other languages are "retained capability," so don't expect the latency/accuracy claims to transfer there.

Try it

git clone https://github.com/netease-youdao/Confucius4-R2T2.git
cd Confucius4-R2T2
pip install -e .   # Python 3.12 recommended; vLLM CUDA pinning applies

./run_example.sh /path/to/audio.wav \
  --model_path netease-youdao/Confucius4-R2T2 \
  --infer_mode stream_vllm \
  --language English \
  --chunk_size_ms 160

Worth an afternoon: stream a noisy, accented meeting recording at 80 ms chunks and watch whether the committed prefix ever goes wrong in a way it can't fix — that's the failure mode append-only decoding trades for stability.