inclusionAI/Realtime-Venus

TL;DR: Realtime-Venus (Ant Group + Tsinghua) is an open release of two 9B full-duplex conversational models — an audiovisual "Omni" and an audio-only variant — plus a runtime called Harness that lets the assistant delegate a task to a background agent (the demo wires in Codex) and keep talking while it runs, with the result spoken back into the same live conversation. It's a working system with weights, code, and a browser demo rather than a paper with a video, which makes it one of the more testable open full-duplex stacks right now.

What's actually new here

Full-duplex speech models — ones that listen while speaking and handle interruptions natively rather than via VAD-based turn-taking — exist (Moshi, various Chinese lab releases). The distinctive framing here is asynchronous delegation: the conversational model isn't the thing that does slow reasoning or tool calls. Instead, it emits a natural-language task to Harness, continues the live interaction uninterrupted, and Harness later hands back a prepared result whose spoken delivery timing is decided by the conversational model. The README explicitly separates "task completion" from "spoken delivery" in the work tracking — a sensible decomposition that most voice-agent stacks fudge by blocking or playing filler audio.

The Omni model also claims proactive interaction: it watches streaming camera + mic input and can initiate a response when the scene warrants it, rather than only reacting to a wake event. And the duplex training reportedly teaches the model to distinguish backchannels ("uh-huh") from genuine interruptions that require revising the ongoing response — the hard part of full-duplex, and the thing worth stress-testing in the demo.

What's in the release

• Weights: Realtime-Venus-Omni (9B, streaming audio+video in, speech out) and Realtime-Venus-Audio (9B, audio-only, separately trained), on Hugging Face (inclusionAI/Realtime-Venus) with a ModelScope mirror.
• Code: Harness as a standalone Python package (context, routing, agents, work state, delivery), inference example scripts (offline chat, full-duplex streaming, long-video memory variants), and a full browser demo (model HTTP API, web server, capture/playback UI).
• Demo: browser-based, camera + mic, with a Codex task backend for delegation. There's also an Android APK marked Beta.
• Evidence: this is the weak spot of the README — the Results section is just two figure references from the paper ("video and audio understanding" and "full-duplex interaction results") with no numbers reproduced. You'd need the arXiv paper for benchmarks, latency figures, or interruption-handling metrics. Treat capability claims as unverified until you run it.

Practicalities

Apache-2.0 for the code (fonts/figures excepted). Python 3.10 for standalone inference, CUDA + FFmpeg required; the demo installer targets Linux with Python 3.11/3.12 and the live demo wants at least one A100, so 9B streaming inference with speech synthesis isn't consumer-GPU territory as shipped. The demo requires a Codex login for the task backend, which is an odd dependency for an open release — Harness is described as generic, but the only wired-in executor is Codex. The repo is a week old, 40 stars; expect rough edges. Notably, the release includes the Omni integration but the standalone Audio model ships with its own scripts, and the "proactive interaction" behavior is easiest to probe via the duplex video examples that stream a recorded video and save subtitled output.

Things to actually test: overlapping-speech behavior (does it correctly ignore backchannels but revise on real interruptions?), delegation round-trip (does a result land mid-conversation gracefully?), and speech-out latency under simultaneous listening.

Shortest path to trying it

Offline/duplex scripts on any CUDA box:

pip install 'huggingface_hub>=0.34' 'PyYAML>=6.0'
python download_models.py --model omni --local-dir .
pip install -r Realtime-Venus-Omni/requirements.txt

export REALTIME_VENUS_MODEL_PATH=./Realtime-Venus-Omni
python frontend/Realtime-Venus-Omni/duplex_speech_in_chat.py   # streams a recorded video, saves subtitled output

For the live browser demo (A100, Linux): bash install.sh, configure config.json (model.type: "audio" or "video"), then bash start.sh --config config.json and SSH-tunnel port 8032. Weights: huggingface.co/inclusionAI/Realtime-Venus.