HarryHsing/OmniAgent

TL;DR: OmniAgent is a 7B Qwen2.5-Omni-based agent that treats long-video understanding as active perception: instead of ingesting every frame, it iteratively decides which frames, audio segments, or clips to fetch, summarizes each percept into text, purges the raw media, and keeps reasoning. The headline claim — beating Qwen2.5-VL-72B on LVBench (50.5 vs 47.3) with ~73% fewer frames (203 vs 768) — is a strong data point that deliberate evidence-gathering beats brute-force context for long video, and the turn-level RL credit assignment (TAURA) is a genuinely interesting training contribution.

What it is and why it matters

Most long-video models are passive: they subsample frames uniformly, stuff them into context, and hope the answer is in there. Interactive alternatives typically either pre-scan the whole video (so cost still scales with duration) or bolt external perception tools onto an LLM. OmniAgent's pitch is that a single native omni model does everything — perception, reasoning, and action selection — in a POMDP-style Observation–Thought–Action loop. The environment is deliberately dumb: it only extracts and returns raw frames, audio, or clips on request.

The key architectural move is memory consolidation: after each percept is distilled into a textual summary, the raw media is dropped from context. Context cost therefore scales with information need, not video length. That's the mechanism behind the frame-efficiency numbers and behind the positive test-time scaling result (raising the max turn budget adds +6.2% on VideoMME-Long, while actual turns adaptively saturate around ~11.7).

Training: SFT cold start, then TAURA

Two stages. First, agentic SFT on 58K synthetic trajectories generated via best-of-N exploration with self-correction, filtered both by outcome (task success) and by "rationality auditing" (dropping trajectories with unsupported reasoning). Second, agentic RL with TAURA: the insight is that in multi-turn rollouts, uniform per-turn advantage assignment ("advantage homogenization") wastes credit — TAURA uses turn-level entropy to steer credit toward the pivotal discovery turns where the agent actually finds evidence. Rewards during RL are verifiable: exact-match MCQ/numerical, temporal IoU, and MRA — the LLM-as-judge free-form reward is evaluation-only.

Evidence

Self-reported SOTA among open-source models across ten benchmarks. Against its own base model (Qwen2.5-Omni-7B, same eval setup): +7.5 on LVBench, +4.8 on VideoMME-Long, +12.9 on VSI-Bench, +7.8 on OmniVideoBench. The temporal grounding jumps are the most dramatic — near-zero baseline IoU to competitive numbers — which makes sense: grounding is exactly where iterative seek-and-inspect should crush uniform sampling.

What's actually there, and caveats

Released: code, a gym-like reset/step active-perception environment (agent_system/), SFT and RL 7B checkpoints, RL launchers (TAURA and a GRPO baseline), the public SFT recipe, batch eval and a web demo, plus slides on HF Spaces. The environment is reusable — you can harness closed-source models like Gemini in the same OTA loop via inference/. Apache-2.0, built on verl/verl-agent, ICML 2026 paper.

Caveats: 70 stars, all numbers self-reported. The LVBench-vs-72B comparison depends on the 72B's frame budget setup — verify Table details in the paper. The 768-vs-203 frames comparison also elides that 203 frames are gathered sequentially over ~12 turns, so wall-clock latency per question is likely worse than a single-pass model even if token cost is lower. Inference needs 1× A100 80GB; full RL training wants 64× A100s. The README's install section is partially garbled (a pip install line is truncated), so expect some friction. Free-form eval scoring requires a DashScope API key.

Trying it

git clone https://github.com/HarryHsing/OmniAgent && cd OmniAgent
conda create -n omniagent python=3.11 -y && conda activate omniagent
pip install -e .   # see README Installation; needs CUDA 12.6, ffmpeg, flash-attn
# single-sample inference and web demo entry points live in demo/

Checkpoints: OmniAgent-RL-7B / OmniAgent-SFT-7B on Hugging Face; slides at huggingface.co/spaces/harryhsing/OmniAgent-Slides. Even if the exact numbers soften under scrutiny, the reusable OTA environment plus TAURA-style turn-level credit assignment is the part most likely to get adopted.