Robbyant/lingbot-video

TL;DR: LingBot-Video is the first open-weights MoE video diffusion model aimed specifically at embodied intelligence: a 30B-parameter DiT with ~3B active per token, pretrained on web video plus 70,000+ hours of robot/egocentric data and post-trained with reward signals for physical plausibility and task completion. The whole stack is released under Apache-2.0 — dense 1.3B and MoE 30B-A3B weights, a refiner, VLM-based prompt rewriters, inference code, and a robotics-flavored eval benchmark (RBench) — making it the most complete open entry in the "video model as robot world model" race so far.

What it is and why it matters

The bet here is the same one Cosmos and Wan are making: video generation is a path to world models for robots, and the bottleneck is physical understanding, not aesthetics. LingBot-Video differentiates on three axes. First, MoE scaling from scratch for video DiTs — 30B total, 3B active — which they claim gives ~3× faster inference than a comparable dense model at similar capacity. Second, a data engine that mixes massive web video with 70k+ hours of embodied data (manipulation, navigation, egocentric). Third, a multi-reward post-training stage that scores not just aesthetics but physical rationality and task completion — i.e., reward models that care whether the gripper actually picks the thing up.

Note what it is not: this is a T2V/TI2V generator, not an action-conditioned world model. There's no action-token interface, no policy rollout loop, and no training code in the README. It's a pretrained visual imagination engine for robotics, plus the eval to measure it.

How the pipeline works

Inference is unusually opinionated. The DiT consumes structured JSON captions, not free-form prompts, so the recommended workflow is a three-stage chain: a Qwen3.6-27B rewriter (base VLM + a LoRA adapter, two-stage semantics) expands your prompt into JSON, an "Auto Negative" step prunes a caption-specific negative prompt, and then the DiT runs at 480p with an optional refiner DiT upscaling to 1080p (8 refiner steps, tiled VAE decode).

Evidence

On their own RBench (released Aug 2026, with a public leaderboard), LingBot-Video leads all open models and edges out closed ones on average, with standout scores on manipulation, long-horizon, and quadruped subsets:

Standard caveat applies: topping your own benchmark, plausibly aligned with your own training distribution, is weak evidence. The margins over Wan 2.6 and Cosmos3 are modest (0.620 vs 0.607/0.581), and closed models win several columns (reasoning, spatial, single/dual arm). The internal quality/domain evals are referenced but figures aren't in the README text. The interesting empirical question — whether the physical-rationality reward actually produces measurably better physics than Wan/Cosmos — isn't settled by anything shown here; probe it yourself.

What's actually released, and rough edges

Weights for the 1.3B dense and 30B-A3B MoE + refiner (HF and ModelScope), both rewriter models, full inference code with diffusers and SGLang backends (FP8 MoE experts, grouped expert execution), FSDP + CP8 multi-GPU scripts, a performance/memory matrix, RBench, and a tech report. No training code, no reward models, no dataset release.

Practical friction points:

• Nightly PyTorch: recommended torch 2.12.0.dev+cu130 — a dev build, which will annoy anyone with a pinned environment.
• Heavy stack for full quality: MoE + refiner realistically wants multi-GPU (CP8 scripts are the default full-quality path), and FSDP loading builds the transformer in host RAM first, so you need substantial system memory for the 30B checkpoint.
• Prompt chain overhead: quality depends on running a 27B VLM twice before you touch the DiT. The bundled transformers backend is single-process; production use means self-hosting the rewriter behind vLLM/SGLang with per-request LoRA switching.

The 1.3B dense model is the low-friction on-ramp for single-GPU experimentation.

Shortest path to trying it

git clone https://github.com/Robbyant/lingbot-video && cd lingbot-video
python -m venv .venv && source .venv/bin/activate
pip install -U pip && pip install -r requirements.txt && pip install -e .

# grab LingBot-Video-Dense (1.3B) from HF, then:
export MODEL_DIR=/path/to/dense
python rewriter/inference.py --backend transformers --mode t2v \
  --prompt "a robot arm picks up a red mug and places it on a shelf" \
  --duration 5 --output prompt.json
MODEL_DIR="$MODEL_DIR" ./scripts/single-gpu/run_dense_t2v.sh

Project page with demos: https://technology.robbyant.com/lingbot-video — worth watching the manipulation clips before committing GPU time; that's where the embodied-data pretraining should show or not.