Westlake-AGI-Lab/WorldinWorld

TL;DR — World in World (Westlake AGI Lab) turns a frozen 14B video world model into a controllable re-renderer for real footage: hand it an ordinary clip and a new camera path, and it "re-films" the same event from the new viewpoint — arcs, dollies, in-place rotations, bullet-time freezes — with zero training or fine-tuning, on one 80 GB GPU. The trick is converting everything you want to control into visual evidence injected into the model's own attention. Code is out and runnable now; this is a strong demonstration that big video models can be steered geometrically without touching their weights.

What it does

Three capabilities are released, all operating on real videos:

• Free-camera re-cinematography: replay a source clip along an arbitrary new camera path (pans, arcs, trucks, free exploration via keyframes, or any poses.npy).
• Bullet time: freeze the action at any frame and sweep the camera around the frozen moment — the tennis example holds a ball mid-air for 5 s while the camera orbits and returns.
• Video editing: edit the first frame in any image editor, and the edit propagates through the whole clip, optionally combined with a new camera path.

Still on the roadmap: large-angle/360° re-shooting with a 3D body proxy, frustum memory for long videos, streaming/interactive generation, and motion transfer.

How it works

The base model is LingBot-World 2.0 (14B), kept entirely frozen. Source frames are lifted to 3D with monocular depth (Video-Depth-Anything), warped into the target camera, and the warped frames — plus a per-token confidence mask indicating how much each warped token should be trusted — are fed to the model as extra keys/values in its own attention layers. The model literally sees an approximation of what the new camera should see, and paints in occlusions, disocclusions, and warping artifacts using its generative prior. Bullet time is just a degenerate case (freeze the source frame, move the camera); editing swaps the visual evidence in the edited region.

An optional correspondence-guided attention mode links each generated token to the source token showing the same physical point at the same moment, using dense CoWTracker point tracks (~1 min per 16 frames to compute). The README says it helps most on fast, articulated motion.

What's actually there

Runnable code with two bundled examples (source video, depth, and prompt embeddings included), a four-command pipeline for your own videos (prep → depth → traj → generate), a rich camera-path DSL, and debug outputs showing exactly what the model was given (*_input.mp4, *_mask.mp4). No quantitative benchmarks in the README — evidence is the project-page videos and the reproducible examples. Paper is on arXiv (2609.11548).

Practicalities and caveats

• Heavy: ~86 GB of weights, ~72 GB peak VRAM on an A100-80GB, 5–20 min model load, ~5 min generation for a 15-chunk (240-frame) clip at 832×464. Clips of 5–15 s work well.
• Geometry limits: the README is upfront that moderate camera moves work best; turn far from the source view and the model must invent unseen regions from the text prompt. The 360°/large-angle case is explicitly future work (via a 3D body proxy).
• Prompt hygiene matters: describe moving subjects only in the first-chunk prompt, not the scene prompt, or the model invents duplicates once the camera looks away.
• Moving source cameras need externally supplied per-frame poses (srccam.npz) — no built-in SfM.
• License: CC BY-NC-SA 4.0 (inherited from LingBot-World 2.0), so non-commercial only.

Why it matters

This joins WorldForge, ReCamMaster, and TrajectoryCrafter in the re-camera space, but the "control = visual evidence in frozen attention" framing is cleanly general: the same mechanism handles re-cinematography, bullet time, and edit propagation with no per-task training, and the roadmap (cross-model memory sharing, streaming) suggests they see it as a generic interface to any video world model. If it holds up, it's a compelling argument that geometric control of large video models is a runtime problem, not a training problem.

Try it

git clone git@github.com:Westlake-AGI-Lab/WorldinWorld.git && cd WorldinWorld
conda create -n wiw python=3.10 -y && conda activate wiw
pip install torch==2.6.0 torchvision==0.21.0 --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txt
# use the pre-built flash-attn wheel from the README (building from source is fragile)

python tools/download_weights.py --only lingbot   # ~86 GB
python tools/run_example.py examples/tennis_bullet_time

Demos and results: https://chenxi-song.github.io/worldinworld