DAGroup-PKU/PhysisForcing

TL;DR: PhysisForcing is a training-time auxiliary-loss framework from PKU + NVIDIA that makes video world models for robotic manipulation physically plausible — supervising interaction-critical regions with a pixel-level trajectory loss and a semantic relational loss on intermediate DiT features — at zero inference cost. The interesting part isn't the ~1–4 point video-benchmark gains; it's that plugging the trained model into a closed-loop action planner lifts WorldArena success from 16.0% to 24.0%, which is real evidence that "physics-shaped" video features transfer to control.

The idea

Video world models (Wan, Cosmos, Veo-class) look good but violate physics precisely where it matters for robotics: contact, grasping, object interaction. The standard fixes are either inference-time (guidance, physics simulators in the loop — slow) or full physics-conditioned architectures (expensive, not backbone-agnostic).

PhysisForcing instead adds supervision only during fine-tuning, focused on interaction-critical regions, at two levels:

• Pixel-level trajectory loss — aligning generated motion with tracked point trajectories (the acknowledgements suggest CoTracker3 provides the pseudo-labels, with Depth-Anything-2 likely supplying geometry).
• Semantic-level relational loss — a constraint on an intermediate DiT feature that enforces object-relational consistency (V-JEPA appears in the acknowledgements, suggesting a self-supervised feature target).

Both losses attach to an intermediate feature of the diffusion transformer, so nothing changes in the deployed model: same architecture, same latency. The README is thin on the exact loss formulations — how "interaction-critical regions" are detected, which DiT layer is used, how the relational loss is defined — so the paper is required reading for the details.

Evidence

On the video-generation benchmarks, PF variants top the tables — but read the deltas over the fine-tuned baselines, which is the honest comparison: R-Bench 61.5 → 63.8 (Cosmos), PAI-Bench 84.03 → 85.17, EZS-Bench 80.29 → 81.08. On Wan the fine-tuning itself does a lot of the work (R-Bench 50.7 → 57.9 ft → 62.0 PF). These are solid but incremental video-quality gains.

The number that matters is the world-model evaluation — using the generated video for closed-loop planning via an inverse dynamics model:

A 50% relative improvement over the identical backbone, purely from training-time supervision, is the strongest claim here — it says the physics losses fix something functionally relevant, not just visually. All numbers are self-reported and the paper's per-metric tables aren't in the README.

What's actually there

• Released: inference code and checkpoints for two self-contained bundles — pf_cosmos/ (Cosmos3-Nano + PF, image-to-video) and pf_wan/ (Wan2.2-A14B + PF), each with its own environment and example inputs. MIT license. Demo videos on the project page.
• Not released: training code and auxiliary model checkpoints ("within a week" as of July 8 — check the repo). Without these you can't apply the method to your own backbone, which is the entire selling point of a "plug-and-play" framework.
• Hardware: unstated, but Wan2.2-A14B inference realistically needs 40–80GB-class GPUs; the Cosmos-Nano bundle should be lighter. The two bundles use different PyTorch/CUDA stacks — separate environments required.

Caveats

The method description in this README is a two-sentence sketch; details live in the paper. The closed-loop result is on one planner setup (WorldArena IDM) with one backbone at 5B scale — worth verifying whether it holds across planners and whether PF_Wan14B shows the same effect. And until training code lands, "drops into existing video backbones" is a promise, not a reproducible fact. The benchmark tables also mix apples (frontier commercial models like Veo 3.1, Sora v2 Pro) with oranges (fine-tuned open backbones), so the "ranks first" framing deserves a squint.

Try it

git clone https://github.com/DAGroup-PKU/PhysisForcing.git
cd PhysisForcing/pf_cosmos   # lighter option; or pf_wan for Wan2.2-A14B
# follow the bundle README: environment -> download weights -> run inference

Watch the repo for the training code release — that's when this becomes a tool rather than two checkpoints.