yaoyao-jpg/PhiZero

TL;DR: PhiZero (CASIA, arXiv 2607.28624) is a world model that splits "predict the physics" from "render the pixels": a VLM reasoner takes a first frame plus a caption and autoregressively predicts 256 discrete Physical Language tokens describing future dynamics, which a frozen Wan2.2-TI2V-5B diffusion model (with a LoRA and a learned conditioner) then renders as video. The interesting bit is that the token bottleneck carries motion but not appearance — which makes zero-shot cross-domain motion transfer (human→robot hand, sim→real) fall out of the design, with checkpoints and runnable pipelines actually released despite the repo blurb saying "coming soon."

The idea

Pixel-space video models are notorious for physically incoherent rollouts — objects merging, momentum vanishing — plausibly because the model must solve dynamics and appearance jointly in a continuous latent. PhiZero's answer is an explicit intermediate representation: a video tokenizer (QFormer + FSQ quantization) compresses a 33-frame, 896×512 clip into exactly 256 discrete symbols. A multimodal LLM ("Physical Language Reasoner") is then SFT'd with plain autoregressive cross-entropy — no diffusion or pixel loss — to predict those 256 symbols from a single first frame and a text caption. Finally, an inverse-FSQ conditioner and a DiT LoRA inject the symbols into a frozen Wan2.2 base to render the video via flow matching.

The compression ratio is the striking design choice: 33 frames at 896×512 down to 256 tokens. That bottleneck is what forces motion/appearance disentanglement, and it's why the same tokens re-rendered from a different first frame produce the same motion on a new embodiment. The released transfer suite covers 3 ego-to-dexterous-robot, 4 human-to-robot, and 4 sim-to-real cases — the robotics angle (predicting robot behavior from human demonstrations, sim-to-real without paired videos) is where this could matter beyond video generation.

What's actually released

Despite the repo description's "checkpoints coming soon," the README contains full download instructions (hf download ShuyaoShang/PhiZero) for the reasoner and a tokenizer checkpoint bundling the tokenizer, conditioner, inverse-FSQ tail, and DiT LoRA. You also get: end-to-end training code (tokenizer via reconstruction flow matching; reasoner via ms-swift SFT), a GT-extraction pipeline, 19 curated inference cases, 6 reconstruction cases, 11 motion-transfer cases, and a Physics-IQ Verified benchmark harness. The README says the 19-case and 6-case runs were verified with the released checkpoints (every output: 256 tokens, 896×512, 33 frames, 8 FPS).

Notable gaps: no quantitative results in the README — no Physics-IQ score, no baselines, so you're taking the paper's word for the physics claims. The release inference entry point only runs the 19 checked-in cases; it accepts no external image or prompt, so generalization beyond curated examples can't be probed without hacking the manifest. No license is stated. The released tokenizer is a "general initialization" that must be reconstruction-fine-tuned on source-domain videos (even just the source clips themselves) before any robot or sim-to-real transfer works.

Practicalities

This is a heavyweight, two-environment setup: separate conda envs for reasoner (ms-swift fork) and decoder (DiffSynth-Studio fork), each with a from-source flash-attn 2.8.3 build pinned to your GPU arch (A100/H100 flags given), CUDA 12.4, Linux x86-64. Base weights add ~21 GB (Wan2.2 DiT shards + VAE) on top of PhiZero's checkpoints. The released reasoner SFT used 64 GPUs (global batch 512, ZeRO-2); the tokenizer default schedule is 100k steps. Inference is far cheaper but still needs a serious GPU to hold the 5B DiT.

Try it

git clone https://github.com/yaoyao-jpg/PhiZero && cd PhiZero
hf download ShuyaoShang/PhiZero --local-dir ckpt
hf download Wan-AI/Wan2.2-TI2V-5B \
  --include 'diffusion_pytorch_model-*.safetensors' \
  --local-dir ckpt/wan2.2-ti2v-5b
# ... build the two conda envs per environments/ ...
export PHIZERO_REASONER_CHECKPOINT="${PWD}/ckpt/reasoner"
export PHIZERO_DECODER_CHECKPOINT="${PWD}/ckpt/tokenizer/tokenizer.safetensors"
bash examples/run_inference_cases.sh --case case_infer_03

Project page with videos: https://phi-zero.github.io/ — worth watching the motion-transfer clips before committing to the multi-hour environment build.