ant-research/4DAnyone

TL;DR: 4DAnyone (Ant Research, SIGGRAPH Asia 2026) takes a single casually-shot phone video of a person and synthesizes synchronized multi-view videos — configurable orbits up to 48 cameras across multiple pitch rings — which then feed standard Gaussian-splatting pipelines to get a free-viewpoint 4D human. It's a real capability step for human capture: what used to need a multi-camera dome now takes one 1080p portrait clip, and inference code, models, and examples are released under Apache-2.0.

What it is

The pitch is "create anyone in 4D from a casual monocular video." Rather than trying to regress 4D geometry directly, the system reframes the problem as multi-view video generation: given one input view of a moving person, hallucinate what N synchronized cameras arranged around them would have seen. Once you have those N consistent videos plus a camera rig (cameras.json), 4D reconstruction reduces to a solved-ish problem — feed it to nerfstudio for 3DGS today, with 4DGS integration on the roadmap.

This is the same "generation as a bridge to reconstruction" pattern that's been paying off elsewhere (CAT3D-style for objects, ReconX for scenes), applied here to dynamic humans with an explicit human-body prior in the loop.

How it works

The README is light on architecture, but the pipeline is legible from the code layout and outputs:

1. Motion recovery. GVHMR (a vendored submodule) recovers world-grounded SMPL-X body motion from the input video. This result is cached and reusable across runs.
2. Pose rendering. The recovered body is rendered as skeleton/pose videos from each target camera in the user-specified rig (skeletons/*.mp4 in the output). This is the geometric scaffold that keeps generated views consistent with each other and with the motion.
3. Sparse-to-dense generation. A pose-conditioned video diffusion model first produces a handful of "RCP proposal" views (6 of 24 by default, at yaws spread around the orbit), then generates the dense set of target views — a coarse-to-fine strategy that presumably anchors global appearance before filling in intermediate angles.
4. Reconstruction. The N videos + camera rig go into nerfstudio for Gaussian-splat training.

The camera rig is fully configurable: views per pitch layer (divisible by 4 or 6), pitch angles per layer, starting yaw, and yaw span — so you can do a compact 6-view orbit, a 48-view three-ring dome, or a 24-view frontal arc.

What's actually there

• Inference code, auto-downloading model weights and examples, Apache-2.0.
• Multi-GPU parallelism for pose/VAE stages and target denoising (--gpu_ids).
• Documented efficiency work: peak GPU memory reduced to 25.4 GiB (fits a 32 GB card; they benchmark H20-3E, H200, RTX 5880 Ada, and A6000), and a 1.42× end-to-end speedup for the 24-view pipeline via cross-GPU parallelization. FlashAttention-3 / SageAttention supported.
• A nerfstudio guide for 3DGS reconstruction.

Not there yet: no training code, no quantitative results in the README (no PSNR/consistency numbers vs. prior human novel-view work like these pipelines usually report — check the paper), and notably 4DGS reconstruction itself is still an open roadmap item. Today the repo delivers multi-view video synthesis plus static-style 3DGS via nerfstudio; the "4D" in the name is the intended endpoint, partially delivered. Speed is also a concern: they're planning few-step distillation for an expected 5× speedup, implying current diffusion-based generation of 24–48 videos is slow.

Input constraints are real: 720p+ (1080p recommended), 9:16 portrait, single person, ≥121 frames, mild camera motion. "Casual" means phone-shot, not arbitrary.

Assessment

The interesting bet is that pose-conditioned multi-view video generation is now consistent enough to serve as pseudo-ground-truth for splat optimization — that's where to probe. Cross-view consistency failures (texture flicker between adjacent yaws, hallucinated back-of-body detail, loose clothing that SMPL-X can't represent) will show up as blur or floaters after reconstruction even if individual generated videos look clean. Fast motion and occlusion will stress GVHMR upstream. Nearly 1K stars in under three weeks and a SIGGRAPH Asia acceptance suggest it's worth the hour to run the 6-view test.

Try it

git clone https://github.com/ant-research/4DAnyone.git
cd 4DAnyone
git submodule update --init third_party/GVHMR
conda create -n 4danyone python=3.11 -y && conda activate 4danyone
pip install -r requirements.txt

# quick test: 6-view orbit on a bundled example
python inference.py \
  --video_path "data/source/pexels/2785536-uhd_2160_3840_25fps.mp4" \
  --views_per_layer 6

You'll want a ~32 GB GPU (25.4 GiB peak); multi-GPU helps throughput. Then follow their nerfstudio guide to reconstruct from the generated dense views.