allenai/molmo-motion

TL;DR — MolmoMotion is Ai2's 4B VLM that, given a few RGB frames, a set of user-picked query points (with initial 3D positions), and a language instruction like "pick up the mug," autoregressively predicts each point's 3D trajectory for ~2 seconds in the camera frame. The interesting bet is the representation: language-conditioned 3D point futures as a general motion prior that sits between video models and robot policies — and the release is unusually complete (weights, a 1M-example corpus, a new benchmark, training recipes, and a robotics finetuning path).

What it is and why it matters

Point tracking (CoTracker, TAPIR, SpatialTracker) tells you where points went. MolmoMotion is the forecasting inverse: where will these points go, conditioned on what someone says will happen. That makes the output an action-conditioned, object-agnostic motion plan in metric 3D — a plausible interface for both robot planning (predict where gripper/object points should move, then solve for actions) and motion-guided video generation (predict tracks, condition a video model on them). The README claims both transfers; the robotics one ships as a concrete recipe (MolmoBot finetuning on a Franka pick-and-place benchmark in robotics/), the video-generation one is deferred to the paper.

How it works

The backbone is Molmo2-4B-Pretrain, finetuned in two stages to emit trajectories as text-like tokens (the quickstart mentions decoding a raw token block into floats, with an anchor added back — so it's coordinate-token AR decoding, not a regression head). Inputs: H history frames (1 or 3), P=8 query points as 2D pixels at t₀ plus their 3D camera-frame history, an action string, and a horizon F. Output: a (P, F, 3) tensor of absolute XYZ in meters.

Training: Stage 1 pretrains 40K steps at F=8 on five human-video datasets (EgoDex, YT-VIS, HD-EPIC, Xperience, Stereo4D) with √-frequency mixing on 16 GPUs; Stage 2 finetunes 10K steps to F=30 or F=32. DROID (robot teleop) and MolmoSpaces (sim) ship in the corpus but are excluded from the default recipe — the released motion prior comes from human video only, which is the notable design choice for the robotics-transfer claim.

What's actually released, and the evidence

Two checkpoints (H3-F30 and H1-F32, both AR variants), the MolmoMotion-1M corpus, the PointMotionBench eval suite (HOT3D + WorldTrack + DAVIS), the annotation pipeline for labeling new video, full training/eval configs, and an HF converter. The README's example summary.json gives the only numbers here:

Read that DAVIS column carefully: ~1.2 m average error and PWT of 0.15 on in-the-wild video versus ~11 cm on HOT3D (tabletop hand-object scenes). The prior clearly works best on the manipulation-style scenes it was mostly trained on; generalization to unconstrained scenes is much rougher. Also note the README contains no baseline comparisons — you'll need the paper for that — and paper numbers are best-of-5 sampling while default eval is greedy.

Practicalities and caveats

• Code and weights are Apache 2.0, but the corpus is partially reconstruct-it-yourself: raw videos are license-restricted and rebuilt via per-dataset scripts. The data-generation pipeline vendors UniDepth (CC BY-NC) and SAM 3 (Meta license) — fine for using the released weights commercially, not for regenerating data.
• Inference is not fast: ~40 s per predict_trajectory() call on an 80 GB A100 (plus ~110 s checkpoint load). This is a planning prior, not a real-time tracker.
• You must supply query points and their initial 3D positions — in practice you need a depth estimate and intrinsics upstream, which the README doesn't solve for you at inference time.
• The full PointMotionBench eval is single-GPU only, ~9 GPU-hours.
• Repo is a week old, 134 stars; expect rough edges (the Stereo4D reconstruction footgun is documented).

Try it

git clone https://github.com/allenai/molmo-motion.git && cd molmo-motion
conda create -n molmo-motion python=3.11 -y && conda activate molmo-motion
pip install -e .[viz]
hf download allenai/MolmoMotion-4B-H3-F30 --local-dir checkpoints/MolmoMotion-4B-H3-F30
python examples/01_quickstart.py          # bundled DAVIS clip, renders an MP4
# no GPU? render from a bundled prediction:
python examples/01_quickstart.py --from-prediction

Worth an hour of your time if you care about the video-model-to-robot-policy interface question; the human-video-only motion prior transferring to Franka manipulation (via robotics/README.md) is the claim to stress-test.