next-state/open-dreamer

TL;DR: Open Dreamer is an open JAX/Flax NNX reimplementation of Dreamer 4's world-model pipeline — causal video tokenizer plus action-conditioned latent dynamics — trained on Minecraft/VPT gameplay and playable in real time in the browser. The original Dreamer 4 release (Hafner et al., 2025) was closed; this fills that gap with a trainable, end-to-end open pipeline, though the agent/RL half of Dreamer 4 is still on the roadmap and the license is currently "all rights reserved."

What it is and why you'd care

Dreamer 4's core claim was that you can train agents entirely inside a scalable learned world model — and that the world model itself can run fast enough to be interactive. Nobody outside the original authors has had the training code. Open Dreamer (from Marti Monso, Sacco, and Hu, under the "next-state" org) gives you the world-model half as open code: the tokenizer, the dynamics model, the VPT-style data pipeline, rollout generation, and FVD evaluation. There's also a live in-browser demo where you play Minecraft and can toggle "Game ⟷ Dream" — handing the video stream over to the world model frame by frame, which is a much more honest demo of real-time interactive generation than pre-rendered rollouts.

If you've wanted to do world-model research (interactive video generation, imagination-based RL, action-conditioned dynamics) without building the whole stack from scratch, this plus the referenced Jasmine codebase is the most direct open starting point for the Dreamer 4 recipe.

How the pipeline works

Two models, trained sequentially:

1. Causal video tokenizer — learns a latent representation of gameplay video. Trained on short windows (e.g., 16 frames) of raw MP4 clips, then applied causally to full 256-frame episodes. Standard reconstruction visualizations are dumped during training.
2. Action-conditioned latent dynamics model — trained on the tokenized latent episodes paired with (shifted) VPT-style action sequences. This is the model that generates new frames given actions, and what runs in the real-time demo.

Evaluation is rollout generation plus FVD against real gameplay. The generation.py module contains denoising schedules, which tells you the dynamics model is diffusion/flow-style in latent space, consistent with the Dreamer 4 design.

What's actually there

• Training code: four scripts (train_tokenizer.py, tokenize_minecraft_dataset.py, train_dynamics.py, eval_fvd.py) with Hydra/OmegaConf configs, Grain/ArrayRecord data pipelines, Orbax checkpointing.
• Live demo: hosted in-browser gameplay with the game/dream toggle, running on "Reactor" cloud infra.
• Inference repo: local rollout from your own MP4 + action sequence lives in a separate repo (reactor-team/open-dreamer).
• Not there: no pretrained weights linked in this README, no FVD numbers or comparisons against Dreamer 4's reported results, no training-cost figures, and no data — you need Minecraft/VPT-style ArrayRecord shards yourself. The Dreamer 4 behavior-cloning/RL agent loop is a roadmap item, so this is currently a world-model repo, not an agent repo.

The citation ("How to Train a Frontier-level World Model", Zenodo) suggests an accompanying writeup on the project site with more detail than the README.

Caveats

The big one is the license: all rights reserved, with a formal license "expected in a future release." Don't build anything you plan to ship on this until that resolves. Second, "Dreamer 4 pipeline" here means the world model, not the full method — the imagination-training half that makes Dreamer interesting as an RL result isn't implemented yet. Third, there are no quantitative results in the README at all, so fidelity to the original is unverified; the real-time demo is compelling evidence the dynamics model works interactively, but rollout quality, horizon stability, and training cost need checking before you invest. Requirements: Python 3.11, CUDA 12 JAX, uv.

Try it

Fastest path is the live demo linked from the project page: https://next-state.github.io/open-dreamer/ (or the reactor-team/open-dreamer inference repo for local rollouts). To train from scratch:

git clone https://github.com/next-state/open-dreamer && cd open-dreamer
pip install uv && uv sync && source .venv/bin/activate

# 1. edit configs/dataset/minecraft_vpt.yaml (paths, pixel stats)
uv run scripts/train_tokenizer.py
uv run scripts/tokenize_minecraft_dataset.py
# 2. copy latent_stats.npz values into minecraft_vpt_latent.yaml
uv run scripts/train_dynamics.py
uv run scripts/eval_fvd.py