Multiplayer Interactive World Models with Representation Autoencoders

TL;DR: MIRA is a 5-billion-parameter latent diffusion world model that simulates 2v2 Rocket League — all four players' first-person views at once — conditioned on all four players' controller inputs, in real time (20 fps on one B200 GPU). Two things make it worth your attention. First, it's the first world model to treat a fast, physics-heavy multiplayer game as a joint prediction problem over multiple action streams, rather than folding other agents into the environment. Second, its ablations deliver a crisp, somewhat counterintuitive finding: the thing that keeps autoregressive rollouts stable for minutes-to-hours isn't the generative objective or the model size — it's building the latent space on a frozen, pretrained self-supervised encoder (DINOv3). A from-scratch encoder reconstructs better and generates worse, and drifts. Code, 10,000 hours of data, and a live demo are released.

Why multiplayer is a different problem, not a bigger one

Every interactive world model you've seen — GameNGen's DOOM, Genie, DIAMOND's CS:GO, the driving models — takes one action stream and predicts the next frame. Other agents, if present, are just texture: part of the environment's stochastic dynamics, to be marginalized over.

That framing breaks the moment you want to do anything multi-agent with the model: self-play RL inside imagination, counterfactuals ("what if player 3 had rotated back?"), or human-in-the-loop play against learned opponents. For those you need a model that conditions on everyone's actions, which imposes a strictly harder requirement than adding input channels. The model must correctly attribute agency — this dent in the ball's trajectory came from player 2's boost, not player 4's — and stay coherent under arbitrary combinations of four action streams, including combinations never seen in training. The new failure modes are distinctly multi-agent: ignoring one player's inputs, entangling two players' effects, mis-assigning who caused what.

Rocket League is a nasty testbed on purpose: continuous rigid-body physics, cars and a ball colliding at speed, high-frequency discrete controls (9 keys at 15 Hz per player), partial observability from each player's camera, and tight coupling — every player's actions affect what every other player sees within fractions of a second.

The architecture in one pass

The system is a two-stage latent video model, but several details depart from the standard recipe.

The codec is a "representation autoencoder": instead of training a VAE from scratch, a frozen DINOv3-L extracts per-frame features (averaged across several intermediate layers), a single learned linear bottleneck compresses them 2×2 in space, 2× in time, and 1024→32 channels (~192× total vs. RGB), and a causal space-time ViT decoder maps latents back to pixels. No GAN loss, no KL term, no noise injection — just L1 + LPIPS + a DINO-feature consistency loss with gradient-norm-balanced weights.

The world model is a flow-matching diffusion transformer with factorized attention: bidirectional spatial attention within a latent frame, causal temporal attention across frames. Trained with diffusion forcing — each frame in a training clip gets its own independent noise level \tau \sim U(0,1), so the model constantly practices predicting from corrupted context, which is exactly what it faces at rollout time when conditioning on its own imperfect outputs. Progressive self-distillation (shortcut-model style) collapses sampling to 1–2 function evaluations for real time.

Multiplayer conditioning is almost embarrassingly simple: stack the four latent views vertically into one grid, concatenate the four action embeddings in the matching fixed order, and inject them via AdaLN broadcast to every spatial position. Nothing tells the model which actions belong to which view — it learns the attribution from the tiling correspondence. During training, each player's action embedding is randomly replaced with a learned "absent" token, so at inference the model itself can drive any car the user doesn't control — a free, built-in opponent policy.

The aha: a pretrained latent is an error-absorbing latent

The most valuable result in the paper is Table 3 plus Figure 7, and it inverts the usual autoencoder intuition. Swap the frozen DINOv3 encoder for an architecturally identical encoder trained from scratch, end-to-end with the codec. The from-scratch encoder wins on reconstruction — PSNR 32.2 vs. 29.7, SSIM 0.923 vs. 0.891. And the world model trained on top of it is dramatically worse at generation, and it drifts: over a five-minute rollout its gFID climbs 1.7× more than the pretrained baseline, while the DINOv3 latent stays nearly flat.

The authors' explanation is geometric, and it's the mental picture worth keeping: a self-supervised encoder trained on all of natural imagery produces a smooth latent space where nearby world states map to nearby latents. When the autoregressive model makes a small prediction error, the resulting latent is still a valid encoding of some plausible nearby state — the rollout absorbs the error and continues. A from-scratch encoder, optimized purely for reconstruction, has no such constraint; its latent manifold can be jagged, so a small error lands off-manifold, the decoder and the next prediction compound it, and the rollout warps.

Two supporting results sharpen this. Distilling DINO features into the from-scratch encoder recovers much of the gap (gFID 15.7 vs. 22.5) — so it really is the feature structure, not frozenness per se. And even a random frozen projection of DINOv3 features already yields a decent generation latent (gFID 11.9 vs. 10.7 for the learned bottleneck): the pretrained features do most of the work; the learned compression is a small bonus.

Predicting in pixels, meanwhile, is not close. At matched training budget, pixel-space models land an order of magnitude worse on every generation metric and lose controllability:

The objective ablation is equally lopsided: teacher forcing (train on clean context, as GameNGen and DIAMOND did) hits gFID 32.5 / gFVD 944 at the 4 s horizon and degrades ~10× past the training window; diffusion forcing gets 10.7 / 163 and stays flat out to five minutes.

Measuring obedience, not just looks

A world model can look gorgeous and ignore your inputs. The paper's cleanest methodological contribution is the Action Recoverability Ratio (ARR): train a probe (frozen DINOv3 + small head) to detect which keys are pressed from video; then, for a rollout driven by real logged actions, compute

the probe's average precision on the generated video divided by its AP on the codec reconstruction of the same clip. The denominator cancels both the probe's imperfection and the codec's visual domain, isolating whether the world model actually rendered the commanded action. ARR = 1 means actions are as legible in the generation as in ground truth. Crucially, they validated it against human raters judging action adherence: Pearson r = 0.84, Spearman ρ = 0.93 across model variants. This metric deserves adoption — it fills a real gap between FVD-style distributional metrics and expensive human studies.

ARR also reveals the training dynamics: visual quality (gFID) converges early, then the model spends the rest of training climbing in ARR — first learning to render plausible frames, then learning what the actions mean. Rare actions (air-rolls, reverse) lag common ones, correlating with training frequency (r = 0.82). And in the data-scaling study, gFID saturates above ~50 hours of unique data while ARR keeps climbing to 10,000 hours — more data buys action fidelity that appearance metrics can't see. That's a useful general lesson for anyone evaluating interactive video models.

Multiplayer results and emergent behavior

Training multiplayer from scratch at the single-player compute budget collapses; warm-starting from a single-player checkpoint rescues it. (Caveat: at 2× budget the collapse disappears and mostly-multiplayer training from scratch wins — so this is a small-budget curriculum effect, not a deep law.) The multiplayer model beats single-player on the failure modes that matter: single-player models forget cars that leave the frame (they fall out of the 20-latent rolling window) and sometimes merge a car into the ball; the multiplayer model, seeing four views, largely avoids both.

The emergent properties are the fun part. Cross-view attention maps (Figure 17) show a query token on player 1's car, placed via player 2's view, lighting up that same car in the other views — the model has learned object correspondence across cameras without any supervision for it. Action dropout yields a functional opponent policy: uncommanded cars keep contesting the ball plausibly, recovering behavior from pixels that the original bots computed from privileged game state. The model handles humans playing at ~half the bots' action rate and holds a fully-stationary scene stable — a state that literally never occurs in training data. When one view desynchronizes into noise (a genuinely out-of-distribution player behavior), the other three views pull it back into consistency within moments.

Failures are telling: an untouched ball drifts toward a goal (resting balls are rare in bot play, so the motion prior wins); the clock and score slip at transitions; the model occasionally boosts at kickoff even when the human doesn't press boost, because every bot match opens identically (~80 uncommanded boosts and ~30 jumps in 40 minutes of human play). Almost every failure traces to short context (a few seconds of memory for a 5-minute match) or data imbalance — not to the core dynamics.

How much to believe

The headline durability claim needs parsing. Quantitatively, distributional metrics are flat out to five minutes — a real result, ~75× the 4-second training window. The "hours with no sign of collapse" is anecdotal and unmeasured; treat it as "doesn't visibly explode," not "remains a faithful simulator." Note also that stability-under-rollout and correctness are different: a model can stay in-distribution forever while its clock runs at half speed. The game-state probe (ball/car positions tracked from rollout activations, error dropping monotonically from 2130 to 1448 Unreal units across 100M→5B) is the right kind of physical-understanding evidence, but there's no absolute calibration for what 1448 units means for, say, training an RL agent inside this model — and no agent is ever trained in it. That downstream test, the stated motivation, is entirely future work.

Other honest caveats: all data comes from one bot policy (Nexto) on three maps, so behavioral and environmental diversity is narrow, and the environment is nearly deterministic given all four action streams — which flatters conditional prediction. The 5B and 2.5B models converge to comparable quality at this data budget, so model scaling has already saturated here. And "multiplayer" means four players with a fixed tiling; nothing here addresses variable or large agent counts.

Still, the engineering result stands on its own: a 5B model plus decoder producing four consistent 20 fps views under four live action streams at ~35 ms/frame on a single GPU, with everything released. And the design findings — pretrained-feature latents for rollout stability, diffusion forcing over teacher forcing, ARR as a controllability metric, action dropout as free agent modeling — are the transferable payload.

Where to spend your time: Section 6.3 (the codec ablations, especially Table 3 and Figures 7–8) is the core scientific contribution; Sections 6.2 and 6.5 for the ARR methodology; Sections 6.8–6.9 for a refreshingly candid catalogue of what emerges and what breaks. Or skip all of it and play the live demo, which is the argument in its most persuasive form.