Driving on Memory

TL;DR: The authors take a state-of-the-art end-to-end driving planner, delete its camera feed, and replace it with latent "memories" of other drives past the same GPS location—weeks earlier or later, with completely different traffic. On NAVSIM, the most popular recent driving benchmark, this blind-but-well-traveled planner scores 91.1 PDMS versus 93.7 for the full camera model, essentially matching the leaderboard frontier without ever seeing the scene it's being graded on. The same trick collapses on closed-loop benchmarks (Bench2Drive, RealEngine's safety tests). The conclusion is uncomfortable and well-supported: a high NAVSIM score does not demonstrate that a model reacts to current traffic, and the benchmark has almost no headroom left to reward that capability.

The recurring nightmare of driving benchmarks

End-to-end driving models map raw sensors to a planned trajectory, and we judge them by benchmark score. The interpretation everyone wants to make is: high score ⇒ the model perceived the road, detected the cars and pedestrians, read the light, and planned accordingly.

This interpretation has been burned before. On nuScenes open-loop planning, AD-MLP and Ego-MLP showed that a network fed only ego status (speed, acceleration, navigation command—no perception at all) matched sensor-based planners, because the metric just measured deviation from the logged human trajectory and most scenes were "keep doing what you're doing."

NAVSIM was designed as the fix. It filters out trivial scenes using a constant-velocity baseline, and instead of trajectory-matching it rolls out your predicted 4-second plan in a lightweight simulation and scores collisions, drivable-area compliance, time-to-collision, progress, and comfort (combined into PDMS). NAVSIMv2 goes further: reactive background traffic, a second evaluation stage from synthetically rendered viewpoints, more subscores (EPDMS). And indeed, the ego-status MLP drops to 66.4 PDMS on NAVSIMv1 and a dismal 14.2 EPDMS on v2, far below camera-based models at 93–94 and 46–48. Problem solved, apparently.

This paper asks a sharper question. The camera image contains two kinds of information: static (road geometry, crosswalks, lane topology, plus quasi-static priors like "this intersection is usually congested") and dynamic (the actual cars, pedestrians, and signal phase right now). The benchmarks are supposed to test reaction to the dynamic part. How much of the score actually depends on it?

The probe: replace eyes with memory

The instrument is called MemoryDrivoR, built on DrivoR, a transformer planner where a ViT encodes multi-camera images into 64 register tokens and two transformer decoders (a trajectory proposer and a scorer) cross-attend to those tokens. That clean interface—planner reads a bag of scene tokens—is exactly the surgical intervention point.

The construction, concretely:

1. Memory bank. Run the frozen DrivoR vision encoder over every frame of the training set. Store each frame's scene tokens Z_i \in \mathbb{R}^{n \times c} together with its global ego pose q_i = (x_i, y_i, \psi_i) (position and heading).
2. Retrieval. At test time, use only the current pose q_t to run a nearest-neighbor search: up to k=10 memories within r=20 m, at most one per traversal for diversity.
3. Injection. Each retrieved memory gets a learned embedding of the relative pose T_{m \to t} = T_t^{-1} T_m (where was this observed relative to where I am now?), added to its tokens. A small Perceiver-style resampler compresses each memory from 64 tokens to \ell = 8. The concatenated compressed memories replace the camera tokens as keys and values in the planner's cross-attention. The planner is fine-tuned from pretrained DrivoR weights; ego status (velocity, acceleration, driving command) stays as an input, as it is for every method on the leaderboard.

query: current pose only

By construction, the planner can access the road layout and location-conditioned regularities ("people usually drive slowly here"), but not the current cars, pedestrians, or signal phase. Whatever score this system achieves is a floor on how much of the benchmark can be earned without dynamic scene understanding.

The result: NAVSIM barely needs eyes

MemoryDrivoR reaches 91.1 PDMS on NAVSIMv1—above UniAD (83.4), DiffusionDrive (88.0), AutoVLA (89.1), and ReCogDrive (90.8), all of which look at the scene, and within 2.6 points of DrivoR itself (93.7). On the supposedly hardened NAVSIMv2, it scores 45.0 EPDMS versus DrivoR's 46.7, above RAP-DINO (39.6) and ZTRS (48.1 is the only camera method meaningfully ahead among those listed). Even the collision-related subscores barely move: no-at-fault-collision goes from 99.0 to 98.3, time-to-collision from 96.7 to 95.3. The only meaningful sacrifice is ego progress (90.0 → 86.0).

An HD-map baseline—vectorized static map tokens injected the same way—gets 86.6/40.8, closing most of the gap but falling short of memory. So memories carry more than geometry; they carry the quasi-static texture of places.

Why does this work? The mechanism of the shortcut

Two ingredients conspire, and the paper's qualitative analysis (Section 4.3 and the Discussion) lays them out clearly.

First, the ego status is a partial answer key. NAVSIM queries the model once and evaluates a 4-second rollout. The initial velocity and acceleration come from the human log. If the human was already braking for a red light or a crosswalk, the model receives that braking as input. It doesn't need to see the light; it needs to continue the maneuver the human already started, and the memory of the static world (there's a crosswalk ahead) provides the context to do so sensibly. Ego status is an intended, legitimate input—every leaderboard method uses it—but over a 4 s horizon it leaks the logged decision.

Second, defensive driving is nearly free when the horizon is short. MemoryDrivoR learns to slow down at locations where its memories suggest traffic is typically dense. Driving slower means fewer interactions within 4 seconds, which means fewer opportunities for the metric to penalize you. The cost shows up exactly where you'd predict: ego progress. A planner that hedges by pace, informed by location priors, converts ignorance of current traffic into a small progress penalty rather than collisions.

Put together: NAVSIM largely tests maneuver continuation plus map-conditioned caution, and there's almost no score left over. With memory alone at 91.1 and the human driver at 94.8, the benchmark has roughly 3–4 points of headroom in which genuine reaction to current traffic could possibly be rewarded.

The control experiments that make this credible

The obvious objections are addressed head-on, and this is where the paper earns its claim.

Is retrieval leaking dynamics? Retrieval is keyed on pose only. The retrieved memories mostly come from drives weeks before or after the evaluated sample (Appendix B.1 shows the timestamp distribution), so they cannot encode the current traffic state. There's a subtler leak channel: raw memory tokens do contain the agents present during the past drive, which could correlate with typical traffic. A probing experiment (Appendix A.6) shows the learned resampler actively filters this: after compression, decodability of map elements from the tokens barely drops (AP 49.7 → 49.1) while decodability of agents drops substantially (AP 10.9 → 7.9, worse than a PCA baseline that preserves agents better). The model keeps the streets and discards the ghosts of old cars—retaining, at most, an aggregate crowdedness prior, which is exactly the "quasi-static" information the audit intends to allow.

Is it just location memorization? Maybe camera-based models score high because they memorized test locations during training (train/test locations overlap in NAVSIM). The geographic-split experiment kills this: train camera-based DrivoR without any Pittsburgh data, evaluate only on Pittsburgh, and the EPDMS goes from 39.0 to 42.7—it gets slightly better. Same story on Bench2Drive held-out towns. So the finding is not "models overfit to places." It's the benchmark itself: a model on genuinely held-out geography can still perceive the static scene at test time and coast on it.

Does the audit generalize, or does it break every benchmark? This is the most important contrast:

On Bench2Drive (CARLA, closed-loop, 220 interactive scenarios averaging ~110 m routes), MemoryDrivoR gets a driving score of 34.7 versus 61.0 for camera DrivoR and a success rate of 9.5% versus 30.5%. It beats the no-camera baselines (DrivoR ego-only manages DS 11.6) but is nowhere near passing. RealEngine is even more diagnostic: on open-loop evaluation of the same scenes, memory actually edges out the camera model (87.1 vs 86.6), but in closed loop the gap opens (72.3 vs 88.2), and when RealEngine inserts counterfactual actors—cars that weren't in the original recording—memory collapses to 28.1 vs 53.8. That's the cleanest single demonstration in the paper: identical scenes, identical models, and the shortcut survives exactly as long as the evaluation protocol lets it.

The failure analysis on Bench2Drive (Appendix A.1) confirms the losses are where they should be: pedestrians/cyclists, cut-ins, and signalized junctions show the biggest gaps. On NAVSIM, by contrast, no interaction category—not lead vehicles, not merges, not red lights—cleanly separates memory from camera. The benchmark's own scenario taxonomy can't tell a blind planner from a sighted one.

One caveat within the contrast: HUGSIM-nuScenes is closed-loop with long horizons, yet memory nearly matches camera there (32.2 vs 36.4 HD-Score)—because its 87 scenarios derive from 12 mostly straight routes. Long horizon alone isn't enough; you need demanded interaction.

What changes if this holds

For anyone reading driving leaderboards: NAVSIM PDMS/EPDMS differences among top methods (say, 90–94) should not be read as differences in dynamic scene understanding. With memory alone reaching 91, the metric is saturated with respect to statics-plus-ego-status, and the remaining spread could reflect anything. This matters practically because NAVSIM is one of the two most-used recent benchmarks and drives paper acceptance and architecture claims—several 2025–2026 methods in Table 1 sit below the blind baseline.

The prescription the paper draws is specific: closed-loop evaluation over routes long enough that ego-status cues decay, plus counterfactual actor insertion à la RealEngine so that reacting to current traffic is necessary, not merely possible. NAVSIMv2's pseudo-simulation stage doesn't achieve this because its Stage-2 rollouts start from rule-compliant states with legitimate ego status—errors never compound.

There's also a nice secondary reading: the fusion ablation (Appendix A.7) shows camera + memory beats camera alone on Bench2Drive (DS 63.2 vs 61.0), hinting that pose-indexed memory banks—6.3 GB for all of NAVSIM's four cities, smaller than a VLA checkpoint—are a cheap prior worth having in real systems, echoing concurrent work like PriorEye and Neural Map Prior. And the sensitivity tests show the audit itself is portable: with consumer-grade pose noise it still hits ~87–89 PDMS, and a single retrieved memory (k=1) already gets 89.8.

What to be skeptical about

The claim is deliberately benchmark-level, and the authors are careful not to overclaim—but a few things to hold in mind. The audit requires repeated traversals with accurate poses, so it can't touch benchmarks like WOD-E2E. The 2.6-point residual gap on NAVSIMv1 isn't zero; something about current perception is being rewarded, just not much. The Bench2Drive DrivoR reimplementation is the authors' own and scores modestly (DS 61.0 vs SimLingo's 85.9), so the memory-vs-camera gap there is measured against a mid-tier baseline—though that makes the collapse of memory more, not less, convincing. And the crowdedness prior retained by the resampler is a mild gray zone: "static" here includes statistical regularities of traffic, which blurs the static/dynamic line slightly, though the RealEngine counterfactual test shows this prior is worthless when traffic deviates from typical.

Where to spend your time: Section 4.3 with Tables 1–2 for the headline result and the qualitative crosswalk/red-light analysis; Table 4 (RealEngine) for the sharpest protocol comparison; and Appendix A.6 (resampler probing) if you want to verify for yourself that the memory pathway isn't smuggling dynamics.