LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics

TL;DR: LeVJEPA takes the recently proposed LeJEPA objective — a single invariance loss plus a provably collapse-free distributional regularizer called SIGReg — and applies it to video for the first time. Because there is no teacher/student asymmetry left to protect, the authors can throw away 95% of the tokens per clip (which improves accuracy) and make the encoder block-causal in time (at zero accuracy cost). The result matches or beats V-JEPA 2 at 5.6–20.8× less pretraining compute, and at matched FLOPs a video-pretrained encoder nearly matches DINOv2 on ImageNet while doubling it on motion understanding. If it holds at scale, the practical conclusion is striking: video pretraining stops being the expensive specialist option and becomes a cheaper, more general substrate than image pretraining.

The problem: collapse, and the scaffolding built to avoid it

Joint-embedding self-supervised learning has a well-known failure mode. If you train an encoder to map two views of the same clip to the same embedding, the globally optimal solution is embarrassingly simple: output a constant vector for everything. Loss zero, representation useless. This is representation collapse.

The field's dominant answer has been architectural asymmetry. BYOL, DINO, I-JEPA, V-JEPA, and V-JEPA 2 all break the symmetry between the two branches: one branch is an exponential-moving-average (EMA) "target" copy of the encoder, a stop-gradient blocks learning signal on that side, and a capacity-limited predictor network sits between them. Empirically this works, but nobody has a clean theory of why it prevents collapse, and every one of those components costs compute and adds fragile schedules (EMA decay curves, predictor sizing, masking patterns designed around the predictor's task). In V-JEPA specifically, a large fraction of every training step is spent on a target-encoder forward pass over the full clip and predictor passes over masked queries — computation that exists purely to stabilize learning, not to improve the objective.

The alternative tradition — masked autoencoding like VideoMAE — sidesteps collapse by reconstructing pixels, which has no trivial solution. But then you pay for a decoder, and you inherit design constraints from the imputation task: VideoMAE needs tube masking (occluding the same spatial location in every frame) because video is so temporally redundant that random masks can be filled in by copying from adjacent frames.

Video already costs an order of magnitude more tokens than images. Stack this machinery on top and you get why video pretraining has been the expensive path.

The key idea: constrain the distribution, not the architecture

LeJEPA (Balestriero & LeCun, 2025) proposed a different fix, previously validated only on images: instead of preventing collapse through architectural asymmetry, directly legislate what the embedding distribution must look like. Their analysis argues that, under mild assumptions, an isotropic Gaussian embedding distribution minimizes worst-case downstream probing risk — and a collapsed solution (zero variance along some direction) is maximally far from an isotropic Gaussian. So enforce Gaussianity and collapse is excluded by construction, with a guarantee rather than a heuristic.

The enforcement mechanism, SIGReg, is elegant. Testing whether a batch of K-dimensional embeddings \{z_i\} is distributed as \mathcal{N}(0, I_K) sounds expensive, but the Cramér–Wold theorem reduces it: a distribution is isotropic Gaussian if and only if every one-dimensional projection of it is standard Gaussian. So at each step you sample M random unit directions a_m, project the batch onto each one, and penalize the deviation of each 1D projection from \mathcal{N}(0,1) using the Epps–Pulley statistic — a bounded, outlier-robust goodness-of-fit test on the empirical characteristic function:

where n is the batch size and the integral is a 17-point quadrature. The cost is linear, and distributed training needs only a single small all-reduce of the averaged characteristic function — independent of batch size and embedding dimension.

The full LeVJEPA training loop is then almost comically simple. Sample a 16-frame clip. Build one global view and V aggressively cropped, photometrically augmented local views (all sharing the same temporal window). Push everything through one shared encoder, read out a [cls] token, map it through a small projector (needed because the encoder's final LayerNorm confines embeddings to a sphere, which conflicts with the Gaussian target). Train with

where z_0 is the global-view embedding and gradients flow through both sides of the MSE — no stop-gradient, no target network, no predictor, no decoder. The trade-off weight \lambda = 0.02 is the objective's single hyperparameter, fixed to the LeJEPA default and never tuned in the paper.

Two consequences that make this more than a simplification

The interesting part of the paper is not the transfer of LeJEPA to video per se — it's what becomes possible once the asymmetry machinery is gone.

Consequence 1: token dropping becomes an augmentation, not an approximation. In masked-prediction methods, the masking pattern is part of the task, so it must be structured (tubes) to be non-trivial. In LeVJEPA, nothing is imputed — the retained tokens are simply the encoder's observation of the clip. So the authors drop a fraction \rho of patch tokens uniformly at random and ask what happens. The naive expectation is graceful degradation. Instead, ImageNet accuracy climbs monotonically with sparsity: 33.9% with all tokens, 47.6% with only 5% of tokens retained. Dropping 95% of the input simultaneously cuts feed-forward cost by up to 20\times and acts as a stochastic augmentation, forcing the clip-level embedding to be inferable from sparse random glimpses.

Even better, the established wisdom inverts: tube masking, mandatory for VideoMAE, hurts here (39.6% vs. 50.7% on ImageNet), because a tube permanently occludes most of the scene in every frame, whereas random dropping gives a spatio-temporally distributed sample from which content stays identifiable. This is a nice example of a design choice that looked like a property of video turning out to be a property of the imputation objective.

Consequence 2: causality for free. Asymmetric methods constrain what the encoder can be, because the two branches must relate in a particular way. With one symmetric branch, attention topology is a free parameter. The authors train with block-causal attention: bidirectional within a frame, causal across frames, so every frame's representation depends only on the past. This should cost something — every token loses its future receptive field — but under frozen probing it matches bidirectional attention (51.2% vs. 50.7% on ImageNet). That means the pretrained encoder is natively streaming: new frames extend the representation at constant incremental cost, without re-encoding history. Current world-model pipelines (e.g., V-JEPA 2-AC) get this by freezing a bidirectional encoder and fitting a separate causal temporal model afterward; LeVJEPA bakes temporal ordering into the encoder during pretraining. For anyone thinking about autoregressive world models or real-time robotic perception, this is the quietly important result.

Two more conventions fall as free bonuses: temporal patch aggregation (\tau=2, the standard trick of merging frame pairs at the input) turns out to be unnecessary — per-frame tokenization at matched token budget beats it on both ImageNet (50.7 vs. 47.4) and, surprisingly, on the motion benchmark SSv2 (30.4 vs. 28.8). And although only the [cls] token is supervised, patch tokens spontaneously acquire semantically organized, spatially precise structure (PCA cleanly separates objects from background) — something V-JEPA 2 lacks and V-JEPA 2.1 needed an explicit auxiliary patch loss to obtain.

The evidence

The experimental design is unusually careful about confounds: all baselines (V-JEPA 2, VideoMAEv2, DINOv2) are retrained on the identical 20% subsample of K710 with official code, matched epochs (240), and matched batch size, then evaluated with the same frozen attentive-probing protocol.

Epoch-matched: across ViT-S/B/L, LeVJEPA matches or exceeds V-JEPA 2 at 5.6–20.8× less total pretraining compute. At ViT-B, the gap is under one accuracy point at 4.8 vs. 36.4 ExaFLOPs; at ViT-L, LeVJEPA beats V-JEPA 2 by 1.9 points at 5.6× less compute — its ViT-L costs less than half of V-JEPA 2's ViT-S.

FLOP-matched: hold total compute fixed instead, letting LeVJEPA's cheap steps buy a longer schedule (1,085 epochs, 10 local views):

LeVJEPA leads ImageNet by 7.6 points and takes K400, but trails VideoMAEv2 by 3.2 points on SSv2 — the motion-centric benchmark. More on that below.

Video vs. image pretraining: the comparison I find most consequential. Video pretraining has historically been justified only by motion understanding, since it lagged image pretraining on appearance. Against a compute-matched DINOv2 trained on frames of the same videos:

A 3.1-point appearance deficit in exchange for nearly double the motion accuracy. Since video strictly contains the appearance information of its frames plus temporal structure, the authors' framing is that once the compute overhead is removed, video becomes the preferable general-purpose pretraining substrate. That's the thesis of the paper's title.

Two smaller results round out the picture. Scaling data (K710 + SSv2 + Walking Tours + PE Video, ViT-L, 100 epochs) lifts ImageNet to 69.5% and SSv2 to 55.0% — +9.5 points over the restricted corpus on a shorter schedule, with no hyperparameter changes, suggesting headroom rather than saturation. And a ViT-Tiny pretrained for 12 hours on a single RTX 5080 on eight unlabeled walking videos reaches 25.2% ImageNet from a frozen encoder; the same GPU fits LeVJEPA at batch size 128 under 8 GB where an equivalent V-JEPA configuration saturates at batch 28. Video SSL on commodity hardware is a real democratization claim.

What to be skeptical about

The SSv2 asterisk. The abstract's headline numbers are ImageNet-heavy, and the motion story has a caveat the authors are commendably upfront about: aggressive dropping hurts SSv2 beyond \rho=0.3 at short schedules. Longer training within the same compute budget largely recovers it, but LeVJEPA still trails VideoMAEv2 by 3.2 points on SSv2 in the FLOP-matched comparison. Random sparse observation plausibly destroys the cross-frame correspondences motion cues depend on. If your primary target is fine-grained motion understanding, the win here is efficiency, not accuracy. The proposed fix — dropping schemes that preserve temporal correspondence at high sparsity — is future work.

Scale. Controlled experiments use 20% of K710 and top out at ViT-L. V-JEPA 2's actual selling point is its internet-scale training run; whether SIGReg's dynamics hold at that scale (very large batches, very large models) is explicitly uncharacterized. The retrained-baseline protocol is the right way to do fair science, but it also means every headline comparison lives in a regime where the baselines were not designed to shine. Baselines were run with "recommended hyperparameters" — possibly tuned for larger data — which could disadvantage them at this scale.

Frozen probing only. All evaluation is attentive probing (plus linear on K400); no fine-tuning, no dense prediction (segmentation, tracking), no actual world-model or planning experiments despite the causal-attention framing. The emergent patch structure is shown qualitatively via PCA, not measured on a dense task. The streaming/world-model claims are architectural affordances, demonstrated to cost nothing, but not yet demonstrated to deliver anything.

Protocol mixing. K400 uses linear probing while other benchmarks use attentive probing (for stated cost reasons, and conservatively so), which slightly muddies cross-benchmark reading of Table 3.

None of these undercut the central claims, but they do mean "matches V-JEPA 2" should be read as "matches V-JEPA 2 retrained in a small controlled regime under frozen probing."

Why this matters, and where to look

The pattern here rhymes with a broader arc in SSL: heuristic collapse-prevention (EMA teachers, stop-grads, predictors, centering/sharpening) getting replaced by explicit, analyzable constraints on the embedding distribution — VICReg constrained second moments; LeJEPA/SIGReg constrains the full distribution with a principled target and a guarantee. LeVJEPA is evidence that once you do this, an entire ecosystem of downstream design conventions (tube masks, temporal aggregation, bidirectional-only encoders, post-hoc temporal models) turns out to have been scaffolding for the heuristics rather than requirements of video itself. Three "obvious" defaults get overturned in one paper: random beats structured masking, per-frame beats aggregated tokenization, and causal costs nothing.

If the compute-matched parity with image pretraining survives replication at scale, the practical recommendation flips: pretrain on video by default, get appearance nearly for free and motion as a bonus, with a natively streaming encoder ready for world modeling. That's a big "if" — but the recipe is simple enough (one loss, one untuned hyperparameter, code released) that replication should be fast.

Where to spend your time in the paper: Section 4 is the heart — the token-dropping analysis (4.1) and the causal-attention result (4.4) are the two findings you'll want to internalize, and the tube-vs-random masking discussion is a genuinely good piece of reasoning about why a convention existed. Then Table 3 and Table 4 in Section 5 for the compute-matched evidence, reading them alongside the SSv2 caveat in 4.1.