MLO-lab/LeVJEPA

TL;DR: LeVJEPA takes LeJEPA's provably collapse-free SSL objective (SIGReg) and applies it to video, throwing out everything V-JEPA needs to avoid collapse — EMA target encoder, stop-gradient, predictor — and everything VideoMAE needs — pixel decoder. One encoder, one projector, one hyperparameter. The kicker is that with no imputation task, dropping 95% of tokens at random becomes a beneficial augmentation (accuracy goes up while step cost falls up to 20×), and block-causal attention comes for free, giving you streaming-ready features. Authors include LeCun and Balestriero; weights for a 303M ViT-L are on HuggingFace.

What it is and why it matters

Every major video SSL method carries anti-collapse machinery whose only job is to keep the trivial constant solution off the table. LeVJEPA's claim is that SIGReg — a sketched regularizer that constrains the embedding distribution to an isotropic Gaussian, with a provable no-collapse guarantee — makes all of it unnecessary. The training loss is just invariance between global and local views of a clip, plus SIGReg with a single λ = 0.02. No target-encoder forward pass, no predictor, so the entire cost of a step is the tokens the encoder actually sees.

Two consequences the paper leans on:

Extreme token dropping helps. In masked-prediction methods, masking must be structured (tubes) to keep imputation non-trivial. Here there's nothing to impute, so dropping 95% of patch tokens uniformly at random improves ImageNet probing (33.9% with all tokens → 47.6%) while cutting step cost ~20×; tube masking instead hurts (50.7% → 39.6%). This is what makes per-frame tokenization (tubelet 1) affordable — the encoder sees ~158 of 3137 tokens per step during training, full tokens at inference.

Block-causal attention is free. Bidirectional within a frame, causal across frames, at no accuracy cost (51.2% vs 50.7%). Each frame's representation depends only on the past, so the encoder itself is streaming-ready state for autoregressive world models — no separate temporal model bolted on afterward.

Evidence

All baselines retrained on identical data, evaluated with frozen attentive probing (V-JEPA protocol). Headline claims: matches or beats V-JEPA 2 across ViT-S/B/L at 5.6–20.8× less pretraining compute (their ViT-L costs under half a V-JEPA 2 ViT-S). FLOP-matched at ViT-B, +7.6 on ImageNet over the best video baseline, best K400 linear probe, within 3.2 on SSv2.

The DINOv2 comparison is the interesting one for the field: compute-matched against an image method trained on frames of the same videos, LeVJEPA gets within 3.1 points on ImageNet while nearly doubling motion-centric SSv2 — the first FLOP-matched case (they claim) of video pretraining reaching near-parity with a SOTA image method on appearance tasks. There's also a nice accessibility datapoint: ViT-Tiny, 12 hours, one RTX 5080, eight YouTube videos → 8.9% → 25.2% frozen ImageNet.

What's actually released

Full pretraining recipe (Hydra + Lightning + uv, slurm scripts, Lance-backed data pipeline for Walking Tours), two notebooks (feature visualization; a workshop that trains a tiny model on CPU/MPS/CUDA), and a pretrained ViT-L/16 (303M) on HuggingFace, trained on a 1.8M-clip mix (K710 + SSv2 + Walking Tours + PE-Video): 69.5% IN-1K / 55.0% SSv2 frozen attentive probing.

Caveats: repo is days old and all numbers are self-reported — the compute-matched claims deserve independent replication. Absolute frozen-probe numbers are well below top image SSL models trained at full budget; the story is compute efficiency, not SOTA. Licensing is mixed: repo is MIT but module.py and the weights are CC BY-NC 4.0 (V-JEPA lineage), so no commercial use of the checkpoint. The model card warns that running the weights with full attention silently degrades features — keep attn_mode="block_causal". Frozen-probing only; no fine-tuning or dense downstream (detection/segmentation) results in the README.

Try it

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "galilai-group/LeVJEPA-VideoMix-Large", trust_remote_code=True
).eval()

video = torch.randn(1, 3, 16, 224, 224)  # ImageNet-normalized, ~7.5 fps
with torch.no_grad():
    out = model(pixel_values=video)
out["pooler_output"]  # (1, 1024) CLS feature

To reproduce training from scratch: uv sync, download Walking Tours via scripts/download_walking_tours.sh, build the Lance store, and sbatch slurm/train_walking_tours_vitb.slurm. Paper: arXiv:2608.27395.