cvlab-kaist/GRACE

GRACE is a KAIST/Kakao method that squeezes a pretrained video DiT's latent space down 8× in token count — 16× spatial, 8× temporal compression — while reportedly matching uncompressed Wan2.1-14B on VBench, giving 11.1× lower latency at 480×832×81 and 15.5× at 736×1280×81. It attacks video diffusion's real bottleneck (attention cost scales quadratically with latent tokens) without retraining the DiT from scratch, and the inference code plus checkpoints are already out.

The idea

The hard part of compressing a video VAE harder is not reconstruction, it's that the DiT was trained on one latent distribution and falls apart on a new one. GRACE's answer has three pieces:

1. Dual-latent representation. A frozen base latent stays in the space the pretrained DiT already models; a learned residual latent carries the fine detail that aggressive compression would otherwise destroy. The DiT adapts to a familiar space plus a correction, not a space learned from scratch.
2. Generation-aware alignment. During training, the compressed latent is matched to the original one inside the frozen DiT's feature space, not just at the pixel level. Their autoencoder table makes the point explicitly: competing AEs that reconstruct 1–2 dB better score lower on VBench-I2V after equal-budget adaptation. "Better reconstruction ≠ better generation" is the paper's genuinely useful claim, and it echoes a growing suspicion that reconstruction-optimized latents are the wrong objective for generative latents (cf. REPA-style alignment work on the image side).
3. Asymmetric denoising. At inference, the base latent is denoised ahead of the residual by a fixed offset (δ=0.15), so detail gets added onto content that's already settled. The repo warns this flag is load-bearing: omit --async_delta 0.15 and you get a plausible but different (worse) video.

What's actually released

Inference code for T2V and I2V, plus checkpoints on HuggingFace (chimaharicox/GRACE: DiT LoRA-merged weights, VAE, decoder — 23 GB, auto-downloaded). Base Wan2.1-T2V-14B and I2V-14B-480P must be fetched separately; budget ~100–130 GB of disk total. Training code (both stages) is not yet released, so you can't apply this to your own backbone yet. No HF demo yet either. MIT license on their code; Wan2.1 weights under Wan's license.

Evidence: VBench/VBench-I2V tables under the official protocol (50 steps, CFG 5, bf16), claimed parity with the uncompressed pipeline at 32.8k → 4.3k tokens (480p) and 77.3k → 10.1k (736×1280). They ship the exact prompt sets and a latency benchmarking script. All numbers are self-reported; the table images don't render in the raw README, so exact VBench scores aren't quotable here.

Caveats and maturity

The README is unusually candid about reproducibility traps, which is a good sign but also a warning: you must use the bundled fork of DiffSynth-Studio (the upstream pip package silently ignores rope_pos_scale and produces different videos), avoid FlashAttention, pin PyTorch 2.5.1 for the tabulated numbers, and keep --async_delta 0.15 on. That's a brittle stack. It's tied to Wan2.1-14B specifically; generality across backbones is asserted by design but not demonstrated in released artifacts. And at 8× token compression, the failure modes to probe are exactly what VBench underweights: fast motion, fine texture over time, temporal flicker (note VBench-I2V's Total even excludes temporal_flickering). One week old, 25 stars — nobody has independently verified parity yet.

Still, if parity holds even approximately, this is the most practically important direction in video gen right now: a 10–15× latency cut without touching the DiT architecture, and a transferable lesson that latent spaces should be optimized against the generator's features, not PSNR.

Try it

git clone https://github.com/cvlab-kaist/GRACE.git && cd GRACE
python3 -m venv .venv && source .venv/bin/activate
pip install torch==2.5.1 torchvision && pip install -r requirements.txt

huggingface-cli download Wan-AI/Wan2.1-T2V-14B --local-dir ./Wan2.1-T2V-14B
huggingface-cli download Wan-AI/Wan2.1-I2V-14B-480P --local-dir ./Wan2.1-I2V-14B-480P
export GRACE_WAN_T2V_DIR=./Wan2.1-T2V-14B GRACE_WAN_I2V_DIR=./Wan2.1-I2V-14B-480P

bash scripts/generate_t2v.sh "a shark is swimming in the ocean" outputs/t2v

Samples and full tables: project page · paper · weights.