Frames2LoRA: Parametric Video Internalization for Vision-Language Models

A hypernetwork watches a frozen vision-language model encode a video, and in a single forward pass emits a LoRA adapter that is the video. Attach the adapter, and the same frozen model answers questions about the clip with zero visual tokens in its context — the video lives in weights, not in the prompt. On SmolVLM2 (500M and 2.2B), this matches direct video-in-context inference on most captioning and QA benchmarks while cutting query-time visual tokens by up to ~1,500× and time-to-first-token by 6–80×. If it holds up at scale, it reframes long-video understanding as a memory-consolidation problem rather than a context-window problem.

Video is rent, not property

The standard way VLMs handle video is to pay rent: every frame becomes hundreds of visual tokens, those tokens sit in the context window, and every single query about the same video re-pays the full encoding and attention cost. A few dozen frames is already tens of thousands of tokens. Worse, the failure mode past the capacity ceiling isn't graceful — models start emitting repetitive gibberish unrelated to the video.

The whole efficient-video literature attacks the size of the rent check: frame subsampling drops temporal coverage, token compression prunes and merges spatial tokens, long-context architectures raise the ceiling, streaming methods keep a rolling buffer. But in all of them, visual tokens are still in context at query time, and the ceiling is still there, just farther away.

The authors' framing is blunt: the capacity ceiling is not a constraint to manage, it's a constraint to eliminate. Their move is to take the video out of context entirely and put it into the model's parameters — as a video-specific LoRA adapter, generated once, reused for every subsequent question.

There's a tidy neuroscience echo here that the paper doesn't make but this reader will appreciate: it's the difference between holding an experience in a replay buffer versus consolidating it into synapses. In-context video is episodic replay — you re-perceive the whole thing on every query. Frames2LoRA is consolidation — encode once into weight changes, then answer from memory. The prior art for the text modality is Doc-to-LoRA (Charakorn et al.), which maps documents to adapters with a feedforward hypernetwork. Video is a qualitatively harder version: the token volume is orders of magnitude larger, the compression is cross-modal (visual semantics must be expressed as perturbations to a language model's weight space), and there's a resolution axis with no textual analog.

The mechanism: read the encoding, write the weights

The pipeline has three frozen-plus-one-trainable pieces. A frozen VLM encoder E processes the video v with an internalization instruction i and yields layerwise hidden states \mathbf{C} = E(v, i) \in \mathbb{R}^{L \times S \times D} — the text-side hidden state at every one of L transformer layers, over the fused sequence of length S, hidden dimension D. Crucially they keep the layer axis rather than pooling: the hypernetwork gets to see how the video representation evolves through the stack and to emit layer-indexed adapters.

A trainable Perceiver-style hypernetwork H_\phi then maps \mathbf{C} to a LoRA adapter \theta(v) = H_\phi(\mathbf{C}). Per layer slice, an encoder resampler attends from learned latent queries into the hidden states (fixed-size summary regardless of video length — this is what buys the frame-count generalization later). A decoder resampler uses one output query per (target module, rank direction), and a shared projection head maps each rank latent to the two LoRA factors \mathbf{A}_{\ell,m} \in \mathbb{R}^{R \times d_\text{in}} and \mathbf{B}_{\ell,m} \in \mathbb{R}^{R \times d_\text{out}}, with \mathbf{B}'s scale initialized to zero so training starts from an identity adapter. They target the MLP down_proj modules of the text decoder at rank R=16.

At query time, the frozen answer model F (same backbone as the encoder) runs with the adapter attached: p_\phi(y \mid p, v) = F(y \mid p; \theta(v)). The prompt p contains only text.

Training is refreshingly minimal: teacher-forced cross-entropy over response tokens, where the targets are cached generations from the same frozen SmolVLM2 conditioned on the actual video frames. Only \phi is updated — no per-video optimization, no gradient-based LoRA fitting, ever. Training data is video spans from FineVideo (60/30/10 mix of single-scene, multi-scene, full-video spans), 12 frames at 384px, captioning and summarization prompts only. Note what this implies: it's self-distillation, so the video-in-context base model is by construction the ceiling. The evaluation question is not "is this better?" but "how much of the base model's video understanding survives the trip through a rank-16 adapter?"

The evidence

On five captioning benchmarks (ActivityNet Captions, PLM-RDCap, PLM-RCap, VDC, CaReBench), Frames2LoRA passes both statistical non-inferiority and equivalence at both scales, under an LLM judge (Qwen3-30B, validated against humans at Spearman ρ = 0.823) and under token-F1. In recovery terms: the 2.2B model retains 91.9% of the base judge score on average, the 500M retains 84.2%.

Video QA is the more interesting test because the hypernetwork never saw QA supervision — it's pure zero-shot transfer from captioning. The judge passes 7 of 8 benchmark-scale pairings. NExT-QA actually improves over the base at both scales (+0.046 at 500M, CI entirely above zero). The one failure, PLM-SGQA at 2.2B (−0.198), passes comfortably at 500M, which the authors read as not a fundamental limit of the approach.

Token-F1 on QA reveals a systematic format mismatch worth understanding: on ActivityNet-QA, F1 crashes to 9–12% of base, yet the judge test passes. The base model answers "yes" or "a dog"; the captioning-trained adapter answers in descriptive sentences that contain the right answer. PLM-SGQA — which has longer descriptive references — flips direction entirely (+0.145 F1 at 500M). So the lexical metric is measuring verbosity, not correctness. Still, keep this in mind: the judge, with its "don't penalize verbosity" rubric, is doing real work in these headline claims.

Out-of-distribution scaling is where the result gets striking. Trained only at 12 frames / 384px, they sweep to 1,024 frames and 1,024px. The Perceiver's fixed-size latent bottleneck means the adapter machinery doesn't care about input length, and it stays stable across the whole grid (average token-F1 change −0.012 at 500M) — while direct in-context inference degenerates into repetitive gibberish at the high end, where Frames2LoRA beats it by +0.12–0.13 F1. Efficiency scales accordingly: geometric-mean TTFT speedups of 6.7× (500M) and 20.1× (2.2B), maxing at 17.2× and 79.1×; answer-time input tokens drop by 150×/302× on average and 713×/1,507× at the largest settings, because the query genuinely contains zero visual tokens.

The amortization story is the practical one. On VidCapBench (~15 questions per video), even counting the one-time internalization cost, average TTFT drops from 6.45s to 0.55s (500M) and 7.06s to 0.58s (2.2B):

Against KV caching and FrameFusion token compression (Figure 4), Frames2LoRA is the only method where query TTFT stays flat as video tokens grow, whose reusable preprocessing is cheaper than cache creation at scale, and whose quality holds at long contexts. A KV cache also amortizes across queries, but it grows with the video; a rank-16 adapter is constant-size, and the two approaches compose (they show FrameFusion + Frames2LoRA works).

Composition, and what's actually in an adapter

Two exploratory results elevate this from "neat trick" to "possible new direction."

Rank-space composition. Split a video into two halves, internalize each independently, concatenate the LoRA ranks, decode. The composed adapter retains 93.1% (500M) and 86.2% (2.2B) of the single-adapter token-F1 on VDC — despite the hypernetwork never being trained for composition. If this holds beyond two chunks, long-video internalization becomes embarrassingly parallel: chunk, internalize, concatenate. That's a memory architecture, not just a compression trick. The caveat is that the current operation doesn't model temporal order at all, and two chunks is the entire experiment.

The adapters have structure. Appendix C decomposes generated rank-16 adapters into rank slices and ablates them. The top 4 of 16 slices (by Frobenius norm product \|A_r\|_F \|B_r\|_F) recover 94.8% of full performance. The rank ordering is stable across all 500 test videos — direction R11 is always most important, R7 always least — meaning the hypernetwork has learned a consistent output coordinate system rather than assigning meaning arbitrarily per video. Most intriguing: removing the single dominant direction gives a higher point estimate than the full adapter (0.1317 vs 0.1262; CIs overlap), suggesting that direction carries a generic captioning prior rather than video-specific content.

Appendix D adds causal probes: layerwise adapter removal and direct-logit-attribution show that early-layer adapter updates have large norms but weak causal effect, while late-layer updates align strongly with the answer direction — the adapter mostly steers logits late in the stack. That norm–function dissociation is a useful methodological warning for anyone doing adapter interpretability.

What to be skeptical of

"Non-inferiority" is doing heavy lifting. The equivalence margin for the judge metric is 0.15 on a [0,1] scale — generous. Real, systematic drops pass it: −0.084 on ActivityNet Captions at 2.2B, −0.085 average on QA at 2.2B, and VDC camera captions at 500M recover only 42% (fine-grained cinematographic attributes resist compression into a rank-16 perturbation, though scale recovers this to 82%). The honest summary is "~85–92% of base quality at effectively zero marginal query cost," which is still a strong trade, but not the parity the headline implies.

Small models, weak baseline. SmolVLM2 at 500M and 2.2B. Whether visual semantics can be written into a 70B model's weight space by a feasible hypernetwork is untested, and the dramatic long-video wins partly reflect that the base model falls apart at 1,024 frames — a stronger long-context VLM would compress that gap. Also, one hypernetwork per backbone, and the 2.2B run took 201 hours on 6 A100s.

Fixed capacity. A rank-16 adapter on down_proj modules is the same size whether the video is 12 frames or 1,024. "Stable" at 1,024 frames means gracefully lossy, not lossless — and there's no retrieval mechanism; whatever the hypernetwork didn't deem caption-relevant is gone. Needle-in-haystack-style temporal QA at long lengths is conspicuously untested.

Self-distillation ceiling. Supervision comes from the frozen base model's own captions, so the method inherits every blind spot of SmolVLM2's video understanding.

Why it matters anyway

The claim that survives the caveats: cross-modal parametric internalization works — a feedforward network can translate a video into a language model's weight space well enough to support open-ended zero-shot QA, at constant query cost, with composable chunks. That's a genuinely different point in the design space from every compression and caching method, and it suggests an agent-memory architecture where experiences are consolidated into adapter libraries rather than replayed as tokens — the systems-consolidation picture, implemented in LoRA.

If you read two things: Section 5.4–5.6 (the OOD scaling sweep, efficiency comparison, and chunk composition — the results that distinguish this from a benchmark-matching exercise) and Appendices C–D (the rank and layer ablations, which are unusually good interpretability work for a systems paper and hint at how a learned "weight-space codebook" for video actually organizes itself).