Video2LoRA: Parametric Video Internalization for Vision-Language Models

From Tokens to Parameters: Internalizing Video into Model Weights

Modern vision-language models (VLMs) struggle with the sheer volume of data required to "watch" a video. To understand a clip, these models typically convert every single frame into a massive sequence of visual tokens—digital building blocks of information. These tokens must be fed into the model's context window (its fixed memory capacity) alongside any text questions. Because the cost of processing these tokens scales with every frame and every repeated query, analyzing long videos quickly becomes computationally prohibitive.

A new paper introduces VIDEO2LORA, a method designed to bypass this token-heavy bottleneck. Instead of forcing the model to re-read every frame every time a user asks a question, the researchers propose turning the video itself into a set of model weights. By converting a video into a compact "adapter," the model can effectively "remember" the visual content. It does this without needing the raw video frames present in its active memory during the conversation.

Moving Beyond the Context Window

The core challenge in video understanding is the "capacity ceiling." Current state-of-the-art approaches attempt to manage this limit through various compression or scaling strategies. These include frame subsampling (discarding frames to save space), visual token pruning (merging spatial pixels), or expanding the context window (increasing the model's literal memory capacity). While these methods improve efficiency, they do not resolve the underlying issue. The visual tokens remain in the context at query time. Furthermore, the model must incur the heavy encoding overhead every time a new question is asked.

VIDEO2LORA takes a fundamentally different approach called parametric internalization. Rather than trying to squeeze more visual information into the context window, the method encodes the video into the model’s own parameters. Specifically, it uses a hypernetwork to predict the weights for a Low-Rank Adaptation (LoRA)—a technique that adds small, trainable mathematical updates to a frozen model—directly from the video. Once this process is complete, the video is essentially "baked into" the model's weights. When a user asks a question, the VLM responds using these specialized weights. This requires zero visual tokens in its context.

The Architecture of Internalization

The researchers implement this via a "perceiver hypernetwork" that acts as a bridge between raw visual data and model parameters. As shown in, the process begins with a frozen VLM encoder.

This encoder processes the input video to produce layer-wise hidden states—the intermediate mathematical representations the model generates as it "thinks" through the video.

The hypernetwork then reads these states and, in a single forward pass, generates the specific LoRA adapter weights. This is distinct from standard LoRA fine-tuning, which requires slow, iterative gradient updates (mathematical corrections made during training). Instead, VIDEO2LORA predicts the necessary weight perturbations directly. The resulting adapter is then attached to the frozen answer model. This allows it to answer text prompts based solely on the "knowledge" stored in the adapter.

To capture the complexity of video, the authors utilize a Perceiver-style resampler architecture. This system uses an encoder resampler to attend to the video-conditioned hidden states. This produces a fixed-size representation. A decoder resampler then uses one output query for each target module and LoRA rank direction. This ensures the vast amount of visual information is distilled into the specific mathematical updates needed for the VLM.

Breaking the Token Bottleneck

The implications of this shift from "tokens in context" to "information in parameters" are most visible in the efficiency gains. The authors report that VIDEO2LORA reduces the answer-time visual-token load by up to 1,500× compared to direct in-context inference. This leads to a dramatic reduction in Time to First Token (TTFT). TTFT is the latency experienced between submitting a query and seeing the first word of the response. Specifically, the study finds query TTFT reductions of 6–80× depending on the model scale.

Crucially, the method proves remarkably stable. Traditional in-context inference often "degenerates"—producing incoherent or repetitive text when overwhelmed by too many tokens. However, VIDEO2LORA remains stable even when scaled to 1,024 frames and 1,024px resolution. This is impressive because it was trained on only 12 frames at 384px. This robustness is illustrated in .

As frame count and resolution increase, the quality (measured by Token-F1) remains steady or even improves. In contrast, the base model's performance would typically collapse.

Furthermore, the method excels in scenarios involving repeated interaction. As demonstrated in, the cost of "internalizing" the video is a one-time setup fee.

As the number of questions per video increases, the amortized cost per question drops significantly. For the 2.2B model, the amortized TTFT falls to approximately 0.80 seconds after ten questions. This makes it ideal for interactive video analysis.

Emergent Compositionality and Limits

One of the more surprising findings involves how these video "memories" can be combined. The researchers discovered that adapters generated for non-overlapping segments of a video can be composed in "rank space." This means one could potentially internalize two different halves of a long video independently. You could then combine their adapters to reason about the whole. This provides a potential path toward processing extremely long videos without seeing them all at once. This is visualized in, where composed adapters retain performance levels close to those of a single, full-video adapter.

However, the framework is not without its edges. The authors note several limitations: 1. Scale Dependency: Currently, a separate hypernetwork must be trained for every different scale of VLM. For example, a 500M model requires a different hypernet than a 2.2B model. 2. Semantic Granularity: Because the method compresses video into weight perturbations, it may prioritize high-level scene and event information. It might overlook fine-grained details, such as precise camera motions or subtle spatial distinctions. 3. Style Mismatch: Since the hypernetwork is trained primarily on captioning tasks, it tends to produce more verbose, descriptive answers. This occurs during question-answering tasks compared to the shorter, more direct responses of the base model.

Future work will likely focus on creating scale-transferable hypernetworks. Researchers may also incorporate mixed supervision—training on both captioning and direct question-answering—to better align the model's response style with user expectations.

Figures from the paper