CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning

When teaching AI new tasks one by one, it often forgets old ones. It may also get confused by different ways of asking questions. This is the central tension in Multimodal Continual Instruction Tuning (MCIT). MCIT is the process of expanding a model's capabilities sequentially as new data arrives.

Engineers currently face a binary choice. They can update all tasks using a shared parameter set. This causes new gradients to overwrite old knowledge, known as catastrophic forgetting. Alternatively, they can allocate dedicated modules for every new task. This prevents the model from reusing learned knowledge. It also leads to unbounded parameter growth. A new paper proposes CRAM (Centroid-Routing and Adaptive MoE) to resolve this. It decouples instruction format isolation from visual capability reuse.

The Problem

The status quo fails because multimodal knowledge is heterogeneous. Some parts of the learning process are "format-sensitive." Other parts are "domain-transferable."

The authors find that instruction formats trigger severe interference. Formats are the specific ways a question is phrased or how an answer is expected. As shown in [Figure 1a], training on one response convention (e.g., providing a letter for multiple-choice) causes a massive drop in accuracy on others. This happens even if the semantic content is identical. This is a "surface-level" failure. The model retrieves the right idea but fails the syntax. This is quantified by the high "format-wrong" ratios in [Figure 1b].

Conversely, visual capabilities are partially transferable. The paper demonstrates in that the model maintains a baseline capability on a different domain (Flickr30k) while training on VizWiz.

Current isolation methods fail here. They treat every task as a silo. This blocks natural reuse. It forces the model to relearn basic visual features for every new task. This wastes parameters.

How It Works

CRAM treats instruction patterns and visual representations differently. This approach is illustrated in .

The method uses three primary mechanisms:

1. Semantic Group Routing: To solve format interference, the model uses a frozen CLIP text encoder (a model that maps text and images to a shared space) to cluster instructions. It calculates a "relative confidence margin" between cluster centers. If an instruction is unique, a new cluster is created. Otherwise, it is routed to an existing group. This isolates conflicting output conventions into separate semantic buckets.
2. Adaptive-Rank Expert Instantiation: Instead of adding fixed-size experts for every task, CRAM probes the "capability gap." During a warm-up phase, the model collects initial gradients into a temporary LoRA buffer (a lightweight trainable module). The authors use Singular Value Decomposition (SVD, a method to decompose matrices into principal directions) to find the main directions of these updates. They apply a "spectral threshold" to discard noise. They also use an "orthogonality filter" to prune directions already covered by existing experts. This ensures new parameters only target novel information. This is seen in the rank pruning in [Figure 5b].
3. Centroid-Guided Orthogonal Learning: To stabilize the Mixture-of-Experts (MoE, an architecture where different parts of the model handle different inputs) structure, the model assigns a learnable "centroid" to each expert. A centroid is a geometric anchor in the feature space. Routing uses a Gaussian Radial Basis Function (RBF, a kernel that measures distance-based similarity). This directs inputs to experts with aligned capabilities. To prevent "magnitude imbalance" when experts co-activate, the authors apply an orthogonality penalty ($L_{dec}$). This forces new updates to stay in the orthogonal complement (the perpendicular subspace) of historical knowledge.

Numbers

The authors report significant gains in accuracy and efficiency. On the UCIT benchmark, CRAM achieves an average accuracy of 74.73%. This outperforms the previous state-of-the-art (SAME). It provides a 13.94% improvement over the MoE-LoRA baseline. On the 10-task TriGap benchmark, CRAM reaches 47.79% accuracy. This surpasses the strongest baseline, DISCO, by 1.25%.

CRAM also shows superior parameter efficiency. As shown in, CRAM achieves these results using significantly fewer parameters than DISCO or HiDe-LLaVA.

The adaptive-rank mechanism scales capacity based on task complexity. Harder tasks like "Roadside" receive higher ranks. Simpler tasks require minimal parameters. The training overhead is also reported to be marginal. There is only a modest increase in time compared to standard MoE-LoRA. This is because the heavy SVD calculations only occur during the discrete expert instantiation phase.

What's Missing

While the results are compelling, there are gaps for practitioners:

• Long-Horizon Stability: The authors admit the method has not been validated on significantly longer task sequences. In a production loop with hundreds of tasks, it is unclear if the cluster-based routing will eventually lead to routing collapse.
• Hardware-Specific Latency: The paper discusses parameter counts and training time. However, it lacks a detailed breakdown of inference-time latency. Adding experts and managing centroid-based routing adds complexity to the forward pass. I would want to see the actual throughput on standard inference hardware.
• Sensitivity to Cluster Initialization: Effectiveness depends on the initial CLIP embeddings and the margin threshold $\theta$. The paper does not deeply explore how sensitive the system is to the quality of the initial semantic clusters.

Should You Prototype This

Yes, if you are building a specialized agent that needs to learn new, domain-specific visual skills sequentially. The core insight is powerful. You can isolate "how to speak" (format) while sharing "what to see" (vision). If you struggle with catastrophic forgetting in a sequential fine-tuning pipeline, the adaptive-rank instantiation via SVD is a concrete technique. It can stop your parameter count from exploding. However, if your task stream is very short (2-3 tasks), the complexity of implementing centroid-routing and orthogonality penalties may not be worth the effort.

Figures from the paper