ICA Lens: Interpreting Language Models Without Training Another Dictionary

Stop Training New Dictionaries for Every Layer

Instead of training massive, expensive neural networks to explain how AI models work, researchers can use a faster classical method called Independent Component Analysis (ICA). This new workflow, ICALens, makes ICA stable and easy to use. It reveals meaningful patterns like grammar, topics, and word meanings without the heavy computational cost of current standards.

Mechanistic interpretability aims to decompose the internal representations of large language models (LLMs) into understandable features. Currently, the industry standard is Sparse Autoencoders (SAEs). These are large, overcomplete dictionaries (sets of features larger than the model's dimensionality) learned via gradient descent. They attempt to map complex activations into sparse, human-readable latents. While powerful, SAEs are computationally punishing. For instance, the Gemma Scope project required hundreds of SAEs. It also used roughly 20% of GPT-3's training compute to map a single model. This creates a massive bottleneck. If you want to explore a new model or a different sparsity setting, you face a prohibitive training and storage tax.

The cost of the SAE status quo

The prevailing assumption in the field is that we need to learn a new, specialized dictionary for every activation site we wish to inspect. This "train-to-understand" paradigm is fundamentally slow. Even when using publicly released SAEs, researchers are limited by the coverage and release cadence of those specific dictionaries.

The authors of this paper argue that we have overlooked a simpler signal. Many interpretable directions are naturally selective on certain tokens. These selective directions should, theoretically, look less Gaussian (meaning they do not follow a standard bell-curve distribution) than random projections. If this is true, we should not need to spend thousands of GPU hours training a neural network to find them. We should be able to extract them directly from the activation geometry using classical statistical methods.

Stabilizing ICA for high-dimensional activations

The core proposal is ICALens. This is a workflow designed to make Independent Component Analysis (ICA) viable for LLM residual streams. ICA is a classical method that searches for a linear basis where the one-dimensional projections are as non-Gaussian as possible. However, applying "off-the-shelf" ICA to LLMs is notoriously brittle. LLM activations are high-dimensional. They contain massive outliers, such as attention-sink tokens (rare tokens that absorb high attention weights). They also feature oscillating components that prevent standard solvers from converging.

The authors implement an optimized, GPU-parallel FastICA pipeline in PyTorch. They introduce three specific "stability recipes" to handle the unique geometry of transformer activations:

1. Row-Normalization: Before whitening (a process that decorrelates the data), each activation vector is normalized by its $\ell_2$ norm. This prevents high-norm outliers from dominating the optimization landscape. As shown in, this significantly stabilizes convergence compared to raw activations.

2. p95-LIM Robust Acceptance: Standard FastICA uses a "worst-case" stopping rule. It rejects a layer if even a single component fails to stabilize. The authors propose a softer fallback. If 95% of components have stabilized according to the Limit (LIM) statistic, the layer is accepted.
3. Adaptive Refit: If a layer refuses to converge, the pipeline automatically reduces the target component count and retries. This ensures the researcher gets the highest possible resolution allowed by the convergence criteria.

Competitive performance without the training tax

The paper demonstrates that ICA is not just a weak baseline. It is a highly efficient "first lens." The authors evaluate ICALens across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 2B Base.

On the SAEBench benchmark, the authors report that ICA is competitive with public SAEs in sparse probing (predicting concepts from feature activations). More importantly, ICA outperforms SAEs in Targeted Probe Perturbation (TPP) under small-to-medium component budgets . TPP is a metric that measures how effectively a feature can be used for selective interventions.

The authors also introduce the Effective Receptive Field (ERF). This measures how much context a component requires to trigger. They find a clear empirical link between non-Gaussianity and interpretability. Components with higher excess kurtosis (a measure of "peakedness" or heavy tails) tend to have smaller ERFs .

Essentially, the most "statistically weird" directions are often the most local and easiest for humans to explain.

Capacity and sign limitations

While ICALens is efficient, it is not a silver bullet. There are two primary constraints a practitioner must keep in mind:

• Capacity Ceiling: Unlike SAEs, which are overcomplete, ICA is inherently compact. It can return at most $d$ components for a $d$-dimensional space. If you need a massive dictionary to capture subtle nuances, ICA will eventually hit a wall that an SAE can bypass.
• Sign Ambiguity: ICA directions are identifiable only up to a sign. A component might represent "presence of X," but the math might return it as "absence of X." The authors address this by treating both sides of the component as separate features. This adds a layer of bookkeeping to any downstream intervention or steering tasks.

The verdict: A mandatory first pass

If you are currently spending significant compute cycles training SAEs just to see if a layer contains interesting features, you should pivot. ICALens is a "probably yes" for rapid exploration. It acts as a high-speed scout. It can tell you which layers or concepts are worth the heavy lifting of full dictionary learning.

The results in prove that ICA successfully recovers the non-Gaussian signatures that SAEs are essentially trying to approximate.

For engineers focused on quick prototyping or resource-constrained interpretability, this is a major win.

The code is reportedly available; see the paper for the canonical link at https://github.com/liusida/ICALens. Checkpoints and an interactive ICA Explorer are also available via the project's Hugging Face collection and web demo.

Figures from the paper