Neuron Populations Exhibit Divergent Selectivity with Scale

Researchers have found that certain "universal" neurons follow predictable patterns as models get bigger. These neurons act similarly across different AI models. As models scale up, these shared neurons become more specialized and clear. Meanwhile, the rest of the neurons become a messy mix of many different things. This discovery bridges the gap between macroscopic scaling laws and microscopic organization. Macroscopic laws tell us how loss drops as we add compute. Microscopic laws describe how neurons actually organize themselves.

The Problem

Current scaling laws are largely black-box observations. We know that increasing parameters, data, and compute reduces loss. These losses follow precise power laws. However, these laws say almost nothing about internal representational structure. We are essentially flying blind. We treat the model as a monolithic function rather than a collection of interacting units.

This lack of visibility makes interpretability a moving target. As we scale, we encounter the problem of superposition. Superposition is when neurons encode multiple, unrelated features simultaneously to maximize capacity. This creates a "polysemantic" (multi-meaning) mess. A single neuron might fire for both "Python code" and "sunset photos." This makes it impossible to isolate specific behaviors. We currently lack a way to predict how this internal chaos evolves. We do not know how it changes from a 100M parameter model to a 30B parameter model. As shown in, the core challenge is understanding feature competition.

We need to know if scaling helps or hurts our ability to isolate features.

How It Works

The authors focus on "Rosetta Neurons." These are a specific population of neurons whose activation patterns recur across independently trained models. They identify these units through a rigorous matching process:

1. Activation Alignment: The authors align activations to a shared canonical space. For language, they use UTF-8 byte boundaries to ensure alignment regardless of tokenization (the process of breaking text into sub-units). For vision, they use bilinear interpolation (a method to resize grids) to map different patch grids to a common spatial resolution .

2. Similarity Scoring: They compute the Pearson correlation (a measure of linear relationship) between neuron pairs across the shared dataset.
3. Mutual Nearest-Neighbor Filtering: They only retain pairs $(u, v)$ if $v$ is the top-$k$ most similar neuron to $u$ in the second model. Simultaneously, $u$ must be the top-$k$ most similar to $v$ in the first model .

Once identified, the authors apply a scaling analysis. They propose an analytical model based on "capacity allocation." The intuition is that a network has a finite budget of "clean" neuron capacity. High-importance features are worth the "cost" of being isolated into a single, monosemantic (single-meaning) neuron. Lower-importance features stay in superposition. They remain mixed in a polysemantic background. This leads to the "Neuron Polarization Effect." As the model grows, the Rosetta population becomes increasingly selective and specialized. The non-Rosetta population becomes a denser, noisier crowd .

Numbers

The paper reports that Rosetta Neurons follow a sublinear power law. This means they grow, but slower than the total model size. In language models (up to 30B parameters) and vision models (up to 5B parameters), the number of these shared neurons grows predictably. The growth exponent $\alpha$ lies between 0.5 and 0.7 .

Consequently, these shared units occupy a shrinking fraction of the total neuron count as models scale.

The most striking practical result is the utility of these neurons in data filtering. The authors used the CodeSearchNet dataset to test this. They identified a single Rosetta Neuron selective for JavaScript. They used this neuron to curate training data. The filter achieved an F1 score of 0.98 for recovering the JavaScript subset [Table 1]. This score indicates nearly perfect recovery of the target domain. Performing continued pretraining on this filtered data reduced test perplexity (a measure of how well a model predicts a sample) to 3.02. This is nearly as good as the "oracle" (perfect knowledge) performance of 3.01. It significantly outperforms random sampling, which yielded a perplexity of 3.59 [Table 1].

What's Missing

There are gaps a practitioner should note. First, Rosetta Neurons only capture a subset of shared computation. They do not account for structures like circuits or attention heads. These elements might carry significant functional weight. If you debug complex logic, looking at individual MLP neurons might miss the bigger picture.

Second, the "selectivity" metrics are modality-dependent. In language, they use vocabulary-space kurtosis (a measure of how concentrated activations are). In vision, they rely on a "VLM-as-a-judge" proxy. This uses a large multimodal model to describe neuron responses. Because these metrics differ, it is hard to compare interpretability across modalities using one standard.

Finally, the scaling law may be sensitive to training objectives. The authors note that the trend is robust across many vision objectives. However, they mention that the trend may not hold for all models, citing DINOv3 as a possible exception.

Should You Prototype This

Yes, specifically for data curation. If you are doing continued pretraining, do not just use generic data. The authors show that Rosetta Neurons can act as high-precision filters. They can select high-quality, domain-specific data for training. This can greatly improve sample efficiency.

However, do not expect a silver bullet for general interpretability. The paper proves that scaling polarizes the population. It does not guarantee that every important feature becomes a Rosetta Neuron. If you implement this, start by running mutual nearest-neighbor matching. Use it to see if you can identify your own "specialist" neurons. Code is reportedly available; see the project page for the canonical link.

Figures from the paper