Fairness Pruning: Locating Demographic Bias in GLU-MLP Layers via Differential Activations

Pinpointing the Bias Neurons

Researchers have found a way to pinpoint specific "biased" neurons in AI models. They do this by comparing how neurons react to different demographic groups. By turning these neurons off, they can change biased behavior. Cruously, they can do this without breaking the model's ability to reason or speak normally.

Large language models (LLMs) often inherit historical inequalities from their training data. This leads them to reproduce or even amplify demographic biases regarding race, gender, or religion. Currently, engineers combat this using three main levers. They filter training data, fine-tune the model with safer examples, or modify outputs during post-processing. However, these methods are blunt instruments. Fine-tuning acts on all weights simultaneously. This makes it difficult to remove bias without triggering "catastrophic forgetting." This is a phenomenon where a model loses general knowledge or reasoning abilities while trying to learn a new behavior.

The central question is whether bias is a diffuse property spread across the entire network. Alternatively, is it concentrated in specific, identifiable circuits? If the latter is true, we might perform surgery on the model's weights. This would be more precise than a total overhaul.

The limits of blunt force mitigation

Existing approaches to fairness suffer from a lack of granularity. Data filtering attempts to fix the problem at the source. But as the authors note, synthetic data used for filtering often inherits the very biases it aims to erase. Post-processing methods attempt to edit the model's internal representations. These methods often struggle to generalize to the complex, generative nature of modern LLMs.

These methods share a structural deficit. They do not distinguish between weights that encode social bias and those that sustain intelligence. This creates a high-stakes tradeoff. The more aggressively you try to "fix" a model's social behavior, the more likely you are to degrade its utility. This might affect tasks like mathematics or scientific reasoning. The researchers argue for a more surgical approach. They use mechanistic interpretability (the study of how neural networks process information) to locate the specific neurons responsible for demographic responses.

A surgical strike via differential activations

The authors propose "Fairness Pruning." This is a lightweight method. It avoids the heavy computational cost of training new models or Sparse Autoencoders (tools used to untangle complex concepts into interpretable features). Instead, the method relies on a four-phase pipeline centered on "minimally contrastive prompt pairs."

1. Contrastive Pairing: Researchers construct pairs of prompts. These prompts are identical except for a single demographic attribute (e.g., "A [man/woman] approached the officer"). To ensure a fair comparison, the authors enforce a strict technical constraint. Both prompts must tokenize—break down into the same number of digital units—identically.
2. Activation Capture: The model processes these pairs. Researchers monitor the internal signals at the input of the down_proj layer. In GLU-based architectures (a common way to build MLP layers in models like Llama), this point captures the integrated signal of the neuron's contribution to the "residual stream." This is the highway of information flowing through the network.
3. Scoring: The authors calculate a BiasScore for each neuron. This is the absolute difference between the average activation caused by the first prompt versus the second. A high score indicates a neuron that reacts sharply to demographic changes.
4. Selective Zeroing: Once the most sensitive neurons are identified, researchers "prune" them. They do this by setting the weights to zero. This effectively silences those specific neurons.

As shown in, the signal is not randomly distributed.

While the raw components of the neurons show activity across various depths, the strongest bias signal consistently concentrates in the model's final layer.

Decoupling bias from intelligence

The effectiveness of this method hinges on whether bias and capability occupy different "circuits." The authors test this by zeroing out a tiny fraction of the model. In the case of Llama-3.2-1B, they zeroed at most 40 neurons. This represents less than 0.031% of the total MLP width. This is a very small portion of the total available neurons.

The results are striking. The authors report a mean retention of 99.49% in reasoning and general knowledge capabilities. This means the model kept almost all its original intelligence despite the pruning. This suggests that bias and capability occupy dissociable circuits. They can be manipulated separately. Furthermore, these biased neurons are highly specific. According to, between 66% and 88% of the candidate neurons for a specific category are unique to that circuit.

They do not overlap with other categories.

However, the intervention is not a perfect "mute" button. Because the BiasScore only measures the magnitude of a neuron's reaction and not its direction, zeroing can have unpredictable results. The authors observe "bidirectional bias destabilization." Removing a neuron might reduce a stereotype in one context. Yet, it might inadvertently amplify it in another. This happens because the selected sets mix neurons that push toward a stereotype with those that push against it.

Navigating the unintended consequences

While the method is efficient, it has clear boundaries. The researchers note that their evaluation relies on the BBQ and EsBBQ benchmarks. These use explicit demographic attributes in multiple-choice formats. Consequently, the method might not catch "implicit" biases. These are biases that emerge from subtle cues like names or occupations rather than overt labels.

Additionally, the technique is currently optimized for models under 4 billion parameters. It specifically targets the SwiGLU architecture. It is unknown if these localized circuits remain as discrete in much larger models. There is also a linguistic hurdle. The requirement for exact tokenization parity is difficult to meet in morphologically rich languages (languages where words change shape based on grammar, like Spanish). This reduces the number of usable prompt pairs.

The verdict: A foundation for steering

Is Fairness Pruning ready for production? Not quite. It provides a vital proof of concept. The current version is a "blind" intervention. It can prove a neuron is involved in bias. However, it cannot yet tell you if that neuron is an amplifier or a regulator.

The real value lies in the methodological foundation. By proving that bias and capability operate on dissociable circuits, the authors have cleared a path. The next step is "directional behavior modulation." This would involve moving from zeroing neurons to "activation steering." This uses asymmetric datasets to calculate a signed score. This would allow engineers to selectively scale a neuron's influence up or down. They could tune the model's social alignment without retraining it. Code and datasets for this approach are reportedly available via the OptiPFair library; see the paper for the canonical links.

Figures from the paper