The Identity Trap in EEG Foundation Models: A Diagnostic Audit

The Identity Trap: Diagnosing Subject-Identity Confounding in EEG Foundation Models

Researchers have found that many AI models for brain signals (EEG) aren't actually learning medical conditions. Instead, they are just recognizing the unique "fingerprint" of each person. They created a new diagnostic tool called FMScope. This tool helps scientists tell the difference between a real medical discovery and a model just memorizing who the patient is. This distinction is vital. If a model predicts Alzheimer’s by recognizing a specific patient's brainwave pattern, it will fail on a new person in a real clinic.

The ambiguity of high accuracy

In the push to build "foundation models"—large-scale AI systems pretrained on massive datasets—electroencephalography (EEG) has become a primary frontier. These models promise to identify clinical biomarkers (measurable indicators of a disease). They aim to find patterns in human brain activity. Currently, the gold standard is subject-disjoint cross-validation. This is a protocol where the model is tested on entirely different people than those in the training set.

However, the authors argue that even this rigorous protocol is insufficient. High accuracy under subject-disjoint splitting can still be deceptive. It might reflect a genuine, cross-subject clinical marker. Or, it might reflect stable physiological subject traits that happen to correlate with labels in a specific cohort. A model might achieve high accuracy by learning to identify the individual. It treats the patient's unique "brain fingerprint" as a shortcut to the label.

Inside the FMScope diagnostic suite

To move beyond mere accuracy scores, the authors propose FMScope. This is a "pre-flight" diagnostic protocol. It audits a model's frozen representations (the internal mathematical vectors produced by the model before task-specific training). The goal is to see if the model's internal view is dominated by patient identity or the clinical signal.

The FMScope framework uses five diagnostic layers, as seen in and .

First, the authors use variance decomposition. This quantifies how much information comes from the clinical label versus the subject's identity. Second, they employ subject-axis erasure (using a method called LEACE). This attempts to mathematically "wipe away" the dimensions representing individual identities. If label prediction improves after this erasure, the identity signal was likely a misleading shortcut.

The third diagnostic is aperiodic 1/f ablation. This seeks the physical source of the identity signal. Researchers look at the aperiodic 1/f background (the broadband, non-oscillatory component of the EEG spectrum). Think of this like the steady hum of an engine. While rhythmic beats might signal a state, the underlying hum is a stable characteristic of that specific machine. Finally, the suite uses layer-wise probing to see where the identity signal emerges in the architecture .

It also uses direction consistency to see if subjects encode tasks along a shared mathematical direction.

Universal dominance and the 1/f carrier

The study reveals that the "Identity Trap" is a pervasive issue. The authors tested three prominent foundation models (LaBraM, CBraMod, and REVE) across four clinical datasets. They demonstrated that subject identity dominates these representations. Specifically, the frozen subject-variance fraction is 13–89× higher than a random-Gaussian null (a theoretical baseline for random noise) .

This means the identity signal is far stronger than random chance.

Crucially, this trap worsens during training. The paper reports that fine-tuning (adjusting a pretrained model for a specific task) amplifies subject-related variance. This increase ranges from 10 to 63 percentage points (pp) in all tested cases. Instead of focusing on the disease, fine-tuning often teaches the model to lean harder into the easiest cue: the patient's identity.

The researchers also pinpoint a physiological culprit. By performing aperiodic ablation described in, they showed a result.

Removing the 1/f background component caused a uniform drop in subject-identification accuracy. This drop was 9 to 19 percentage points in the LaBraM and CBraMod models. This suggests the model uses the stable, broadband spectral shape of the EEG to recognize individuals. The model's ability to decode the clinical label improves significantly once this identity axis is erased [Table 3]. This is especially true in datasets where the label varies within the same subject.

Limits of the audit

The FMScope framework provides a powerful diagnostic. However, the authors acknowledge several constraints. The study uses a $2 \times 2$ sampling layout with only one dataset per clinical category. Therefore, specific conclusions for datasets like "Stress" are empirical observations for those cohorts. Furthermore, the datasets are relatively small ($N \leq 65$). This leads to wide confidence intervals. Readers should be cautious when interpreting the exact magnitude of performance gains.

There is also a structural limitation regarding certain clinical data. For "trait" labels (conditions like dementia that are fixed characteristics), the identity signal and disease signal are entangled. In these cases, the model cannot easily distinguish between the person and the disease. The person is the disease state in the context of the data. The paper notes that for such labels, clinicians must validate predictions against independently established physiological markers. Relying on the model's internal logic alone is not enough.

Verdict: A mandatory pre-flight check

The evidence suggests that current EEG foundation models are highly susceptible to shortcut learning via subject identity. For engineers building clinical AI, the takeaway is clear. High accuracy on a benchmark is necessary but not sufficient for clinical utility.

If you are developing or deploying an EEG model, do not rely solely on subject-disjoint cross-validation. Instead, implement a diagnostic pass like FMScope. This ensures your model is not just a sophisticated biometric scanner. The authors provide the FMScope toolkit as an open-source resource. Code is reportedly available at https://github.com/Jimmy110101013/fmscope. Before committing heavy compute resources to fine-tuning, use these tools. Verify that your model has actually captured a biological signal worth pursuing.

Figures from the paper