BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language

Researchers have discovered a way to "cheat" at brain decoding. By simply padding visual data with zeros, a model can achieve near-perfect reconstruction scores without ever learning a single biological mapping . This "Padding Hacking" vulnerability reveals a critical flaw in how we evaluate neural synthesis. Current methods often mistake simple information leakage for true understanding of the brain.

A new paper introduces BrainJanus to solve this and broader modeling problems. Instead of building separate tools for every task, the authors propose a single, unified framework. This model treats brain, vision, and language as parts of one interconnected system. By teaching a model to speak a shared "language" of discrete tokens, they have created an engine for any-to-any generation. While currently focused on brain, vision, and language pairs, the architecture allows for seamless movement between these three specific modalities.

Beyond Fragmented Translation Pipelines

The status quo in neural modeling suffers from a fundamental misunderstanding of the brain. Neuroscience shows that the human brain is an intrinsic multimodal integration system [Figure 1a]. A single visual stimulus triggers visual responses and associated linguistic concepts across the cortex. Existing machine learning approaches often overlook this interplay. They typically rely on unimodal alignment. This forces brain signals to match only visual features while ignoring the rich linguistic context in neural activity.

This creates a "lossy" modeling problem. Most current methods treat encoding and decoding as independent, unidirectional tasks. They struggle to capture the bidirectional correspondence inherent in biological thought. To compensate for this lost information, researchers often lean heavily on massive, frozen external models. They use tools like Stable Diffusion or LLaMA to "hallucinate" missing semantic detail. As shown in [Figure 1b], this results in task-specific pipelines. These are computationally expensive and cannot share knowledge between seeing, reading, and thinking.

Mapping the Omni Space

To resolve this, the authors propose BrainJanus. It uses two primary architectural innovations to bring disparate data types into a common mathematical territory.

1. The Unified Brain Tokenizer: Continuous neural dynamics are difficult for computers to process directly. The authors developed a VQ-style (Vector Quantized) tokenizer. This component takes continuous neural signals and "quantizes" them. It snaps them to the nearest entry in a discrete "codebook" of learned embeddings. This transforms fluid brain activity into a sequence of discrete tokens. It is much like how a digital camera turns light into pixels.

2. The All-in-One Autoregressive Model: Once all modalities become discrete tokens, they reside in a shared "Omni space." The researchers employ a single Transformer backbone. This model uses next-token prediction (predicting the most likely next piece of data in a sequence). In this setup, the model does not care about the source of a token. It simply learns the probability of what follows .

This enables seamless transitions, such as turning brain tokens into image tokens.

Through supervised fine-tuning (SFT), the model learns to navigate this space bidirectionally. It can switch between encoding and decoding without changing its underlying structure.

Performance Across the Sensory Spectrum

The authors evaluated BrainJanus using the Natural Scenes Dataset (NSD). The results suggest the unified approach provides a significant lift over specialized models. In brain-to-text decoding, the paper reports a BERTScore of 38.12 and a CLIP score of 96.2%. This surpasses previous state-of-the-art models like MindLLM [Table 2]. Qualitatively, the generated captions are more detailed. They capture specific objects and actions rather than generic descriptions .

In brain-to-image decoding, BrainJanus achieved a CLIP semantic similarity of 94.4%. This metric measures how well the image matches the intended concept. Remarkably, BrainJanus is the only autoregressive model tested for direct brain-to-image generation. It outperformed several diffusion-based baselines in high-level semantic alignment [Table 3]. Even without a diffusion mechanism, it produces reconstructions with better structural consistency .

The model also demonstrates zero-shot generalization. A model trained only to generate text from brain signals can also attempt to generate images.

The Risks of Semantic Hacking

The paper identifies a critical vulnerability in how we measure progress. The authors highlight "Padding Hacking," which exposes a flaw in current evaluation protocols.

In visual-to-fMRI synthesis, researchers judge models by how well reconstructed images match the original. However, the authors show a "hacker" model can achieve near-perfect scores easily. It simply takes the ground-truth visual embedding and "zero-pads" it . This hides the answer inside the brain signal's dimensions. The decoder then retrieves the exact information without learning any biological mapping. This trivial strategy produced near-perfect scores in the authors' tests [Table 4].

Beyond this warning, the model has physical constraints. Currently, BrainJanus is restricted to fMRI data within the visual cortex. It cannot yet model whole-brain cognitive processes. There is also a risk of "hallucination." Powerful generative priors might prioritize making a visually pleasing image over staying faithful to the biological signal.

A Verdict on Unified Neuro-Modeling

BrainJanus shifts from "translating" brain signals to "modeling" them as a holistic ecosystem. By proving that discrete tokenization can bridge biological neurons and digital bits, the authors provide a new blueprint.

Is it a complete replacement for current methods? Not yet. The computational costs of running a 7B-parameter Transformer are high. Also, the scope is currently limited to visual processing. However, the ability to perform any-to-any generation within one framework is a major step. For researchers building integrated systems that understand perception and language, this methodology is an essential starting point. Code is reportedly available at the official GitHub repository linked in the paper.

Figures from the paper