Effective Biological Representation Learning by Masking Gene Expression

TxFM: Surpassing Atlas-Scale Models via Curated Data and Masked Autoencoding

Researchers have developed a new AI model called TxFM. It learns to understand gene expression by "filling in the blanks" in genetic data. Unlike previous models that used massive, messy datasets, TxFM uses a high-quality dataset. This allows it to better predict how cells react to genetic changes.

Common wisdom in transcriptomics suggests that bigger is always better. It assumes scaling foundation models to massive, "atlas-scale" datasets is the only way to capture cellular biology. This paper finds the opposite. The authors report that TxFM, trained on a curated 1.4 million sample corpus, outperforms models trained on datasets over 100× larger. This contradicts the belief that sheer volume trumps data quality in self-supervised representation learning (a method where the model learns from the data itself without manual labels).

The Problem

Modeling RNA sequencing (RNA-seq) data is difficult. Technical noise and experimental batch effects (systematic variations from different lab protocols) create significant hurdles. Current transcriptomic foundation models (FMs) attempt to solve this using self-supervised learning (SSL). However, they frequently hit a performance ceiling. Many recent studies find these models fail to outperform simple linear baselines like Principal Component Analysis (PCA) on held-out tasks.

This creates a dilemma for engineers. Either raw gene expression counts contain almost all the recoverable biological signal. Or, current pretraining approaches are fundamentally misaligned with the data. Most existing models treat gene expression as a sequence. This ignores the fact that a gene expression profile is an unordered "bag-of-words" of molecular abundances. By forcing a sequential inductive bias onto non-sequential data, current FMs struggle to build truly transferable representations.

How It Works

TxFM is an asymmetric Masked Autoencoder (MAE) designed for the non-sequential, count-based nature of transcriptomics. The architecture, shown in, operates in three stages:

1. Asymmetric Encoding: The model takes a gene expression vector $x$ and masks a large portion of it. Only the unmasked gene tokens pass through a Transformer encoder. Each token combines a learnable gene embedding with its preprocessed expression value. A special [CLS] token (a special token used to aggregate sample-level information) is appended to the set.
2. Bottlenecked Representation: The encoder produces a single, sample-level representation ($e$) from the [CLS] token. This forces the model to compress high-dimensional biological signals into a dense latent space.
3. Domain-Specific Decoding: A lightweight Multi-Layer Perceptron (MLP) decoder takes this representation and predicts the full expression vector $\hat{x}$. The authors introduce a novel rectified tanh activation function, $\phi(z)$, in the decoder. This function respects the non-negative, library-bounded nature of normalized RNA-seq data. It prevents the mathematical divergences often seen with standard ReLU activations.

The training uses a Poisson-based reconstruction loss. Unlike Mean Squared Error (MSE), the Poisson loss generates gradients proportional to the relative prediction error. This prioritizes the reconstruction of low-to-moderate expression genes. These genes comprise the majority of the signal in single-cell data.

Numbers

The authors demonstrate that data curation is more impactful than model scale. They report that TxFM-B, trained on the curated DiverseRNA-1.4M dataset, achieves an overall perturbation representation score of 39.11. This score measures how well the model organizes samples and perturbations into meaningful clusters. This result significantly outperforms models like scGPT and STATE-SE, which were trained on much larger corpora.

The performance gains extend to recovering biological logic. The paper finds that the model's learned parameters intrinsically capture functional gene relationships. The authors measure "gene-gene relationship recall"—how well the model's internal weights identify known protein-protein interactions. TxFM-B's decoder achieved a recall of 43.9% after PCA post-processing .

This means the model successfully recovers nearly 44% of known biological links from its weights alone.

From a compute perspective, the training is efficient. The authors report that training the TxFM-B model on the 1.4M dataset required approximately 864 GPU-hours on H100s. This represents a significant saving compared to atlas-scale models.

What's Missing

There are gaps that a practitioner should note. First, the curation of DiverseRNA-1.4M is heavily oriented toward oncology and perturbational biology. The authors admit this distributional bias might limit generalization to other biological contexts. This includes areas like developmental biology or immunology.

Second, the paper lacks a formal exploration of scaling laws for transcriptomics. Moving from a "Small" to a "Base" backbone helped. However, scaling to a "Large" backbone did not yield inductive gains on the 1.4M dataset .

This suggests that the current dataset may saturate the larger model capacity.

Finally, the paper does not address the inference-time latency overhead. The encoder's self-attention mechanism scales quadratically with the number of unmasked tokens. This could become a bottleneck in high-throughput production pipelines.

Should You Prototype This

Yes, if your goal is high-fidelity representation learning for drug discovery or genetic perturbation prediction. The evidence shows that the combination of Poisson loss, rectified tanh activation, and careful data curation beats brute-force scaling. The authors have released a checkpoint and benchmarking tools at https://github.com/recursionpharma/opentxfm.

If you are struggling with foundation models that fail to beat PCA, do not just throw more data at the problem. Look at your preprocessing and your loss function. Implementing a Poisson-based reconstruction objective and a constrained activation function may yield more immediate results than migrating to a larger, uncurated model.

Figures from the paper