Evolutionary Curriculum Learning Improves Biological Sequence Modeling

Instead of training AI models on biological sequences in a random order, this paper proposes a method that introduces sequences gradually. It starts with very similar sequences and moves to more diverse ones. This "curriculum" approach aims to help models better understand the complex, hierarchical rules governing proteins and RNA. Standard training treats all sequences in a Multiple Sequence Alignment (MSA)—a collection of homologous sequences used to identify conserved patterns—as equally interchangeable. The authors argue this ignores the inherent evolutionary structure of the data.

The core tension in biological sequence modeling is that evolution is not a uniform process. It creates a spectrum of similarity. Some sequences are nearly identical and share almost all functional constraints. Others have diverged significantly over time. Current state-of-the-art generative models, such as EVE for proteins or RfamGen for RNA, typically use Variational Autoencoders (VAEs) to learn these distributions. However, these models usually sample training batches uniformly. They might also use phylogenetic reweighting (adjusting weights to prevent overrepresented lineages from dominating the gradient). Neither method considers the "difficulty" of the sequence. This paper suggests that evolutionary distance is a useful, unsupervised proxy for training difficulty.

The failure of exchangeability in MSAs

The fundamental assumption in most MSA-based training is that sequences are exchangeable. This means the order in which you see them should not matter. The authors argue this is a mistake. Sequences in an MSA represent a continuum of evolutionary history. Close homologs provide clear, locally consistent signals regarding structural and functional constraints. In contrast, distant homologs introduce massive compositional variation. This variation can be difficult for a model to reconcile early in training.

By treating all sequences as equally likely to appear in a minibatch, standard stochastic gradient descent fails to exploit this natural hierarchy. This lack of structure can lead to extreme instability. For example, the baseline performance for the PTEN protein shows this risk. While some seeds might perform well, others can "collapse," yielding wildly inaccurate predictions .

The implicit problem is that the model is asked to learn broad, divergent rules before it has mastered the local, highly conserved motifs that define basic function.

Scaling complexity via anchor-based sampling

To resolve this, the authors introduce Evolutionary Curriculum Learning (ECL). This is a plug-in strategy that requires no changes to the underlying model architecture. The mechanism relies on an "anchor-and-neighbors" procedure that evolves during training.

1. Anchor Selection: At each training step $t$, the algorithm selects an "anchor" sequence from the training set. This selection is still governed by phylogenetic weights $\pi$ to ensure lineage diversity.
2. Neighborhood Construction: Instead of sampling the rest of the batch randomly, the model identifies the $k$-nearest neighbors of that anchor. These neighbors are chosen based on precomputed evolutionary distances. For proteins, this uses the WAG+$\Gamma$ substitution model (a maximum-likelihood estimate of amino-acid change rates). For RNA, it uses distances in the Covariance Model (CM) feature space (a probabilistic grammar representing RNA structure).
3. Power-Law Expansion: This is the "curriculum" element. The neighborhood size $k(t)$ is not fixed. It starts small ($k_0$) and expands according to a log-linear schedule toward the full size of the dataset $N$.

As illustrated in, the model begins by mastering the nuances of closely related sequences.

As training progresses, the "window" of allowed evolutionary distance widens. Eventually, the model must integrate the high-variance, divergent sequences it encountered only in the later stages.

Stability gains and improved bit scores

The empirical results suggest that this curriculum acts as a stabilizer and a performance booster. The authors evaluate the method across two distinct domains: protein variant effect prediction and RNA sequence generation.

In the protein domain, the impact on the PTEN protein is notable. The baseline EVE model was highly unstable. It averaged an AUROC of 0.905 but dropped as low as 0.54 in some seeds. ECL achieved a perfect 1.000 AUROC across every single seed . This represents a gain in stability rather than just a shift in average performance. Even on the more stable p53 protein, ECL raised the mean ClinVar classification AUROC from 0.981 to 0.989 [Table 1]. This indicates a slightly better ability to classify disease-causing variants. ECL also leads to better-fitting generative models. This is evidenced by lower validation negative ELBO (the Evidence Lower Bound, a measure of how well the model fits the data) and reconstruction loss .

For RNA generation using RfamGen, the authors measure success using "CM bit scores." These are log-likelihood scores that indicate how well a generated sequence matches the natural grammar of an RNA family. The paper reports that ECL improves mean bit scores across all three tested RNA families .

When aggregating the results across 15 different training runs, ECL outperformed its seed-matched baseline in 12 of them. This passed an exact sign test with $p = 0.035$.

Complexity bottlenecks and limited breadth

There are several reasons to maintain caution regarding the immediate generalizability of ECL.

First, the computational cost of the preprocessing step is $O(N^2)$. The algorithm requires a precomputed pairwise distance matrix for all $N$ sequences in an MSA. Scaling this to extremely large datasets will become a significant bottleneck. The authors acknowledge this. While training itself has no extra per-step overhead, the initial quadratic cost is the primary obstacle to scalability.

Second, the evidence for RNA improvement is statistically thin at the family level. The authors only tested three RNA families. Therefore, they cannot claim that ECL improves RNA modeling generally. They can only claim it improves the specific training runs performed. A family-level test would require a much larger sample size to reach significance.

Finally, the protein results are somewhat constrained by the evaluation sets. For PTEN, the AUROC is "saturated." There were only two benign variants available in the intersection of the MSA and the ClinVar labels. In such a regime, an AUROC of 1.000 means the model successfully ranked those two specific points correctly. It does not necessarily imply a profound leap in general intelligence.

The verdict: A robust inductive bias

Is Evolutionary Curriculum Learning worth implementing? If you are working with existing MSA-based VAE architectures, the answer is likely yes. The method is "orthogonal" to the model itself. This means it is a pure training-side intervention that does not require redesigning encoders or decoders.

The ablation studies provide compelling evidence. Simply sampling from local neighborhoods (the "fixed-$k$" variant) is actually harmful. This was particularly true for p53, where it performed worse than the baseline .

This happens because fixed-size sampling starves the model of global variation. The benefit is explicitly tied to the expansion of the curriculum. This confirms that the power-law schedule provides a principled way to increase task difficulty.

For practitioners, the trade-off is clear. You pay a one-time $O(N^2)$ preprocessing cost. In return, you may gain significant training stability and better downstream performance. Code for the implementation is reportedly available at https://github.com/KentoNishi/icml26-ecl. Expect this to be a standard component in specialized biological foundation models, provided the $N^2$ scaling issue is managed via approximate methods.

Figures from the paper