ARSENAL: Learning Transferable Regulatory DNA Representations with Targeted Short-Context Language Models

Researchers have developed a new AI model called ARSENAL that focuses specifically on regulatory DNA instead of the whole genome. By training on small, functional pieces of DNA, the model becomes much better at identifying important genetic patterns. It also improves predictions of how mutations affect gene activity.

Most DNA language models (DNALMs) aim to learn general-purpose representations of genomic sequence. These models help scientists interpret genetic variants or design new sequences. Current state-of-the-art methods typically rely on massive scaling. They train on entire genomes using extremely long sequence contexts to capture the vast landscape of the nucleus. However, this "bigger is better" approach faces a fundamental hurdle in regulatory genomics.

Regulatory DNA—the instructions that tell a cell when and where to express a gene—is incredibly sparse. While protein-coding regions make up only about 1.5% of the genome, regulatory elements like enhancers and promoters are scattered throughout a massive sea of "neutral" background DNA. The actual functional logic is encoded in short, precise patterns called transcription factor motifs (specific DNA sequences recognized by regulatory proteins). Trying to learn these delicate, local grammars by looking at millions of base pairs of mostly irrelevant background sequence is like trying to learn a specific dialect by reading an entire continent's worth of random noise.

The signal-to-noise problem in genome scaling

Current DNALMs often struggle with regulatory tasks. Their training objectives can become dominated by the statistical properties of non-functional, repetitive background DNA. When a model predicts the next nucleotide across a whole genome, it may prioritize general genome composition. It might miss the specific, combinatorial "syntax" (the rules governing how motifs are arranged) of regulatory elements.

Recent benchmarks show that even as models grow in size, they often perform poorly on critical tasks. One such task is variant effect prediction (predicting how a single mutation changes a regulatory element's function). The authors of the ARSENAL study argue that scaling context length and corpus size alone does not guarantee learning. The sparsity of functional signals in a whole-genome training set acts as a dilutant. This washes out the subtle regulatory grammar that researchers actually need to capture.

Targeting the regulatory signal

To solve this, the authors propose a shift in strategy: instead of training on everything, train on what matters. ARSENAL (A Regulatory SEquence Nucleic Acid Learner) is a short-context masked DNA language model. It is designed to concentrate learning on functionally enriched regions.

The mechanism follows these core steps:

1. Enriched Pretraining: Rather than using the whole genome, the authors pretrained ARSENAL on approximately 2.3 million ENCODE candidate cis-regulatory elements (cCREs). These are high-confidence regions, such as promoters and enhancers, that are known to be biologically active.
2. Local Context Windowing: The model operates on a 350 bp sequence context. This length matches the physical scale at which many transcription factor motifs and their combinations occur.
3. Masked Language Modeling (MLM): During training, the model performs a standard MLM objective. It attempts to predict a hidden (masked) nucleotide based on its neighbors. This forces the model to learn local dependencies and "syntax."
4. Noise Suppression: To prevent the model from memorizing repetitive junk DNA, the authors excluded loss contributions from long, soft-masked repeat stretches during pretraining.

As shown in, this architecture focuses the model's attention on the local regulatory environment.

It treats DNA as a collection of high-density functional modules rather than a continuous long-range string.

Recovering motifs and predicting variants

The effectiveness of this targeted approach is evident in how well the model reconstructs biological reality. In zero-shot tests—tasks where the model makes predictions without specific fine-tuning—ARSENAL successfully recovers known transcription factor binding sites.

The authors report that ARSENAL produces highly structured likelihood reconstructions at representative loci, such as the $\beta$-globin enhancer .

Furthermore, when analyzing nucleotide dependency maps (which quantify how much one nucleotide constrains another nearby), ARSENAL shows clear, diagonal dependency blocks . These blocks align perfectly with recovered motifs. This indicates the model has learned the internal "spelling rules" of regulatory elements.

On the DART-EVAL benchmark, which measures how well models identify known binding motifs, the authors find that ARSENAL outperforms other prominent DNALMs [Figure 3A]. This translates directly to practical utility in variant scoring. When evaluating how single-nucleotide changes affect chromatin accessibility (the "openness" of DNA that allows proteins to bind), the authors report that ARSENAL achieves higher Pearson correlations with experimental effect sizes than prior models .

Remarkably, the authors achieve these results using only a 350 bp window. Previous evaluations used much larger 2,114 bp windows.

Limits of the short-context view

Despite these successes, ARSENAL is a specialized tool with inherent trade-offs. Because the model is restricted to a 350 bp context, it is fundamentally blind to long-range genomic interactions. In biology, an enhancer located thousands of base pairs away can loop over to contact a promoter. ARSENAL cannot model this "enhancer-promoter coupling" because those elements fall outside its narrow field of view.

Furthermore, the model is cell-type agnostic. It learns the general grammar of regulatory DNA. However, it does not inherently know which motifs are active in a specific cell without downstream supervision. Finally, the authors note that the model's generative capabilities are currently evaluated in silico. This means the sequences it "designs" are judged by other AI models rather than being tested in a wet lab. Until these synthetic sequences are validated in biological assays, the "design" aspect remains theoretical.

A verdict for regulatory modeling

If your goal is to build a model that understands the global architecture of a chromosome, ARSENAL is likely not the right tool. However, if you are building tools for variant interpretation, motif discovery, or local sequence design, the paper provides a compelling case for targeted pretraining.

The authors demonstrate that you do not need to ingest the entire genome to learn its most important functional rules. By focusing on the "signal" of the cCREs, ARSENAL achieves superior performance on regulatory-specific benchmarks. It does this while using significantly less computational context. For practitioners looking to implement this, code is reportedly available at https://github.com/kundajelab/regulatory_lm. The verdict: for local regulatory syntax, targeted short-context models are a highly efficient and effective alternative to brute-force scaling.

Figures from the paper