Genomic Next-Token Predictors are In-Context Learners

Abstract: In-context learning (ICL) -- the capacity of a model to infer and apply abstract patterns from examples provided within its input -- has been extensively studied in LLMs trained for next-token prediction on human text. In fact, prior work often attributes this emergent behavior to distinctive statistical properties in human language. This raises a fundamental question: can ICL arise organically in other sequence domains purely through large-scale predictive training? To explore this, we turn to genomic sequences, an alternative symbolic domain rich in statistical structure. Specifically, we study the Evo2 genomic model, trained predominantly on next-nucleotide (A/T/C/G) prediction, at a scale comparable to mid-sized LLMs. We develop a controlled experimental framework comprising symbolic reasoning tasks instantiated in both linguistic and genomic forms, enabling direct comparison of ICL across genomic and linguistic models. Our results show that genomic models, like their linguistic counterparts, exhibit log-linear gains in pattern induction as the number of in-context demonstrations increases. To the best of our knowledge, this is the first evidence of organically emergent ICL in genomic sequences, supporting the hypothesis that ICL arises as a consequence of large-scale predictive modeling over rich data. These findings extend emergent meta-learning beyond language, pointing toward a unified, modality-agnostic view of in-context learning.

• The paper demonstrates that genomic models trained on DNA sequences exhibit emergent in-context learning, reaching over 40% accuracy at 128 shots.
• It employs parallel symbolic transformation tasks across both genomic and language modalities to reveal modality-agnostic log-linear scaling in few-shot performance.
• The findings challenge the notion that in-context learning is unique to language, suggesting it arises from autoregressive prediction over structured, complex sequences.

Emergence of In-Context Learning in Genomic Next-Token Predictors

Introduction and Motivation

The phenomenon of in-context learning (ICL)—the ability of models to generalize and induce abstract patterns from a set of in-context demonstrations—has been framed as an emergent property of LLMs optimized for next-token prediction on natural language. The prevailing debate concerns whether ICL is unique to the distributional properties of human language or a byproduct of large-scale predictive modeling on any sufficiently complex and structured symbolic sequence. The paper "Genomic Next-Token Predictors are In-Context Learners" (2511.12797) directly addresses this by empirically analyzing ICL in large autoregressive models trained exclusively on genomic (DNA) sequences and comparing them with LLMs.

Experimental Framework and Task Design

To conduct an unbiased, cross-modality comparison of ICL, the authors design parallel symbolic reasoning tasks that abstract away linguistic or biological semantics. They focus on program synthesis tasks involving bitstring transformations encoded into both nucleotide (A/T/C/G) and digit-based representations, allowing for a direct assessment of sequence pattern induction in both genomics and LLMs.

Figure 1: Parallel symbolic reasoning tasks enable direct comparison of ICL in linguistic (digit-based) and genomic (nucleotide-based) models.

The key experimental desiderata are cross-domain comparability and the use of a minimal vocabulary. By using fixed-length bitstrings (8 bits), tasks become equally tractable in genomic and LLMs without domain-specific biases. The task suite comprises 100 deterministic transformation functions realized as compositions of fundamental bit-level primitives (e.g., bitwise NOT, shifts, rotations, masks). Prompts for both modalities are constructed as randomized mappings of bits to respective alphabets, removing memorization artefacts.

Model Families and Evaluation

The study benchmarks ICL on two compute-matched model families: (1) Qwen3 LLMs ranging from 0.6B to 14B parameters, and (2) Evo2 genomic models at 1B, 7B, and 40B. All are base models with no instruction tuning, ensuring the assessment isolates in-context generalization from pretraining or explicit meta-ICL. The evaluation protocol uses a Monte Carlo approach over transformation functions, context sets, and queries, reporting exact-match accuracy as the primary metric.

Empirical Results

Scaling Laws of In-Context Learning

Few-shot performance in both model families increases roughly log-linearly with the number of in-context demonstrations (shots), with Evo2 models displaying particularly monotonic, substantial gains. Notably, the largest Evo2 models reach over 40% accuracy at 128 shots, with Qwen3-14B approaching 35%, both far exceeding the mode guessing baseline.

Figure 2: Few-shot performance of Qwen3 and Evo2 models demonstrates consistent log-linear improvement with respect to $\log(\text{shots})$.

Critical claims substantiated include:

• Organically emergent ICL in genomic models: This is the first direct evidence that models pretrained on DNA (without meta-learning signals) display robust ICL scaling with shot number.
• Modality-agnostic scaling: The log-linear scaling of ICL is observed regardless of modality, contravening the hypothesis that language is uniquely structured for ICL (contradicts H₁, supports H₂ in the paper’s taxonomy).

Parameter Scaling and Task Complexity

Increasing parameter count in both families boosts ICL, but performance saturates in Evo2 beyond 7B, indicating capacity thresholds for these symbolic ICL tasks. At fixed high shot counts, Evo2 models outperform size-matched Qwen3 on the designed tasks.

Sensitivity to Task Complexity

To quantify task difficulty, the BitLoad metric is computed, measuring the number of input bits that affect the output. Evo2 models maintain higher accuracy on high-BitLoad (complex) transformations compared to Qwen3, suggesting more robust pattern induction over functions with deeper input-output dependencies.

Figure 3: BitLoad versus accuracy for Evo2 models over all shot numbers, illuminating model sensitivity to task complexity.

Role of Output Diversity

Analysis of output entropy (BitDiversity) further emphasizes that models struggle most when the required output distribution exhibits high entropy, i.e., with patterns requiring resolution of more ambiguous output structures.

Figure 4: BitDiversity versus accuracy for Evo2 at all shot numbers, showing performance degrades as output entropy increases.

Qualitative Behavioral Differentiation

Detailed per-task analyses reveal divergent inductive biases. Evo2 models excel on full bitstring manipulations (identity, NOT, complex masks), which correspond to high-BitLoad operations, whereas Qwen3 has relative strength on tasks involving positional shifts or aggregation (e.g., majority/minority functions) with lower or sparser dependencies. These behavioral asymmetries may reflect the regularities of their respective training corpora, with Evo2’s DNA source naturally favoring local structure, while text has richer examples of global and summary operations.

Theoretical and Practical Implications

The results challenge theoretical accounts that restrict ICL’s emergence to linguistic data, motivating a generalization toward compression-based or architecture-agnostic explanations. The empirical confirmation that hybrid genomic architectures (combining convolution and attention) yield similar ICL profiles to Transformers suggests that ICL is not contingent on architectural particulars but rather on autoregressive prediction over rich, compressible sequence distributions.

Practically, the presence of ICL in genomic models predicates future advancements in data-driven biological inference, prompt-based genomic feature engineering, and few-shot sequence annotation. Mechanistically, these findings argue for deeper, cross-modal interpretability research into the circuit-level origins of ICL.

Future Directions and Open Questions

The study raises several avenues for subsequent investigation:

• Wider cross-modality generalization: Can ICL be confirmed in other symbolic sequence domains, such as time series, system logs, or physics simulations, if next-token predictors are sufficiently scaled?
• Mechanistic interpretability: Dissection of Evo2's attention and convolutional patterns for ICL will elucidate universal versus domain-specific circuit mechanisms.
• Task design: Extending beyond bitstring tasks to more complex, structured challenges will probe the limits and transferability of emergent ICL.
• Tokenizer and pretraining effects: Decoupling the effects of restricted vocabularies and training distributions on symbolic pattern learning remains essential.

Conclusion

"Genomic Next-Token Predictors are In-Context Learners" (2511.12797) provides rigorous empirical evidence that autoregressive models trained on non-linguistic, biologically derived datasets can develop in-context learning capacities analogous to those of LLMs. The observed log-linear scaling of ICL, robustness to function complexity, and lack of reliance on linguistic structure reinforce a modality-agnostic, scale-centric view of in-context meta-learning. This work provides a critical foundation for unified theories of sequence modeling, contextual learning, and the mechanistic basis of adaptation in AI systems.