DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA (2412.05430v1)

Abstract: Recent advances in self-supervised models for natural language, vision, and protein sequences have inspired the development of large genomic DNA LLMs (DNALMs). These models aim to learn generalizable representations of diverse DNA elements, potentially enabling various genomic prediction, interpretation and design tasks. Despite their potential, existing benchmarks do not adequately assess the capabilities of DNALMs on key downstream applications involving an important class of non-coding DNA elements critical for regulating gene activity. In this study, we introduce DART-Eval, a suite of representative benchmarks specifically focused on regulatory DNA to evaluate model performance across zero-shot, probed, and fine-tuned scenarios against contemporary ab initio models as baselines. Our benchmarks target biologically meaningful downstream tasks such as functional sequence feature discovery, predicting cell-type specific regulatory activity, and counterfactual prediction of the impacts of genetic variants. We find that current DNALMs exhibit inconsistent performance and do not offer compelling gains over alternative baseline models for most tasks, while requiring significantly more computational resources. We discuss potentially promising modeling, data curation, and evaluation strategies for the next generation of DNALMs. Our code is available at https://github.com/kundajelab/DART-Eval.

• The paper introduces DART-Eval, a benchmark that rigorously tests DNA language models using zero-shot, probed, and fine-tuning across five key regulatory tasks.
• The study reveals that simpler, integration-free models often outperform DNALMs, especially in counterfactual genetic variant predictions.
• The paper highlights that while DNALMs perform well on basic regulatory detection, their efficacy declines on complex tasks, urging improvements in fine-tuning and modular training.

Evaluation of DNALMs Using the DART-Eval Benchmark

The paper presents a rigorous framework called DART-Eval, devised to evaluate the capabilities of DNA LLMs (DNALMs) with a keen focus on non-coding regulatory elements crucial for gene expression. As advancements in self-supervised models in computing, such as LLMs for NLP, inspire similar strides in genomics, the research aims to establish a benchmark for examining DNALMs' performance across several biologically relevant tasks.

Objectives and Methodology

DART-Eval addresses the performance of DNALMs like Caduceus, DNABERT-2, GENA-LM, HyenaDNA, Mistral-DNA, and Nucleotide Transformer, comparing them against traditional, supervised "ab initio" models. These DNALMs are assessed in three settings: zero-shot, probed, and fine-tuned. The suite comprises five tasks: regulatory sequence detection, transcription factor motif sensitivity, cell-type-specific feature learning, quantitative prediction of regulatory activity, and counterfactual prediction of genetic variants.

Key Findings

A pivotal highlight of the paper is the observation that simpler integration-free models surpass DNALM-based approaches in several benchmarks. Specifically, baseline models like ChromBPNet outperformed DNALMs in counterfactual predictions, a critical task in understanding genetic variant impacts. Additionally, the evaluation demonstrated that while DNALMs exhibit adequate performance on more straightforward tasks (distinguishing regulatory from non-regulatory DNA), their efficacy declines on complex tasks.

1. Regulatory DNA Discrimination: All evaluated DNALMs in a zero-shot setting prioritized regulatory elements over compositionally matched controls, albeit with varied accuracy scores. Fine-tuning showed slight improvements, yet ab initio models maintained competitive performance.
2. Transcription Factor Motif Sensitivity: DNALMs demonstrated capacity in identifying TF motifs, though with notable variability between instances. Embedding-based results were less reliable, reinforcing that leverage of full model expressivity is crucial for intricate identification tasks.
3. Cell-Type Specific Differential Activity: The analysis exposed the inadequacy of DNALM embeddings to offer insightful cell-type distinction without fine-tuning. Interestingly, supervised methods like fine-tuning offered enhancements over probing but failed to surpass the baseline CNN models consistently.
4. Quantitative Activity Prediction: Fine-tuned DNALMs paralleled ChromBPNet in regression tasks, though not universally outperforming it. The results underscored the challenge of predicting precise activity levels from localized sequences without extensive fine-tuning.
5. Variant Effect Prediction: Nucleotide Transformer excelled in zero-shot evaluations, but fine-tuned models lagged the ChromBPNet standard in predicting allelic effects, indicating a need for incorporating more sophisticated evaluation tasks.

Implications and Future Directions

The research underscores the need for continued advancements in data annotation and sequence modeling to improve DNALM outcomes, particularly for predicting distal interactions and more nuanced regulatory functions. The observed limitations call for modular training methods to enhance fine-tuning efficiency and the development of context-sensitive models incorporating evolutionary principles.

Given DART-Eval's adaptability, future versions could integrate longer-context evaluations and more intricate functional element representation beyond focal points in regulatory syntax. A broader incorporation of unexplored species might illuminate conserved patterns transcendental to academic and practical biotechnological applications.

In conclusion, while DNALMs hold significant promise, especially in leveraging genomic data, this paper highlights vital areas for improvement and optimization, suggesting potential pathways towards a comprehensive understanding and leveraging of DNA's non-coding functional anatomy.