DNAMotifTokenizer: Domain-Informed DNA Tokenization

• DNAMotifTokenizer is a domain-informed tokenization strategy that integrates biologically validated DNA motifs with k-mer and monomer tokens.
• It employs a greedy tokenization algorithm and motif extraction from PWM sources to preserve sparse, high-value sequence features.
• By improving motif recovery in language models and DNA storage decoding, it outperforms traditional k-mer and BPE methods on genomic benchmarks.

DNAMotifTokenizer is a domain-informed tokenization strategy for DNA sequences, directly embedding functional motifs into the vocabulary alongside short k-mer and monomer tokens. It was designed to address the inadequacies of general-purpose tokenization approaches such as k-mer and Byte-Pair Encoding (BPE) in capturing sparse, biologically meaningful sequence features. DNAMotifTokenizer has been deployed both for language-model pretraining in genomics and as a critical component in end-to-end models for motif-based DNA storage signal decoding (Zhou et al., 18 Dec 2025, Agarwal et al., 2024).

1. Rationale and Motivations

DNA sequence language modeling frequently relies on breaking sequences into fixed-length tokens (k-mers) or learning a vocabulary of frequent substrings (BPE). However, functional DNA motifs—such as transcription factor binding sites—are characteristically sparse, of variable length (5–12 bp), and unevenly distributed, covering approximately 60% of the human genome (Zhou et al., 18 Dec 2025). K-mer tokenization treats all positions uniformly and may arbitrarily fragment rare motifs. BPE, trained on the frequency of substrings in the entire genome, can split or ignore functional motifs due to their low global frequency. This fragmentation increases cross-entropy for downstream masked language modeling (MLM) and impairs interpretability and accuracy.

Let $N$ denote genome length and $M$ the set of all motif occurrences:

$f_M = \frac{|M|}{N} \ll 1$

With such low frequency, motif-spanning tokens are under-sampled, leading LLMs to misrepresent motif regions, which are often those of highest biological interest (Zhou et al., 18 Dec 2025).

2. Methodological Framework

2.1 Motif Token Extraction

DNAMotifTokenizer curates its motif vocabulary from biologically validated sources (e.g., JASPAR 2024 vertebrate PWM collection, 879 motifs). PWMs are binarized with a probability threshold $\theta=0.5$ (positions with $p_{i,b}\le0.5$ set as a wildcard and trimmed at boundaries). Consensus sequences are constructed by taking the highest $p_{i,b}$ at each retained position.

2.2 Vocabulary Construction

The vocabulary comprises:

• All motifs (827) and their reverse complements (827)
• All 3-mers ($4^3=64$) for non-motif sequence spans
• Mononucleotide tokens $\{A, C, G, T, N\}$ (5)
• Five special tokens ([PAD], [UNK], [CLS], [SEP], [MASK])

This aggregates to $|V|=1{,}728$, although ~900 are routinely active (Zhou et al., 18 Dec 2025).

2.3 Greedy Tokenization Algorithm

For a DNA sequence $s[1..n]$, greedy tokenization proceeds as follows:

• For each position and offset $o\in\{0,1,2\}$, scan a Trie (motif search structure) for motif matches within windows of length $4..12$.
• If motifs match, prioritize selection to maximize motif coverage; if not, fallback to 3-mer or 1-mer as appropriate.
• Advance by the corresponding token length.

The greedy objective is to maximize the number of motif tokens:

$$\max_{\mathcal{T}} \sum_{j=1}^K \mathbf{1}\bigl[t_j\in V_{\mathrm{motif}}\bigr]$$

Fallback mechanisms ensure sequence coverage with no gap, correlating to standard subword vocabulary practices. The MLM objective and network architecture remain conventional; only the input embedding and segmentation differ (Zhou et al., 18 Dec 2025).

3. Integration in Machine-Learning Architectures

The DNAMotifTokenizer concept generalizes beyond sequence tokenization for LLMs, extending to the direct output of deep neural networks from sequencing signals in DNA storage applications. In Motif Caller (Agarwal et al., 2024), the input is a normalized, windowed 1D sequencing "squiggle" (ionic current measurements) which is processed into overlapping frames. Each frame passes through convolutional layers, followed by a stack of bidirectional GRUs, yielding per-window softmax probabilities over the motif vocabulary plus CTC blank.

Formally, for input $s$ and windows $\{x_1, ..., x_{T'}\}$:

• Feature extraction: $a^{(t)} = f_{\text{conv}}(x_t)$
• Sequence modeling: $h^{(t)} = f_{\text{rnn}}(a^{(1:t)})$
• Token prediction: $p^{(t)} = \mathrm{Softmax}(W_o h^{(t)} + b_o)$

Alignment is determined via Connectionist Temporal Classification (CTC), collapsing repeated and blank tokens to yield a predicted motif sequence. The full framework enables end-to-end motif-level decoding, making classical basecalling and motif search unnecessary for motif-based storage formats (Agarwal et al., 2024).

4. Empirical Performance and Benchmarks

Extensive downstream evaluations demonstrate the impact of domain-informed tokenizers:

• Genome Understanding Evaluation (GUE): Mean MCC 0.6815$\pm$0.123 for DNAMotifTokenizer vs 0.6730 for BPE(1024) and 0.6617 for best k-mer tokenizer.
• SCREEN (cCRE annotation): MCC 0.8850 for DNAMotifTokenizer, exceeding BPE(1024) (0.8781) and approaching k-mer methods (0.9159).
• DART-EVAL (synthetic benchmarks): Task-specific gains are less marked, reflecting the underrepresentation of natural motif distribution.
• Motif Caller in DNA storage: Direct motif-token prediction increased per-read motif yield from 8.8% to 15.9% in empirical data; synthetic results showed five-fold improvement in identity (from 26.1% for Motif Search to 78.8% for Motif Caller). Reads-per-block required for 100% motif recovery halved from 66 to 37 (Agarwal et al., 2024, Zhou et al., 18 Dec 2025).

Statistical validation with Wilcoxon tests confirmed the significance of gains on GUE/SCREEN and NT when comparing DNAMotifTokenizer with BPE(1024) (p < 0.01).

5. Interpretability and Biological Insight

DNAMotifTokenizer facilitates interpretability:

• In single-nucleus ATAC-seq from human brain cell types, motif tokens accounted for outsize attribution scores by Integrated Gradients.
• Motif tokens showed enrichment patterns concordant with known biological regulation (e.g., WT1, EGR, KLF, SOX) in cell-type-specific analyses (Zhou et al., 18 Dec 2025).
• The shared and cell-type-specific motif usage reflects biologically meaningful regulatory programs.

The explicit presence of functional motifs in the vocabulary allows model attributions to be traced to discrete regulatory elements, supporting hypothesis generation and mechanistic interpretation.

6. Implementation and Best Practices

Key recommendations for deploying DNAMotifTokenizer:

1. Curate high-quality motif libraries (e.g., JASPAR), binarized and trimmed by sensible PWM thresholds (e.g., $p\geq0.5$).
2. Build vocabularies combining motif tokens, short subwords (3-mers), and monomers, targeting total size $\lesssim 1,000$ to minimize redundancy.
3. Apply greedy tokenization using motif-aware windowing and local offsets.
4. For DNA storage, set window length and stride on empirical dwell-time; align model output to motif sequences via CTC.
5. Pretrain with masked token objectives under controlled compute budgets; validate both accuracy and interpretability using attribution techniques.

For computational efficiency with large motif libraries ($>100$ motifs), options such as hierarchical softmax or adaptive output sampling can be justified. In motif-based DNA storage, model throughput, sequence normalization, and overlap window tuning are critical for practical deployment (Agarwal et al., 2024, Zhou et al., 18 Dec 2025).

7. Comparative Analysis and Domain Significance

DNAMotifTokenizer consistently delivers improved accuracy and interpretability on genomic tasks relative to k-mer and BPE alternatives, especially for functional-annotation and regulatory genomics benchmarks. Gains are most pronounced when rare, high-information-content motifs are critical for task performance. The strategy establishes a paradigm in which biologically meaningful elements define the boundaries of symbolic decomposition for both language modeling and high-throughput signal decoding, advancing the field toward more interpretable and generalizable DNA representations (Zhou et al., 18 Dec 2025, Agarwal et al., 2024).