gtars-tokenizers: Genomic Interval Tokenization Library

• gtars-tokenizers is a genomic interval tokenizer that maps arbitrary input intervals to a fixed vocabulary, enabling standardized ML processing.
• It is implemented in Rust for speed and memory efficiency and offers multi-language support via Python, R, CLI, and WebAssembly for broad integration.
• Benchmark results show up to 2–3x faster performance than traditional tools on large datasets, reducing preprocessing latency in ML pipelines.

gtars-tokenizers is a specialized library for mapping genomic interval data into a fixed discrete vocabulary so that interval-based genomics data can be used in modern machine learning pipelines in a manner analogous to text tokenization in natural language processing. It takes arbitrary input intervals and assigns them to a predefined genomic universe—a consensus set of genomic regions that acts as the model vocabulary—thereby enabling genomic intervals to be represented as token IDs and processed with standard token-based ML workflows. The package is implemented in Rust for speed and memory efficiency, exposed through Python, R, CLI, WebAssembly, and Rust interfaces, and is designed to integrate with Hugging Face-style tokenization workflows as well as PyTorch and TensorFlow (LeRoy et al., 3 Nov 2025).

1. Problem setting and conceptual role

gtars-tokenizers addresses a central obstacle in genomic machine learning: genomic interval datasets are heterogeneous because each experiment typically defines its own regions of interest. The paper frames this with assays such as ATAC-seq, ChIP-seq, and Hi-C, whose outputs are commonly summarized as genomic intervals. Even when two datasets measure similar biology, they may produce different coordinate sets, which makes them difficult to compare, combine, and feed into models that require a consistent, discrete vocabulary (LeRoy et al., 3 Nov 2025).

The library treats this as a tokenization problem. In the same way that NLP systems convert raw text strings into tokens from a fixed vocabulary, gtars-tokenizers converts intervals into token IDs associated with a predefined vocabulary of genomic regions. This supports consistent representation across experiments, feature alignment across datasets, direct use of embedding layers, compatibility with transformers and other token-based architectures, and less ad hoc preprocessing. The paper explicitly connects this design to work on Atacformer, where interval tokenization was required to make deep learning practical (LeRoy et al., 3 Nov 2025).

A common misunderstanding would be to view the package as only a general-purpose interval-overlap utility. The paper instead emphasizes that it is specifically an ML-aware tokenizer: it is intended to produce Hugging Face-style token outputs, especially input_ids, so that interval data can flow into modern training and inference stacks with minimal custom glue (LeRoy et al., 3 Nov 2025).

2. The genomic universe as vocabulary

The target vocabulary is described as a shared vocabulary, consensus set of genomic intervals, or predefined universe of regions. Conceptually, this universe plays the role of a text tokenizer’s vocabulary. Each region in the universe becomes a discrete token, and new query intervals are mapped onto those predefined regions through interval overlap (LeRoy et al., 3 Nov 2025).

The universe provides a fixed token space, a reproducible basis for representing many datasets, a common interface between preprocessing and downstream models, and interoperability between experiments that did not originally share coordinates. In the paper, the universe is assumed to be externally defined rather than learned inside the library. A standard instantiation method is loading a BED file:

tokenizer = Tokenizer.from_bed("path/to/file.bed")

This BED file defines the universe, and each row corresponds to a region that can become a token (LeRoy et al., 3 Nov 2025).

The paper also makes clear that a predefined universe is both an enabling abstraction and a modeling commitment. Its implied advantages include model compatibility, reproducibility, interoperability, and scalability. Its implied costs include dependence on the chosen universe, potential information loss when heterogeneous intervals are discretized into predefined regions, boundary effects when intervals overlap multiple universe regions, assembly dependence, and the fact that inter-dataset comparability depends on a shared vocabulary choice rather than on raw coordinates alone (LeRoy et al., 3 Nov 2025).

3. Architecture, interfaces, and overlap backends

A defining architectural feature of gtars-tokenizers is a single Rust core that is exposed through multiple environments. The paper lists the following interfaces:

Interface	Role
Rust crate	Native systems-level implementation
Python	ML and scientific workflows
R	Bioinformatics workflows
CLI	Shell-based preprocessing
WebAssembly	Browser and web deployment

This architecture is presented as a way to avoid duplicated implementations while preserving consistent semantics and performance across user communities (LeRoy et al., 3 Nov 2025).

The package is described as compatible with the Hugging Face tokenizers API and as a near-drop-in replacement for standard Hugging Face tokenizers. The intended outputs follow familiar structures such as input_ids, allowing interoperability with downstream ecosystems including PyTorch, TensorFlow, PyTorch Lightning, AllenNLP, Evaluate, PEFT, and Weights & Biases (LeRoy et al., 3 Nov 2025).

At the overlap-computation level, gtars-tokenizers implements two Rust-based backends: BITS via gtars/bits and AIList via gtars/alist. The manuscript identifies them as interval-overlap methods and presents them as the core engines used to compare query intervals against the genomic universe. However, it does not provide pseudocode, mathematical overlap equations, or asymptotic complexity formulas for either method in the text supplied. This absence is itself notable because the paper’s emphasis is on practical tokenization infrastructure and ML integration rather than a formal algorithmic comparison between overlap data structures (LeRoy et al., 3 Nov 2025).

4. Tokenization workflow and API surface

The intended workflow begins with tokenizer initialization from a predefined universe, typically in BED format. Query intervals are then supplied as tuples of chromosome, start, and end coordinates. The paper’s example is:

query_intervals = [("chr1", 100, 200), ("chr2", 300, 400)]

The core operation is overlap computation between these query intervals and the indexed universe. The tokenizer then assigns token identities corresponding to overlapping universe regions and returns an object containing at least Hugging Face-style input_ids (LeRoy et al., 3 Nov 2025).

The paper’s stepwise operational picture is as follows. First, a BED-defined universe is loaded. Second, query intervals are provided. Third, overlap is computed through BITS or AIList. Fourth, overlapping universe intervals are mapped to token IDs. Fifth, those IDs are returned in tokenizer-like output structures. This design allows genomic intervals to serve as inputs to embedding layers, transformers, and other token-centric downstream modules (LeRoy et al., 3 Nov 2025).

The Python usage pattern shown in the manuscript is:

import torch import gtars.tokenizers as Tokenizer tokenizer = Tokenizer.from_bed("path/to/file.bed") network = torch.nn.Embedding(tokenizer.vocab_size, 64) query_intervals = [("chr1", 100, 200), ("chr2", 300, 400)] tokens = tokenizer.tokenize(query_intervals)["input_ids"] out = network(torch.tensor(input_ids))

The paper notes a likely variable-name mismatch between tokens and input_ids, but the intended flow is unambiguous: build the tokenizer from a BED vocabulary, tokenize intervals into input_ids, and feed those IDs into an embedding layer (LeRoy et al., 3 Nov 2025).

Several assumptions are implicit in this workflow. Intervals are represented by chromosome, start, and end; coordinates must be compatible with the universe; chromosome naming should match the universe convention; and the same genome assembly should be used for vocabulary and query intervals. Because overlap is coordinate-based, mismatched naming conventions or assemblies would compromise tokenization fidelity (LeRoy et al., 3 Nov 2025).

5. Performance characteristics and benchmarking

The paper benchmarks gtars-tokenizers against bedtools, bedops, and bedtk, describing those tools as general-purpose genomic interval arithmetic systems rather than ML-specific tokenizers. Benchmark workloads span 10K, 100K, and 1M query regions, and the paper explicitly discusses universes with more than 1 million intervals as a realistic scale for genomic interval machine learning (LeRoy et al., 3 Nov 2025).

Its clearest quantitative claim is that for large universes with more than 1 million intervals, gtars-tokenizers is around 2–3x faster than bedtools and bedops, while being comparable to bedtk. The paper states that this pattern holds across all three query sizes. It presents this as evidence that the library is practical for large-scale preprocessing, repeated tokenization over many samples, and training or inference settings in which preprocessing latency matters (LeRoy et al., 3 Nov 2025).

The manuscript repeatedly emphasizes memory efficiency, especially in connection with the Rust implementation and overlap engines, but it does not provide explicit memory benchmark numbers in the supplied text. Likewise, it does not report a numerical head-to-head comparison between BITS and AIList. The benchmark conclusions are therefore strongest at the package level rather than at the level of internal backend selection (LeRoy et al., 3 Nov 2025).

From an ML-systems perspective, the benchmark results support a specific claim: tokenization of genomic intervals can remain in-memory and library-native rather than being delegated to I/O-heavy shell pipelines. A plausible implication is that this reduces friction when interval tokenization is embedded inside training pipelines that expect Python-native or Hugging Face-style preprocessing components.

6. Limitations, assumptions, and broader research context

The principal assumption of gtars-tokenizers is dependence on a predefined universe. If the universe is too narrow or otherwise poorly chosen, biologically relevant intervals may be represented suboptimally. The paper also identifies unresolved or underspecified cases: ambiguous overlaps when a query interval matches multiple universe regions, no-overlap cases, chromosome naming incompatibilities, genome assembly mismatches, and the absence of discussion of strand-aware tokenization. BED is the only explicit file format shown in the manuscript (LeRoy et al., 3 Nov 2025).

These constraints delimit the meaning of tokenization in this setting. The library standardizes heterogeneous interval data into a shared discrete feature space, but that standardization is not lossless in the strong sense of preserving every property of raw interval geometry. Instead, it creates a consistent token space suitable for embeddings, batching, transfer learning, and model interoperability (LeRoy et al., 3 Nov 2025).

In a broader research context, gtars-tokenizers belongs to a larger shift in which tokenizers are treated as modality-specific interfaces rather than incidental preprocessing code. Work on graph tokenization similarly formulates a tokenizer as the layer that converts non-sequential structures into discrete representations consumable by standard Transformers (Guo et al., 11 Mar 2026). Work on multilingual language modeling argues that tokenization and especially pre-tokenization affect representational fairness across scripts, indicating that tokenizer design can materially shape what a model can efficiently represent (Velayuthan et al., 2024). This suggests that gtars-tokenizers is part of a wider movement toward treating tokenizer design as an independent systems problem.

Within genomics, the package’s broader significance lies in moving interval-based analysis away from bespoke feature engineering and toward general-purpose token-based ML representations. Its central contribution is not a new predictive architecture, but a standardized mapping from genomic coordinates to input_ids, implemented as a fast Rust library with multi-language bindings and direct compatibility with mainstream ML software ecosystems (LeRoy et al., 3 Nov 2025).