GeneTuring Benchmark: Genomics QA Evaluation
- GeneTuring Benchmark is a genomics QA framework that assesses LLMs using NCBI tools on structured tasks like nomenclature, location, functional analysis, and sequence alignment.
- It organizes 450 questions into nine categories, emphasizing exact-match accuracy, partial credit scoring, and error analysis for API grounding and retrieval precision.
- The benchmark underpins extensions like GeneHop and protocols like GeneGPT, showcasing evolving multi-hop query decomposition and rigorous performance evaluation.
Searching arXiv for GeneTuring benchmark and closely related genomics QA papers. GeneTuring is a genomics-focused question answering benchmark introduced by Hou and Ji for evaluating LLMs on precise biomedical information access through the National Center for Biotechnology Information (NCBI) suite of databases and tools. In its original description, it is presented as a benchmark of single-hop genomics QA organized into four modules with 50 question–answer pairs per task, and it is paired in the same work with GeneHop, a complementary multi-hop dataset built upon GeneTuring (Jin et al., 2023). The benchmark is notable for emphasizing database-grounded retrieval, identifier normalization, locus resolution, functional association, and sequence alignment rather than free-form biomedical narration, which makes it particularly sensitive to hallucination and tool-use capability.
1. Definition, scope, and corpus organization
The original GeneTuring description evaluates LLMs on genomics-focused question answering using the NCBI suite of databases and tools. It groups the benchmark into four modules: Nomenclature, Genomic Location, Functional Analysis, and Sequence Alignment, with 50 question–answer pairs per task (Jin et al., 2023). These modules target short-answer problems such as official gene-symbol recovery, chromosome identification, disease-linked gene retrieval, and BLAST-mediated sequence mapping.
The source description contains an internal counting discrepancy: it states that GeneTuring “consists of eight single-hop tasks,” but the accompanying task list enumerates nine task definitions. Later works consistently evaluate “nine tasks from the GeneTuring benchmark” or describe GeneTuring as “450 total questions, evenly divided into nine categories,” which suggests that the nine-task interpretation became the operational convention in subsequent benchmarking practice (Abedini et al., 15 Jan 2026, Hong et al., 23 Sep 2025).
A compact summary of the original task grouping is as follows.
| Module | Tasks listed in the original description |
|---|---|
| Nomenclature | Gene alias; Gene name conversion |
| Genomic Location | Gene SNP association; Gene location; SNP location |
| Functional Analysis | Gene disease association; Protein-coding genes |
| Sequence Alignment | DNA→human genome; DNA→multiple species |
This structure makes the benchmark unusual among biomedical QA datasets because the answer space is dominated by canonical symbols, coordinates, binary labels, and database-derived lists rather than explanatory prose. A plausible implication is that GeneTuring functions as a stress test for API grounding and schema-aware extraction rather than for generic language modeling alone.
2. Task semantics and the GeneHop extension
In the original specification, the Nomenclature module contains two tasks. Gene alias asks for the official symbol given a non-official gene synonym. Gene name conversion asks for conversion from an Ensembl gene ID to its official symbol. The Genomic Location module contains Gene SNP association, which returns the associated gene for a given SNP; Gene location, which returns the chromosome of a given gene; and SNP location, which returns the chromosome of a given SNP. The Functional Analysis module contains Gene disease association, which lists genes implicated in a given disease and is measured by recall, and Protein-coding genes, which decides whether a gene is protein-coding, with outputs TRUE or NA. The Sequence Alignment module contains DNA→human genome, which maps a DNA sequence to human chromosomal coordinates, and DNA→multiple species, which maps a DNA sequence to its species of origin (Jin et al., 2023).
The same work introduces GeneHop as a multi-hop extension motivated by the observation that real-world genomics questions often require composing multiple database calls. GeneHop contains three tasks, each with 50 instances: SNP→gene→function, phrased as “What is the function of the gene associated with SNP rs…?”; Disease→genes→locations, phrased as “List chromosome locations of genes related to Disease X.”; and Sequence→gene→aliases, phrased as “What are the aliases of the gene that contains this sequence…?” (Jin et al., 2023).
Subsequent papers preserve the use of GeneTuring as a genomics QA substrate but sometimes alter task semantics. One reproduction evaluates nine tasks named GeneAlias, NameConversion, SNPAssociation, GeneLocation, SNPLocation, DiseaseAssociation, ProteinCodingGenes, DNA→Human, and DNA→Species; however, some definitions differ from the original benchmark. For example, that work defines GeneAlias as retrieving all known aliases for a gene official symbol, DiseaseAssociation as retrieving diseases linked to a gene, and ProteinCodingGenes as identifying genes that code for a given protein (Abedini et al., 15 Jan 2026). Another later study summarizes GeneTuring as spanning nine categories over nomenclature, genomic location, functional analysis, and sequence alignment, again under a nine-task formulation (Hong et al., 23 Sep 2025). This suggests that “GeneTuring” can denote both the original benchmark specification and a family of closely related task suites used in later agentic-genomics evaluations.
3. Evaluation protocol and scoring conventions
The original GeneTuring evaluation uses exact-match accuracy for all single-hop tasks except gene disease association, recall for gene disease association and multi-hop tasks, and partial credit of 0.5 for DNA→human genome when only the chromosome matches (Jin et al., 2023). For retrieval-like tasks, the original paper defines the score
This metric design reflects the heterogeneity of the tasks. Identifier conversion and chromosome lookup are naturally exact-match problems, whereas association tasks involve set-valued answers for which recall or is more appropriate. The partial-credit rule for DNA→human genome acknowledges that sequence alignment errors can be partially correct at the chromosome level even when coordinate endpoints are mismatched.
Later evaluations adopt different scoring normalizations. One reproduction normalizes all task-specific scores to , defines overall performance as the macro-average of the nine task scores, and retains partial credit of 0.5 for the human alignment task when the chromosome is correct but positions differ (Abedini et al., 15 Jan 2026). A still later study reports only per-task and overall accuracy, states that no or other metrics were used, and explicitly removes the partial “0.5” scoring for human-genome alignment by fixing a reference genome, GRCh38, and enforcing exact coordinate matches (Hong et al., 23 Sep 2025).
These variants matter for interpreting leaderboard claims. This suggests that direct numerical comparison across GeneTuring papers requires caution, because later works change not only model architecture but also task definitions, answer formats, and scoring rules.
4. API-grounded benchmarking and the GeneGPT protocol
GeneTuring became closely associated with GeneGPT, which augments Codex with NCBI Web APIs via in-context prompting plus an augmented decoding loop (Jin et al., 2023). In the original setup, the prompt contains an instruction header, documentation snippets for NCBI E-utils and the BLAST URL API, four demonstrations showing a benchmark question, an explicit API URL embedded in brackets, the pasted JSON/XML response, and a final “Answer: …” line, followed by the test question. The instruction header is explicitly: “Your task is to use NCBI APIs to answer genomic questions.” The documented endpoints include E-utils base URLs for esearch, efetch, and esummary over gene, snp, and omim, as well as BLAST Put and Get calls (Jin et al., 2023).
The augmented decoding loop detects two control tokens: “→” to trigger an API call and “\n\n” to mark answer completion. The algorithmic sketch is: build the prompt from header, documentation, demonstrations, and test question; let the model generate the next token; when the token is “→”, extract the last URL in the prompt, execute an HTTP GET, and append the result to the prompt; when the prompt ends with an answer followed by a double newline, stop and extract the answer. Temperature is fixed at 0 for deterministic behavior (Jin et al., 2023).
For GeneHop, the protocol appends the instruction “Let’s decompose the question to sub-questions and solve them step by step.” GeneGPT then generates a chain-of-thought in which each sub-question is answered via an API call, the response is appended, and the next sub-question conditions on prior answers. The paper reports that this demonstrates capability to generalize to 3+ sub-questions and 4+ API calls, beyond the single-hop demonstrations (Jin et al., 2023).
Concrete examples illustrate the benchmark’s operational character. In a single-hop gene-alias example, the question “What is the official gene symbol of LMP10?” is answered by first issuing an esearch query over the gene database, then an efetch call over candidate gene IDs, after which GeneGPT extracts the official symbol PSMB10. In a multi-hop example, the question about the function of the gene associated with SNP rs1241371358 is decomposed into a query that maps the SNP to gene LRRC23 via esummary?db=snp, followed by a gene summary lookup via esummary?db=gene. In a BLAST-based alignment example, a DNA sequence is submitted with Blast.cgi?CMD=Put, the result is retrieved with Blast.cgi?CMD=Get&FORMAT_TYPE=Text, and the answer is extracted as chr15:91950805–91950932 (Jin et al., 2023).
5. Reported performance and benchmark evolution
On the original GeneTuring evaluation, GeneGPT achieves state-of-the-art performance on eight tasks in the GeneTuring benchmark with an average score of 0.83, and the detailed table reports overall averages of 0.80 for GeneGPT-full and 0.83 for GeneGPT-slim. The same table reports 0.44 for the new Bing, 0.08 for BioMedLM, 0.04 for BioGPT, 0.16 for GPT-3, and 0.12 for ChatGPT (Jin et al., 2023). GeneGPT-slim uses only two demonstrations, Dm.1 alias and Dm.4 BLAST, and matches or exceeds full demo performance, which the paper interprets as evidence of strong cross-task generalizability.
At the task level in the original paper, GeneGPT-full reaches 1.00 on Name conversion, 1.00 on SNP association, 1.00 on SNP location, 0.76 on Gene–disease association, 0.76 on Protein-coding gene, 0.44 on DNA→human genome, and 0.86 on DNA→multiple species; GeneGPT-slim reaches 1.00 on Name conversion, 1.00 on SNP association, 0.98 on SNP location, 1.00 on Protein-coding gene, 0.44 on DNA→human genome, and 0.88 on DNA→multiple species (Jin et al., 2023). On GeneHop multi-hop QA, the reported averages are 0.24 for New Bing and 0.50 for GeneGPT.
Later benchmark-oriented systems report substantially higher scores under their own GeneTuring formulations. GenomAgent evaluates GeneGPT and GenomAgent on nine tasks from the GeneTuring benchmark, reporting overall macro-averages of 0.83 for GeneGPT-Slim and 0.93 for GenomAgent, corresponding to an approximately 12.05% improvement under that paper’s calculation (Abedini et al., 15 Jan 2026). The Nano Bio-Agent framework reports that its best model-agent combination achieves 98% accuracy on the GeneTuring benchmark, with representative 3–10B models scoring from 85.9% to 95.7% overall and an overall summary range of 86–97% (Hong et al., 23 Sep 2025).
A concise trajectory is shown below.
| System or setting | Reported overall result |
|---|---|
| GeneGPT-full | 0.80 |
| GeneGPT-slim | 0.83 |
| New Bing on GeneHop | 0.24 |
| GeneGPT on GeneHop | 0.50 |
| GenomAgent | 0.93 |
| NBA best model-agent combination | 98% accuracy |
Because these results arise from different task formulations and scoring conventions, they should be read as evidence of benchmark influence and methodological evolution rather than as a single homogeneous leaderboard.
6. Error structure, limitations, and interpretive issues
The original GeneGPT analysis manually classifies errors into five types: E1, wrong API; E2, right API but wrong arguments; E3, fetched data but failed to extract the answer; E4, API returns no relevant entry; and O, other, mostly BLAST positional mismatches (Jin et al., 2023). The task-wise counts are diagnostically uneven. Gene disease association shows 15 E1 errors, indicating substantial difficulty in choosing the correct database. Sequence gene alias in GeneHop shows 30 E2 and 8 E3 errors, indicating that multi-hop parameter propagation and argument formatting are fragile. SNP gene function shows 29 E3 errors, indicating that answer extraction rather than retrieval dominates that failure mode. DNA→human genome shows 42 errors in category O and 7 in E3, reflecting the special difficulty of BLAST output interpretation (Jin et al., 2023).
The original paper draws three concrete directions for future work from these counts: improve argument parsing and answer extraction with structured post-processing, handle unindexed items by graceful fallbacks or alternative data sources, and refine BLAST parsing to map coordinates under a consistent reference (Jin et al., 2023). Later work echoes these concerns. The GenomAgent paper notes that attribution of gains remains unclear because it lacks an ablation suite isolating each agent’s contribution, and it argues for broader genomics corpora and real-world workflows beyond GeneTuring (Abedini et al., 15 Jan 2026). The NBA study reports that sequence alignment remains the most challenging category, with errors clustering around parsing multi-hit BLAST results for non-human species, incorrect coordinate rounding or failure to specify reference genome version, and rare mis-extraction of SNP alleles; it also reports that failures in the Task Classification step are below 2% above 3B parameters (Hong et al., 23 Sep 2025).
One recurrent source of confusion is whether GeneTuring should be regarded as a fixed benchmark specification. The original paper defines tasks in one way and couples them to NCBI-centric QA and GeneHop (Jin et al., 2023), whereas later works sometimes reinterpret the task inventory, alter input–output semantics, introduce additional sources such as HGNC, UCSC, or AlphaGenome, and modify scoring procedures (Abedini et al., 15 Jan 2026, Hong et al., 23 Sep 2025). This suggests that GeneTuring now functions both as a named benchmark and as a benchmark family for evaluating grounded genomics QA systems. Its enduring methodological role is to test whether model outputs can be anchored to authoritative biomedical databases, whether multi-step retrieval chains can be executed reliably, and whether exact-answer genomics QA can be evaluated with enough granularity to expose API selection, parameterization, extraction, and alignment errors.
