BioinfoMCP Benchmark for Agentic Bioinformatics

• The paper establishes a quantitative framework that evaluates MCP-converted bioinformatics tools and workflows, reporting 95.6% tool reliability and 94.7% pipeline success.
• The benchmark tests 38 bioinformatics tools and 5 complex workflows across 3 agent hosts to ensure cross-host robustness and execution fidelity.
• The system leverages Docker containerization and structured MCP protocols to verify that agent-driven pipelines produce expected outputs under realistic conditions.

Searching arXiv for the primary BioinfoMCP paper and nearby benchmark papers to ground the article in current literature. BioinfoMCP Benchmark is the validation component of BioinfoMCP, a unified platform for enabling Model Context Protocol (MCP) interfaces in agentic bioinformatics. Whereas BioinfoMCP Converter automates the transformation of legacy bioinformatics command-line tools into MCP-compliant servers, the Benchmark provides a systematic, quantitative framework for validating that the converted tools and the agentic pipelines they compose actually work when called by large-language-model-driven AI agents. In the reported implementation, it evaluates 38 MCP-converted bioinformatics tools and five representative end-to-end workflows across three MCP Hosts, with an overall tool reliability of approximately 95.6% and a quoted complex-workflow success rate of 94.7% (Widjaja et al., 2 Oct 2025).

1. Position within the BioinfoMCP platform

BioinfoMCP Benchmark is one of two pillars of the BioinfoMCP platform, the other being BioinfoMCP Converter. The Converter is designed to automatically generate robust MCP servers from tool documentation using LLMs, while the Benchmark is designed to test whether those servers function correctly in realistic agent-mediated execution settings. This division of labor is central to the platform’s logic: server generation and server validation are treated as separate problems rather than collapsed into a single conversion pipeline (Widjaja et al., 2 Oct 2025).

The benchmark is therefore not a task benchmark in the narrow sense of scoring biological answers alone. Its primary object of evaluation is the operational reliability of MCP-converted bioinformatics tools and of the multi-step workflows built from them. The paper characterizes this as a way to validate that converted tools and agentic pipelines “actually work” when invoked through MCP, which places emphasis on execution fidelity, interoperable interfaces, and cross-host robustness rather than only on offline schema conformance.

A plausible implication is that BioinfoMCP Benchmark formalizes a systems-level notion of correctness for agentic bioinformatics. In this framing, an MCP server is useful only if an agent can discover it, call it through JSON, receive a valid response, and obtain the expected output artifacts under realistic host constraints.

2. Architecture and execution model

The Benchmark component comprises four layers: a test-case repository, an AI-agent host layer, a benchmarking orchestrator, and a report generator (Widjaja et al., 2 Oct 2025).

The test-case repository contains 38 MCP-converted bioinformatics tools. Each tool is packaged in its own Docker container and exposes a standardized FastMCP 2.0 server interface. This packaging decision makes the benchmark simultaneously an interface benchmark and a deployment benchmark, since containerization, MCP exposure, and tool invocation are evaluated together rather than in isolation.

The AI-agent host layer includes three representative MCP Hosts. These are a locally deployed Python-based AI agent (LAI) running on a 64 GB-RAM remote machine, Anthropic Claude Desktop (CD) on a 16 GB local machine, and Cursor (CR) on a 16 GB local machine. The use of three hosts is significant because it turns the benchmark into a cross-host robustness test instead of a single-agent demonstration.

The benchmarking orchestrator uses structured prompts for two regimes: individual tool testing and pipeline testing. It drives each agent to issue MCP function calls and logs the full JSON request, the JSON response, including stdout, stderr, and output_files, any error message, and a pass/fail verdict. The report generator then aggregates per-tool and per-pipeline outcomes into summary tables and computes success rates, reliability, and versatility.

An illustrative example given in the paper uses FastQC on Claude. The user prompt requests FastQC on /data/SRR8405197.fastq, the MCP request JSON includes fields such as "input_file": "/data/SRR8405197.fastq" and "threads": 4, and the response JSON reports the executed command and output files such as SRR8405197_fastqc.html and the corresponding zip archive. The agent then returns a human-readable summary. This example makes explicit that the benchmark observes the entire request-response-output chain rather than merely whether a server endpoint exists.

3. Validation protocol

The benchmark proceeds in two stages: individual tool validation and pipeline validation (Widjaja et al., 2 Oct 2025).

In individual tool validation, for each of the 38 MCP-converted tools $T$, the orchestrator sends a generic prompt of the form “please run tool $T$ on test input file $F_T$” to each of the three agents $A \in \{\mathrm{LAI}, \mathrm{CD}, \mathrm{CR}\}$. The agent translates the prompt into one or more MCP JSON calls. A pass is recorded if the MCP server is invoked without internal syntax or runtime errors and the expected output files are produced. Failures due to exhaustion of memory or wall-time are flagged separately as tool-intrinsic resource limits.

In pipeline validation, the benchmark specifies five representative end-to-end tasks spanning core sequencing modalities: RNA-seq, WGS assembly, ChIP-seq, ATAC-seq, and WGS/WES variant calling. For each pipeline $P$, the agent is prompted to execute the full sequence of MCP tool calls required by that workflow. A pipeline is considered successfully executed only if all constituent steps complete with valid outputs and the agent returns a final summary report.

All experiments were run in triplicate, once per agent host, to assess cross-platform robustness. This design matters because it distinguishes host-dependent failures from failures intrinsic to the MCP server or to the workflow definition. The paper also explicitly separates resource-related failures from intrinsic correctness of the MCP interface, which helps interpret negative results without conflating interface semantics with hardware constraints.

4. Formal metrics and their interpretation

BioinfoMCP Benchmark defines three primary metrics: per-tool average success, overall tool reliability, and pipeline success rate (Widjaja et al., 2 Oct 2025).

For tool success, the benchmark uses the binary indicator

$$s_{t,a} = \begin{cases} 1 & \text{if tool } t \text{ succeeds on agent } a,\ 0 & \text{otherwise.} \end{cases}$$

The per-tool average success, termed versatility, is

$V_t = \frac{1}{|A|}\sum_{a \in A} s_{t,a},$

with $|A| = 3$.

Overall tool reliability aggregates across all tools and all agents:

$$R_{\mathrm{tools}} = \frac{\sum_{t \in T}\sum_{a \in A} s_{t,a}}{|T| \times |A|}.$$

With $|T| = 38$ and $T$0, the paper reports $T$1.

For complex workflows, the benchmark defines a pipeline-level success indicator

$T$2

which equals 1 only if every step in pipeline $T$3 succeeds on agent $T$4. Aggregate pipeline success is then

$T$5

where $T$6. The paper reports $T$7 as the key complex-workflow benchmark outcome.

No confidence intervals were computed explicitly. The paper also states that resource-related failures are annotated separately and are not counted against the intrinsic correctness of the MCP interface. This metric design reveals the benchmark’s central methodological choice: reliability is treated as an execution statistic over tool-host pairs and workflow-host pairs, not as a downstream biological accuracy measure.

5. Coverage: converted tools and representative workflows

The benchmark covers 38 MCP conversions. These are bcftools, Bedtools:coverage, Bedtools:intersect, Bowtie2, BWA, Cell-ranger, Cutadapt, deepTools:bamCoverage, deepTools:computeGCBias, deepTools:correctGCBias, deepTools:plotCorrelation, fastp, FastQC, UCSC-FaToTwoBit, Flye, freebayes, GATK:ApplyBQSR, GATK:BaseRecalibrator, GATK:HaplotypeCaller, GATK:SelectVariants, Gunzip, HISAT2, Kallisto, MACS3:callpeak, MACS3:hmmratac, Minimap2, MAFFT, MEME, MultiQC, Qualimap, Quast, Salmon, Samtools, Seqtk, SPAdes, STAR, Trim-galore, and Trimmomatic (Widjaja et al., 2 Oct 2025).

The workflow layer consists of five complex pipelines. These are not abstract task labels; each is defined as an ordered sequence of tool calls.

Workflow	Tool sequence	Task type
RNA-seq differential expression	FastQC → samtools sort/index → Qualimap → MultiQC	RNA-seq
WGS de novo assembly	FastQC → fastp filtering → SPAdes assembly → Quast assessment → MultiQC	WGS assembly
ChIP-seq motif discovery	FastQC → Bowtie2 alignment → samtools processing → MACS3 peak calling → deepTools analysis → MultiQC → R sessions	ChIP-seq
ATAC-seq open-chromatin profiling	FastQC → Trim-galore trimming → Bowtie2 alignment → samtools QC → MACS3 callpeak → MultiQC	ATAC-seq
WGS/WES somatic SNV calling	FastQC → fastp trimming → Bowtie2 alignment → samtools sort/index → GATK → FreeBayes → bcftools	WGS/WES variant calling

The selection spans several core sequencing modalities and mixes QC, alignment, assembly, peak calling, quantification, bias correction, variant calling, and reporting steps. This breadth is important because MCP conversion quality can be stress-tested by workflows that differ substantially in parameterization style, artifact types, and intermediate-file conventions.

6. Reported outcomes, interpretation, and limitations

For individual tools, Table 1 of the paper reports that 36 out of 38 tools succeeded on all three agents. Cell-ranger, described as memory-heavy, failed on all three agents, and STAR failed on two agents due to runtime constraints. This yields the reported overall tool reliability of approximately 95.6% (Widjaja et al., 2 Oct 2025).

For complex workflows, the paper reports that across 5 pipelines and 3 agents, corresponding to 15 end-to-end trials, 14 succeeded in full completion. That gives

$T$8

The same section then states that, when weighted by step-level reliability, the quoted complex-workflow success rate is 94.7%. One potential source of confusion is therefore that two different summaries appear in the paper: a simple end-to-end completion count of 14/15 and a weighted step-level reliability figure of 94.7%. Both are reported, but they quantify slightly different aspects of workflow success.

The benchmark’s significance is framed in three ways. First, it demonstrates that automatically converted MCP servers are not merely syntactically correct but semantically robust when orchestrated by real AI agents. Second, it bridges the gap between heterogeneous bioinformatics tools and agent-based workflows. Third, it establishes a reusable benchmark suite consisting of 38 tools, 5 pipelines, and 3 agents that can track progress as conversion methods and agent capabilities evolve.

The limitations are explicit. Resource-intensive tools such as Cell-ranger and STAR can fail due to current agent-host constraints. No formal statistical uncertainty quantification is provided, since confidence intervals are not computed. The benchmark is also restricted to a limited set of pipelines and datasets; the paper notes that extending to additional modalities such as metagenomics and proteomics would further validate generality. A plausible implication is that the present benchmark is strongest as a proof of interoperable execution under MCP, and less comprehensive as a survey of all bioinformatics modalities.

7. Relation to adjacent bioinformatics benchmarks

BioinfoMCP Benchmark occupies a distinct niche within recent bioinformatics benchmark design. BioAgent Bench is an evaluation suite for AI agents in common bioinformatics tasks such as RNA-seq, variant calling, and metagenomics, with automated artifact-based grading and robustness stress tests under corrupted inputs, decoy files, and prompt bloat (Fa et al., 29 Jan 2026). BixBench evaluates LLM-based agents on 53 real-world analytical trajectories and 296 open-answer questions designed to test dataset exploration, long multi-step analysis, and interpretation in computational biology (Mitchener et al., 28 Feb 2025). BIOME-Bench targets two literature-grounded tasks, Biomolecular Interaction Inference and Multi-Omics Pathway Mechanism Elucidation, with instance-level supervision and multi-granular evaluation (Wei et al., 31 Dec 2025). PhysiBench provides 612 executable intracellular Boolean regulatory network variants and a dataset of 120,000 time-resolved multiscale stochastic simulations for systems biology benchmarking (Masera et al., 16 Jun 2026).

Against that landscape, BioinfoMCP Benchmark is notable for evaluating interface standardization and execution fidelity rather than biological answer correctness, literature-grounded mechanistic reasoning, or multiscale simulation behavior. This suggests that it occupies an infrastructure layer within the broader BioinfoMCP evaluation stack: it asks whether MCP-converted tools are invocable, composable, and cross-host reliable. Other contemporary benchmarks focus more directly on reasoning quality, robustness to perturbation, or scientific interpretation.

The benchmark also advances beyond prior work that either assessed LLM performance on code generation in isolation or integrated pipelines in rigid, hand-coded frameworks. Its MCP-centric design standardizes the tool interface across agents and removes the need for per-agent wrappers. The suite is open-sourced through the BioinfoMCP repository, which positions it as an extensible foundation for future evaluation of agentic bioinformatics under standardized protocol-level tooling (Widjaja et al., 2 Oct 2025).