SP-Bench: Spatial Proteomics Workflow Benchmark

• SP-Bench is a comprehensive benchmark for evaluating autonomous agent performance on multi-stage spatial proteomics workflows.
• It operationalizes full pipeline orchestration by assessing tasks from image correction to clustering with a detailed hierarchical task taxonomy.
• The benchmark facilitates failure analysis and performance comparison, highlighting execution accuracy across various task difficulties and imaging modalities.

Searching arXiv for the cited SP-Bench papers to ground the article. SP-Bench is a benchmark for evaluating autonomous agents on spatial proteomics workflows. In the formulation introduced alongside SP-Mind, it is presented as the first comprehensive standardized benchmark for agentic spatial proteomics analysis, designed to test whether an agent can orchestrate the full analysis pipeline from raw multiplexed tissue images through downstream phenotyping rather than merely execute isolated tools (Yuan et al., 23 Jun 2026). The same name has also been used for a benchmark suite of real-world constrained multi-objective optimization problems for battery thermal management system design (Ouyang et al., 29 Oct 2025). In current usage, however, the designation most directly refers to the spatial proteomics benchmark associated with SP-Mind (Yuan et al., 23 Jun 2026).

1. Definition and scope

SP-Bench evaluates workflow orchestration competence in spatial proteomics. The benchmark is motivated by the observation that spatial proteomics analysis is inherently multi-stage and operationally complex, with typical workflows requiring illumination correction, registration or stitching, background subtraction, tissue microarray dearraying, probability mapping, segmentation, quantification, and clustering or phenotyping (Yuan et al., 23 Jun 2026). Existing automated pipelines are described as reproducible but too rigid, because they require expert configuration and do not adapt well to different queries, tissue types, or imaging modalities. SP-Bench was therefore created to test whether an LLM-based agent can understand the task, choose tools, respect data dependencies, propagate intermediate outputs correctly, and finish successfully (Yuan et al., 23 Jun 2026).

A central feature of the benchmark is that it is not just a dataset of images or labels. Instead, it operationalizes the full analysis chain as a benchmark of agentic execution. This means the primary object of evaluation is not static prediction quality on a fixed supervised task, but the ability to translate natural-language requests into correct multi-step analytical procedures. This suggests that SP-Bench occupies a different methodological niche from conventional biomedical image benchmarks, which usually isolate one stage such as segmentation or cell typing.

The benchmark is organized around 102 distinct natural-language queries, 18 task categories, 8 core spatial proteomics analysis stages, and 4 difficulty tiers (Yuan et al., 23 Jun 2026). This hierarchical structure is intended to distinguish failures in single-tool usage from failures in sequencing, state management, and long-horizon execution.

2. Workflow model and task taxonomy

SP-Bench is built around eight core operations that mirror the spatial proteomics pipeline (Yuan et al., 23 Jun 2026):

Core stage	Function
Illumination	correcting microscope-induced shading or uneven illumination
Registration	stitching and aligning tiled or multi-cycle acquisitions
Background Subtraction	removing autofluorescence and imaging artifacts
Tissue Dearray	extracting individual tissue cores from TMA scans
Probability Mapping	generating nuclear probability maps
Segmentation	producing cell and nuclear masks
Quantification	extracting per-cell marker intensities and morphology
Clustering	grouping cells into biologically meaningful populations

These stages are embedded in a four-tier taxonomy that increases workflow depth from isolated operations to end-to-end pipelines (Yuan et al., 23 Jun 2026). The benchmark divides tasks into Basic, Intermediate, Advanced, and Challenging tiers. The Basic tier contains 8 categories and 40 queries and consists of single-stage tasks. The Intermediate tier contains 4 categories and 28 queries and covers two-stage workflows. The Advanced tier contains 3 categories and 21 queries and covers three-stage workflows. The Challenging tier contains 3 categories and 13 queries and covers workflows of four or more stages (Yuan et al., 23 Jun 2026).

The category design is explicit. Examples include Signal Calibration $\rightarrow$ Illumination, Image Registration $\rightarrow$ Registration, Artifact Removal $\rightarrow$ Background Subtraction, Cell Segmentation Pipeline $\rightarrow$ Probability Mapping $\rightarrow$ Segmentation, Downstream Interpretation $\rightarrow$ Segmentation $\rightarrow$ Quantification $\rightarrow$ Clustering, and Grand Pipeline $\rightarrow$ all 8 stages (Yuan et al., 23 Jun 2026). This taxonomy permits stage-local and pipeline-level diagnosis. A plausible implication is that the benchmark is designed as much for failure analysis as for leaderboard ranking.

The expected workflow logic is described as:

raw multiplexed image $\rightarrow$ illumination correction $\rightarrow$0 registration $\rightarrow$1 background subtraction $\rightarrow$2 dearraying or probability mapping $\rightarrow$3 segmentation $\rightarrow$4 quantification $\rightarrow$5 clustering or annotation (Yuan et al., 23 Jun 2026).

3. Benchmark construction and data sources

The benchmark construction procedure has two principal components: synthetic query generation followed by expert curation. For each task category, the authors first used Gemini 3 Pro to generate realistic natural-language prompts intended to resemble how a researcher would request analysis (Yuan et al., 23 Jun 2026). These prompts include both generic instructions and context-specific variants specifying output paths, marker channels, or tissue or core structure.

The resulting prompts were then reviewed by three bioinformatics experts, who evaluated each query for biological plausibility, correct module dependencies and data flow, and alignment with realistic spatial proteomics requests (Yuan et al., 23 Jun 2026). Queries judged problematic by at least two of the three experts were removed. The paper reports that 32 of the initial 134 generated queries were excluded, leaving the final 102 (Yuan et al., 23 Jun 2026).

SP-Bench also aims to test generalization across biological and technological heterogeneity. It covers 5 major tissue environments—Meningioma, GI Stromal Tumor, Normal Colon, Classic Hodgkin Lymphoma, and Pancreatic Ductal Adenocarcinoma—and spans 4 mainstream imaging technologies—CyCIF, MIBI, CODEX, and IMC (Yuan et al., 23 Jun 2026). The benchmark inputs were derived from the MCMICRO exemplar dataset, the MAPS cHL dataset, and the PDAC IMC dataset. For stages 1 to 7, the primary source is the MCMICRO Exemplar-002 dataset, and domain experts manually executed the workflow to create validated intermediate outputs. For clustering, the benchmark uses four quantification CSVs: cHL_1_MIBI, cHL_2_MIBI_5, cHL_CODEX, and PDAC_IMC (Yuan et al., 23 Jun 2026).

If a CSV exceeded 100 MB, it was subsampled with stratified random sampling to preserve class balance (Yuan et al., 23 Jun 2026). This suggests that benchmark construction balances realism with tractable execution, particularly for downstream clustering stages.

4. Inputs, outputs, and success conditions

SP-Bench is framed around practical analysis artifacts rather than abstract benchmark instances. Example task prompts specify inputs such as raw OME-TIFF images, illumination profile files, stitched mosaics, marker metadata CSVs, segmentation masks, quantification CSVs, and output directories (Yuan et al., 23 Jun 2026). The agent is expected to use the correct tool or code, preserve the correct order of execution, pass outputs to subsequent stages, save files in the requested directory, and return an accurate summary (Yuan et al., 23 Jun 2026).

This means the benchmark evaluates operational correctness, not just endpoint numerical similarity. The paper uses a strict execution accuracy metric. A query counts as successful only if the agent satisfies a set of nine verification criteria, including the following requirements (Yuan et al., 23 Jun 2026):

• Tool and code selection: uses the appropriate tool or code.
• Parameter fidelity: follows all user-specified parameters.
• Workflow integrity: preserves correct sequence and intermediate data flow.
• Output compliance: produces outputs in the required format and location.
• Execution reliability: completes without runtime error or timeout.
• Input protection: does not modify input data or directories.
• Reporting fidelity: summarizes the work accurately.
• Transparency: reports any errors or warnings transparently.

In practice, a task is successful only if the requested analysis is completed end-to-end and the result is both scientifically and operationally valid (Yuan et al., 23 Jun 2026). This is stricter than standard benchmark protocols that score only the final prediction artifact. A common misconception would be to treat SP-Bench as a pure image-analysis benchmark; the paper instead defines it as a benchmark of agentic workflow execution.

5. Role in evaluating SP-Mind

SP-Bench is the principal benchmark used to validate SP-Mind, which the paper presents as an autonomous reasoning and execution agent for spatial proteomics (Yuan et al., 23 Jun 2026). SP-Mind is described as an LLM-driven system with a modular library of 10+ spatial proteomics tools, expert-curated Spatial BioSkill Templates, a ReAct-style reasoning loop, and code execution with sandboxed file persistence (Yuan et al., 23 Jun 2026).

Within this system, SP-Bench functions as the benchmark that determines whether these capabilities translate into real analytical performance. The benchmark is used to test whether the agent can identify which spatial proteomics stage is needed, choose the right tool, configure parameters properly, recover from errors, and continue through long multi-step pipelines (Yuan et al., 23 Jun 2026). The benchmark thus directly instantiates the paper’s central claim that an autonomous agent can unify the end-to-end spatial proteomics pipeline.

The benchmark also enables ablation-style comparison. In the reported experiments, performance is compared across SP-Mind, SP-Mind (no skill), Biomni-SP, ToolUniverse-SP, AutoGen, Biomni, and ToolUniverse (Yuan et al., 23 Jun 2026). The distinction between domain-adapted variants and general-purpose agent frameworks is important, because it separates the value of general tool-use competence from the value of domain specialization.

6. Reported performance and observed limitations

The paper reports execution success rates over the 102 queries, averaged over 3 runs with mean and standard deviation (Yuan et al., 23 Jun 2026). The reported average execution accuracies are 68.9% for SP-Mind, 62.4% for SP-Mind (no skill), 55.4% for Biomni-SP, 50.5% for ToolUniverse-SP, 19.3% for Biomni, 15.0% for ToolUniverse, and 14.7% for AutoGen (Yuan et al., 23 Jun 2026).

Performance by difficulty tier is also reported (Yuan et al., 23 Jun 2026):

Tier	SP-Mind	SP-Mind (no skill)	Strongest non-SP-Mind baseline
Basic	95.8 ± 1.4	95.0 ± 2.5	ToolUniverse-SP: 90.8 ± 1.4
Intermediate	84.5 ± 7.4	82.1 ± 7.1	Biomni-SP: 73.8 ± 4.1
Advanced	61.9 ± 4.8	44.4 ± 7.3	Biomni-SP: 36.5 ± 5.5
Challenging	33.3 ± 4.4	28.2 ± 4.4	Biomni-SP: 23.1 ± 0.0

The paper highlights that the performance gap widens on harder tasks, with 61.9% for SP-Mind on Advanced tasks versus 36.5% for Biomni-SP, and 33.3% on Challenging tasks versus 23.1% for Biomni-SP (Yuan et al., 23 Jun 2026). The authors interpret this as evidence that skill templates and task-conditional reasoning improve long-horizon workflow execution. That interpretation is explicitly presented in the source.

The paper also identifies several failure modes and limitations. SP-Mind still depends on human-curated skill templates and cannot yet autonomously synthesize new reusable skills from successful traces (Yuan et al., 23 Jun 2026). Reported practical failure modes include file-management and stale-path errors, difficulty with background subtraction input formatting, and occasional bookkeeping mistakes in long multi-stage chains (Yuan et al., 23 Jun 2026). More generally, the benchmark reveals that generalist agents collapse on complex spatial tasks, often approaching zero on the hardest tiers, whereas even the best-performing domain-specialized agents remain far from perfect (Yuan et al., 23 Jun 2026).

7. Naming ambiguity and broader significance

The label SP-Bench is not unique across the literature. In a different context, it denotes a benchmark suite of 12 real-world constrained multi-objective optimization problems for battery thermal management system design, intended to evaluate constrained multi-objective evolutionary algorithms on surrogate-based engineering formulations (Ouyang et al., 29 Oct 2025). That use is unrelated to spatial proteomics. The shared naming illustrates a broader pattern in recent benchmark literature, where short benchmark names are often overloaded across domains.

In the spatial proteomics context, however, SP-Bench has a specific methodological significance. It defines a benchmark whose unit of analysis is a natural-language request coupled to an executable biomedical workflow rather than a static predictive task. This suggests a shift from conventional supervised evaluation toward assessment of sequencing, dependency management, tool invocation, parameter handling, and error recovery in domain-specific scientific pipelines.

The benchmark’s broader significance therefore lies in its attempt to operationalize what it means for an autonomous agent to behave like a competent spatial biology analyst. By emphasizing workflow integrity, heterogeneous inputs, strict execution criteria, and multi-stage difficulty scaling, SP-Bench provides an evaluation framework for the end-to-end analytical competence required in spatial proteomics research (Yuan et al., 23 Jun 2026).