scBench: Multi-Domain Benchmark Overview

• scBench is an overloaded benchmark label applied across different domains, including LLM evaluation, compliance reasoning, sports commentary, spatial reasoning, and single-cell biology.
• The evaluation frameworks use varied methodologies such as KV cache lifecycle analysis, deterministic graders, GPT-based scoring, and structured JSON outputs.
• Interpreting scBench scores requires contextual clarity as each domain employs unique protocols, evaluation tasks, and performance metrics.

“scBench” and “SCBench” are not unique benchmark names in the contemporary arXiv literature. The label is used for several unrelated evaluation frameworks spanning compliance-oriented Text-to-SQL and retrieval-augmented generation, KV-cache-centric long-context inference, sports video commentary generation, spatial reasoning, and multiple strands of single-cell biology evaluation. In one benchmark-evaluation context, “scBench” is explicitly identified with “ScenarioBench” (Atf et al., 29 Sep 2025), but other papers use “SCBench” for “SharedContextBench” (Li et al., 2024), a sports commentary benchmark for Video LLMs (Ge et al., 2024), and the “Spatial Competence Benchmark” (Vira et al., 5 Mar 2026), while single-cell papers introduce distinct benchmarks under the names “scBench” (Workman et al., 9 Feb 2026), “scBench-Long” (Diks et al., 25 Jun 2026), and a scPilot-associated “scBench” suite (Gao et al., 12 Feb 2026).

1. Nomenclature and scope

The reused name reflects a genuine naming collision across fields rather than a single benchmark family. In practice, correct interpretation depends on disciplinary context, task definition, and cited paper.

Name in paper	Domain	Defining focus
ScenarioBench	Compliance evaluation	Trace-grounded Text-to-SQL and RAG
SCBench (SharedContextBench)	Long-context LLM inference	KV cache lifecycle under shared context reuse
SCBench	Video LLMs	Sports commentary generation
SCBench (Spatial Competence Benchmark)	Spatial reasoning	Executable spatial outputs with deterministic or simulator-based verification
scBench	Single-cell RNA-seq analysis	Verifiable workflow-step recovery
scBench-Long	Long-horizon single-cell biology	Final scientific conclusions from raw or near-raw data
scBench in scPilot	Omics-native reasoning	Annotation, trajectory, and TF–gene reasoning
scSSL-Bench	Single-cell SSL	Representation learning across downstream tasks

A common source of confusion is that several of these benchmarks are technically sophisticated and benchmark-like in similar ways—structured tasks, formal graders, or realistic workflows—yet they are methodologically incompatible and measure different capabilities. “SCBench” in a long-context systems paper usually refers to SharedContextBench (Li et al., 2024), whereas “scBench” in a single-cell biology paper may refer either to the 394-problem scRNA-seq workflow benchmark (Workman et al., 9 Feb 2026) or to the 9-dataset scPilot evaluation suite (Gao et al., 12 Feb 2026). In a compliance setting, the same shorthand is explicitly mapped to ScenarioBench (Atf et al., 29 Sep 2025).

2. SCBench as SharedContextBench for long-context LLMs

“SCBench: A KV Cache-Centric Analysis of Long-Context Methods” defines SCBench as “SharedContextBench,” a benchmark for evaluating long-context methods from a KV-cache-centric perspective across four lifecycle stages: KV cache generation, KV cache compression, KV cache retrieval, and KV cache loading (Li et al., 2024). Its central design choice is to replace single-request evaluation with test sessions built around a shared context and multiple follow-up queries. The benchmark has two shared-context modes: multi-turn and multi-request. In multi-turn mode, follow-up turns use golden answers as context rather than model-generated outputs, specifically to avoid error propagation. The released benchmark contains 931 multi-turn sessions and 4,853 queries, averaging about 5 turns per session.

SharedContextBench covers 12 tasks organized into four capability categories: string retrieval, semantic retrieval, global information, and multi-task. The tasks are Retr.KV, Retr.Prefix-Suffix, Retr.MultiHop, Code.RepoQA, En.QA, Zh.QA, En.MultiChoice, Math.Find, ICL.ManyShot, En.Sum, Mix.Sum+NIAH, and Mix.RepoQA+KV. The evaluation spans eight categories of long-context solutions, including Gated Linear RNNs, Mamba-Attention hybrids, sparse attention, KV cache dropping, quantization, retrieval, loading, and prompt compression, and it is conducted on eight long-context LLMs.

The main empirical claims are methodological rather than leaderboard-centric. Methods with sub-$O(n)$ decoding memory degrade sharply in shared-context scenarios; sparse encoding with dense decoding is much more robust; dynamic sparse patterns are more expressive than static patterns; and exact retrieval tasks often require full or near-full KV retention. The paper also identifies attention distribution shift in long-generation scenarios. These findings recast SCBench as a systems benchmark for shared-context durability rather than a generic long-context accuracy suite (Li et al., 2024).

Later KV-cache papers use SCBench in that sense. “Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs” uses SCBench as a long-context understanding benchmark “adapted from RULER and InfiniteBench to evaluate long-context understanding of KV cache compression methods under the same context but different queries” (Bui et al., 3 Dec 2025). In that evaluation, TRIM-KV is reported as competitive with full cache overall at a 32768 budget, with Full KV overall at 49.4 and TRIM-KV overall at 48.2, while consistently outperforming StreamingLLM, H2O, and SnapKV. The same paper explicitly notes that retrieval-heavy tasks such as Retr.KV and Code.RepoQA remain difficult for all eviction methods because “the context is incompressible” (Bui et al., 3 Dec 2025).

“DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity” likewise treats SCBench as a benchmark for realistic multi-turn dialogue and complex multi-turn scenarios (Hao et al., 8 Feb 2026). Under that usage, SCBench is reduced to four representative tasks: Retr.KV, En.QA, ICL.ManyShot, and Mix.Sum+NIAH. The paper’s reported pattern is that static eviction baselines such as SnapKV collapse on Retrieval KV, whereas DeltaKV remains substantially more robust while reducing memory footprint relative to full cache and, in some settings, even exceeding full-cache performance on Many-Shot or mixed tasks (Hao et al., 8 Feb 2026).

3. ScenarioBench as “scBench” in compliance evaluation

“ScenarioBench: Trace-Grounded Compliance Evaluation for Text-to-SQL and RAG” states explicitly that, in the benchmark-evaluation context of that paper, “ScenarioBench” is the intended meaning of “scBench” (Atf et al., 29 Sep 2025). ScenarioBench is a policy-grounded, trace-aware benchmark for evaluating Text-to-SQL and retrieval-augmented generation in compliance settings. It is built for scenarios such as regulated communications, where a system must output both a decision—such as allow, block, safe-rewrite, or escalate—and a trace showing which policy clauses justify that decision.

Each benchmark item is a YAML scenario compiled into synchronized representations of the same policy canon: Prolog facts for deterministic rule execution and Policy_DB (SQL) for retrieval and NLQ-to-SQL. For each YAML scenario, the evaluator holds a no-peek gold-standard package containing the gold decision, a minimal witness trace, the governing clause set, and the canonical SQL query. The gold trace is defined as the minimal sufficient set of clauses needed to justify the decision, and SQL correctness is judged by result-set equivalence on clause IDs rather than by string match.

ScenarioBench scores both the decision and the justification. Decision quality is evaluated with accuracy and macro-F1. Trace quality is decomposed into completeness, correctness, and order, with order measured using Kendall-$\tau$. The benchmark further imposes a grounding constraint: trace clause IDs must come only from the system’s own retrieved clauses. The reported metrics include decision accuracy, macro-F1, trace completeness, trace correctness, trace order via Kendall-$\tau$, retrieval effectiveness through Recall@k, MRR, and nDCG, SQL correctness via result-set equivalence on clause_id, policy coverage, latency, and explanation-hallucination rate. The discussion also distinguishes a strict hallucination rate in which any citation outside the gold closure counts as hallucination (Atf et al., 29 Sep 2025).

The benchmark introduces a normalized Scenario Difficulty Index,

$$\mathrm{SDI}=w_D(1-\mathrm{Acc})+w_T\!\big(1-\tfrac{1}{3}(T_c+T_k+T_o)\big)+w_R(1-\mathrm{nDCG}@k),$$

with default weights $(w_D,w_T,w_R)=(0.5,0.3,0.2)$ (Atf et al., 29 Sep 2025). This formulation couples decision quality, trace quality, and retrieval quality in a single normalized difficulty score. Compared with Spider-, BIRD-, or KILT-style benchmarks, ScenarioBench is stricter about clause-level provenance, no-peek gold packaging, and grounding explanations in the same canon used for scoring.

4. scBench in single-cell biology

In single-cell biology, “scBench” names more than one benchmark family. “scBench: Evaluating AI Agents on Single-Cell RNA-seq Analysis” introduces a benchmark of 394 verifiable problems derived from practical scRNA-seq workflows spanning 6 sequencing platforms and 7 task categories: QC, Normalization, Dimensionality Reduction, Clustering, Cell Typing, Differential Expression, and Trajectory Analysis (Workman et al., 9 Feb 2026). Each problem combines a data snapshot, usually an AnnData .h5ad file, a natural-language prompt specifying the required JSON output format, a deterministic grader, metadata, and hidden internal notes. The benchmark emphasizes “specify what, not how,” except in explicitly procedural tasks.

This scRNA-seq scBench defines three evaluation types—Scientific, Procedural, and Observational—and five deterministic grader families: NumericTolerance, MultipleChoice, MarkerGenePrecisionRecall, LabelSetJaccard, and DistributionComparison. The agent harness is mini-SWE-agent, with a maximum of 100 action steps, a 300 second timeout per command, a 600 second total evaluation timeout, and scoring zero if the agent never writes eval_answer.json. Each model-evaluation pair is run 3 times. Across eight frontier models, reported overall accuracy ranges from 29.2% to 52.8%, with Claude Opus 4.6 at 52.8% and Gemini 2.5 Pro at 29.2%. One of the paper’s main claims is that platform choice affects accuracy as much as model choice, with 40+ percentage point drops on less-documented technologies (Workman et al., 9 Feb 2026).

“scBench-Long: Verifiable Benchmarking of Long-Horizon Single-Cell Biology” addresses a different level of difficulty (Diks et al., 25 Jun 2026). Instead of asking for recovery of a single local analysis step, it evaluates whether agents can recover a final scientific conclusion from raw or near-raw data. The benchmark contains 21 evaluations across five study systems—melanoma CD8 T-cell reactivity, CD8 RNA+ATAC regulatory inference, human–monkey chimera development, KRAS-driven lung tumor aging, and lethal COVID-19 lung pathology—and reports 1,068 completed trajectories. Candidate claims are reproduced, reviewed, and converted into controlled answer vocabularies with deterministic grading and companion trajectory rubrics. The strongest model–harness pair passes 16/63 runs, or 25.4%, with 8/21 evaluations having at least one passing replicate, 6/21 a majority of passing replicates, and 2/21 all replicates passing (Diks et al., 25 Jun 2026).

A third usage appears in “scPilot: LLM Reasoning Toward Automated Single-Cell Analysis and Discovery,” which releases a “scBench” suite of 9 expertly curated datasets and graders for three workflows: cell-type annotation, developmental-trajectory reconstruction, and transcription-factor targeting or GRN prediction (Gao et al., 12 Feb 2026). Here the benchmark is tied directly to “omics-native reasoning,” in which the model converses in natural language while inspecting single-cell RNA-seq data and on-demand bioinformatics tools. The paper reports that iterative omics-native reasoning lifts average accuracy by 11% for cell-type annotation, Gemini-2.5-Pro cuts trajectory graph-edit distance by 30% versus one-shot prompting, and scPilot improves AUROC over direct prompting by an average of +0.098 (Gao et al., 12 Feb 2026).

A related but differently named benchmark, “scSSL-Bench,” evaluates 19 self-supervised learning methods on 9 datasets for batch correction, cell type annotation, and missing modality prediction (Ovcharenko et al., 10 Jun 2025). It is not itself called “scBench,” but it illustrates how the same naming space has expanded within single-cell research to cover agent evaluation, long-horizon biological reasoning, and representation learning.

5. SCBench in video understanding and spatial reasoning

“SCBench: A Sports Commentary Benchmark for Video LLMs” defines SCBench as a benchmark for sports video commentary generation (Ge et al., 2024). Its dataset, CommentarySet, contains 5,775 annotated video clips across athletics, basketball, soccer, gym, table tennis, and tennis, with 4,908 training clips and 867 test clips. The benchmark’s evaluation framework is SCORES, a six-dimensional metric whose dimensions are Situation, taCtic, emOtion, backgRound, key Events, and techniqueS. The authors argue that conventional automatic metrics such as BLEU, ROUGE, METEOR, CIDEr, and SPICE are poorly aligned with sports commentary quality, so they propose a GPT-based evaluation method grounded in SCORES and using a 0-to-10 score. In the reported experiments, InternVL-Chat-2 achieves the best overall SCORES value, 5.44, surpassing the second-best model by 1.04 (Ge et al., 2024).

This SCBench is open-ended and commentary-centric rather than deterministic in the sense of executable verification. Its human validation study uses 36 human evaluators, and the paper reports that SCORES has a 60% overlap with human judgments, about 1.5× higher than Vanilla LLM/GPT evaluation (Ge et al., 2024). The benchmark therefore occupies a different methodological position from SharedContextBench or the single-cell scBench variants: it stresses fine-grained temporal visual understanding and professional-style generation rather than structured recovery of a formally checkable artifact.

A separate use of the acronym appears in “Spatial Competence Benchmark,” abbreviated SCBench (Vira et al., 5 Mar 2026). This benchmark defines spatial competence as “the quality of maintaining a consistent internal representation of an environment and using it to infer discrete structure and plan actions under constraints.” It spans three hierarchical capability buckets—axiomatic inference, constructive synthesis, and planning—and nearly every task requires a structured executable output, such as coordinates, edge sets, action sequences, voxel sets, or paths. Tasks are checked by deterministic verifiers or simulator-based evaluators, with partial credit for some complex tasks. The benchmark contains 22 tasks and 285 subtasks.

The reported bucket-level results show monotonically decreasing performance up the capability ladder for all three frontier models evaluated. The ordering Axiomatic $>$ Constructive $>$ Planning holds for Claude Sonnet 4.5, Gemini 3 Pro Preview, and GPT-5.2, and the strongest models reach 57.6% overall (Vira et al., 5 Mar 2026). The paper further reports that accuracy gains from increased output-token budgets saturate quickly and that the dominant failure mode is “Local-Only,” where models produce locally plausible geometry but fail to enforce global constraints. Unlike SharedContextBench, the current evaluation is limited to single-turn, zero-shot prompts (Vira et al., 5 Mar 2026).

6. Cross-cutting design patterns and interpretive issues

Although these benchmarks share a name fragment, they operationalize evaluation in markedly different ways. SharedContextBench formalizes the full lifecycle of the KV cache under multi-turn and multi-request reuse (Li et al., 2024). ScenarioBench formalizes clause-level evidence grounding with no-peek gold packaging and falsifiable clause IDs (Atf et al., 29 Sep 2025). The scRNA-seq scBench and scBench-Long rely on deterministic graders, controlled answer surfaces, and structured JSON or typed outputs (Workman et al., 9 Feb 2026, Diks et al., 25 Jun 2026). The Spatial Competence Benchmark emphasizes structured executable outputs checked programmatically or in simulation (Vira et al., 5 Mar 2026). By contrast, the sports commentary SCBench relies on a label-conditioned GPT judge grounded in SCORES (Ge et al., 2024).

This divergence matters for interpretation. A score from SharedContextBench is about shared-context robustness under KV-cache reuse; a score from ScenarioBench is about correctness plus trace-grounded compliance justification; a score from single-cell scBench is about recovery of a biological result from workflow state; a score from scBench-Long is about long-horizon claim recovery; and a score from sports SCBench is about commentary quality under a six-dimensional rubric. A plausible implication is that numerical results across these benchmarks are not cross-comparable even when the acronym is identical.

A second recurring issue is the distinction between local competence and long-horizon competence. The scRNA-seq scBench evaluates a snapshot immediately prior to an analysis step (Workman et al., 9 Feb 2026), whereas scBench-Long evaluates whether an agent can move beyond local analysis steps and make complex scientific claims supported by single-cell data (Diks et al., 25 Jun 2026). SharedContextBench exposes degradation across repeated reuse of the same context (Li et al., 2024), and ScenarioBench shifts gains toward justification quality under explicit time budgets (Atf et al., 29 Sep 2025). This suggests that “scBench” has become associated, in several subfields, with evaluation settings that are harder to shortcut than standard single-step QA or single-request inference.

The most important encyclopedic point is therefore terminological rather than taxonomic: “scBench” is an overloaded benchmark label. Correct interpretation requires the full paper title, domain, and benchmark protocol, because the name now refers to multiple unrelated but technically mature evaluation frameworks across LLM systems, compliance reasoning, multimodal generation, spatial reasoning, and single-cell biology.