SuiteEval: Unified IR Benchmark Evaluation

• SuiteEval is a unified evaluation framework that standardizes multi-benchmark IR experiments with consistent dataset selection, metric computation, and artifact management.
• It employs a user-defined pipeline generator along with DatasetContext and Suite abstractions to enable shared dynamic indexing and efficient data handling.
• The framework reduces storage footprint and configuration discrepancies by grouping datasets by corpus and applying official metrics uniformly across evaluations.

SuiteEval is a unified framework for end-to-end retrieval benchmark evaluation that standardizes multi-benchmark information retrieval experiments around a small user-facing interface: the user supplies a retrieval pipeline generator, and the framework handles data loading, grouping datasets by corpus, indexing, ranking, metric computation, and result aggregation. It is motivated by fragmentation in retrieval evaluation—different dataset subsets, different aggregation methods, different pipeline configurations, and different artifact-management choices—which can undermine reproducibility and comparability, especially for foundation embedding models expected to generalize out of domain (Parry et al., 20 Feb 2026).

1. Motivation and problem setting

SuiteEval was introduced to address a concrete methodological problem in modern IR benchmarking: benchmark reporting increasingly spans suites such as BEIR, LoTTE, MS MARCO, NanoBEIR, and BRIGHT, yet nominally comparable evaluations often differ in dataset membership, aggregation conventions, and pipeline engineering details. The framework’s stated goal is therefore standardization rather than merely convenience: it defaults to complete coverage of all datasets and splits for a benchmark suite, applies official metrics and aggregation procedures uniformly, and eliminates boilerplate orchestration so that the user specifies only the IR pipeline (Parry et al., 20 Feb 2026).

The paper identifies four fragmentation sources. First, different studies may evaluate on different subsets or splits within the same suite. Second, even when they use the same datasets, they may aggregate results differently. Third, existing benchmark codebases often support only limited retrieval architectures, so new rerankers or hybrid pipelines require custom scripts, leading to inconsistent preprocessing and indexing choices. Fourth, practical differences in index and artifact management can change evaluation behavior across large suites. SuiteEval is positioned as a benchmark-suite controller that sits above retrieval toolkits and standardizes evaluation across benchmark suites rather than replacing the underlying retrieval stack (Parry et al., 20 Feb 2026).

A recurring misconception is that SuiteEval is primarily a productivity wrapper. The paper instead frames it as infrastructure for reproducible and comparable IR research. Its emphasis on complete suite coverage, official measures, benchmark-aware aggregation, and corpus-level orchestration is intended to reduce opportunities for silent divergence between published evaluations (Parry et al., 20 Feb 2026).

2. Core abstractions and system architecture

The architecture revolves around three main ideas: a user-defined pipeline generator, a DatasetContext abstraction, and a Suite controller. The pipeline generator is the only mandatory user contribution. It accepts a dataset context and yields one or more PyTerrier pipelines. SuiteEval then executes those pipelines across the entire benchmark suite in a standardized way (Parry et al., 20 Feb 2026).

DatasetContext encapsulates dataset-specific execution state. It provides a workspace path, a get_corpus_iter() iterator used for indexing, and a text_loader() utility for attaching document text when required by rerankers. Filesystem writes are scoped to context.path, and these artifacts are treated as ephemeral unless a persistent index_dir is supplied. This isolates dataset-specific details from retrieval logic and lets the user write pipelines against a common interface rather than against benchmark-specific loading code (Parry et al., 20 Feb 2026).

A Suite instance acts as the controller. It bundles dataset identifiers, benchmark-specific contexts, official evaluation measures, aggregation rules, and orchestration logic. Its job is to group datasets by underlying corpus, instantiate a DatasetContext for each corpus group, invoke the user’s pipeline generator once per corpus group, execute the returned pipelines across the associated test collections, compute evaluation measures, and return detailed results together with suite-level aggregates (Parry et al., 20 Feb 2026).

Construct	Role	Concrete elements
Pipeline generator	User-defined system specification	Yields one or more PyTerrier pipelines
DatasetContext	Dataset wrapper and workspace	path, get_corpus_iter(), text_loader()
Suite	Benchmark-suite controller	Dataset IDs, official measures, aggregation rules

This decomposition is one of the framework’s central design choices. It preserves flexibility at the retrieval-pipeline level while enforcing standardization at the suite-execution level. The paper explicitly positions SuiteEval above tools such as PyTerrier, Anserini, ir_datasets, and TIRA: those systems help with reproducible experimentation, data processing, or containerization, but do not fully automate end-to-end suite evaluation with consistent dataset selection, metric handling, aggregation rules, and index lifecycle management (Parry et al., 20 Feb 2026).

3. Execution model and dynamic indexing

The execution workflow begins with a user-written function of the form def systems(context: DatasetContext): ... yield pipeline1 .... The user is responsible for defining the retrieval, reranking, or hybrid pipeline and for constructing an index if needed, typically inside the context workspace. SuiteEval automates the rest of the benchmark execution process (Parry et al., 20 Feb 2026).

At runtime, the chosen suite resolves dataset identifiers using ir_datasets, obtaining the corpus, topics, and relevance judgments. Datasets are then grouped by underlying corpus rather than treated independently. This is a critical systems abstraction: indexing should happen once per corpus, after which multiple query or test collections sharing that corpus can reuse the same index. The paper’s MS MARCO passage example makes this explicit: one index for MSMARCOv1 is reused for DL-2019 and DL-2020, then released, and a separate index is created for MSMARCOv2 and used for DL-2022 (Parry et al., 20 Feb 2026).

systems(context) is called once per corpus group, not once per test collection. All pipelines yielded for that corpus therefore share the same index instance. Inside the generator, the user can instantiate an index such as PisaIndex(context.path / "index.pisa") and call index.index(context.get_corpus_iter()). SuiteEval ensures that the workspace is correctly scoped, that an existing on-disk index is reused if present, and that otherwise the index is built once. This is the paper’s notion of dynamic indexing: indices are materialized only when needed, reused across all test collections sharing a corpus, optionally persisted if index_dir is provided, and otherwise treated as ephemeral (Parry et al., 20 Feb 2026).

This execution model is also why the framework can support multi-stage systems without special-case orchestration. The paper’s examples include BM25 retrieval composed with a text loader, followed by MonoT5 or Electra-based reranking, but the framework is described more generally as supporting IR pipelines expressed in the PyTerrier ecosystem. The number of index materializations is therefore bounded by the number of distinct corpora rather than by the number of test collections or pipeline variants (Parry et al., 20 Feb 2026).

4. Benchmarks, metrics, and suite extension

SuiteEval ships with pre-registered benchmark suites including LoTTE, MSMARCOPassage, MSMARCODocument, BEIR, NanoBEIR, and BRIGHT. The abstract summarizes built-in support as BEIR, LoTTE, MS MARCO, NanoBEIR, and BRIGHT, while the body distinguishes passage and document variants of MS MARCO. These suites are handled through the same abstraction: each is represented as a Suite object, each defines dataset IDs and metadata, datasets are resolved through ir_datasets, and metric computation and aggregation are suite-specific but orchestrated through one controller (Parry et al., 20 Feb 2026).

The framework does not impose one global metric or one global aggregation rule. Instead, official measures are declared in suite metadata and then applied consistently across all datasets in that suite. This prevents metric drift while still respecting benchmark-specific conventions. The paper gives a custom suite example in which metadata={"official_measures": [nDCG@10]} is attached at registration time. It also notes benchmark-specific aggregation behavior: BEIR additionally applies post-hoc filtering and corrects aggregation, NanoBEIR uses similar post-processing, and a custom MS MARCO passage suite returns per-collection nDCG@10 values together with the geometric mean of the three collections for each pipeline (Parry et al., 20 Feb 2026).

Extension is declarative rather than script-heavy. New benchmark suites can be added in a single line through a registry mechanism built around Suite.register(...), which takes a suite name, a list of dataset identifiers, and metadata such as official measures and aggregation behavior. The paper emphasizes that this registration happens at import time and does not require custom subclassing or bespoke benchmark-runner scripts (Parry et al., 20 Feb 2026).

The result object is a long-form pandas DataFrame with one row per dataset, system, and evaluation measure. Because all datasets and metrics in a suite are orchestrated together, SuiteEval can also compute suite-level aggregates in a benchmark-aware way. This output format encodes the framework’s broader methodological position: comparison should happen over an explicitly defined suite surface rather than through manually stitched-together per-dataset scripts (Parry et al., 20 Feb 2026).

5. Empirical evidence and practical impact

The paper is a short systems/demo paper, so its empirical evidence is selective. The clearest quantitative result concerns storage footprint. In a NanoBEIR end-to-end versus re-ranker example, storage without SuiteEval is reported as 249.85 MB and with SuiteEval as 18.29 MB. In a BEIR BM25 grid-search example, storage without SuiteEval is 22888.07 MB and with SuiteEval is 4889.15 MB. These reductions are attributed to corpus-level grouping, dynamic indexing, and reuse of on-disk indices (Parry et al., 20 Feb 2026).

The paper also reports practical demonstration content rather than a broad benchmarking study. The demonstration includes a grid search over a large benchmark suite, a walkthrough for creating a simple suite, and an extension with task-specific custom aggregation. Runtime behavior is discussed conceptually—systems are invoked once per corpus, indices are built only when needed, and test collections sharing a corpus reuse the same index—but the paper does not provide detailed runtime tables, latency curves, ablation studies, or evaluation-consistency audits across prior baselines (Parry et al., 20 Feb 2026).

Several important negative statements clarify the scope of the evidence. SuiteEval does not introduce a new retrieval model or new evaluation metric. It also does not provide explicit equations for nDCG@10, averaging, geometric mean, significance testing, or retrieval scoring functions. Claims about reduced boilerplate and improved reproducibility are qualitative and are supported by the API design and examples rather than by a formal lines-of-code analysis or user study (Parry et al., 20 Feb 2026).

A second misconception is therefore worth addressing: SuiteEval is not a retrieval method paper disguised as infrastructure. Its contribution is evaluation orchestration, benchmark normalization, and artifact lifecycle management. The paper’s strongest evidence is infrastructural rather than model-comparative (Parry et al., 20 Feb 2026).

6. Reproducibility context and relation to other evaluation suites

SuiteEval belongs to a broader movement toward suite-centric evaluation infrastructure in which the benchmark is treated as an executable, versioned, and policy-bearing object rather than as a loose collection of datasets. In software engineering experiments more generally, different test suites intended to measure the same attribute can produce response values differing by as much as about $\pm 60\%$, and the associated measurement-analysis literature argues that the disclosure of datasets and code is insufficient when the suite itself functions as the measurement instrument (Dieste et al., 2021). This broader result suggests why SuiteEval treats official dataset coverage, official measures, aggregation rules, and execution configuration as first-class entities rather than incidental scripts.

In information retrieval specifically, SimEval-IR makes a similar suite-level move for simulator evaluation: it standardizes a canonical session schema, validated adapters, executable benchmarks, and provenance-rich configurations so that realism and tester reliability become comparable under a common infrastructure (Zerhoudi, 30 Apr 2026). For tool-using agents, an executable benchmarking suite has likewise been proposed around task manifests, declared drivers, replay classes, freeze metadata, and an evidence-admission contract that separates paper-facing evidence from smoke or diagnostic runs (Zhong et al., 10 May 2026). SuiteEval differs in domain and mechanism, but it occupies the same methodological space: evaluation is treated as an orchestrated surface with explicit dataset membership, metric policy, artifact management, and reporting semantics rather than as ad hoc experimental glue (Parry et al., 20 Feb 2026).

This comparison also clarifies what SuiteEval does not attempt. Unlike SimEval-IR, it does not define new benchmark tasks or validity criteria for a novel evaluation object. Unlike evidence-gated executable suites for agents, it does not formalize an admission contract over run artifacts. Its niche is narrower and more practical: consistent multi-benchmark IR execution with benchmark-aware measures and aggregation (Parry et al., 20 Feb 2026).

7. Limitations and outlook

The framework has several explicit limitations. It is built around PyTerrier pipelines, so its design is most natural for systems expressible in that ecosystem. Dataset resolution assumes ir_datasets or objects compatible with the same object structure. The paper also mentions benchmark-specific handling such as BEIR post-hoc filtering and corrected aggregation without fully documenting those procedures in the text, which constrains auditability from the paper alone. Empirical evaluation is limited to storage-footprint comparisons and demonstration notebooks rather than extensive runtime, consistency, or usability studies (Parry et al., 20 Feb 2026).

A further limitation is that SuiteEval focuses on orchestration rather than benchmark theory. It does not develop new metrics, formal statistical models, or a comprehensive reproducibility protocol beyond standardized execution and aggregation. This suggests that its main contribution is infrastructural normalization. That is nonetheless consequential in a literature where papers can both claim “BEIR” or “MS MARCO” results while differing in subsets, metrics, or aggregation choices (Parry et al., 20 Feb 2026).

The intended trajectory is extension rather than conceptual overhaul. The registry mechanism is designed to support newly released benchmark suites, and the paper’s framing implies continued growth in suite coverage and benchmark-specific handling. In that sense, SuiteEval is best understood as a framework for consolidating retrieval benchmarking practice around explicit suite definitions, corpus-aware execution, and reusable artifact management at a time when broad, out-of-domain evaluation has become standard for modern retrieval systems (Parry et al., 20 Feb 2026).