ABC-Bench: Multi-Domain Benchmark Suite

• ABC-Bench is a suite of rigorously defined benchmark frameworks that evaluates algorithms in community detection, fairness, autonomous coding, and Bayesian inference.
• It leverages tailored simulation methods such as scalable random graph generation, surrogate modeling, and task-specific pipelines to ensure precise and transparent evaluation.
• The frameworks offer measurable insights by quantifying performance trade-offs and methodological impacts across network science, machine learning, software engineering, and industrial inference.

The term "ABC-Bench" denotes several distinct, rigorously defined benchmark frameworks across different research domains. It appears most notably as (1) Artificial Benchmark for Community Detection (ABCD), (2) hierarchical Approximate Bayesian Computation test bench for industrial inference, (3) ABCFair—an adaptable fairness method benchmarking suite, and (4) ABC-Bench for agentic backend coding in repository-level software engineering. Each instantiation emphasizes precise, parameterized evaluation methodologies tailored to the structural and practical challenges of its field. This entry surveys the main ABC-Bench frameworks, articulating their formal design, methodological advances, and impact on benchmarking research.

1. Artificial Benchmark for Community Detection (ABCD, "ABC-Bench") Model

ABCD ("ABC-Bench") is a scalable random graph generator for evaluating community detection algorithms in complex networks (Kamiński et al., 2020). It supports:

• Power-law degree distribution ($P(w_i=k)\propto k^{-\gamma}$) and community-size distribution ($P(s_\ell=x)\propto x^{-\beta}$).
• A global mixing parameter $\xi\in[0,1]$ interpolating between pure communities ($\xi=0$) and random graph noise ($\xi=1$).
• Efficient construction: Assigns vertices to communities by admissibility constraints ($x_i=\lceil(1-\xi\phi)w_i\rceil\leq s_{f(i)}-1$) and creates edge sets via the Chung-Lu or configuration model.

The design avoids the analytic ambiguities of classic LFR benchmarks (where local mixing $\mu$ is non-transparent) and instead implements direct control over inter- and intra-community edge distributions:

• Each vertex $i$ has weights split as $y_i=(1-\xi)w_i$, $z_i=\xi w_i$.
• Final graph $G=G_0\cup G_1\cup\cdots\cup G_k$ where $G_0$ is the background (noise) graph, $G_\ell$ community-specific random graphs.

ABCD supports $O(n+m)$ scalability, linear modularity-phase transitions, and offers transparent mapping between desired community detectability and model parameters. It is empirically and theoretically validated as producing comparably structured synthetic graphs as LFR, but with improved speed, parameter clarity, and analytical tractability.

2. ABCD+o and ABCD+o²: Outliers and Overlapping Extensions

The ABCD+$o^2$ model generalizes ABCD to encompass outliers and overlapping communities (Barrett et al., 5 Jun 2025). Its construction comprises:

• Truncated power-law sampling for degrees $(d_i)$ and community sizes $(s_j)$.
• Explicit selection of outliers ($s_0$ nodes with all-degree assigned to the background), enforcing feasibility on high-degree nodes.
• Latent geometric layer in $\mathbb{R}^d$ allows for primary and secondary community assignments; average node membership is $\eta$, and degree-membership correlation is tuned via weighting parameter $\alpha$.
• Edge formation distributes stubs to both intra-community and background components, with loops and multi-edges pairwise rewired to produce simple graphs.

This framework better reflects empirical overlap-size and membership-count distributions found in real networks and supports scalable, tunable benchmarking for algorithms sensitive to overlaps and noise. Analytical properties such as modularity phase transitions and self-similarity of degree distributions are formally established.

3. ABCFair: Adaptable Benchmark for Fairness Methods

ABCFair (occasionally "ABC-Bench") presents a fully parameterized pipeline for benchmarking fairness mitigations in supervised learning (Defrance et al., 2024). Core components:

• Data formalization: $(\X, \Y, \R, \S, D)$ where $D=\{(x_i, s_i, y_i)\}_{i=1}^n$.
• Configuration tuple $$\Psi=(\psi_{\rm stage},\psi_{\rm sens},\psi_{\rm notion},\psi_{\rm out},\psi_{\rm \lambda},\psi_{\rm split})$$ controlling intervention stage (pre/in/post-processing), sensitive feature format (binary, categorical, intersectional), fairness notion (demographic parity, equalized odds, calibration), output mode (hard/soft), utility-fairness trade-off, and data splits.
• For each pipeline instance: outputs model $h_\Psi$, standard utility metrics (accuracy, AUROC), and fairness-violation $\Delta(h_\Psi)$ calculated for the specified fairness notion.

ABCFair enables systematic evaluation across interventions and parameter grids, capturing Pareto trade-offs in accuracy-fairness and providing statistical analysis with confidence ellipses. Empirical studies show differing fairness-accuracy relationships in biased versus unbiased label regimes and highlight the interaction effects of method, data granularity, and fairness notion.

4. ABC-Bench for Agentic Backend Coding

ABC-Bench in the context of backend software engineering is a benchmark suite for evaluating autonomous agentic coding capabilities in realistic, repository-level deployment scenarios (Yang et al., 16 Jan 2026). Central elements include:

• Task pool: 224 practical backend tasks curated from 2,000 open-source repositories, covering 8 languages and 19 frameworks, representing real-world API endpoints.
• Benchmark pipeline: Three-stage process of repository exploration (with augmented shell access), automated environment synthesis (including Dockerfile/docker-compose generation and dependency resolution), and task instantiation (patch masking and external API test validation).
• Performance metrics: Pass@1 (fraction of tasks solved on first attempt), environment-stage success $S_1$, and functional-stage success $S_2$. Results tabulated by task, language, and model (LLMs including open-source and proprietary).
• Bottleneck analysis: Environment configuration errors dominate failures, particularly for smaller models. Larger models, once capable of deployment, mainly fail in logic and API usage.

This framework isolates the challenges of full-stack service deployment that are not captured by prior static, code-edit benchmarks (HumanEval, MBPP, SWE-Bench).

ABC-Bench Variant	Domain	Key Mechanism
ABCD/ABCD+$o^2$	Network science	Power-law sampling + mixing
ABCFair	ML Fairness	Configurable pipeline
Agentic Backend Coding	Software Eng.	Full lifecycle execution
ABC Bench (industrial)	Bayes inference	Surrogate ABC test bench

5. Surrogate-Based ABC-Bench for Industrial Bayesian Inference

In industrial systems (e.g., electric motor parameter inference), "ABC-Bench" refers to the implementation of Approximate Bayesian Computation enhanced with hierarchical modeling and polynomial-chaos surrogates (John et al., 2021):

• Hierarchical Bayesian structure: Hyperparameters $\theta=(\mu_V, \mu_T, \sigma_V, \sigma_T)$ drive aleatoric parameters for each run; observed time series processed into low-dimensional summary statistics.
• Surrogate-driven ABC: Replaces expensive forward simulation (ODE solving) with polynomial chaos expansion (PCE)—yielding $2,000\times$ speedup per evaluation.
• Consistency: Under mild regularity (PCE convergence; summary statistics approximate sufficiency; $\epsilon$ tolerance scheduling), ABC posterior converges to the true posterior in Hellinger distance.
• Benchmarked on real test-bench data and synthetic simulations: SMC–ABC and ABC outperform MCMC in speed; SMC–ABC matches MCMC in bias and mean squared error at 20% of evaluations.

This suggests ABC-Bench is advantageous for inverse problems in engineering when analytic likelihoods are unavailable and simulator-based inference is required.

6. Comparative Analysis and Impact

Across instantiations, ABC-Bench frameworks share core principles: parameterized, scalable evaluation, explicit control over generative or operational regimes, and attention to the practical and theoretical bottlenecks in benchmarking. Their impact includes:

• Network science: Enabling the study of community detection under controlled, analytically transparent settings (ABCD, ABCD+$o^2$), with fast generation of large benchmarks for reproducible algorithm comparison.
• Fairness in ML: Allowing fine-grained, use-case adapted assessment of bias mitigation strategies, accurately capturing trade-offs and ensuring comparability.
• Autonomous software engineering: Establishing benchmarks reflective of production complexity—deployment, orchestration, and live API testing—thereby revealing LLM agent weaknesses not measurable via isolated code generation.
• Bayesian inference: Providing efficient methods for high-dimensional, simulator-based inverse problems in industrial contexts.

A plausible implication is that ABC-Bench methodologies, by closing the gap between isolated and holistic evaluation, will drive the refinement of algorithmic and agentic approaches in their respective fields and set new standards for benchmark realism and extensibility.