AbdoBench: Abduction & MRI Segmentation Benchmarks

• AbdoBench is a dual-context benchmark suite that evaluates symbolic abduction and MRI abdominal segmentation, enabling reproducible performance testing.
• In the abduction context, it employs finite domains, default–exception frameworks, and SMT solvers to verify rule validity and minimize abnormality costs.
• In the imaging context, it standardizes performance assessment of deep learning models using metrics like Dice and HD95 across heterogeneous MRI datasets.

In the supplied literature, AbdoBench appears in two distinct contexts. In community discussions around "ABD: Default Exception Abduction in Finite First Order Worlds," the label is used informally for the ABD benchmark suite—ABD-Full, ABD-Partial, and ABD-Skeptical—for default–exception abduction over finite first-order relational worlds [2602.18843]. In "Benchmarking of Deep Learning Methods for Generic MRI Multi-Organ Abdominal Segmentation," AbdoBench denotes a publicly released benchmark for generic abdominal multi-organ segmentation in MRI, including evaluation code, public datasets, and the ABDSynth baseline [2507.17971]. The term therefore requires contextual disambiguation.

1. Terminological scope

The two uses of AbdoBench differ in problem domain, output object, and evaluation protocol.

Usage of “AbdoBench”	Problem setting	Core benchmark artifacts
Informal alias for ABD	Default–exception abduction over finite first-order worlds	ABD-Full, ABD-Partial, ABD-Skeptical; dataset, task definitions, and evaluation protocol
Public MRI benchmark	Generic abdominal multi-organ segmentation in MRI	Three public datasets, evaluation code, and ABDSynth inference code and weights

For the symbolic benchmark, the paper itself consistently uses the name ABD and does not adopt the AbdoBench alias in the text; the alias informally refers to the ABD benchmark suite and its associated dataset, task definitions, and evaluation protocol [2602.18843]. For the imaging benchmark, AbdoBench is the repository and evaluation framework released with the MRI abdominal segmentation study [2507.17971].

This naming overlap is not merely cosmetic. In the ABD setting, the benchmark output is a single quantified first-order exception rule $\alpha(x)$; in the MRI setting, the benchmark output is a collection of segmentation scores such as Dice and HD95 computed for pretrained deep models on unseen datasets. A plausible implication is that citations to “AbdoBench” should be interpreted by task context rather than by name alone.

2. AbdoBench as ABD: finite-world default–exception abduction

ABD formalizes exception discovery in finite relational worlds. The signature is fixed as
$$
\Sigma = {P, Q, R, S, =},
$$
where $P(\cdot)$ and $Q(\cdot)$ are unary predicates, $R(\cdot,\cdot)$ and $S(\cdot,\cdot)$ are binary predicates, and equality is included. A world $W$ has a finite domain $D_W = {a_1,\dots,a_n}$. In ABD-Full, each world specifies complete interpretations for $P$, $Q$, $R$, and $S$, and any unlisted atom is false. In ABD-Partial and ABD-Skeptical, the world additionally specifies a set $\Omega_W$ of unknown ground atoms for $R$ and $S$ only; $P$ and $Q$ are always fully observed [2602.18843].

Each instance fixes a first-order theory $\Theta$ over $\Sigma \cup {Ab}$, where $Ab(\cdot)$ is a unary abnormality predicate. Default-like rules use the schema
$$
\forall x \, (Ante(x) \land \neg Ab(x) \rightarrow Cons(x)),
$$
and the model must define abnormalities by supplying a first-order formula $\alpha(x)$ such that
$$
\forall x \, (Ab(x) \leftrightarrow \alpha(x)).
$$
The hypothesis language is tightly controlled: $\alpha(x)$ is a single first-order formula with exactly one free variable $x$, written in a strict S-expression grammar using and, or, not, forall, and exists. Object constants are disallowed; only variables may appear. Each instance also enumerates AllowedAlphaPredicates and ForbiddenAlphaPredicates, with $Ab$ always forbidden and repaired-consequent predicates typically forbidden to preclude trivial shortcuts [2602.18843].

The benchmark distinguishes three observation regimes. In ABD-Full, validity on a world $W$ is
$$
W \models_{\mathrm{ABD}} \alpha \iff SAT(\Theta \land Facts(W) \land \forall x(Ab(x)\leftrightarrow \alpha(x))).
$$
The cost on a world is the number of elements labeled abnormal by $\alpha$, and instance cost sums this quantity over training worlds. In ABD-Partial, validity is existential over completions:
$$
W \models_{\exists comp} \alpha \iff \exists C:\; SAT(\Theta \land Facts(W^C) \land \forall x(Ab(x)\leftrightarrow \alpha(x))).
$$
Its parsimony objective uses the best-case abnormal count over satisfying completions. In ABD-Skeptical, validity is universal over completions:
$$
W \models_{\forall comp} \alpha \iff \forall C:\; SAT(\Theta \land Facts(W^C) \land \forall x(Ab(x)\leftrightarrow \alpha(x))),
$$
and the cost is the worst-case abnormal count over completions [2602.18843].

The lower-bound baselines use a free-Ab relaxation. In ABD-Full, $\mathrm{OptCost}(W)$ minimizes $|Ab|$ subject to satisfiability of $\Theta \land Facts(W)$ when $Ab \subseteq D_W$ is allowed to vary freely per world. The reported gap,
$$
Gap(\alpha)=Cost(\alpha;W_{tr})-\mathrm{OptCost}(W_{tr}),
$$
therefore conservatively overestimates suboptimality because the lower bound need not be definable by a single formula. The benchmark also reports gap vs. gold, where a planted rule $\alpha^\star$ serves as a generator anchor rather than as a guaranteed optimum [2602.18843].

3. ABD benchmark construction, verification, and empirical behavior

ABD is solver-checkable because all domains are finite and all quantifiers in $\Theta$ and $\alpha$ are grounded over $D_W$. In partial worlds, unknown atoms are modeled as free Boolean variables. ABD-Full validity is a plain SAT check over Z3. ABD-Partial validity reduces to SAT with unknowns as freely assignable Booleans. ABD-Skeptical validity is reduced to the absence of a counterexample completion: the negation of the grounded axiom conjunction is asserted with unknowns free, and UNSAT certifies skeptical validity. Costs and lower bounds are exact cardinalities, and Z3 Optimize is used to minimize or maximize abnormality counts under the validity constraints [2602.18843].

The practical regime is deliberately finite and exact. Signatures use arity $\leq 2$, and domains are 9–12 elements. The benchmark uses seven default theories, all of the form $\forall x(\phi(x)\land \neg Ab(x)\rightarrow \psi(x))$. T1–T5 appear in ABD-Full and ABD-Partial, while T1–T7 appear in ABD-Skeptical. Predicate scoping for $\alpha(x)$ is chosen per theory to preclude trivial repairs [2602.18843].

Benchmark composition is substantial but controlled. The suite contains 600 instances: 195 in ABD-Full with 9.0 average training worlds, 243 in ABD-Partial with 8.3 average training worlds, and 162 in ABD-Skeptical with 6.0 average training worlds. Each instance has five holdout worlds sampled from the same generator without adversarial filtering. World generation draws predicate truth rates from scenario-specific intervals, masks a fraction of $R$ and $S$ ground atoms as unknown in the partial regimes, and enforces acceptance filters so that abnormality is nontrivial and remains sparse, with abnormal rates kept at or below 20% [2602.18843].

To block shortcut hypotheses, the generator uses CEGIS-like competitor elimination. A pool of small competitor formulas with AST $\leq 15$, hand-curated plus mined from model outputs and simple mutants of the gold rule, is tested against the current worlds. If a competitor remains valid with cost close to, or better than, gold, an additional world is adversarially added to break it. This continues until competitors are eliminated or a world budget of 12–15 is reached, with averages of 6–9 [2602.18843].

Evaluation covers ten frontier LLMs: Opus-4.6, Grok4, Grok4.1-fast, Gemini-3.1-Pro, Gemini-3-Pro, GPT-5.2, DeepSeek-Reasoner, Kimi-K2-Thinking, GPT-4o, and Hermes4. Prompts are zero-shot, with a strict one-line JSON output containing the S-expression formula and a one-sentence description; there are no in-context examples, temperature is 0.1, and “thinking” modes are used when available with large budgets. Strong models achieve high training validity across scenarios: Opus-4.6 is reported at ~100%, Gemini-3.1-Pro at ~98–99%, and DeepSeek-Reasoner and Grok4.1-fast at ~95%. Parsimony remains a separator among valid predictions: normalized gap against the free-Ab lower bound is reported as approximately 1.05 for Opus-4.6, 1.30 for Gemini-3.1-Pro, 1.53 for Grok4.1-fast, and 1.56 for DeepSeek-Reasoner, in extra exceptions per world [2602.18843].

Holdout evaluation reveals distinct failure modes. Across all scenarios, holdout gaps exceed training gaps by about +1 extra exception per world on survivors, with $\Delta Gap \approx +0.9$ to $+1.0$ for most strong models. Holdout validity drops sharply, for example from 98.8% train-valid to 61.8% holdout-valid for Opus-4.6, and from 98.0% to 69.7% for Gemini-3.1-Pro. In ABD-Full and ABD-Partial, the dominant failure is parsimony inflation; in ABD-Skeptical, the dominant failure is validity brittleness, while survivors often exhibit smaller $\Delta Gap$ values. Theories with nested or universal consequents, such as T4 and T5, are particularly challenging, and in Skeptical T6 many models “beat gold,” indicating that gold rules may be conservative in worst-case semantics [2602.18843].

A qualitative example under Skeptical T1 illustrates the difference between robust and brittle repairs. The robust rule
$$
\alpha(x)=P(x)\land (\exists y\, R(x,y))\land (\forall z(\neg R(x,z)\lor P(z)))
$$
is reported for Opus-4.6 and remains stable because the universal guard controls worst-case completions. By contrast, the GPT-5.2 rule
$$
\alpha(x)=\exists y \, (R(x,y)\land P(y)\land \forall z(\neg R(x,z)\lor z=y))
$$
fits training worlds but fails on holdouts when multiple $R$-related $P$-elements occur. In ABD-Partial, a compact rule such as $\alpha(x)=\exists y(R(x,y)\land P(y))$ can achieve validity by “getting lucky” with completions, yet still incur a normalized gap of approximately 1.5 per world in the cited instance [2602.18843].

4. AbdoBench as an MRI abdominal segmentation benchmark

In the MRI segmentation literature, AbdoBench is a comprehensive benchmark for generic abdominal multi-organ segmentation in MRI. Its motivation is the contrast between abdominal CT, where signal is highly standardized, and abdominal MRI, where performance is impaired by large variability in acquisition parameters, lack of inherent intensity normalization, and the scarcity of large annotated training sets. The benchmark standardizes out-of-the-box evaluation across diverse MRI vendors, sequences, voxel resolutions, fields-of-view, and subject conditions, without retraining or sequence-specific tuning [2507.17971].

The benchmark evaluates four models—MRSegmentator, MRISegmentator-Abdomen, TotalSegmentator MRI, and ABDSynth—on three public MRI datasets not seen during training by any evaluated method: AMOS MRI, CHAOS MRI, and LiverHCCSeg. Together these datasets span three manufacturers (Philips, GE, and Siemens), five MRI sequences, and a range of healthy and pathological cases [2507.17971].

The datasets are heterogeneous by design. AMOS MRI contains 60 subjects from combined training and validation sets used as a single test cohort, with abdominal cancer and other abnormalities, 15 annotated abdominal organs, resolution ranges from [0.69×0.69×0.82] to [1.95×3.00×3.00] mm, and dimensions from [192×60×64] to [576×468×512]. CHAOS MRI contributes 20 subjects per sequence, for 60 volumes total, across T1 dual in-phase, T1 dual out-phase, and T2 SPIR, with consensus masks for liver, spleen, and kidneys. LiverHCCSeg provides 17 subjects with hepatocellular carcinoma scanned in T1 arterial phase, with liver segmentations from two independent raters [2507.17971].

Evaluation uses Dice and HD95, complemented by qualitative analysis and volume repeatability analysis across CHAOS sequences. Results are reported per organ and per sequence; macro/micro averaging and weighting are not applied. Jaccard index and ASSD are explicitly not part of the reported metrics. Statistical analysis uses Bonferroni-corrected Wilcoxon signed-rank tests and Friedman chi-square tests when appropriate. Generalization is assessed across sequences, vendors, voxel resolutions, and FOVs [2507.17971].

5. Benchmarked MRI methods and the ABDSynth baseline

The MRI benchmark compares three nnU-Net-based systems and one SynthSeg-derived synthetic-training baseline. MRSegmentator is nnU-Net-based and trained on 2,649 volumes across multiple modalities and sequences: 1,200 UK Biobank T1 Dixon sequences, 221 in-house MRI scans with kidney tumors, and 1,228 TotalSegmentator CT scans. It supports diverse T1 and T2 sequences, predicts 40 regions, uses 96×128×160 sliding-window patches, has 31M trainable parameters, and reports a mean inference time of 57.95 ± 53.15 s [2507.17971].

MRISegmentator-Abdomen is also nnU-Net-based. It is trained on 780 T1-weighted scans including pre-contrast and contrast-enhanced phases, with additional data from University Hospital Basel, predicts 62 regions, and is trained solely on T1 scans. Its inference uses 48×160×192 patches, the model has 31M parameters, and mean inference time is 98.19 ± 69.83 s. The version used is v1.0.0 (June 2024, Python 3.11.10) [2507.17971].

TotalSegmentator MRI extends TotalSegmentator to MRI with an nnU-Net backbone. It uses 1,088 MRI scans from the Imaging Data Commons plus TotalSegmentator CT data, predicts 59 regions, is designed to be sequence-independent, uses 112×128×160 patches, has 31M parameters, and reports 39.60 ± 10.34 s mean inference time. The version used is v2.2.0 (May 2024, Python 3.10.13) [2507.17971].

ABDSynth is the distinctive contribution of the benchmark. It is a supervised 3D U-Net identical to SynthSeg, configured for abdominal segmentation and trained on 128 CT segmentations from the TotalSegmentator training set, with no real images used. CT label maps are center-cropped or padded to 300×300×250 voxels at 1.5 mm isotropic resolution. The model predicts 33 regions. Synthetic MRI-like volumes are generated with a segmentation-conditioned Gaussian mixture model, with wide uniform-range parameter sampling for domain randomization. Training includes affine and non-linear spatial transforms, bias field corruption, contrast augmentation, noise injection, resolution modeling, EM-based label subdivision, and pose simulation specific to trunk MRI, with arms removed with 0.5 probability using 3D Slicer Sandbox. The loss is soft Dice loss. Training runs for 500,000 iterations on an Nvidia A100 40GB GPU, taking approximately two weeks on Jetstream2 resources. At inference, ABDSynth uses whole-volume inference without patching, has 13M parameters, and reports 21.17 ± 19.30 s mean inference time. Its weights and inference code are available in the AbdoBench repository [2507.17971].

The four methods differ not only in architecture lineage but also in supervision regime. The three nnU-Net systems were built through iterative pseudo-labeling and expert refinement procedures, whereas ABDSynth is trained purely from CT label maps using domain randomization. This suggests that the benchmark is testing both supervised MRI generalization and the viability of synthetic-training pipelines under complete absence of real MRI training images.

6. Empirical findings, limitations, and availability

The principal MRI finding is that MRSegmentator is the most consistent and generalizable model across datasets, sequences, and vendors. It achieves high Dice and low HD95 across organs and sequences and is statistically superior in many comparisons. MRISegmentator-Abdomen often attains high Dice on AMOS—for example, liver Dice 0.97 (0.02)—but exhibits very large HD95 values and extreme outliers, indicating implausible or spatially incoherent masks, and degrades on unseen CHAOS sequences. TotalSegmentator MRI is competitive overall, usually somewhat below MRSegmentator in mean accuracy but with better spatial coherence than MRISegmentator-Abdomen on unseen sequences. ABDSynth is slightly less accurate overall, yet remains competitive on high-contrast organs such as liver, spleen, and kidneys despite having seen no real images during training; it is weaker on small or deformable structures and can fail on more variable sequences [2507.17971].

Selected quantitative examples illustrate these trends. On AMOS, MRSegmentator leads on both kidneys and achieves liver Dice 0.96 (0.01) with HD95 2.87 (1.35), while MRISegmentator-Abdomen reaches liver Dice 0.97 (0.02) but with HD95 18.86 (40.15). On CHAOS T1 dual in-phase, MRSegmentator attains liver Dice 0.93 (0.01) with HD95 2.05 (0.41), whereas MRISegmentator-Abdomen drops to Dice 0.81 (0.12) and HD95 40.86 (25.42). On LiverHCCSeg, relative to Rater 1, MRSegmentator reaches liver Dice 0.93 (0.03) and HD95 4.93 (3.94), while mean inter-rater Dice is 0.95 and inter-rater HD is approximately 15.7 mm [2507.17971].

The MRI benchmark also identifies several caveats. Annotation conventions differ across datasets—for example, in CHAOS the ground truth includes the renal pelvis within kidneys, whereas automated models exclude it—so volumetric discrepancies may partly reflect protocol mismatch rather than model failure. Pathology stratification is limited by dataset characteristics, model coverage is incomplete, and Dice degrades more rapidly for small or deformable structures such as the pancreas, adrenals, gallbladder, and duodenum [2507.17971].

The symbolic ABD benchmark has its own explicit limitations. Small domains are necessary for exact SMT grounding, but this also makes extensional case-splitting feasible; accordingly, the paper reports size-conditioned metrics and holdouts to detect memorization. The free-Ab lower bound relaxes the single-formula constraint and therefore overestimates suboptimality. Planted gold rules are generator anchors rather than guaranteed optima, and models sometimes beat gold. Future work is described as tightening lower bounds, scaling domains with approximate verification while preserving solver-backed checks, integrating solver-in-the-loop training, and expanding theory and template diversity [2602.18843].

Availability differs across the two uses of the name. For the MRI benchmark, the repository is provided at https://github.com/deepakri201/AbdoBench, with evaluation code, datasets for benchmarking, and ABDSynth inference code and weights [2507.17971]. For the ABD benchmark, the paper states that code, data, and cached results will be released after the review period, and no persistent URL or DOI is provided in the paper body [2602.18843].

Taken together, the two benchmark traditions associated with the label AbdoBench occupy very different technical spaces. One is a rigorously finite, solver-verified benchmark for synthesizing sparse first-order exception rules under closed-world, existential-completion, and universal-completion semantics. The other is a reproducible benchmark for out-of-the-box MRI abdominal multi-organ segmentation across vendors, sequences, resolutions, and fields-of-view. Their shared name masks a substantial conceptual divide: one evaluates quantified relational abduction with exact SMT verification, while the other evaluates deep segmentation models with organ-wise Dice and HD95 on unseen MRI data.