PySODMetrics: Segmentation Metrics Framework

• PySODMetrics is a library that redefines binary segmentation evaluation by decomposing metrics into five fundamental stages.
• The framework clarifies evaluation assumptions by dissecting prediction representation, target extraction, matching, score computation, and reporting.
• Its protocol-oriented approach exposes common pitfalls like class imbalance and target structure bias, guiding more accurate segmentation performance measurement.

Searching arXiv for papers on PySODMetrics and related segmentation metric frameworks. PySODMetrics is a metric library associated with binary target segmentation, especially salient object detection and related tasks, and is most precisely understood through a protocol-oriented view of evaluation rather than as a collection of isolated formulas. In the framework introduced in "Beyond Pixel Overlap: A Framework for Decomposing Segmentation Evaluation Metrics," the target is the task-defined positive region to be segmented rather than a generic foreground object; it may be salient, camouflaged, transparent, glass-like, mirror-like, shadow-like, lesion-like, or defined by other application-specific semantics. Within this view, metrics implemented in the PySODMetrics ecosystem—including MAE, F-measure, IoU, Dice, S-measure, E-measure, weighted F-measure, and related variants—are compositions of modular design choices across five stages: prediction representation, target extraction, target matching, score computation, and metric reporting (Pang et al., 1 Jul 2026).

1. Conceptual scope

PySODMetrics is situated in a line of work that treats evaluation metrics for binary segmentation as protocol definitions. The central claim is that a metric is not merely “a formula,” but a pipeline whose stages encode assumptions about what prediction is evaluated, what target entity is extracted, how prediction and ground truth are aligned, what notion of quality is computed, and how the result is summarized. The framework was introduced not as a new scalar score, but as a method for interpreting metric behavior and for making the assumptions of existing protocols visible (Pang et al., 1 Jul 2026).

This perspective is consequential because two metrics may share most of their evaluation path and differ in only one stage, while other metrics may appear algebraically distinct yet operationally implement similar protocols. In this sense, PySODMetrics is not only a software artifact but also a concrete instantiation of a broader methodological position: metric choice defines what counts as success. The reference code for the framework is available at https://github.com/lartpang/PySODMetrics (Pang et al., 1 Jul 2026).

2. Five-stage decomposition

The framework decomposes representative binary segmentation metrics into five stages. These stages are not ancillary bookkeeping; they are the metric itself in operational form.

Stage	Function	Representative options
1. Prediction representation	Determines how model output enters evaluation	Soft grayscale map, fixed-threshold binary map, adaptive-threshold binary map, dynamic threshold sweep
2. Target extraction	Defines what the “target” is and at what granularity	Whole-map target, separated components, region, edge, skeleton
3. Target matching	Aligns prediction-side and GT-side elements	Pixelwise correspondence, threshold-based deterministic matching, globally optimized matching
4. Score computation	Computes a notion of quality	Pixel error, confusion-matrix scoring, importance weighting, structural similarity, enhanced alignment, multiscale AUC, correction effort, context relevance
5. Metric reporting	Summarizes the result	Raw scalar, raw curve, threshold-wise mean, threshold-wise maximum, threshold-wise AUC

The analytical value of this decomposition is that it makes visible where two nominally similar evaluations diverge. A threshold policy in Stage 1, a connected-component parsing rule in Stage 2, or a reporting convention in Stage 5 can alter what is being measured as much as a change in the score formula itself. The framework therefore treats reporting, thresholding, and target definition as constitutive parts of the metric, not as implementation details (Pang et al., 1 Jul 2026).

3. Prediction representation, target extraction, and matching

Stage 1 decides how the prediction enters evaluation. Four common representations are identified. A soft grayscale map keeps the prediction as a continuous probability map $P$ with values in $[0,1]$ and is used by metrics such as MAE, weighted $F_\beta$, $S_m$, $C_\beta$, and SI-MAE. A fixed-threshold binary map binarizes the prediction at a chosen threshold,

$B_t = \mathbf{1}[P > t],$

with $t=\frac{1}{2}$ given as a common choice, though explicitly characterized as a protocol choice rather than a universal truth. An adaptive-threshold binary map uses the image-dependent threshold

$t_{\mathrm{adp}} = \min\{2\bar{P},1\},$

where $\bar{P}$ is the mean prediction value. A dynamic threshold sweep evaluates the family

$\{B_t=\mathbf{1}[P \ge t]\mid t \in \mathcal{T}\},$

yielding a curve over thresholds rather than a single score. The practical effect is that Stage 1 determines whether evaluation emphasizes calibration or confidence quality, one binary operating point, image-adaptive behavior, or threshold robustness (Pang et al., 1 Jul 2026).

Stage 2 defines the target entity. The framework separates this into granularity and type. Granularity may be whole-map, in which the image-level mask is treated as one target, or separated components, in which connected components are parsed individually. Type may be region, edge, or skeleton. Whole-map region evaluation is associated with MAE, classical $[0,1]$0, IoU, Dice, BER, weighted $[0,1]$1, $[0,1]$2, $[0,1]$3, and $[0,1]$4. Separated-target region evaluation is used by SI-MAE, size-invariant $[0,1]$5, and hIoU. Edge and skeleton targets appear in MSIoU and HCE, respectively. The practical consequence is that Stage 2 changes what counts as a unit of evaluation, with direct consequences for fairness toward small objects, fragmentation sensitivity, boundary emphasis, and structural or topological concerns (Pang et al., 1 Jul 2026).

Stage 3 specifies how prediction-side and GT-side elements are matched. Pixelwise correspondence is the default in many metrics, including MAE, weighted $[0,1]$6, $[0,1]$7, $[0,1]$8, $[0,1]$9, IoU, Dice, MSIoU, and $F_\beta$0. For separated components, threshold-based deterministic matching may be used to accept a pair if overlap exceeds a threshold. More elaborate protocols use globally optimized matching. The framework uses hIoU as an example: it parses connected components from prediction and ground truth, uses overlap qualification, uses Hungarian assignment with centroid distance, and handles unmatched objects explicitly. This stage becomes critical in crowded scenes, merged objects, fragmented predictions, and small sparse targets, where object correspondence assumptions can dominate the interpretation of an overlap score (Pang et al., 1 Jul 2026).

4. Score computation families

Stage 4 formalizes what kind of quality is being computed. PySODMetrics-related evaluation spans several scoring families rather than one unified notion of correctness.

Pixel error is exemplified by MAE:

$F_\beta$1

This measures average soft disagreement without thresholding, with lower values preferred. The framework notes that it is weak when the target is tiny because background dominates the average.

Confusion-matrix scoring includes $F_\beta$2, IoU, Dice, OA, BER, PR curves, and ROC curves. With TP, FP, TN, and FN as primitives, the paper gives

$F_\beta$3

$F_\beta$4

$F_\beta$5

and explicitly notes that

$F_\beta$6

Dice is therefore a monotonic transform of IoU, with the same ranking and a different numeric scale. OA is

$F_\beta$7

and, for binary prediction and binary ground truth, $F_\beta$8. BER is

$F_\beta$9

which compensates for class imbalance by weighting target and background equally.

Importance weighting is represented by weighted $S_m$0. The method forms an error map $S_m$1, smooths it using a Gaussian kernel, weights errors by distance or importance, computes weighted precision and recall, and combines them with the $S_m$2 formula. The weighted confusion quantities are given as

$S_m$3

This makes boundary-adjacent, interior, and distant false positives contribute differently from one another.

Structural similarity is exemplified by $S_m$4:

$S_m$5

with a common default $S_m$6. Here $S_m$7 is object-aware similarity and $S_m$8 is region-aware similarity. The object-aware foreground and background terms are

$S_m$9

$C_\beta$0

and

$C_\beta$1

The region-aware component is

$C_\beta$2

The metric is therefore sensitive to coherent foreground response, background suppression, object layout, and block-wise structural preservation.

Enhanced alignment is represented by $C_\beta$3. The per-pixel alignment is

$C_\beta$4

and the score is

$C_\beta$5

This rewards local binary agreements that also align with the global centered structure of the ground truth.

Multiscale AUC appears in MSIoU, which changes the evaluated target to edge maps and computes scale-wise coverage:

$C_\beta$6

followed by

$C_\beta$7

This makes the metric sensitive to thin structures, fine boundary fragments, and multi-resolution edge support.

Correction effort is exemplified by HCE, which estimates manual correction work rather than overlap:

$C_\beta$8

The question being answered is therefore how many edits a human would need, not how many pixels are wrong.

Context relevance is represented by $C_\beta$9, designed for camouflaged object detection. The framework gives

$B_t = \mathbf{1}[P > t],$0

$B_t = \mathbf{1}[P > t],$1

with normalized forms

$B_t = \mathbf{1}[P > t],$2

and combination

$B_t = \mathbf{1}[P > t],$3

A camouflage-aware weighted version is also given:

$B_t = \mathbf{1}[P > t],$4

The practical effect is scene-conditioned difficulty: correct recovery is credited more when the target is visually similar to its surroundings (Pang et al., 1 Jul 2026).

5. Reporting conventions and task-aware protocol design

Stage 5 determines how the score is reported. The framework distinguishes raw scalar, raw curve, threshold-wise mean, threshold-wise maximum, and threshold-wise AUC. MAE is typically a raw scalar; IoU, Dice, OA, and BER are usually raw scalars at one threshold; PR and ROC are raw curves; AP and ROC-AUC are area summaries; and $B_t = \mathbf{1}[P > t],$5 and $B_t = \mathbf{1}[P > t],$6 are often reported as mean or maximum over thresholds, depending on protocol. The reporting rule changes the question being asked: a scalar at one threshold asks how good the chosen operating point is, a maximum asks for best possible threshold performance, a mean emphasizes average threshold robustness, and an AUC integrates performance over a curve (Pang et al., 1 Jul 2026).

The same framework is used to analyze limitations of earlier protocols and the motivations for later metrics. Older evaluation protocols mainly emphasized pixelwise error, overlap, or whole-image accuracy. The paper argues that these can miss structure, boundary quality, small-object fairness, component-level correspondence, manual correction cost, and context-dependent difficulty. Several explicit examples are given. OA can be dominated by background under extreme class imbalance, whereas BER balances target and background error. Standard overlap can hide failures on tiny objects, whereas SI-MAE, size-invariant $B_t = \mathbf{1}[P > t],$7, and hIoU address target-size bias. Ordinary IoU underweights thin edges, motivating MSIoU. Pixel counts may ignore object coherence, motivating $B_t = \mathbf{1}[P > t],$8 and $B_t = \mathbf{1}[P > t],$9. Overlap does not measure editing burden, motivating HCE. Ordinary metrics ignore scene difficulty in camouflage-like settings, motivating $t=\frac{1}{2}$0 (Pang et al., 1 Jul 2026).

A recurrent misconception addressed by the framework is that metric names alone suffice for comparison. The paper explicitly argues that benchmark reporting should document prediction representation, threshold policy, target entity, matching rule, score formula, empty-case convention, and reporting rule. This suggests that direct comparison across papers is unreliable when nominally identical metrics are instantiated under different protocol choices.

6. Relation to adjacent evaluation libraries and name disambiguation

PySODMetrics belongs to binary segmentation evaluation, not to the evaluation of Self-Organizing Maps. This distinction matters because name-based similarity can obscure a categorical methodological difference. The paper "A Survey and Implementation of Performance Metrics for Self-Organized Maps" describes SOMperf, an open-source Python library for evaluating SOMs through clustering validity indices and topographic indices, including quantization error, distortion, topographic error, combined error, neighborhood preservation or trustworthiness, topographic product, topographic function, and class scatter index. That paper explicitly does not mention a package called PySODMetrics and does not establish any relation between PySODMetrics and SOMperf (Forest et al., 2020).

The contrast also clarifies scope. SOMperf addresses two questions specific to SOMs: whether the map approximates the data distribution well, and whether it preserves neighborhood or topological relationships. PySODMetrics, by contrast, is framed around binary target segmentation and the decomposition of segmentation metrics into prediction representation, target extraction, target matching, score computation, and reporting. A plausible implication is that the two libraries occupy different evaluation regimes even though both are concerned with metric design, software implementation, and the interpretation of quantitative scores.