Outlier-Aware Segmentation

• Outlier-aware segmentation is a framework that treats atypical observations as separate entities to improve overall segmentation by avoiding forced inlier fits.
• Empirical studies demonstrate that manual relabeling of failure cases yields higher Dice improvements compared to traditional inlier augmentation, highlighting its efficiency.
• Techniques combining uncertainty estimation with robust optimization allow joint modeling of inlier structures and explicit outlier detection across diverse domains.

Outlier-aware segmentation denotes a family of segmentation methods that explicitly account for samples, pixels, voxels, points, or temporal observations that violate the assumptions of the underlying model. Taken together, the literature suggests that the term functions as an umbrella over several related tasks: using failure cases to improve a segmentor, jointly segmenting known structure and unknown content, propagating distributional uncertainty through a dense predictor, and formulating segmentation objectives in which outliers are represented by dedicated latent variables or corruption terms (Xu et al., 2020, Popescu et al., 2021, Grcić et al., 2023, Katz et al., 2014). Across these settings, the central technical theme is the same: segmentation quality deteriorates when atypical observations are forced into the inlier model, and performance improves when such observations are either isolated, down-weighted, explicitly labeled, or modeled as a separate source of uncertainty.

1. Scope, terminology, and problem classes

The meaning of outlier is task-dependent. In abdominal CT, outliers are subjects exhibiting global failure of a baseline multi-organ segmentor, especially in smaller or more variable organs (Xu et al., 2020). In open-set semantic segmentation, outliers are pixels that do not belong to the training taxonomy and therefore should be assigned to an unknown or void class (Bevandić et al., 2019, Grcić et al., 2023, Nayal et al., 2024). In model-based face reconstruction, outliers are image regions that cannot be expressed by the parametric face model, such as occluders or make-up (Li et al., 2021). In sequential segmentation, outliers are observations that would otherwise induce spurious changepoints or distort piecewise-constant segment means (Katz et al., 2014, Fearnhead et al., 2016). In subspace segmentation, outliers are columns of the data matrix that do not lie in the union of low-dimensional subspaces (Liu et al., 2011). In robotic mapping, outliers include reflected near-ground LiDAR returns, ambiguous ground points for registration, moving objects in static map building, and unknown objects in large outdoor point clouds (Lim, 2024, Faulkner et al., 25 Aug 2025).

Setting	Outlier notion	Representative paper
Abdominal CT	Global failure cases of a baseline segmentor	(Xu et al., 2020)
Brain MRI OoD segmentation	Voxels with high distributional uncertainty away from the training manifold	(Popescu et al., 2021)
Road-scene semantic segmentation	Unknown or void pixels outside the known class set	(Bevandić et al., 2019, Grcić et al., 2023, Nayal et al., 2024)
Face reconstruction	Regions not expressible by a 3D Morphable Model	(Li et al., 2021)
Sequential segmentation	Samples absorbed by sparse outlier variables or bounded robust losses	(Katz et al., 2014, Fearnhead et al., 2016)
Subspace segmentation	Column-sparse corruptions in the data matrix	(Liu et al., 2011)
3D robotic mapping	Reflections, moving objects, or unknown point-cloud structures	(Lim, 2024, Faulkner et al., 25 Aug 2025)

This task-relative definition is important because outlier-aware segmentation is not restricted to open-set semantic segmentation. It also includes procedures in which outliers are the training signal used to refine a segmentor, as well as formulations in which segmentation and outlier detection are solved jointly.

2. Failure-driven data curation and active retraining

A direct formulation of outlier-aware segmentation appears in abdominal multi-organ CT, where the problem is cast as single-pass active learning through human quality assurance rather than iterative uncertainty sampling (Xu et al., 2020). A pre-trained 3D U-Net was run on 6,317 deidentified clinical portal venous phase CT scans, and predictions were triaged into success, usable, and global failure. Manual QA identified 817 failures; 47 of these outlier scans were manually relabeled for pancreas segmentation, while a matched comparison set consisted of 37 inliers whose baseline pancreas masks were used as labels.

The baseline model itself was trained on 220 abdomen CT scans: 100 manually segmented multi-organ volumes and 120 splenomegaly scans with manually labeled spleens. Preprocessing included body-part regression to localize the abdomen, soft-tissue windowing to $[-175, 275]$ HU, and down-sampling to $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$ with dimensions $160 \times 160 \times 64$. Augmented models were initialized from the baseline weights, trained with learning rate $0.0001$, and optimized with Dice loss,

$$\mathrm{Dice}(A,B)=\frac{2|A\cap B|}{|A|+|B|}, \qquad L_{\mathrm{Dice}}=1-\mathrm{Dice}.$$

The empirical result was that outlier labels had higher marginal value than inlier additions. Baseline pancreas Dice on the 47 outlier test scans was $0.4196$ with standard deviation $0.211$. Inlier-augmented models improved this to $0.4536$ at $X=5$ and to $0.4881$ at $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$0, whereas outlier-augmented models improved it to $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$1 at $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$2 and to $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$3 at $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$4. Manual labeling of outliers increased Dice by $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$5, compared with $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$6 for inliers, with $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$7 by two-tailed paired $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$8-test. The per-sample gains were likewise larger for targeted failures: approximately $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$9 per added sample for $160 \times 160 \times 64$0 outliers versus $160 \times 160 \times 64$1 for $160 \times 160 \times 64$2 inliers, and approximately $160 \times 160 \times 64$3 versus $160 \times 160 \times 64$4 when averaged over $160 \times 160 \times 64$5 and $160 \times 160 \times 64$6 added samples respectively (Xu et al., 2020).

The study reports that this improvement in single-organ pancreas performance was obtained without diminishing multi-organ performance or significantly increasing training time; convergence occurred in an average of $160 \times 160 \times 64$7 epochs, with some models reaching validation optima in $160 \times 160 \times 64$8 to $160 \times 160 \times 64$9 epochs and all in fewer than $0.0001$0. This suggests that, in data-limited clinical pipelines, outlier-aware segmentation may be implemented as targeted relabeling of baseline failure cases rather than indiscriminate dataset expansion.

3. Probabilistic OoD segmentation and likelihood-based unknown scoring

A second line of work treats outlier-aware segmentation as dense prediction under epistemic and distributional uncertainty. In a hierarchical convolutional Gaussian Process architecture, feature maps are modeled as Gaussian distributions at each spatial location, and the standard convolving GP is replaced by an affine operator on moments followed by a Distributional Gaussian Process layer operating in Wasserstein-2 space (Popescu et al., 2021). For Gaussian features $0.0001$1 and affine parameters $0.0001$2, the propagated distribution is

$0.0001$3

while the Wasserstein-2 distance between Gaussians is

$0.0001$4

The key OoD score is the variance of the final-layer “distributional uncertainty” component $0.0001$5, denoted $0.0001$6. On UK Biobank brain MRI, DistGP-Seg achieved Dice $0.0001$7 for CSF, $0.0001$8 for GM, and $0.0001$9 for WM, approaching deterministic U-Net performance of $$\mathrm{Dice}(A,B)=\frac{2|A\cap B|}{|A|+|B|}, \qquad L_{\mathrm{Dice}}=1-\mathrm{Dice}.$$0, $$\mathrm{Dice}(A,B)=\frac{2|A\cap B|}{|A|+|B|}, \qquad L_{\mathrm{Dice}}=1-\mathrm{Dice}.$$1, and $$\mathrm{Dice}(A,B)=\frac{2|A\cap B|}{|A|+|B|}, \qquad L_{\mathrm{Dice}}=1-\mathrm{Dice}.$$2. On BraTS tumor OoD detection, the Dice overlap between the OOD mask and tumor labels reached $$\mathrm{Dice}(A,B)=\frac{2|A\cap B|}{|A|+|B|}, \qquad L_{\mathrm{Dice}}=1-\mathrm{Dice}.$$3 at $$\mathrm{Dice}(A,B)=\frac{2|A\cap B|}{|A|+|B|}, \qquad L_{\mathrm{Dice}}=1-\mathrm{Dice}.$$4 and $$\mathrm{Dice}(A,B)=\frac{2|A\cap B|}{|A|+|B|}, \qquad L_{\mathrm{Dice}}=1-\mathrm{Dice}.$$5 at $$\mathrm{Dice}(A,B)=\frac{2|A\cap B|}{|A|+|B|}, \qquad L_{\mathrm{Dice}}=1-\mathrm{Dice}.$$6, exceeding OVA-DM, OVNNI, DUQ, VAE-LG, and AAE-LG (Popescu et al., 2021).

A more recent modular approach attaches a lightweight Unknown Estimation Module to frozen foundation-model features, avoiding any retraining of the original encoder or segmentation head (Nayal et al., 2024). The module learns a generic inlier distribution and an outlier distribution over projected features and combines them with the inlier segmentor’s confidence through a log-likelihood ratio,

$$\mathrm{Dice}(A,B)=\frac{2|A\cap B|}{|A|+|B|}, \qquad L_{\mathrm{Dice}}=1-\mathrm{Dice}.$$7

The UEM adds approximately $$\mathrm{Dice}(A,B)=\frac{2|A\cap B|}{|A|+|B|}, \qquad L_{\mathrm{Dice}}=1-\mathrm{Dice}.$$8 parameters, less than $$\mathrm{Dice}(A,B)=\frac{2|A\cap B|}{|A|+|B|}, \qquad L_{\mathrm{Dice}}=1-\mathrm{Dice}.$$9 of a $0.4196$0-parameter base model. On DINOv2 features, the likelihood-ratio formulation improved RoadAnomaly performance from AUROC $0.4196$1, AP $0.4196$2, FPR $0.4196$3 for inlier-density scoring alone to AUROC $0.4196$4, AP $0.4196$5, FPR $0.4196$6, and improved Fishyscapes LostAndFound from AUROC $0.4196$7, AP $0.4196$8, FPR $0.4196$9 to AUROC $0.211$0, AP $0.211$1, FPR $0.211$2. The abstract additionally reports a $0.211$3 average-precision improvement over the previous best method while leaving inlier performance unaffected; Cityscapes mIoU remains at the Stage-1 baseline, approximately $0.211$4 for DINOv2-b (Nayal et al., 2024).

An earlier shared-feature formulation used a ladder-style encoder-decoder with a semantic head and a binary outlier head trained in a single forward pass under domain shift (Bevandić et al., 2019). Negative supervision came from ImageNet-1k, including bounding-box-based object pasting into inlier road scenes. On WildDash validation, a DenseNet-169 two-head model achieved AP $0.211$5 on WD-LSUN, AP $0.211$6 on WD-Pascal, and mIoU $0.211$7 on WildDash, outperforming max-softmax, ODIN, and several alternative negative-training schemes on outlier patch detection (Bevandić et al., 2019). Architecturally, this line established the practical template that later work preserves: dense shared features, a dedicated unknown score, and explicit calibration of the trade-off between inlier segmentation and outlier rejection.

4. Mask-level aggregation and weakly supervised structured masking

A recurrent criticism of pixel-wise outlier scoring is that semantic borders generate high-entropy responses even for inlier pixels. Mask-level recognition addresses this by aggregating pixel evidence into non-exclusive region hypotheses and then redistributing uncertainty back to pixels (Grcić et al., 2023). In the Mask2Former-based formulation, masks are

$0.211$8

and closed-set per-pixel prediction is obtained by mask ensemble,

$0.211$9

The proposed dense anomaly score is ensemble over anomaly scores of mask-wide predictions,

$0.4536$0

which is lower than the anomaly of ensembled mask predictions and therefore reduces false positives at semantic borders where mask assignment confidences are low. Empirically, with negative data, M2F-EAM reached AP $0.4536$1 and FPR95 $0.4536$2 on SMIYC AnomalyTrack, AP $0.4536$3 and FPR95 $0.4536$4 on ObstacleTrack, and AP $0.4536$5 with FPR $0.4536$6 on Fishyscapes Static, while keeping Cityscapes validation mIoU at $0.4536$7 (Grcić et al., 2023).

A related but distinct use of structured masking appears in model-based face reconstruction. FOCUS jointly trains a face autoencoder and an outlier segmentation network so that reconstruction losses are gated by a learned inlier mask $0.4536$8 (Li et al., 2021). The reconstruction objective includes masked photometric loss,

$0.4536$9

together with perceptual, landmark, and coefficient regularization terms. The segmentation module is updated with a neighbor loss, a perceptual distance on masked images, an area loss, a preserve loss, and a binarization regularizer in an EM-type alternation. Because the target mask is not given, initialization uses residual thresholding with $X=5$0, and ambiguity between true outliers and systematic model misfit is addressed with a synthetic Misfit Prior. On the NoW test set, FOCUS achieved median $X=5$1 mm, mean $X=5$2 mm, std $X=5$3 mm; FOCUS-MP improved this to median $X=5$4 mm, mean $X=5$5 mm, std $X=5$6 mm. On the AR database, the unsupervised segmentation network obtained F1 $X=5$7 on unoccluded faces, $X=5$8 with glasses, and $X=5$9 with scarves (Li et al., 2021).

These two strands share a common structural claim: outlier-aware segmentation improves when the model reasons over coherent regions rather than independent pixels. In one case the regions are learned masks over semantic scene structure; in the other they are latent inlier regions induced by a generative renderer.

5. Explicit robust optimization in sequential and subspace segmentation

Long before open-set dense vision became prominent, segmentation with explicit outlier variables had been formalized as a convex optimization problem for sequential data. Outlier-Robust Convex Segmentation introduces piecewise-constant segment means $0.4881$0 and sparse outlier variables $0.4881$1 through

$0.4881$2

The fused penalty induces consecutive homogeneous segments, while the $0.4881$3 penalty assigns atypical samples to $0.4881$4 instead of letting them pull the segment means. The paper derives an exact convex solver and a top-down greedy method with complexity $0.4881$5, proves a consistency result for the two-segment noiseless setting, and reports strong robustness on speech tasks. On TIMIT biphone boundary detection, TD-ORCS achieved mean error $0.4881$6 ms in the clean setting and outperformed baselines at contamination levels around $0.4881$7 (Katz et al., 2014).

A related sequential formulation replaces explicit sparse outlier variables by bounded segment losses. For changepoint detection, the robust penalized objective is

$0.4881$8

with segment cost

$0.4881$9

The central theorem is that only bounded losses are robust to arbitrarily extreme outliers; unbounded losses can always justify singleton segments around sufficiently extreme observations. The biweight loss imposes a minimum segment length $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$00, yields consistency with the correct number of changepoints and $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$01 localization error under stated assumptions, and supports the exact dynamic-programming algorithm R-FPOP with online capability (Fearnhead et al., 2016). The method is demonstrated on well-log data, copy number variation, and wireless tampering.

In subspace segmentation, outliers appear as column-sparse corruptions rather than temporal deviations. Low-Rank Representation models the data matrix as $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$02 and solves

$(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$03

Under the paper’s visibility, relatively well-defined dictionary, incoherence, and bounded outlier-fraction conditions, any optimal solution exactly recovers the row space of $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$04 and the support of $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$05, hence enabling exact subspace segmentation and outlier detection (Liu et al., 2011). On the Yale-Caltech benchmark, LRR achieved segmentation accuracy $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$06 and outlier-detection AUC $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$07, outperforming PCA and RPCA variants.

Taken together, these optimization-based formulations show that outlier-aware segmentation need not be framed as uncertainty estimation. It can instead be posed as a joint inference problem in which segmentation variables and outlier variables are solved simultaneously.

6. 3D scenes, robotic mapping, and open-set point-cloud segmentation

In robotics, outlier-aware segmentation often appears first as preprocessing for downstream registration and mapping. A ground-segmentation method for long-term robotic mapping partitions LiDAR scans into polar bins, reuses plane estimates from neighboring bins in a near-to-far cascade, and validates local plane normals with an upright gravity constraint (Lim, 2024). Ground points are then removed because they are featureless for localization and induce ambiguous correspondences. The method runs at more than $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$08 Hz on a $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$09-channel LiDAR, and the paper reports that combining ground segmentation with graduated non-convexity enables registration to overcome up to $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$10–$(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$11 outliers. The same logic is extended to hierarchical multi-session SLAM, where GNC downweights false loop closures, and to instance-aware static map building, where objects in contact with the ground and inconsistent under temporal height-difference tests are rejected from the static map (Lim, 2024).

Open-set point-cloud segmentation adds an explicit anomaly mask. A reconstruction-based approach for large outdoor LiDAR scenes trains a scene autoencoder on default context and uses the discrepancy

$(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$12

as privileged input to an anomaly-aware segmentor built on a modified Point Mamba backbone (Faulkner et al., 25 Aug 2025). Geometry is reconstructed relative to octree leaf centers, and the network introduces a 3D-to-1D serialization strategy that partitions $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$13 into $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$14 blocks, orders them by a Z-order curve, sorts points by $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$15 within each block, and alternates the $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$16-sorting direction between adjacent blocks. On SemanticKITTI, Mamba AD with Rubik’s augmentation and reconstruction preprocessing trained on objects only achieved AUROC $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$17, AUPR $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$18, and mIoU $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$19. On ECLAIR, Mamba AD reached AUROC $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$20, AUPR $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$21 on known anomalies, AUROC $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$22, AUPR $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$23 on unknown anomalies, and mIoU $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$24; with reconstruction preprocessing, the unknown-anomaly AUPR increased to $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$25 while mIoU remained $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$26 (Faulkner et al., 25 Aug 2025).

These 3D results illustrate two distinct uses of segmentation under outlier pressure. One is rejection segmentation, in which nuisance structures such as ground or dynamic instances are removed before registration. The other is open-set segmentation, in which the anomaly mask is itself a target prediction.

7. Recurrent design trade-offs and limitations

Across the literature, several trade-offs recur. The first is supervision cost. Manual QA and relabeling of abdominal CT failures required approximately $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$27 hour per pancreas and introduced potential selection bias because outlier identification depended on human triage (Xu et al., 2020). FOCUS avoids dense mask annotation but pays for this with a complex EM-type training procedure and residual ambiguity between true outliers and systematic misfit regions such as eyebrows or specularities (Li et al., 2021).

A second trade-off is representation preservation versus outlier supervision. The likelihood-ratio UEM explicitly freezes the foundational encoder and inlier segmentor so that outlier training does not compromise the learned feature representation (Nayal et al., 2024). Earlier road-scene systems reached strong OoD detection through negative training, but the choice between a separate outlier head and outlier exposure inside the segmentation head altered semantic-border behavior, image-wide OoD response, and mIoU under hazard conditions (Bevandić et al., 2019). Mask-level recognition reduces border false positives, but its strongest results depend on negative data such as pasted ADE20K instances labeled as void (Grcić et al., 2023).

A third trade-off concerns formal robustness versus sensitivity. In changepoint detection, bounded losses are robust to arbitrarily extreme outliers, but the same boundedness induces a minimum detectable segment length controlled by $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$28 and $(2 \text{ mm}, 2 \text{ mm}, 6 \text{ mm})$29 (Fearnhead et al., 2016). In convex sequential segmentation, stronger outlier penalties preserve boundaries but may miss subtle local variation; in subspace segmentation, exact recovery requires visibility, incoherence, and a bounded outlier fraction (Katz et al., 2014, Liu et al., 2011).

A final recurrent issue is generalization to truly unknown anomalies. DistGP, LR-based unknown estimation, and Mamba-based reconstruction all improve unknown detection, yet the gap between performance on known or proxy anomalies and performance on genuinely unknown anomalies remains substantial (Popescu et al., 2021, Nayal et al., 2024, Faulkner et al., 25 Aug 2025). This suggests that outlier-aware segmentation is not a single solved problem but a collection of design strategies for preserving segmentation quality when atypical structure is unavoidable.