Stereo 3D Anomaly Detection (S3AD)

• Stereo-based 3D Anomaly Detection (S3AD) is a framework that identifies rare and unseen road objects using stereo vision and geometry-driven foreground detection.
• It decouples 2D and 3D supervision while integrating stereo disparity features to improve scale estimation and anomaly scoring.
• KITTI-AR benchmark augmentation and targeted evaluation metrics validate S3AD’s effectiveness in detecting out-of-distribution anomalies in autonomous driving.

Stereo-based 3D Anomaly Object Detection (S3AD) denotes a stereo-vision framework for autonomous driving that addresses the failure of closed-set 3D detectors on rare, unseen, or otherwise anomalous road objects by coupling category-agnostic foreground discovery with 3D localization and scale estimation. In the formulation introduced in "Stereo-based 3D Anomaly Object Detection for Autonomous Driving: A New Dataset and Baseline" (Mu et al., 12 Jul 2025), S3AD is both an algorithm and an evaluation setting: the algorithm decouples 2D and 3D supervision, adds a foreground confidence branch driven by stereo disparity features, and performs target-level anomaly scoring; the evaluation setting is anchored by KITTI-AR, a synthetic augmented-reality stereo benchmark designed to expose open-world failures that standard KITTI-style closed-set evaluation does not adequately reveal.

1. Safety problem and open-world formulation

S3AD is motivated by a safety problem that arises when 3D detection technology moves from enclosed highways to open conventional roads. In that setting, rare anomaly categories, out-of-distribution (OoD) objects never seen during training, and anomalous configurations of otherwise familiar categories can appear on the road. The core claim is that conventional closed-set 3D detectors are not reliable on such objects: they may either fail to produce a box or detect the object while assigning a wrong known class, such as mistaking an unusual object for a pedestrian or traffic cone (Mu et al., 12 Jul 2025).

The framework is explicitly situated in the Safety of the Intended Functionality (SOTIF) perspective. Under that perspective, perception failures need not originate from sensor malfunction; they can arise from distributional mismatch and limited model generalization. In autonomous driving, these failures are operationally significant because downstream planning requires object presence, distance, and scale. Pure 2D anomaly detection is therefore insufficient: it can indicate appearance-level irregularity, but it does not provide the 3D position and size needed for braking or avoidance (Mu et al., 12 Jul 2025).

The S3AD formulation identifies two factors that limit generalization of 3D detectors. The first is the coupled training of 2D and 3D tasks, in which semantic classification, 2D box regression, and 3D estimation are entangled. The second is insufficient diversity in the scale distribution of training samples, which biases size prediction for novel objects toward the statistics of known classes. This suggests that open-world 3D anomaly detection is not only a recognition problem but also a supervision-design and scale-generalization problem (Mu et al., 12 Jul 2025).

2. Core architecture and stereo-geometric representation

S3AD is built on a stereo 3D detector similar to YOLOStereo3D, but modifies the detection pipeline by inserting a binary foreground branch that uses stereo disparity features only. The stereo input is written as

$(x_L, x_R), \quad x \in \mathbb{R}^{W \times H}.$

The model contains a feature backbone $F_b(\cdot)$, stereo correlation fusion $F_s(\cdot)$, classification heads $H_{cls}(\cdot)$ and $H_{fg}(\cdot)$, regression heads $H_{reg2D}(\cdot)$ and $H_{reg3D}(\cdot)$, and a disparity reconstruction head $H_{dis}(\cdot)$ (Mu et al., 12 Jul 2025).

At anchor level, the shared stereo feature is

$f_s = F_s(f_L, f_R),$

with

$$C_{norm} = H_{cls}([f_s, f_L]), \quad Box_{2D} = H_{reg2D}([f_s, f_L]), \quad Box_{3D} = H_{reg3D}([f_s, f_L]).$$

Here $F_b(\cdot)$0 is the stereo correlation feature, $F_b(\cdot)$1 is the normal-class prediction over $F_b(\cdot)$2 known categories and $F_b(\cdot)$3 anchors, $F_b(\cdot)$4 contains $F_b(\cdot)$5, and $F_b(\cdot)$6 contains

$F_b(\cdot)$7

The defining novelty is the foreground head

$F_b(\cdot)$8

where $F_b(\cdot)$9 is a binary foreground confidence for each anchor. This branch uses only stereo disparity-related features $F_s(\cdot)$0, rather than left-view appearance features, because the framework assumes that foreground/background separation is more tightly tied to depth geometry than to object semantics. A disparity auxiliary task is added through

$F_s(\cdot)$1

which enforces stereo geometry learning through disparity reconstruction (Mu et al., 12 Jul 2025).

A plausible implication is that S3AD decomposes the open-world problem into a geometry-dominant foreground decision and a semantics-conditioned normal-class decision. That separation is central to its later anomaly-scoring rule.

3. Decoupled supervision, training objective, and anomaly scoring

A central design choice in S3AD is the decoupling of 2D and 3D supervision so that the detector can exploit additional 2D-only annotations even when 3D labels are unavailable. The method introduces two rounds of sampling. In 2D sampling, anchors with valid 2D boxes are used for 2D box regression, classification, and foreground classification. In 3D sampling, anchors with valid 3D boxes are used for depth, orientation, and 3D scale regression. If the annotated sets are $F_s(\cdot)$2 and $F_s(\cdot)$3, then after filtering and sampling the anchor sets become $F_s(\cdot)$4 and $F_s(\cdot)$5, with

$F_s(\cdot)$6

This design makes cheap 2D annotations from synthetic or open-vocabulary sources operationally useful for 3D anomaly detection (Mu et al., 12 Jul 2025).

The total training objective is

$F_s(\cdot)$7

The normal-category classification term is

$F_s(\cdot)$8

with focal weighting

$F_s(\cdot)$9

The foreground classification term is

$H_{cls}(\cdot)$0

where $H_{cls}(\cdot)$1 is constructed from original annotations and pseudo-labels from an open-vocabulary detector; GroundingDINO is explicitly cited for that role. The disparity supervision uses a stereo focal loss following YOLOStereo3D. A further detail is scale normalization: instead of the class-specific mean and standard deviation used by YOLOStereo3D, S3AD uses a unified mean and standard deviation across known classes for 3D box scale normalization, and applies the same values at test time for anomaly categories (Mu et al., 12 Jul 2025).

Anomaly scoring is also defined at target level rather than pixel level. S3AD compares against MSP,

$H_{cls}(\cdot)$2

and RbA,

$H_{cls}(\cdot)$3

where in RbA

$H_{cls}(\cdot)$4

The proposed foreground-aware scores are

$H_{cls}(\cdot)$5

and

$H_{cls}(\cdot)$6

where $H_{cls}(\cdot)$7 is the foreground confidence and, for S3AD,

$H_{cls}(\cdot)$8

The scoring logic is that an anchor with high foreground confidence but low confidence in every known class is likely anomalous, whereas an anchor that is not foreground should not be treated as a valid object candidate. The reported ablations state that RbAF performs best among the scoring variants (Mu et al., 12 Jul 2025).

4. KITTI-AR and the construction of a stereo anomaly benchmark

S3AD is accompanied by KITTI-AR, a synthetic augmented-reality stereo dataset built by rendering new 3D models onto real KITTI stereo backgrounds using Blender. The dataset extends KITTI by adding 97 new categories and totals about 6k stereo image pairs. Its purpose is to expose generalization failures that are difficult to study on standard KITTI, which mainly evaluates Car, Pedestrian, and Cyclist and is therefore not sufficient to test arbitrary anomaly detection (Mu et al., 12 Jul 2025).

The construction pipeline uses the original KITTI stereo background, 3D model rendering, and camera parameters inferred from KITTI annotations. The process includes six steps: setting virtual left/right stereo cameras; computing possible insertion regions so placed objects are not occluded by existing 3D objects; scaling and positioning 3D models using mesh statistics; converting objects into camera coordinates $H_{cls}(\cdot)$9; rendering under random lighting; and overlaying rendered foreground objects onto real KITTI stereo images. The resulting data preserve real-world background style while adding new object categories (Mu et al., 12 Jul 2025).

KITTI-AR is partitioned into two subsets with distinct roles. KITTI-AR-ExD is used as extra training data, contains 39 common categories, and is intended to address sparse sample distribution and enrich scale diversity; examples include sofas, refrigerators, and printers. KITTI-AR-OoD is used only for evaluation, contains 58 rare categories, and is excluded from training entirely in order to simulate zero-shot or open-world anomaly detection; examples include elephants, goats, fire hydrants, wheelchairs, and trash bins. The paper emphasizes that the OoD categories must be strictly held out from training for a fair anomaly evaluation (Mu et al., 12 Jul 2025).

This split operationalizes two different generalization questions. KITTI-AR-ExD tests whether broader geometric and scale coverage can improve 3D foreground detection. KITTI-AR-OoD tests whether that broader coverage transfers to previously unseen categories.

5. Experimental behavior, baselines, and ablation evidence

The experimental setup follows YOLOStereo3D’s framework and uses PyTorch, a single NVIDIA 4090D GPU (24GB), Adam, an initial learning rate of 0.0001, a cosine learning-rate schedule, batch size 8, top-100-pixel cropping, and resizing to $H_{fg}(\cdot)$0. Data augmentation is the same as YOLOStereo3D. Reported metrics include $H_{fg}(\cdot)$1, $H_{fg}(\cdot)$2, BEV AP in some tables, anomaly evaluation at 11 recall positions, and IoU thresholds such as $H_{fg}(\cdot)$3 and $H_{fg}(\cdot)$4. Baselines include Det-PointCLIP, Det-PointCLIPv2, OV-Uni3DETR, 3D-CLIP, OV-Mono3D, and Mask2Anomaly for qualitative 2D comparison (Mu et al., 12 Jul 2025).

On the KITTI single OoD setting, S3AD reports OoD AP 21.37, Car AP 80.93, and Cyclist AP 23.09, and is described as competitive with open-vocabulary or multimodal 3D methods while using only stereo vision. On KITTI-AR-OoD, the staged effect of additional supervision is more pronounced. Training only on original KITTI yields $H_{fg}(\cdot)$5. Adding KITTI-AR-ExD 2D annotations only raises this to $H_{fg}(\cdot)$6. Adding KITTI-AR-ExD 3D annotations yields $H_{fg}(\cdot)$7. The comparison with OV-Mono3D further isolates the stereo advantage on depth and scale: OV-Mono3D reports $H_{fg}(\cdot)$8 and $H_{fg}(\cdot)$9 at IoU $H_{reg2D}(\cdot)$0, while S3AD reports $H_{reg2D}(\cdot)$1 and $H_{reg2D}(\cdot)$2 (Mu et al., 12 Jul 2025).

The ablation studies support three claims. First, decoupling 2D and 3D supervision materially improves OoD 3D AP, and pseudo-labels from GroundingDINO are reported to be nearly as good as manual 2D ground truth. Second, foreground-aware anomaly scoring improves target-level anomaly detection: on KITTI-AR-OoD, MSP scores 56.96, MSPF 60.42, RbA 59.14, and RbAF 74.35. Third, stereo disparity features alone are effective for foreground detection and are more robust than relying on left-view appearance alone. In addition, as more KITTI-AR-ExD data are added, OoD performance steadily improves and the predicted 3D scale distribution becomes much closer to the actual distribution of unseen objects (Mu et al., 12 Jul 2025).

A common misconception is that anomaly detection in driving can be reduced to open-vocabulary 2D recognition. The reported KITTI-AR-OoD comparison indicates otherwise: very strong $H_{reg2D}(\cdot)$3 does not imply usable $H_{reg2D}(\cdot)$4. In the S3AD formulation, 3D anomaly detection is constrained as much by geometry and scale estimation as by foreground recall.

6. Subsequent developments, scope boundaries, and limitations

The immediate research trajectory after S3AD includes efforts to retain the open-set stereo objective while improving efficiency. DDStereo is explicitly framed as addressing the same core safety problem—detecting unknown or out-of-distribution obstacles from stereo images in autonomous driving—but with an end-to-end Transformer detector, a dual-decoder design, and shared object-level queries for target-level alignment. It is also explicitly positioned as a more efficient alternative to S3AD, noting that S3AD is accurate but too slow for real-time deployment. DDStereo reports 23.5 ms inference time, 62.65 GFLOPs, 19.6M parameters, and stronger open-set results on KITTI-AR-OoD, including OoD AP values of $H_{reg2D}(\cdot)$5 under R11 for 3D / BEV / 2D and $H_{reg2D}(\cdot)$6 under R40, while also citing S3AD at 75.4 ms runtime (Mu et al., 23 Jun 2026). This suggests that S3AD established a task definition and benchmark that later work could preserve while redesigning the detection architecture.

At the same time, not every 3D anomaly-detection framework belongs to the same problem class. BridgeNet, for example, is a unified multimodal industrial anomaly detector that converts aligned point clouds into depth images, processes RGB and depth with a shared 2D backbone, and targets anomaly detection and localization rather than 3D object bounding boxes, instance detection, or semantic 3D object detection. Its own limitations section states that it does not propose a stereo matching or depth-estimation network and that “stereo-based” in the strict vision sense is not its focus (Xiang et al., 25 Jul 2025). The contrast helps delimit S3AD’s scope: S3AD is specifically about stereo-based 3D object-level anomaly detection for driving, not multimodal defect localization in industrial inspection.

S3AD’s limitations are partly explicit and partly implicit. The original KITTI dataset is described as insufficient to fully expose OoD failures because some anomalies are too similar to normal categories. Monocular depth is described as weak for true 3D anomaly detection. The method also depends on synthetic augmentation for diverse scale coverage. Future work is directed toward faster, more accurate, real-time open-vocabulary 3D detection for arbitrary 3D objects in stereo settings (Mu et al., 12 Jul 2025). A plausible implication is that S3AD should be understood less as a final deployment architecture than as a benchmark-setting formulation: it defines how stereo geometry, decoupled supervision, and target-level anomaly scoring can be combined to evaluate open-world obstacle perception in 3D.