KITTI-AR: Augmented Stereo 3D Anomaly Dataset

• KITTI-AR is a synthetic augmented reality stereo dataset built on real KITTI imagery, designed for 3D anomaly detection in autonomous driving.
• It includes a dual-split design with KITTI-AR-ExD for extended in-distribution training and KITTI-AR-OoD for zero-shot anomaly evaluation.
• Coupled with the S3AD framework, the dataset employs a stereo-based pipeline with decoupled 2D/3D supervision and foreground detection to enhance anomaly scoring.

KITTI-AR is a synthetic augmented-reality stereo dataset derived from KITTI for stereo-based 3D anomaly object detection in autonomous driving. It was introduced together with the S3AD method to address a central limitation of closed-set 3D detection: models trained on a small number of known classes often misdetect or fail to detect rare, unknown, or out-of-distribution road objects. KITTI-AR preserves real KITTI background imagery and camera geometry while injecting rendered 3D foreground objects from many additional categories, thereby providing a benchmark for generalization to arbitrary foreground objects and for anomaly-oriented evaluation in stereo 3D detection (Mu et al., 12 Jul 2025).

1. Definition and problem setting

KITTI-AR extends KITTI by adding 97 new categories and producing about 6k stereo image pairs in total. Its purpose is to expose weaknesses of conventional 3D detectors under category shift and to support methods that can separate generic foreground detection from closed-set class prediction (Mu et al., 12 Jul 2025).

The dataset is motivated by two limitations identified in prior 3D anomaly detection resources. First, KITTI has only a small number of annotated categories, and standard 3D evaluation uses only a few of them. Second, realistic 3D anomaly benchmarks are scarce: some methods omit labels, some rely on simulation such as CARLA, and others use image editing. The paper argues that simulation can have a large domain gap, while category omission is constrained by the small original label set. KITTI-AR addresses both limitations by retaining the real KITTI background and enriching it with rendered 3D foreground objects and 3D annotations.

A plausible implication is that KITTI-AR occupies a specific intermediate position between purely simulated datasets and purely real closed-set datasets: it retains real-scene appearance and camera geometry while expanding category coverage through controlled augmentation. The paper presents this as a way to study anomaly detection under more realistic stereo conditions than fully synthetic alternatives.

2. Dataset composition and benchmark splits

KITTI-AR is divided into two subsets with different roles in training and evaluation. The structure is explicitly designed to distinguish between expanded in-distribution supervision and zero-shot anomaly evaluation (Mu et al., 12 Jul 2025).

Subset	Classes / size	Role
KITTI	8 classes, 3712 train, 3769 val	Base benchmark
KITTI-AR-ExD	8 + 39 classes, 4038 train	Extra training data
KITTI-AR-OoD	8 + 58 classes, 2347 val/test	OoD evaluation only

KITTI-AR-ExD contains 39 common categories and is used as extra training data. Its stated purpose is to address the sparse scale distribution problem in KITTI training data. These categories function as extra in-distribution or extended data for better generalization.

KITTI-AR-OoD contains 58 rare categories and is used only for testing/evaluation. These categories are excluded from training in order to simulate zero-shot / out-of-distribution (OoD) conditions in real autonomous driving.

This split design is central to the benchmark. The ExD subset tests whether broader category and scale coverage improves generic 3D foreground modeling, whereas the OoD subset tests whether those improvements transfer to anomaly detection on unseen categories. The paper’s organization therefore distinguishes ordinary data augmentation from anomaly-oriented evaluation in a controlled way.

3. Augmented-reality synthesis pipeline

KITTI-AR is constructed with a Blender-based rendering pipeline that inserts rendered 3D objects into real KITTI stereo images. The synthesis procedure is described as preserving the realistic background style of KITTI better than fully simulated datasets (Mu et al., 12 Jul 2025).

The rendering pipeline has six explicit stages. First, the method infers stereo camera parameters from KITTI, including left and right virtual camera positions, angles, field of view, and resolution. Second, it finds valid object placement regions based on existing KITTI 3D objects, with the constraint that a newly inserted object should not be occluded by existing 3D objects in the scene. Third, for each 3D model, the method computes scale $(w,h,l)$ and geometric center from mesh statistics, then scales the object to a plausible size and translates it to a target position and height. Fourth, the object is transformed into the camera coordinate system with $(x,y,z)$. Fifth, it is rendered under random lighting. Sixth, the rendered foreground is composited with the original KITTI stereo pair to create an augmented-reality stereo scene.

The paper emphasizes that this procedure is designed to preserve real KITTI background appearance and camera geometry while introducing many new foreground categories. This suggests that KITTI-AR is intended not merely as synthetic data generation, but as a stereo-consistent perturbation of a real benchmark. In that sense, it differs from datasets built by fully re-rendering scenes, such as "Virtual KITTI 2" (Cabon et al., 2020), which clone KITTI tracking sequences in a game engine and provide dense modalities including RGB, depth, segmentation, flow, and scene flow. KITTI-AR instead targets anomaly object detection by compositing rendered 3D objects into real stereo imagery.

4. S3AD: stereo-based 3D anomaly detection framework

KITTI-AR was introduced together with S3AD: Stereo-based 3D Anomaly Detection, a detector built on the stereo framework of YOLOStereo3D and specialized for anomaly-oriented 3D detection (Mu et al., 12 Jul 2025). The method is structured around a stereo backbone and correlation fusion module, a normal-category multi-class classification head $H_{cls}$, a category-agnostic foreground binary classification head $H_{fg}$, a 2D box regression head $H_{reg2D}$, a 3D box regression head $H_{reg3D}$, and a disparity reconstruction head $H_{dis}$.

The main stereo pipeline is written as

$f_s = F_s\big(f_L, f_R\big),$

$C_{norm} = H_{cls}\big([f_s, f_L]\big),$

$Box_{2D} = H_{reg2D}\big([f_s, f_L]\big),$

$(x,y,z)$0

where $(x,y,z)$1 and $(x,y,z)$2 are left and right image features, and $(x,y,z)$3 is the stereo correlation feature.

The paper uses the term decoupling in two distinct senses. First, it decouples normal-class classification from foreground detection. The standard classifier predicts the known classes, while a separate binary foreground branch predicts whether an anchor is object or background. Crucially, the foreground head uses only stereo disparity features:

$(x,y,z)$4

According to the paper, this is intended to learn generic objectness rather than appearance-specific class cues.

Second, it decouples 2D and 3D supervision. Classification and 2D box regression can be trained with 2D labels only, whereas 3D box regression is trained only where 3D annotations exist. A dual sampling strategy is used, with one round of sampling for 2D supervision and one round for 3D supervision. The authors explicitly argue that this reduces annotation cost and helps release generalization to arbitrary foreground objects.

The total training objective is

$(x,y,z)$5

The classification losses are focal-loss style BCE terms. The paper also notes that 3D scale normalization uses a unified mean and standard deviation for all known classes rather than class-specific normalization, because novel classes do not have an associated class mean or standard deviation at test time.

5. Anomaly scoring and evaluation protocol

The anomaly score in S3AD is based on the idea that an unknown object should receive strong foreground evidence but weak support for all known classes (Mu et al., 12 Jul 2025). Let $(x,y,z)$6 denote the foreground confidence and $(x,y,z)$7 the vector of normal-class logits or confidences. The paper defines two foreground-aware anomaly scores.

The first is MSPF:

$(x,y,z)$8

The second is RbAF:

$(x,y,z)$9

For comparison, the paper also gives non-foreground baselines:

$H_{cls}$0

$H_{cls}$1

The interpretation given in the paper is direct: $H_{cls}$2 indicates whether an anchor is a foreground object, while the second term measures how strongly that anchor belongs to any known class. A high anomaly score therefore corresponds to strong foreground evidence and weak evidence for all known classes. Among the scoring methods evaluated, the paper reports that RbAF is the best-performing method.

The benchmark is therefore not limited to ordinary 3D detection AP; it is designed to assess whether a model can maintain object-level 3D localization while identifying targets that fall outside the trained label space. This places KITTI-AR in the broader class of open-world or anomaly-oriented detection benchmarks, but with explicit stereo geometry and anchor-level scoring.

6. Empirical findings and benchmark significance

The reported experiments are framed as evidence that KITTI-AR is both challenging and diagnostically useful for stereo 3D anomaly detection (Mu et al., 12 Jul 2025). When a model is trained only on standard KITTI and evaluated on KITTI-AR-OoD, the resulting OoD AP$H_{cls}$3 is 9.09. The paper interprets this as evidence that standard KITTI training does not generalize well to novel object shapes and scales.

Adding KITTI-AR-ExD as extra training data improves performance substantially. With 2D labels only, OoD AP$H_{cls}$4 rises to about 21.05, and OoD AP$H_{cls}$5 rises to 87.89. With 3D labels, OoD AP$H_{cls}$6 reaches 74.35, and OoD AP$H_{cls}$7 reaches 90.06. The paper uses these results to argue that the augmented-reality data materially improves generalization rather than acting merely as a synthetic artifact.

The paper also compares S3AD to OV-Mono3D on KITTI-AR-OoD. At IoU $H_{cls}$8, OV-Mono3D obtains AP$H_{cls}$9 = 90.91 and AP$H_{fg}$0 = 0.64, while S3AD obtains AP$H_{fg}$1 = 90.06 and AP$H_{fg}$2 = 74.35. At IoU $H_{fg}$3, OV-Mono3D yields AP$H_{fg}$4 = 5.39, whereas S3AD yields AP$H_{fg}$5 = 87.04. The paper presents this as evidence that stereo-based depth estimation is much better suited to OoD 3D localization than monocular open-vocabulary detection in this setting.

The anomaly-scoring ablation further differentiates methods: MSP gives 56.96 OoD AP, MSPF gives 60.42, RbA gives 59.14, and RbAF gives 74.35. A related ablation shows that foreground detection with disparity features is stronger than using only left-image appearance: $H_{fg}$6 only: 72.73, $H_{fg}$7 only: 74.35, and $H_{fg}$8 for OoD AP$H_{fg}$9.

The sample-size study is also central. OoD AP$H_{reg2D}$0 increases from 9.09 with KITTI only to 24.45 with 10% of ExD, then to 35.08, 55.93, 66.36, 67.62, and finally 74.35 with full ExD. The paper interprets this as evidence that richer synthetic scale diversity improves 3D anomaly detection generalization. Qualitative findings further indicate that KITTI-trained models often miss anomalies or classify them as known categories with low confidence, and that a common failure mode is inaccurate size estimation even when approximate position remains correct.

7. Position within KITTI-derived research

KITTI-AR belongs to a broader family of KITTI-derived resources that use controlled augmentation or simulation to study robustness, transfer, and generalization, but its emphasis is narrower and more explicitly anomaly-centric (Mu et al., 12 Jul 2025). Unlike "Virtual KITTI 2" (Cabon et al., 2020), which is a re-rendered clone benchmark for tasks such as multi-object tracking, stereo matching, monocular depth estimation, camera pose estimation, and semantic segmentation, KITTI-AR focuses on stereo 3D anomaly object detection using real KITTI backgrounds plus inserted rendered 3D objects. Unlike "Synth It Like KITTI" (Marcus et al., 20 Feb 2025), which generates CARLA-based LiDAR data for synthetic-to-real transfer to KITTI, KITTI-AR is centered on stereo image pairs and zero-shot anomaly categories.

This positioning matters because the benchmark’s technical design is tied to its evaluation objective. Real KITTI backgrounds reduce the appearance gap relative to fully simulated environments; stereo consistency supports direct 3D localization; and the ExD/OoD split makes it possible to measure whether additional category and scale diversity improves open-set performance. A plausible implication is that KITTI-AR is best understood not simply as a data augmentation resource, but as an instrument for analyzing how closed-set stereo 3D detectors behave when generic foreground detection, class prediction, and 3D localization are no longer aligned.

In that sense, KITTI-AR formalizes a specific benchmark regime: training may use expanded but still controlled category diversity, while evaluation isolates rare categories never seen during training. The paper’s results indicate that this regime exposes substantial weaknesses in standard KITTI-trained models and provides a concrete testbed for foreground-aware, stereo-based anomaly detection.