RICO: Realistic Incremental Object Detection Benchmarks

• The paper introduces two benchmark suites—D-RICO and EC-RICO—for evaluating sequential object detection under realistic domain and sensor shifts.
• It employs heterogeneous datasets with diverse environmental, sensor, and annotation conditions to expose challenges in continual learning.
• Empirical results show that replay methods partially recover performance, highlighting the critical stability–plasticity trade-off in incremental learning.

Searching arXiv for the RICO benchmark paper and closely related incremental object detection papers. Realistic Incremental Object Detection Benchmarks (RICO) denotes two benchmark suites for incremental learning in object detection: Domain RICO (D-RICO) and Expanding-Classes RICO (EC-RICO). They were introduced to evaluate sequential detector training under conditions that include domain shifts, heterogeneous sensors, varying weather and time-of-day conditions, perspective changes, and inconsistent labeling policies across datasets, rather than the synthetic and simplified settings common in earlier evaluations. In formal terms, RICO studies a sequence of tasks $T_t$ with dataset $D_t=(X_t,Y_t)$, where $X_t=\{x_i^t\}_{i=1\ldots n_t}$ are images and $Y_t=\{y_i^t\}_{i=1\ldots n_t}$ are annotations $y_i^t=\{(c_{i,j}^t,b_{i,j}^t)\}_{j=1\ldots m_i^t}$, and the detector after learning up to task $t$ is denoted $M_t$ (Neuwirth-Trapp et al., 19 Aug 2025).

1. Problem setting and research context

Incremental Learning (IL) trains models sequentially on new data without full retraining, and in object detection it must balance adaptation to new data against retention of previously acquired knowledge. RICO was proposed from the premise that existing evaluations often rely on synthetic, simplified benchmarks and therefore obscure real-world IL performance; its purpose is to expose failure modes that arise when incremental detection is confronted with realistic heterogeneity in domains, sensors, and annotation conventions (Neuwirth-Trapp et al., 19 Aug 2025).

Earlier work had already distinguished among category shift, domain shift, and joint domain-plus-category shift in incremental object detection. In particular, "Multi-Task Incremental Learning for Object Detection" formalized three scenarios across seven datasets and reported that domain gaps have smaller negative impact on incremental detection, while category differences are problematic (Liu et al., 2020). RICO operates in the same general problem family, but it organizes evaluation around longer benchmark sequences and explicitly incorporates shifts in camera type, weather, time of day, indoor versus outdoor scenes, real versus synthetic data, and labeling policy differences (Neuwirth-Trapp et al., 19 Aug 2025).

A nearby but distinct line of work arises in open-world or open-class detection. "YOLOOC: YOLO-based Open-Class Incremental Object Detection with Novel Class Discovery" constructed a benchmark in which novel classes are only encountered at the inference stage and proposed a detector for that setup (Wan et al., 2024). RICO, by contrast, is centered on incremental object detection under realistic domain changes and, in EC-RICO, supervised class expansion with label masking for future classes (Neuwirth-Trapp et al., 19 Aug 2025).

2. Benchmark definitions and sequence design

RICO consists of two benchmarks with different incremental-learning regimes. D-RICO is a domain-incremental learning benchmark with a fixed label space of three super-classes: person for all pedestrians, bicycle for all bicycles with rider merged, and vehicle for all non-bicycle motor vehicles. It contains 15 tasks $T_1\ldots T_{15}$, each drawn from a distinct dataset source except that one dataset is used twice with different splits, for a total of 14 independent public datasets. Each task has 2,782 train, 423 val, and 1,399 test images. The fixed class set is $$C_t=\{\text{person},\text{bicycle},\text{vehicle}\}$$, although some tasks restrict to subsets for complexity (Neuwirth-Trapp et al., 19 Aug 2025).

EC-RICO is defined as expanding-classes domain-incremental learning (EC-DIL). At each incremental step $t$, one new class is introduced into $D_t=(X_t,Y_t)$0 while the domain also shifts. The sequence has 8 tasks $D_t=(X_t,Y_t)$1, each from a distinct dataset among the same 14 sources. Its class expansion strategy is explicitly staged: $D_t=(X_t,Y_t)$2; $D_t=(X_t,Y_t)$3 adds car; $D_t=(X_t,Y_t)$4 adds bicycle; $D_t=(X_t,Y_t)$5 adds motorcycle; subsequent steps add truck, bus, traffic light, and street sign, with $D_t=(X_t,Y_t)$6 adding street sign. Each EC-RICO task has 3,040 train, 511 val, and 1,417 test images. Only labels for classes in $D_t=(X_t,Y_t)$7 are supervised, and future-class objects appear unlabeled (Neuwirth-Trapp et al., 19 Aug 2025).

The task ordering in D-RICO is randomized under constraints: “daytime” (nuImages) is first, “gated” and “inclement” are not consecutive, and no two similar sensors appear back-to-back. This sequence construction is meant to maximize sensor and environmental diversity. A plausible implication is that the benchmark is designed to prevent trivial curriculum effects in which adjacent tasks are excessively similar.

Benchmark	Sequence structure	Label-space behavior
D-RICO	15 tasks from 14 public datasets; 2,782 train / 423 val / 1,399 test per task	Fixed $D_t=(X_t,Y_t)$8
EC-RICO	8 tasks from distinct datasets among the same 14; 3,040 train / 511 val / 1,417 test per task	One new class per step, with future classes unlabeled

3. Source datasets, modality diversity, and annotation heterogeneity

The two RICO benchmarks are built from 14 diverse datasets spanning real and synthetic domains, multiple imaging modalities, and differing annotation conventions. The source set includes vehicle-mounted RGB data such as nuImages (Daytime) and BDD100K (Nighttime); thermal data such as FLIR Thermal; gated active-NIR data from DENSE in “Gated” and “Inclement” variants; fisheye data from FishEye8K, WoodScape, LOAF, and TIMo; drone RGB data from VisDrone; fused RGB-thermal data from SMOD; simulated or game-generated data from SHIFT, Sim10K, and Synscapes; and event-plus-RGB data from DSEC (Neuwirth-Trapp et al., 19 Aug 2025).

The variation dimensions are explicit: weather, time of day, indoor versus outdoor, urban versus rural density, real versus synthetic generation, perspective changes, and sensor differences including fisheye, thermal, event, drone, and gated active-NIR. RICO also encodes substantial label-policy heterogeneity. The benchmark description notes, among other examples, tight visible boxes versus amodal boxes, riders merged into bicycles or labeled separately and then merged, ignore regions, group annotations, rotated-to-axis-aligned boxes, boxes derived from segmentation masks, excluded small or occluded objects, and event overlays requiring box adjustment (Neuwirth-Trapp et al., 19 Aug 2025).

This annotation heterogeneity is central rather than incidental. The analysis associates contradictory labeling policies with false negatives and false positives, giving examples such as bicycle racks, split bus segments, and separate rider labels. It also identifies inconsistent treatment of small objects and occluded instances as a source of evaluation mismatch across tasks (Neuwirth-Trapp et al., 19 Aug 2025). This suggests that RICO is not only a domain-shift benchmark but also a benchmark for cross-dataset supervision inconsistency.

Dataset	Task role(s)	Sensor / annotation cue
nuImages (Daytime)	D task 1; EC task 3	Vehicle-mounted RGB; tight visible boxes; riders inside
FLIR Thermal	D task 2; EC task 7	Thermal; separate rider and bike merged
FishEye8K (Fisheye Fix)	D task 3; EC task 4	4-cam fisheye RGB; bicycle and motorcycle conflation
VisDrone (Drone)	D task 4; EC task 6	Drone RGB; amodal boxes; ignore regions
SHIFT (Simulation)	D task 5; EC task 5	Simulated RGB; reference policy
WoodScape (Fisheye Car)	D task 6; EC task 1	Vehicle fisheye RGB; boxes from seg masks
SMOD	D only	Fused RGB; bike labeled as bicycle
Sim10K	D only	GTA V RGB; excludes rider from motorbike
BDD100K (Nighttime)	D task 9; EC task 8	Vehicle-mounted RGB; separate rider merged
LOAF	D only	Indoor fisheye RGB; rotated to axis-aligned boxes
DENSE (Gated)	D task 11; EC task 2	Gated active-NIR; group labels and small objects removed
Synscapes	D only	Synthetic RGB; boxes from seg; occluded removed
TIMo	D only	Thermal fisheye; log-signal normalized; 1/6 frames kept
DENSE (Inclement)	D only	Gated active-NIR; fog, rain, snow; small/group removed
DSEC	D only	Event+RGB; event overlaid on RGB; boxes adjusted

4. Evaluation protocol and experimental configuration

RICO adopts COCO-style mean Average Precision averaged over IoU thresholds from .50 to .95 in steps of .05:

$D_t=(X_t,Y_t)$9

where $X_t=\{x_i^t\}_{i=1\ldots n_t}$0 is the area under the precision–recall curve for class $X_t=\{x_i^t\}_{i=1\ldots n_t}$1 (Neuwirth-Trapp et al., 19 Aug 2025).

The benchmark then defines incremental-learning metrics on $X_t=\{x_i^t\}_{i=1\ldots n_t}$2, the mAP on the test set of task $X_t=\{x_i^t\}_{i=1\ldots n_t}$3 after finishing task $X_t=\{x_i^t\}_{i=1\ldots n_t}$4 for $X_t=\{x_i^t\}_{i=1\ldots n_t}$5. Average Incremental Accuracy (AA) is

$X_t=\{x_i^t\}_{i=1\ldots n_t}$6

The Forgetting Measure (FM) is

$X_t=\{x_i^t\}_{i=1\ldots n_t}$7

Plasticity is characterized by Forward Transfer (FWT) and Intransigence Measure (IM). With $X_t=\{x_i^t\}_{i=1\ldots n_t}$8 denoting the mAP of an isolated model trained on $X_t=\{x_i^t\}_{i=1\ldots n_t}$9 alone, and $Y_t=\{y_i^t\}_{i=1\ldots n_t}$0 the mAP of a joint model trained on $Y_t=\{y_i^t\}_{i=1\ldots n_t}$1, the benchmark uses

$Y_t=\{y_i^t\}_{i=1\ldots n_t}$2

and

$Y_t=\{y_i^t\}_{i=1\ldots n_t}$3

The protocol states that higher FM implies more forgetting and higher IM implies greater plasticity deficit relative to joint training. All experiments are run with three random seeds and reported as mean $Y_t=\{y_i^t\}_{i=1\ldots n_t}$4 standard deviation; final-task aggregates are abbreviated AA$Y_t=\{y_i^t\}_{i=1\ldots n_t}$5AA$Y_t=\{y_i^t\}_{i=1\ldots n_t}$6, FM$Y_t=\{y_i^t\}_{i=1\ldots n_t}$7FM$Y_t=\{y_i^t\}_{i=1\ldots n_t}$8, FWT$Y_t=\{y_i^t\}_{i=1\ldots n_t}$9FWT$y_i^t=\{(c_{i,j}^t,b_{i,j}^t)\}_{j=1\ldots m_i^t}$0, and IM$y_i^t=\{(c_{i,j}^t,b_{i,j}^t)\}_{j=1\ldots m_i^t}$1IM$y_i^t=\{(c_{i,j}^t,b_{i,j}^t)\}_{j=1\ldots m_i^t}$2 (Neuwirth-Trapp et al., 19 Aug 2025).

The detector architecture uses an EVA-02-L Vision Transformer backbone, pre-finetuned on Objects365 and COCO, with a Cascade Faster R-CNN head in Detectron2 with IL modifications; the backbone is frozen by default. General training hyperparameters are: input size 1536$y_i^t=\{(c_{i,j}^t,b_{i,j}^t)\}_{j=1\ldots m_i^t}$31536, batch size 20, 700 iterations per task, AdamW with 0.1 warm-up and cosine learning-rate scheduling, base learning rate 0.001, end learning rate 0, and augmentations including random flip, scale $y_i^t=\{(c_{i,j}^t,b_{i,j}^t)\}_{j=1\ldots m_i^t}$4, fixed-size crop, brightness/contrast/saturation jitter, and lighting noise (Neuwirth-Trapp et al., 19 Aug 2025).

The compared incremental-learning methods are Naïve Fine-Tuning (FT); replay with 1%, 10%, or 25% of previous tasks’ images in a growing buffer; ABR with mixup and mosaic replay and distillation on ROI features and class logits from the teacher’s final cascade stage; Meta-ILOD with per-layer warp matrices $y_i^t=\{(c_{i,j}^t,b_{i,j}^t)\}_{j=1\ldots m_i^t}$5 that precondition gradients; BPF with pseudo-labels from a frozen previous model, disabled in DIL, plus distillation on logits and boxes; and LDB with task-specific domain bias terms and output heads, with task ID predicted during inference by a nearest-mean-classifier on global image features (Neuwirth-Trapp et al., 19 Aug 2025).

RICO also measures task affinity (TA), adapted from Taskonomy, by fine-tuning only the head of $y_i^t=\{(c_{i,j}^t,b_{i,j}^t)\}_{j=1\ldots m_i^t}$6 on $y_i^t=\{(c_{i,j}^t,b_{i,j}^t)\}_{j=1\ldots m_i^t}$7 and computing

$y_i^t=\{(c_{i,j}^t,b_{i,j}^t)\}_{j=1\ldots m_i^t}$8

where $y_i^t=\{(c_{i,j}^t,b_{i,j}^t)\}_{j=1\ldots m_i^t}$9 is the performance of $t$0 adapted to task $t$1 and $t$2 is scratch performance (Neuwirth-Trapp et al., 19 Aug 2025).

5. Reported performance and empirical patterns

The final metrics show that all evaluated IL methods underperform both joint training and individual training, and that replay constitutes the strongest baseline under both D-RICO and EC-RICO. On D-RICO, Joint Training reaches AA $t$3, Individual Train reaches $t$4, Naïve FT reaches $t$5 with FM $t$6, and Replay 25% reaches AA $t$7 with FM $t$8. On EC-RICO, Joint Training reaches AA $t$9, Individual Train $M_t$0, Naïve FT $M_t$1 with FM $M_t$2, and Replay 1% yields the highest reported AA among incremental methods at $M_t$3 (Neuwirth-Trapp et al., 19 Aug 2025).

Method	D-RICO	EC-RICO
Joint Training	AA 43.75±0.03	AA 38.46±0.06
Individual Train	AA 49.37±0.13	AA 45.54±0.01
Naïve FT	AA 30.60±0.35; FM 19.43±0.30; FWT −0.63±0.08; IM 4.95±0.07	AA 37.51±0.10; FM 8.62±0.05; FWT −0.23±0.10; IM 6.52±0.06
Replay 1%	AA 40.20±0.19; FM 8.91±0.26; FWT −0.79±0.15; IM 4.74±0.13	AA 38.09±0.06; FM 5.83±0.23; FWT −2.17±0.16; IM 4.66±0.16
Replay 10%	AA 43.06±0.14; FM 4.40±0.10; FWT −2.23±0.07; IM 3.39±0.07	AA 37.20±0.10; FM 3.82±0.12; FWT −5.24±0.02; IM 2.02±0.01
Replay 25%	AA 43.43±0.58; FM 3.35±0.64; FWT −2.88±0.06; IM 2.77±0.05	AA 37.53±0.35; FM 2.21±0.30; FWT −6.44±0.14; IM 0.93±0.10
ABR	AA 32.81±0.66; FM 15.26±0.71	AA 38.04±0.11; FM 7.05±0.22
Meta-ILOD	AA 38.51±0.17; FM 9.26±0.25	AA 36.99±0.16; FM 6.42±0.25
BPF	AA 30.74±1.00; FM 19.13±1.04	AA 37.32±0.13; FM 8.07±0.12
LDB	AA 42.49±0.17; FM 1.21±0.13	AA 28.88±0.25; FM 2.16±0.14

Several empirical conclusions are stated explicitly. All incremental methods underperform individual training, with a reported 19% AA drop for Naïve FT on D-RICO. Even small 1% replay recovers 10% AA on D-RICO and surpasses most state-of-the-art IL methods. Replay 25% nearly matches joint training on D-RICO but still lags individual training by about 6%. Distillation-based methods—ABR, Meta-ILOD, and BPF—yield only marginal improvements over Naïve FT, and LDB achieves minimal forgetting but very low plasticity, with its model extension described as insufficient for EC-RICO (Neuwirth-Trapp et al., 19 Aug 2025).

The analysis attributes much of the difficulty to weak teachers and the stability–plasticity dilemma. Average next-task mAP before new training is approximately 25–29%, with Meta-ILOD best at 28.70% on D-RICO, and the benchmark argues that this low teacher performance explains distillation failure. The reported curves show that no method achieves both low FM and high FWT simultaneously. Task-order sensitivity is substantial for Naïve FT, whose AA varies by $M_t$4 points across five designed orders, whereas Replay 10% is highly robust to order with AA $M_t$5 and FM $M_t$6 (Neuwirth-Trapp et al., 19 Aug 2025).

The benchmark also reports a strong negative correlation between FM and AA. It identifies a 6% gap between joint and individual training on D-RICO and a 7% gap on EC-RICO, and heuristically attributes this to weak teachers in distillation, single models’ inability to manage diverse tasks, and insufficient plasticity. A plausible implication is that RICO exposes limits not only of existing continual-learning algorithms but also of the single-model design assumption under long heterogeneous task sequences.

6. Relation to adjacent formulations and open research directions

RICO occupies a specific position within the broader landscape of continual detection benchmarks. Relative to earlier multi-task incremental detection work, it replaces short paired scenarios with longer sequences and a systematically heterogeneous source pool. Relative to open-world or open-class benchmarks, it does not focus on discovery of classes that are only encountered at inference time; instead, its realism derives from cross-domain sequential learning and, in EC-RICO, class growth under partial supervision through label masking (Liu et al., 2020).

This distinction matters because superficially similar benchmark labels can conceal different problem definitions. In YOLOOC, the central challenge is to detect novel classes and then incrementally learn them without forgetting previously known classes, under a setup where novel classes are only encountered at the inference stage (Wan et al., 2024). In EC-RICO, future-class objects appear unlabeled until their class enters the current supervised label set (Neuwirth-Trapp et al., 19 Aug 2025). These are complementary rather than interchangeable formulations of realism.

The open directions proposed for RICO are correspondingly broad: efficient replay with budget-constrained and diversity-aware sampling or generative rehearsal for object detection; adaptive architectures with dynamic capacity expansion or conditional modules for heterogeneous domains; domain-adaptive distillation through domain translation or multi-task teacher ensembles; joint optimization of plasticity and stability via meta-learning strategies that explicitly trade off forgetting against adaptation; and benchmark extensions to online IL, few-shot IL, continual addition of classes beyond eight, and real-time event-camera IL (Neuwirth-Trapp et al., 19 Aug 2025). The benchmark report also states that RICO’s code, dataset scripts, and IL-optimized Detectron2 are publicly released, positioning it as a reference framework for realistic evaluation in incremental object detection.