EC-RICO Incremental Object Detection Benchmark

• EC-RICO is a benchmark that simulates continual learning by introducing a new domain and object class at each incremental step.
• It standardizes evaluation using eight tasks drawn from varied automotive and surveillance datasets with consistent preprocessing and annotation protocols.
• The benchmark emphasizes the stability–plasticity trade-off, highlighting challenges in retention, adaptability, and the limitations of single-model approaches.

Searching arXiv for the EC-RICO benchmark paper and baseline incremental object detection methods. Expanding-Classes RICO (EC-RICO) is a realistic incremental object detection benchmark introduced in “RICO: Two Realistic Benchmarks and an In-Depth Analysis for Incremental Learning in Object Detection” (Neuwirth-Trapp et al., 19 Aug 2025). It models a continual-learning setting in which each incremental learning step introduces both a new domain and a new object class. The benchmark is designed to evaluate object detectors under coupled distribution shift and label-space expansion, with an emphasis on the stability–plasticity trade-off: retaining performance on previously learned classes while adapting to newly introduced classes in new domains. In the formulation reported for EC-RICO, the benchmark comprises eight tasks drawn from distinct automotive or surveillance datasets, each standardized through a common preprocessing and evaluation protocol (Neuwirth-Trapp et al., 19 Aug 2025).

1. Concept and problem setting

EC-RICO simulates “a realistic continual-learning scenario in which, at each step, a new domain and a new object class are introduced” (Neuwirth-Trapp et al., 19 Aug 2025). This distinguishes it from evaluations that isolate either domain shift or class expansion. The benchmark therefore targets a more demanding regime than fixed-class domain-incremental evaluation, because the detector must simultaneously absorb a new category and adapt to a new visual distribution.

At step $t$ with $t=1,\dots,8$, EC-RICO introduces one new object class alongside all previously seen classes. The training label set is defined as

$C_t = \{\text{classes introduced in } 1\ldots t\},$

and “all objects of future classes remain unlabeled (‘background’)” (Neuwirth-Trapp et al., 19 Aug 2025). This labeling rule is central to the benchmark’s difficulty: future categories are present in images before they are officially introduced, but are treated as background until their designated step. A plausible implication is that EC-RICO stresses both open-world ambiguity and annotation incompleteness in a sequential setting.

The benchmark is positioned within incremental learning for object detection, where “Incremental Learning (IL) trains models sequentially on new data without full retraining” and must balance “adaptability to new data with retention of old knowledge” (Neuwirth-Trapp et al., 19 Aug 2025). EC-RICO operationalizes this balance under real-world heterogeneity rather than synthetic simplifications.

2. Dataset composition and task sequence

Each EC-RICO task is drawn from “a distinct automotive or surveillance dataset,” and “all images are reformatted to RGB (3 channels) at 1536×1536 and annotated with axis-aligned bounding boxes for the classes in that step” (Neuwirth-Trapp et al., 19 Aug 2025). The eight tasks span real and synthetic domains, multiple sensor modalities, multiple viewpoints, and heterogeneous annotation styles.

Step	Dataset	New class
1	WoodScape	person
2	DENSE gated	car
3	nuImages daytime	bicycle
4	FishEye8K	motorcycle
5	SHIFT simulation	truck
6	VisDrone drone	bus
7	FLIR thermal	traffic light
8	BDD100K night	street sign

The domain structure is explicitly cumulative. Step 1 begins with WoodScape and the class set $\{person\}$. Step 2 expands to $\{person, car\}$ and adds the gated domain. Step 3 adds bicycle and “+{multi-camera}.” Step 4 adds motorcycle and “+{surveillance}.” Step 5 adds truck and “+{synthetic}.” Step 6 adds bus and “+{aerial}.” Step 7 adds traffic light and “+{thermal}.” Step 8 adds street sign and “+{nighttime}” (Neuwirth-Trapp et al., 19 Aug 2025). This organization makes domain complexity monotone in step index, rather than keeping environmental conditions fixed while classes accumulate.

The underlying datasets and domain characteristics are as follows: WoodScape is real, fisheye RGB, vehicle-mounted, with “segmentation→tight bounding box”; DENSE gated is real, with “gated RGB & Depth,” vehicle-mounted, and “instance boxes, remove ‘group’ classes”; nuImages is real, “multi-camera 6×RGB,” vehicle-mounted, with “tight, visible-only boxes”; FishEye8K is real, fisheye RGB, surveillance, with “loose overhead boxes”; SHIFT is synthetic, “RGB (CARLA),” vehicle-mounted, with “tight boxes from simulator”; VisDrone is real, “nadir RGB (drone),” aerial, with “amodal boxes”; FLIR thermal is real, thermal, vehicle-mounted, with “visible-only boxes, merged rider+bicycle”; and BDD100K night is real, RGB, vehicle-mounted, with “tight, visible-only boxes (night)” (Neuwirth-Trapp et al., 19 Aug 2025).

This diversity is not incidental. The benchmark is “built from 14 diverse datasets covering real and synthetic domains, varying conditions (e.g., weather, time of day), camera sensors, perspectives, and labeling policies,” although the EC-RICO task sequence itself is summarized through eight tasks/domains (Neuwirth-Trapp et al., 19 Aug 2025). This suggests that the benchmark is intended as a stress test for robustness across acquisition conditions, geometry, and annotation conventions rather than a narrowly scoped class-incremental protocol.

3. Preprocessing and data protocol

EC-RICO uses a harmonization pipeline to reduce purely superficial incompatibilities while preserving substantive domain differences. The preprocessing protocol includes annotation alignment, image selection and cleaning, resizing and channel formatting, and training-time augmentation (Neuwirth-Trapp et al., 19 Aug 2025).

Under annotation alignment, the protocol specifies: “Merge rider + bicycle boxes when IoU > 0.25 (FLIR, VisDrone),” “Recompute boxes from segmentation masks (WoodScape, Synscapes),” and “Discard group annotations and very small boxes (< 7×7 px)” (Neuwirth-Trapp et al., 19 Aug 2025). These operations standardize incompatible annotation policies across source datasets without collapsing their distinct semantics entirely.

Image selection and cleaning includes “Subsample high-frequency frames (every 50th in SHIFT, every 6th in TIMo),” “Filter by mean grayscale value to remove night/day as needed,” and “Remove near-duplicate or low-visibility images” (Neuwirth-Trapp et al., 19 Aug 2025). The goal is not merely resizing or relabeling, but curation of a sequence appropriate for incremental evaluation.

Resizing and channel formatting impose a common representation: “Resize/pad/crop to 1536×1536,” “Duplicate grayscale→RGB for thermal tasks,” and “Fuse RGB+thermal via IFCNN network (SMOD)” (Neuwirth-Trapp et al., 19 Aug 2025). Training-only augmentation further includes “Random horizontal flip; random brightness/contrast/saturation (0.6–1.4×); random lighting jitter (±0.1),” as well as “Random scale (0.1–2×) and fixed-size crop to 1536×1536” (Neuwirth-Trapp et al., 19 Aug 2025).

Dataset splits are fixed per task as “Training: 3040 images,” “Validation: 511 images,” and “Testing: 1417 images” (Neuwirth-Trapp et al., 19 Aug 2025). The splits preserve “scene-level integrity (no scene overlap) in a 60 / 10 / 30 ratio via Monte Carlo assignment” (Neuwirth-Trapp et al., 19 Aug 2025). For incremental object detection, scene-level separation is important because near-duplicate frames can otherwise inflate apparent retention and transfer.

4. Evaluation methodology and metrics

EC-RICO evaluates models after every incremental step. Let $mAP_{k,j}$ denote “mAP on task $j$’s test set after learning step $k$ ($j \le k$),” let $mAP'_j$ denote the “individual model mAP trained only on $t=1,\dots,8$0,” and let $t=1,\dots,8$1 denote the “joint-training mAP on $t=1,\dots,8$2 (all tasks together)” (Neuwirth-Trapp et al., 19 Aug 2025).

The benchmark reports four metrics. Overall Accuracy, or Average Incremental mAP, is

$t=1,\dots,8$3

The Forgetting Measure is

$t=1,\dots,8$4

Forward Transfer is

$t=1,\dots,8$5

The Intransigence Measure is

$t=1,\dots,8$6

The reported interpretation is: “Higher $t=1,\dots,8$7, FWT, IM indicate better performance; lower FM indicates less forgetting” (Neuwirth-Trapp et al., 19 Aug 2025). The inclusion of both $t=1,\dots,8$8 and $t=1,\dots,8$9 baselines is significant. The individual-model baseline isolates per-task learnability without interference, while the joint-training baseline estimates the attainable performance of a single model trained on the aggregate data. A plausible implication is that EC-RICO distinguishes three distinct failure modes: catastrophic forgetting, poor forward adaptation, and capacity-limited multi-domain compromise.

The recommended experimental setup uses “≥1 GPU with 24 GB VRAM,” an “EVA-02-L (frozen) + Cascade Faster R-CNN head,” “AdamW; lr=0.001; warm-up=10% iters; cosine LR decay to 0,” “Iterations per task: 700,” and “batch size=20” (Neuwirth-Trapp et al., 19 Aug 2025). The replicable evaluation protocol requires: a joint-training run to collect $C_t = \{\text{classes introduced in } 1\ldots t\},$0; individual-task runs to collect $C_t = \{\text{classes introduced in } 1\ldots t\},$1; an incremental run over $C_t = \{\text{classes introduced in } 1\ldots t\},$2 with checkpointing at iteration 700; evaluation on all test sets $C_t = \{\text{classes introduced in } 1\ldots t\},$3 for $C_t = \{\text{classes introduced in } 1\ldots t\},$4 after each task; computation of $C_t = \{\text{classes introduced in } 1\ldots t\},$5, $C_t = \{\text{classes introduced in } 1\ldots t\},$6, $C_t = \{\text{classes introduced in } 1\ldots t\},$7, and $C_t = \{\text{classes introduced in } 1\ldots t\},$8; and repetition “3× with different seeds; report mean ± std” (Neuwirth-Trapp et al., 19 Aug 2025).

5. Baselines and benchmark operating point

EC-RICO includes several baselines and incremental learning configurations. The baseline set comprises “Naïve Finetuning (FT): sequential fine-tuning without any anti-forgetting,” “Replay {1%, 10%, 25%}: random sample from all previous data in a growing buffer,” and four named methods: ABR, Meta-ILOD, BPF, and LDB (Neuwirth-Trapp et al., 19 Aug 2025).

The reported configurations are specific. ABR uses “Mixup+mosaic, distillation weights $C_t = \{\text{classes introduced in } 1\ldots t\},$9, $\{person\}$0, buffer=2000 images.” Meta-ILOD uses “Warp last conv layer, lr=1e-4, feature buffer=5000/class, distill weight=1.” BPF uses “Pseudo-labels disabled (DIL), distill $\{person\}$1, $\{person\}$2, buffer=2000.” LDB “Adds task-specific bias+output layers, batch=5, training iterations=2784” (Neuwirth-Trapp et al., 19 Aug 2025).

These baselines correspond to methods identified in the benchmark document as Liu et al. for ABR, Joseph et al. for Meta-ILOD, Leonardis-Trapp et al. for BPF, and Song et al. for LDB (Neuwirth-Trapp et al., 19 Aug 2025). Within the benchmark narrative, they represent different anti-forgetting strategies: replay, distillation, architectural adjustment, and bias-balancing. The results reported for EC-RICO indicate that increasing methodological sophistication does not necessarily translate into superior performance under the combined domain-and-class incremental regime.

Because the benchmark standardizes backbone, detector head, input resolution, and training schedule, the operating point is intentionally demanding. A plausible implication is that the frozen EVA-02-L backbone and shared detection head expose limitations that might remain hidden in lower-resolution or less heterogeneous protocols.

6. Empirical findings and interpretation

The principal empirical result is that “all IL methods underperform in adaptability and retention, while replaying a small amount of previous data already outperforms all methods. However, individual training on the data remains superior” (Neuwirth-Trapp et al., 19 Aug 2025). The benchmark quantifies the gap between upper baselines and incremental methods.

For joint versus individual training, the reported values are “Joint $\{person\}$3 vs. Individual $\{person\}$4 ⇒ $\{person\}$5 (single model limitation)” (Neuwirth-Trapp et al., 19 Aug 2025). Among incremental methods, “Naïve FT: $\{person\}$6, $\{person\}$7, $\{person\}$8, $\{person\}$9”; “Replay 1%: $\{person, car\}$0, $\{person, car\}$1, $\{person, car\}$2, $\{person, car\}$3”; and “Best SOTA (LDB): $\{person, car\}$4, $\{person, car\}$5, $\{person, car\}$6, $\{person, car\}$7” (Neuwirth-Trapp et al., 19 Aug 2025). The summary conclusion is explicit: “All IL methods remain below Individual baseline; simple replay outperforms all complex methods” (Neuwirth-Trapp et al., 19 Aug 2025).

Three EC-RICO-specific challenges are identified. First, “Weak Distillation Teachers”: the “Previous model’s next-task mAP (average) ≈23.5%; distillation cannot transfer knowledge across diverse domains” (Neuwirth-Trapp et al., 19 Aug 2025). Second, a pronounced “Stability-Plasticity Trade-off,” where “Strong anti-forgetting (low FM) often coincides with low plasticity (negative FWT)” (Neuwirth-Trapp et al., 19 Aug 2025). Third, a “Single-model Bottleneck”: “The fixed backbone + shared head cannot adapt to diverging domain-class combinations (Δ7% joint vs. individual)” (Neuwirth-Trapp et al., 19 Aug 2025).

These findings directly shape the benchmark’s stated recommendations: future incremental detectors should “enhance plasticity (beyond mere forgetting minimization),” “incorporate domain-aware modules or parameter expansion,” and “explore smarter replay selection to maximize transfer per sample” (Neuwirth-Trapp et al., 19 Aug 2025). This suggests that EC-RICO functions not only as an evaluation suite but also as an argument against assessing incremental object detection primarily through forgetting-oriented metrics on simplified benchmarks.

7. Relation to broader benchmark design and common misunderstandings

EC-RICO is one of two RICO benchmarks introduced in the same work, the other being Domain RICO (D-RICO), which “features domain shifts with a fixed class set,” whereas EC-RICO “integrates new domains and classes per IL step” (Neuwirth-Trapp et al., 19 Aug 2025). The distinction is methodologically important. D-RICO isolates adaptation under a fixed ontology; EC-RICO adds label-space growth and unlabeled future classes. Confusing the two would obscure the specific source of difficulty in reported EC-RICO results.

A common misunderstanding is to interpret low forgetting alone as evidence of superior incremental learning. EC-RICO’s results explicitly counter that reading: the best reported “SOTA” result in terms of low $\{person, car\}$8 is LDB with $\{person, car\}$9, yet its $mAP_{k,j}$0 and $mAP_{k,j}$1 are substantially worse than replay and naïve fine-tuning on overall accuracy and forward adaptation (Neuwirth-Trapp et al., 19 Aug 2025). In EC-RICO, low forgetting can coincide with severe intransigence.

A second misunderstanding is to treat joint training as an upper bound that incremental learning should approach closely if forgetting is solved. EC-RICO reports a nontrivial gap even between joint and individual training, $mAP_{k,j}$2, attributed to “single model limitation” (Neuwirth-Trapp et al., 19 Aug 2025). This indicates that some of the challenge is not sequentiality per se, but the representational burden of compressing diverging domain-class combinations into one detector.

A third misunderstanding is that sophisticated distillation frameworks should dominate replay-based baselines. EC-RICO instead reports that “simple replay outperforms all complex methods” (Neuwirth-Trapp et al., 19 Aug 2025). The accompanying interpretation is heuristic: the gap is attributed to “weak teachers in distillation, single models’ inability to manage diverse tasks, and insufficient plasticity” (Neuwirth-Trapp et al., 19 Aug 2025). This suggests that the benchmark primarily exposes limitations of current anti-forgetting mechanisms when domain shift is entangled with semantic expansion.

In that sense, EC-RICO occupies a specific place in incremental object detection research. It is not merely a larger benchmark, but a benchmark whose construction forces evaluation under simultaneous domain drift, class accretion, heterogeneous sensors, heterogeneous annotation styles, and partially unlabeled future semantics. Its significance lies in demonstrating that methods that appear competitive in synthetic or simplified protocols may degrade sharply once these factors are combined (Neuwirth-Trapp et al., 19 Aug 2025).