Cumulative Consensus Score (CCS)

• Cumulative Consensus Score (CCS) is a label-free, model-agnostic metric that assesses object detector reliability by measuring spatial consistency across mild, non-geometric augmentations.
• It aggregates Intersection over Union (IoU) scores from multiple augmented views into a single scalar per image, enabling continuous monitoring without ground-truth annotations.
• The method is computationally efficient and is especially useful in deployment settings such as autonomous driving and robotics where real-time performance tracking is crucial.

Cumulative Consensus Score (CCS) is a label-free, model-agnostic metric for evaluating object detectors in deployment when ground-truth bounding-box annotations are unavailable. It measures the spatial consistency of detector outputs under mild, non-geometric test-time data augmentation (TTDA): predicted boxes are collected across augmented views of the same image, overlaps are computed with Intersection over Union (IoU), and the resulting agreement is aggregated into a single scalar per image. Higher CCS indicates more stable predictions and, by proxy, higher reliability, enabling continuous monitoring and comparison of detectors in real-world settings (Manoharan et al., 16 Sep 2025).

1. Deployment Motivation and Conceptual Basis

The central problem addressed by CCS is that, in deployment settings such as autonomous driving and robotics, ground-truth bounding-box annotations are unavailable. Traditional evaluation metrics such as mAP and F1-score require labels, so continuous monitoring of detector quality “in the wild” is impossible under standard supervised protocols (Manoharan et al., 16 Sep 2025).

CCS is introduced as a label-free proxy for detector reliability. Its premise is that a robust detector should produce very similar boxes on slightly perturbed versions of the same image. The perturbations are deliberately mild and non-geometric, so the underlying object locations should not systematically change. Under this assumption, spatial agreement across augmented views serves as an operational proxy for reliability. The score is computed at the case level, producing one scalar per image, which makes it suitable not only for aggregate comparison but also for identifying under-performing scenarios in deployment (Manoharan et al., 16 Sep 2025).

A plausible implication is that CCS shifts evaluation from annotation-dependent correctness to perturbation-dependent consistency. The method does not replace supervised metrics where labels exist; rather, it provides a mechanism for ongoing assessment when labels do not.

2. Formal Definition

Let a deployment dataset be indexed as $\{x_i\}_{i=1}^N$. For each image $x_i$, apply $M$ mild, non-geometric augmentations, yielding

$x_i^{(1)}, x_i^{(2)}, \dots, x_i^{(M)}.$

Running a detector on each augmented image produces sets of predicted bounding boxes

$$B_i^{(j)} = \{b_{i,1}^{(j)}, b_{i,2}^{(j)}, \dots, b_{i,N_i^{(j)}}^{(j)}\}.$$

For any two augmentations $j \neq k$, CCS constructs a pairwise IoU matrix

$$\Omega_i^{(j,k)} \in \mathbb{R}^{N_i^{(j)} \times N_i^{(k)}},$$

whose $(p,q)$ entry is

$$\Omega_i^{(j,k)}(p,q) = \mathrm{IoU}\bigl(b_{i,p}^{(j)}, b_{i,q}^{(k)}\bigr) = \frac{\mathrm{area}(b_{i,p}^{(j)} \cap b_{i,q}^{(k)})} {\mathrm{area}(b_{i,p}^{(j)} \cup b_{i,q}^{(k)})}.$$

To ignore spurious small overlaps, the matrix is thresholded at $\beta$:

$x_i$0

For each row $x_i$1, CCS computes the row-wise maximum IoU,

$x_i$2

These maxima are summed and normalized to obtain the pairwise consensus score

$x_i$3

The normalizer $x_i$4 can be chosen as a constant such as $x_i$5, the number of detections in the original unaugmented image, or $x_i$6 itself. Choosing $x_i$7 ensures that $x_i$8 (Manoharan et al., 16 Sep 2025).

Finally, CCS averages pairwise consensus over all ordered augmentation pairs:

$x_i$9

The paper also gives an optional form based on averaging over the $M$0 unordered pairs $M$1 by symmetrizing:

$M$2

In compact notation,

$M$3

where $M$4 zeroes out values below $M$5 (Manoharan et al., 16 Sep 2025).

3. Algorithmic Procedure and Computational Profile

For each image, CCS follows a four-stage procedure. First, it generates augmentations $M$6. Second, it runs the detector on each augmented image and collects the corresponding box sets $M$7. Third, for every ordered pair $M$8 with $M$9, it constructs the IoU matrix $x_i^{(1)}, x_i^{(2)}, \dots, x_i^{(M)}.$0, applies thresholding at $x_i^{(1)}, x_i^{(2)}, \dots, x_i^{(M)}.$1, computes the row-wise maxima, and normalizes their sum to obtain $x_i^{(1)}, x_i^{(2)}, \dots, x_i^{(M)}.$2. Fourth, it averages over all ordered pairs to produce $x_i^{(1)}, x_i^{(2)}, \dots, x_i^{(M)}.$3 (Manoharan et al., 16 Sep 2025).

The complexity per image is

$x_i^{(1)}, x_i^{(2)}, \dots, x_i^{(M)}.$4

The paper reports that, in practice, with $x_i^{(1)}, x_i^{(2)}, \dots, x_i^{(M)}.$5 and at most $x_i^{(1)}, x_i^{(2)}, \dots, x_i^{(M)}.$6 boxes per augmentation, CCS post-processing takes approximately $x_i^{(1)}, x_i^{(2)}, \dots, x_i^{(M)}.$7–$x_i^{(1)}, x_i^{(2)}, \dots, x_i^{(M)}.$8 ms on CPU (Manoharan et al., 16 Sep 2025).

This computational profile is important because CCS is intended for continuous monitoring rather than one-off offline benchmarking. A plausible implication is that the dominant deployment cost is repeated detector inference under TTDA, while the overlap aggregation itself remains lightweight in the reported regime.

4. Hyperparameters and Design Choices

The number and type of augmentations are a primary design choice. The default configuration uses $x_i^{(1)}, x_i^{(2)}, \dots, x_i^{(M)}.$9 mild photometric transforms: noise, blur, brightness/contrast shifts, sharpen, and color shift. Geometric transforms are explicitly avoided so that boxes do not systematically translate. This restriction is central to the interpretation of overlap as consistency rather than transformed geometry (Manoharan et al., 16 Sep 2025).

The IoU threshold is set to $$B_i^{(j)} = \{b_{i,1}^{(j)}, b_{i,2}^{(j)}, \dots, b_{i,N_i^{(j)}}^{(j)}\}.$$0 by default, following common detection practice. This threshold suppresses weak overlaps and focuses the score on box pairs that exhibit substantial spatial agreement (Manoharan et al., 16 Sep 2025).

The normalizer $$B_i^{(j)} = \{b_{i,1}^{(j)}, b_{i,2}^{(j)}, \dots, b_{i,N_i^{(j)}}^{(j)}\}.$$1 controls scale and symmetry. With $$B_i^{(j)} = \{b_{i,1}^{(j)}, b_{i,2}^{(j)}, \dots, b_{i,N_i^{(j)}}^{(j)}\}.$$2, no normalization is applied and $$B_i^{(j)} = \{b_{i,1}^{(j)}, b_{i,2}^{(j)}, \dots, b_{i,N_i^{(j)}}^{(j)}\}.$$3 can exceed $$B_i^{(j)} = \{b_{i,1}^{(j)}, b_{i,2}^{(j)}, \dots, b_{i,N_i^{(j)}}^{(j)}\}.$$4. With $$B_i^{(j)} = \{b_{i,1}^{(j)}, b_{i,2}^{(j)}, \dots, b_{i,N_i^{(j)}}^{(j)}\}.$$5, the pairwise score lies in $$B_i^{(j)} = \{b_{i,1}^{(j)}, b_{i,2}^{(j)}, \dots, b_{i,N_i^{(j)}}^{(j)}\}.$$6, but symmetry is broken because $$B_i^{(j)} = \{b_{i,1}^{(j)}, b_{i,2}^{(j)}, \dots, b_{i,N_i^{(j)}}^{(j)}\}.$$7 in general. With $$B_i^{(j)} = \{b_{i,1}^{(j)}, b_{i,2}^{(j)}, \dots, b_{i,N_i^{(j)}}^{(j)}\}.$$8, the number of boxes in the unaugmented image, the baseline is shared across directions, but the score can exceed $$B_i^{(j)} = \{b_{i,1}^{(j)}, b_{i,2}^{(j)}, \dots, b_{i,N_i^{(j)}}^{(j)}\}.$$9 if augmentations discover more boxes than the baseline image (Manoharan et al., 16 Sep 2025).

For cross-model comparison, the paper introduces an indifference margin $j \neq k$0. When comparing CCS deltas against supervised metric deltas, differences satisfying $j \neq k$1 or $j \neq k$2 are labeled “tie” (yellow) and excluded from strict congruence counts. This design avoids spurious sign flips near zero and makes the congruence analysis less sensitive to negligible differences (Manoharan et al., 16 Sep 2025).

5. Experimental Validation

The experimental study trains detectors on Open Images and evaluates them on KITTI as an unseen dataset. The detector families include Faster R-CNN as a two-stage architecture and RetinaNet and SSD as single-stage architectures (Manoharan et al., 16 Sep 2025).

The supervised reference metrics are F1-score at IoU threshold $j \neq k$3, Probabilistic Detection Quality (pPDQ) with a focus on spatial quality, and Optimal Correction Cost (OC-cost) with $j \neq k$4. For two models $j \neq k$5 and $j \neq k$6, the paper computes per-image $j \neq k$7 and the analogous $j \neq k$8 for each supervised metric. A sample is deemed congruent when

$j \neq k$9

outside the indifference margin $$\Omega_i^{(j,k)} \in \mathbb{R}^{N_i^{(j)} \times N_i^{(k)}},$$0. The reported statistics are the fraction of congruent images and Spearman’s $$\Omega_i^{(j,k)} \in \mathbb{R}^{N_i^{(j)} \times N_i^{(k)}},$$1 (Manoharan et al., 16 Sep 2025).

The key results reported in Table III are as follows.

Comparison	Congruence	Spearman’s $$\Omega_i^{(j,k)} \in \mathbb{R}^{N_i^{(j)} \times N_i^{(k)}},$$2
$$\Omega_i^{(j,k)} \in \mathbb{R}^{N_i^{(j)} \times N_i^{(k)}},$$3 vs. $$\Omega_i^{(j,k)} \in \mathbb{R}^{N_i^{(j)} \times N_i^{(k)}},$$4	93.26%	$$\Omega_i^{(j,k)} \in \mathbb{R}^{N_i^{(j)} \times N_i^{(k)}},$$5
$$\Omega_i^{(j,k)} \in \mathbb{R}^{N_i^{(j)} \times N_i^{(k)}},$$6 vs. $$\Omega_i^{(j,k)} \in \mathbb{R}^{N_i^{(j)} \times N_i^{(k)}},$$7	94.37%	$$\Omega_i^{(j,k)} \in \mathbb{R}^{N_i^{(j)} \times N_i^{(k)}},$$8
$$\Omega_i^{(j,k)} \in \mathbb{R}^{N_i^{(j)} \times N_i^{(k)}},$$9 vs. $(p,q)$0	91.28%	$(p,q)$1

These results indicate over 90% congruence between CCS and the supervised baselines in the reported controlled experiments (Manoharan et al., 16 Sep 2025).

The robustness analysis varies augmentation random seeds across 15 seeds. Spearman’s $(p,q)$2 remains stable within $(p,q)$3 for F1, $(p,q)$4 for pPDQ, and $(p,q)$5 for OC-cost. Across architectures and training regimes, as summarized in Table IV, congruence reaches up to 98% on F1, remains above 90% on pPDQ and OC-cost, and yields $(p,q)$6 up to 0.81 for F1 (Manoharan et al., 16 Sep 2025).

6. Model-Agnostic Deployment Use, Limitations, and Future Work

CCS is explicitly model-agnostic. It treats the detector as a black box that outputs bounding boxes and scores, requiring no architectural changes, no ensembles, and no internal feature access. The same procedure applies to single-stage and two-stage detectors and to any backbone (Manoharan et al., 16 Sep 2025).

Its output is a per-image score, which supports case-level monitoring. Images with unusually low CCS can be flagged automatically as unstable or unreliable detections. The paper situates this within a DevOps-style pipeline: periodic TTDA checks can be run, CCS can be tracked over time, new and old models can be compared through $(p,q)$7, and alerts can be triggered when consensus drops below a threshold (Manoharan et al., 16 Sep 2025).

The method also has explicit limitations. CCS is sensitive only to appearance changes and does not certify robustness to geometric shifts or adversarial attacks. The choice and intensity of augmentations remain design decisions, and the paper notes that more deployment-realistic augmentations may further improve correlation. The indifference margin $(p,q)$8 is user-tunable, and more systematic selection methods are left for future work. Computational cost may be reduced further through parallelization, including randomized smoothing or GPU-accelerated IoU computations. Proposed extensions include broader datasets, varied operating conditions such as weather and night/day, additional object classes, and integration with uncertainty-aware decision making (Manoharan et al., 16 Sep 2025).

Taken together, these properties position CCS as a mechanism for ongoing assessment of deployed object detectors under label scarcity. This suggests a specific role for CCS: not as a replacement for annotation-based evaluation, but as an operational proxy for reliability when deployment conditions preclude continuous supervised measurement.