---
title: 'COCO-ReM: Refined COCO Mask Annotations'
url: https://www.emergentmind.com/topics/coco-rem
type: topic
---

# COCO-ReM: Refined COCO Mask Annotations

COCO-ReM (COCO Refined Masks) is a comprehensive re-annotation of the COCO-2017 instance segmentation dataset, designed to address critical annotation errors and inaccuracies that have emerged as limiting factors in benchmarking high-performing object detectors. By refining mask boundaries, ensuring more exhaustive object coverage, and correcting label inconsistencies, COCO-ReM delivers a cleaner set of annotations with the goal of restoring the reliability of COCO benchmarking while maintaining compatibility with prior research conventions [2403.18819].

## 1. Motivation and Scope

COCO-2017, despite its foundational role in object detection research, contains pervasive errors affecting both model training and evaluation. Key issues include:

- **Imprecise mask boundaries**: The polygon-based annotation interface uses straight-line segments to approximate object contours, resulting in coarse mask edges.
- **Absence of holes in masks**: Annotation procedures precluded internal voids, so masks for objects like cup handles or scissor blades are often filled solidly, not reflecting true structure.
- **Inconsistent occlusion handling**: There is no standardization; some masks are amodal (covering occluders), others modal (cutting around occluders).
- **Non-exhaustive instance coverage**: Some images have incomplete instance annotations, and “grouped” masks may merge objects of a given class.
- **Near-duplicate masks across categories**: Approximately 410 pairs in val-2017 (2.3% of instances) share IoU $>0.8$ but carry different category labels.

These deficiencies degrade detector performance and evaluation, particularly as model Mask AP surpasses 70%. Addressing them while preserving COCO’s 80-class structure and open-source flexibility was prioritized to ensure research continuity [2403.18819].

## 2. Annotation Correction Methodology

COCO-ReM employs a three-stage, semi-automatic pipeline to revise both train and validation splits:

**Stage 1: Mask Boundary Refinement**
- Utilizes SAM (Segment Anything Model) prompting with bounding boxes and error-region points for each mask (10 repeats per mask, majority-voted pixel labels).
- Manual inspection and interactive correction for all ∼36K val masks, with ∼900 requiring further intervention for precision, targeting categories with frequent occlusions or holes.

**Stage 2: Exhaustive Instance Annotation**
- **LVIS imports**: Where LVIS provides more non-crowd instances for an (image, category) pair, the more complete LVIS annotations replace COCO’s.
- **Model-assisted discovery**: An ensemble of three LVIS-trained ViTDet models proposes high-confidence extra masks, with manual review yielding 1,056 new val instances, each with refined boundaries.

**Stage 3: Label Correction (val only)**
- Manual disambiguation of near-duplicate, high-IoU category masks.
- Remaining grouped-instance masks (~100) retagged as "crowd".

The validation split is fully author-verified; automated-only steps are applied to the training split [2403.18819].

## 3. Dataset Characteristics and Format

COCO-ReM increases both accuracy and exhaustiveness relative to COCO-2017. Key statistics:

| Split      | COCO-2017 | COCO-ReM   |
|------------|-----------|------------|
| val        | 36,781    | 40,689     |
| train      | 860,001   | 1,093,027  |

- **Median IoU** between COCO-ReM val masks and source (COCO/LVIS) ≈ 0.97: the majority of shapes persist, with boundary definition significantly sharpened.
- **Mask “holes”**: ≈2,000 val masks now include true holes, compared to zero in COCO.
- **Annotation Quality**: Strong boundary displacement reductions; AP gains of 5–7 points at IoU thresholds ≥0.75.
- **Distribution and Format**: Available at https://cocorem.xyz. Annotations are in standard COCO JSON with polygons replaced by bitmasks. All images remain under the original Flickr-CC license [2403.18819].

## 4. Evaluation Metrics and Sensitivity

COCO-ReM retains the original COCO evaluation protocol:

- **IoU calculation**: $\operatorname{IoU}(G,P) = \frac{|G \cap P|}{|G \cup P|}$
- **TP/FP/FN** at threshold $t$
- **Precision–Recall**: $P(R;t) = \frac{TP(t)}{TP(t)+FP(t)}$, with recall $R = \frac{TP(t)}{TP(t)+FN(t)}$
- **Average Precision**: $AP_t = \int_0^1 P(R;t)dR$
- **COCO mAP**: $AP = \frac{1}{10}\sum_{k=1}^{10} AP_{0.45 + 0.05k}$, with special points $AP_{50}$, $AP_{75}$

No new metrics are introduced, but COCO-ReM’s higher mask fidelity sharpens sensitivity to boundary accuracy, especially at higher IoU thresholds [2403.18819].

## 5. Experimental Results and Comparative Analysis

Extensive benchmarking of 50 object detectors across COCO-ReM and COCO-2017 reveals:

- **AP Improvements Across Architectures**: Every model class—region-based (Mask R-CNN, Cascade R-CNN, ViTDet) and query-based (Mask2Former, OneFormer)—improves in AP. State-of-the-art OneFormer InternImage-H shows a +7.7 AP gain.
- **Model Ranking Shifts**: Query-based models overtake region-based on COCO-ReM, corresponding with human preference for sharper masks.
- **Threshold Sensitivity**: At IoU=0.5, AP increases modestly (+0.2–0.5); at IoU≥0.75, gains reach +3–8, confirming improvements are primarily due to more precise boundaries.
- **Category-specific Effects**: 69/80 categories see AP gains, largest for fine detail (e.g., fork, carrot, orange). 11 categories show decreased AP, mainly where prior “no-hole” annotation bias penalized correct predictions.
- **Annotation Stages (Ablation)**:
    - Stage 1 (boundary refinement) alone recovers ~7.7 AP of noise.
    - Exhaustiveness (Stage 2) adds little net AP; slightly down due to increased set size.
- **Training Efficiency**:
    - Models trained on COCO-ReM converge faster and achieve higher AP at equivalent or smaller model size.
    - Example: Mask R-CNN ViTDet-B: 100 epochs on COCO-2017 yields AP ≈ 49.1; COCO-ReM achieves AP ≈ 52.4. COCO-ReM-trained models match the performance of much larger (3×) models trained on COCO-2017 [2403.18819].

## 6. Best Practices, Limitations, and Future Directions

**Recommended Usage**:
- Employ COCO-ReM masks for both training and validation.
- Retain original image splits and hyperparameters; no architecture or evaluation code change needed.

**Limitations**:
- SAM may hallucinate small spurious components. Niche, fine-detail categories may benefit from additional review.
- COCO-ReM does not re-annotate the private test set; thus, official test-dev still uses original COCO-2017 masks.

**Refinement Directions**:
- Extension to COCO stuff and panoptic splits.
- Human re-annotation focusing on largest boundary outlier categories (±500 instances with IoU<0.8).
- Broadening of external annotation imports (e.g., OpenImages) to increase exhaustiveness.
- Community-driven corrections via the public GitHub repository (https://github.com/kdexd/coco-rem) [2403.18819].

**Significance**:  
COCO-ReM provides mask annotations significantly closer to perceptual ground truth, rectifies critical ranking pathologies in high-performing segmenters, and substantially improves both detector training efficiency and final benchmark AP—all without disrupting prevalent codebases or evaluation methodologies. The adoption of COCO-ReM is advised for future research in object detection and instance segmentation [2403.18819].

Source: https://www.emergentmind.com/topics/coco-rem