RoHOI: Robustness Benchmark for HOI

• RoHOI is a benchmark that defines robust evaluation for human–object interaction models by testing them under 20 corruption types at multiple severity levels.
• It introduces quantitative metrics like MRI and CRI to assess model performance and stability under various image degradations.
• The SAMPL training strategy employs semantic masking and progressive curriculum learning to improve localization and interaction detection in corrupted conditions.

RoHOI is a robustness benchmark for Human–Object Interaction (HOI) detection, designed to evaluate and promote resilience of HOI models to a comprehensive set of real-world corruptions. HOI detection—identifying human–object pairs and their interaction classes in images—is fundamental for robust robot-human assistance and context-aware artificial intelligence. RoHOI establishes standardized methods for assessing the degradation of HOI models under diverse visual corruptions and proposes new metrics and training strategies to enhance robust feature learning (Wen et al., 12 Jul 2025).

1. Structure of the RoHOI Benchmark

RoHOI augments the standard HICO-DET and V-COCO HOI detection benchmarks with a suite of 20 synthetic corruption types, each rendered at five severity levels, resulting in 100 distinct test sets. The benchmark is built on established HOI datasets:

• HICO-DET: 47,776 images, 600 HOI categories (117 verbs × 80 objects), annotated with human–object pairs. Standard evaluation uses mean Average Precision (mAP) with IoU ≥ 0.5, partitioned into Full, Rare, and Non-Rare splits.
• V-COCO: 10,346 images, 29 verbs, 81 object categories, ~16,000 human instances. Evaluation uses role-based AP: AP^#1_role (verb only) and AP^#2_role (verb plus object localization at IoU ≥ 0.5).

The 20 corruption types are categorized as follows:

Corruption Group	Examples	Scaling Parameter
Optical-System (OS)	Motion Blur, Defocus, Gaussian/Glass	kernel size, PSF radius, σ
Sensor/Compression/Transmission	Gaussian, Shot, S&P, Speckle, JPEG, Packet Loss	noise std, block loss, JPEG Q
Environmental (EI)	Exposure, Rainbow, Occlusion, Vignette	exposure intensity, occlusion ratio
Geometric & Scene (GS)	Moiré, Screen Crack, Elastic, Perspective, Pixelation, Zoom Blur	density, transformation extent

Each corruption is parameterized for severity (ℓ = 1 … 5); for example, kernel sizes or noise variances increase linearly with ℓ.

2. Robustness Metrics

RoHOI introduces two quantitative measures to summarize both the average robustness and stability of a model’s predictions across all corruption types and severities:

• Mean Robustness Index (MRI):

$$\mathrm{MRI} = \frac{1}{C}\sum_{c=1}^C \left( \frac{1}{L_c}\sum_{\ell=1}^{L_c} \mathrm{mAP}_{c,\ell} \right)$$

where $C$ is the number of corruptions (20), $L_c$ is the number of severity levels (typically 5), and $\mathrm{mAP}_{c,\ell}$ is the performance at corruption $c$ and severity $\ell$. MRI averages performance equally over all corruptions and severities.

• Composite Robustness Index (CRI):

$$\mathrm{CRI} = \frac{1}{C}\sum_{c=1}^{C} \frac{\overline{\mathrm{mAP}_c}} {\,\mathrm{mAP}_\text{clean} /\left( \log(1+\sigma_c)+1 \right)}$$

where $\overline{\mathrm{mAP}_c}$ is the mean mAP across severities for corruption $c$, $\mathrm{mAP}_\text{clean}$ is the clean (uncorrupted) baseline, and $C$0 is the standard deviation of mAP over severities for that corruption. CRI penalizes instability in performance across severity levels.

3. Analysis of HOI Detector Robustness

RoHOI benchmarks contemporary HOI detection models, including both two-stage (QPIC, UPT, CDN, DiffHOI) and one-stage (FGAHOI, GEN-VLKT, QAHOI, SOV-STG, MUREN, RLIPv2) architectures.

Key empirical findings:

• Transformer-based vision–LLMs (RLIPv2) yield higher clean and corrupted set scores (HICO-DET mAP = 38.60, MRI = 24.58, CRI = 0.28).
• Performance degrades markedly under certain corruptions. Severe pixelation and packet loss result in over 50% relative drop in accuracy.
• Two-stage methods propagate localization errors from blur/noise, while one-stage query-based models like RLIPv2 show relatively more stable—though still degraded—robustness, particularly under distortions such as moiré or zoom blur.
• RoHOI thus demonstrates that despite significant advances, all current techniques experience substantial vulnerabilities under realistic corruptions.

4. Semantic-Aware Masking-based Progressive Learning (SAMPL)

RoHOI introduces SAMPL, a training strategy designed to increase robustness by curriculum-based exposure to semantic, instance-wise occlusions:

• SAM-Guided Semantic Masking: Using the Segment-Anything Model (SAM), instance-specific masks are generated and expanded (dilate, convex hull, resize to predefined ratios). These masks are applied to input images, simulating partial occlusion aligned with the object(s) of interest, thus forcing the detector to exploit both holistic and local features.

$C$1

• Score-Guided Progressive Curriculum: Curriculum advances to more severe occlusions when plateauing is detected in validation gains. Progression is controlled by a dynamic threshold, using a memory bank $C$2 and a “severity score” to adaptively select the epoch’s masking severity.
• Loss Optimization: Standard HOI objectives (object/verb cross-entropy, bounding box L1+GIoU, Kullback–Leibler divergence) are optimized at each step, always with the dynamically chosen mask severity.

Ablation studies confirm:

• Structured, semantic masking induces greater robustness than random pixelation or feature-level noise.
• Optimal coverage ratios (40–60%) yield the best compromise between clean accuracy and robustness (MRI, CRI).
• Overly large occlusions (>70%) compromise clean-set performance.

5. Experimental Results and Implications

• SAMPL achieves an improvement in composite robustness (HICO-DET CRI = 0.29 vs. 0.28 for RLIPv2 baseline; V-COCO MRI = 48.83 vs. 45.05; AP^#2_role = 71.13 vs. 70.76).
• Qualitative results show that SAMPL improves localization and interaction detection of small or distorted actors under challenging corruptions such as screen crack, pixelation, salt-and-pepper, and moiré.
• Both RLIPv2 and SAMPL-trained models are reliably challenged by severe glass blur and high-frequency elastic distortions, highlighting open problems in robust perception.
• Test set example: under domain-randomized conditions, policies trained with SAMPL show an absolute increase in unseen-background success rates (+25%).

6. Impact and Research Directions

RoHOI exposes the limitations of state-of-the-art HOI models in the presence of real-world image corruptions and provides a standardized framework—datasets, metrics, and protocol—for benchmarking and driving the development of robust algorithms. With its semantically aligned, curriculum-based masking, SAMPL offers a directionally valid solution and current state of the art in robustness for HOI detection.

A plausible implication is that future model architectures and training paradigms will increasingly incorporate structured, semantic perturbation and adaptive curriculum learning as integral components for deployment in open-world environments. However, the persistent performance drops under extreme corruptions indicate unresolved challenges, suggesting continued research is necessary on both the algorithmic and representational levels (Wen et al., 12 Jul 2025).