Hopkins RFOs Bench Dataset
- Hopkins RFOs Bench is a dataset and benchmark featuring 144 critical retained foreign object cases in chest X-rays collected over 18 years.
- It provides detailed image- and object-level annotations with rigorous radiologist validation, ensuring high inter-rater agreement and precise localization.
- The benchmark evaluates state-of-the-art detectors and synthetic augmentation strategies to overcome rare-event challenges in clinical imaging.
Searching arXiv for the target paper and closely related benchmark context. Hopkins RFOs Bench is a dataset and benchmark for critical retained foreign object detection in clinical chest radiographs. It was introduced in "Dataset and Benchmark for Enhancing Critical Retained Foreign Object Detection" as the first and largest dataset of its kind, containing 144 chest X-ray images of critical retained foreign object cases collected over 18 years from the Johns Hopkins Health System, and it is accompanied by benchmarking of several state-of-the-art object detection models and two synthetic image generation methods intended to mitigate data scarcity (Wang et al., 9 Jul 2025). The benchmark is centered on clinically consequential objects such as surgical sponges and needles, in contrast to prior datasets dominated by non-critical foreign objects.
1. Clinical scope and dataset definition
Critical retained foreign objects (RFOs) include surgical instruments such as sponges and needles, and they pose serious patient safety risks while carrying significant financial and legal implications for healthcare institutions. Detecting such objects with artificial intelligence is difficult because critical RFO cases are rare and because chest X-ray datasets specifically featuring critical RFOs have been limited. Existing datasets only contain non-critical RFOs, such as necklace or zipper, which constrains their value for developing clinically impactful detection algorithms (Wang et al., 9 Jul 2025).
Hopkins RFOs Bench was constructed to address that gap. The dataset contains 144 chest X-ray images with confirmed critical RFOs collected over an 18-year period from 2007 to 2024. To create a balanced benchmark, it also includes 150 "No RFOs" control images and 150 images with non-critical RFOs, yielding 444 images in total. Critical RFO types comprise surgical sponges, needles, wires, sutures and rings. The average number of critical RFOs per critical image is 1.0 with , whereas non-critical images contain on average 2.7 objects.
| Subset | Images | Description |
|---|---|---|
| Critical RFOs | 144 | Confirmed critical retained foreign object cases |
| No RFOs | 150 | Control images |
| Non-critical RFOs | 150 | Images containing non-critical RFOs |
This composition makes the benchmark simultaneously a detection task, a clinically oriented classification task, and a study in rare-event learning. A plausible implication is that the relatively small number of positive critical cases is not a peripheral property of the benchmark but the central technical constraint around which the entire evaluation protocol is organized.
2. Annotation model and data semantics
The benchmark provides both image-level and object-level supervision. Image-level labels are defined as $0=$ no RFO, $1=$ non-critical RFO, and $2=$ critical RFO. Object-level labels are provided in JSON, with each instance encoded as:
$2=$7
Within this schema, RFO-type is $0=$ non-critical and $1=$ critical, while shape-type is $0=$ rectangle and $1=$ polygon. The coordinate system uses origin at the top-left, with increasing to the right and $0=$0 increasing downward (Wang et al., 9 Jul 2025).
Quality control is a defining part of the benchmark. Each image was reviewed by two board-certified radiologists, and the final inter-rater agreement exceeded 0.95 with $0=$1 for bounding boxes. This matters because the target objects can be small, sparse, and visually subtle, so annotation noise can dominate model comparisons if left uncontrolled. The use of polygons in addition to rectangles also indicates that the benchmark is not restricted to coarse box-level localization when finer geometric delineation is needed.
3. Image acquisition and preprocessing pipeline
The source modality is posterior-anterior chest radiography in DICOM format. For model training, images are resampled to $0=$2 pixels. The preprocessing pipeline consists of de-identification, intensity normalization, and patient-wise splitting. All protected health information is removed from DICOM headers and pixel data following HIPAA; pixel intensities are scaled to $0=$3 and then standardized using ImageNet mean $0=$4 and standard deviation $0=$5; and the data are split patient-wise into 70% train, 10% validation, and 20% test (Wang et al., 9 Jul 2025).
These details are operationally important. A patient-wise split prevents leakage across train, validation, and test partitions, which is especially consequential in a limited-size medical imaging benchmark. The use of DICOM as the source format, followed by fixed-resolution resampling and standardized normalization, also makes the benchmark compatible with mainstream object detection pipelines while preserving a clinically conventional acquisition modality.
4. Benchmark methodology and detector configurations
The benchmark evaluates four state-of-the-art object detectors: Faster R-CNN, RetinaNet, FCOS, and YOLOv5. The specified configurations are Faster R-CNN as a 2-stage detector with a ResNet-50 FPN backbone and $0=$6 input; RetinaNet as a 1-stage detector with ResNet-50 FPN and focal loss; FCOS as an anchor-free detector with a ResNet-50 backbone; and YOLOv5 with a CSPDarknet-53 backbone and $0=$7 input (Wang et al., 9 Jul 2025).
Training uses stochastic gradient descent with learning rate $0=$8, momentum $0=$9, and weight decay $1=$0. The scheduler is step decay with $1=$1 every 5 epochs. Batch size is 8 for training and 1 for validation. Training lasts 50 epochs for pretraining or standalone training, with 5 additional epochs for fine-tuning. The total loss is the sum of classification and localization components; classification uses cross-entropy or focal loss, and localization uses smooth-$1=$2. Evaluation includes IoU, AP, mAP, FROC, and image-level ACC, FNR, and AUC.
Two training setups are reported: training on Hopkins RFOs Bench alone, and pretraining on Object-CXR followed by fine-tuning on Hopkins RFOs Bench.
| Training setup | Model | ACC / FNR / AUC / FROC |
|---|---|---|
| Hopkins RFOs Bench only | Faster R-CNN | 74.3% / 0.29 / 0.73 / 49.8 |
| Hopkins RFOs Bench only | FCOS | 71.4% / 0.32 / 0.67 / 45.1 |
| Hopkins RFOs Bench only | RetinaNet | 70.2% / 0.29 / 0.70 / 49.0 |
| Hopkins RFOs Bench only | YOLOv5 | 75.0% / 0.26 / 0.74 / 50.5 |
| Object-CXR $1=$3 Hopkins fine-tune | Faster R-CNN | 74.5% / 0.29 / 0.80 / 53.3 |
| Object-CXR $1=$4 Hopkins fine-tune | FCOS | 73.0% / 0.26 / 0.75 / 51.7 |
| Object-CXR $1=$5 Hopkins fine-tune | RetinaNet | 73.5% / 0.23 / 0.77 / 53.1 |
| Object-CXR $1=$6 Hopkins fine-tune | YOLOv5 | 75.7% / 0.24 / 0.78 / 50.2 |
The reported numbers show that no detector is near saturation on this task. Baseline detectors achieve only moderate performance, with $1=$7 and $1=$8, which underscores the difficulty of critical RFO detection. Pretraining on the larger non-critical dataset followed by fine-tuning consistently improves both classification and localization.
5. Synthetic image generation strategies
Because critical RFO cases are scarce, the benchmark also studies synthetic augmentation using two advanced image synthesis methods: DeepDRR-RFO and RoentGen-RFO. DeepDRR-RFO is physics-based. Its pipeline consists of CT volume segmentation into air, soft tissue, and bone using TotalSegmentator; 3D reconstruction of RFO meshes using TripoSR; X-ray simulation via the Beer-Lambert law with material attenuation $1=$9; and automatic projection of 3D RFO coordinates into 2D bounding boxes (Wang et al., 9 Jul 2025).
RoentGen-RFO is diffusion-based and derived from RoentGen. Its forward noising process is Gaussian, its reverse denoising process is parameterized by $2=$0 via a U-Net, and its prompting mechanism encodes RFO-type and approximate coordinates. The training objective is a denoising loss. In other words, the two methods differ not only in image synthesis mechanism but also in how explicitly they encode radiographic formation and object geometry.
| Method | Realism / cost | $2=$1FROC (@2k aug) |
|---|---|---|
| DeepDRR-RFO | Moderate realism; $2=$2 GPU-hours for 1k images | $2=$3 |
| RoentGen-RFO | High visual realism; $2=$4 GPU-hours for 1k images | $2=$5 |
DeepDRR-RFO yields a large $2=$6 point FROC gain when adding approximately 2,000 images, but it incurs slower mesh-based simulation and has limited anatomical diversity. RoentGen-RFO produces more photorealistic chest anatomy, yet it fails to capture critical RFO signatures reliably, resulting in a net performance drop when synthetic data are naively added. This directly counters a common misconception that higher visual realism necessarily implies better downstream detection performance.
6. Empirical interpretation, practical guidance, and nomenclature
The benchmark’s principal empirical findings are internally consistent. First, critical RFO detection remains difficult even for strong contemporary detectors. Second, pretraining on Object-CXR and then fine-tuning on Hopkins RFOs Bench improves both classification and localization. Third, physics-based synthetic augmentation is effective when used in moderation, specifically at around 2,000 synthetic images, whereas excessive augmentation can cause overfitting to simulation artifacts. Fourth, diffusion-based synthetic images require careful domain-specific fine-tuning, because zero-shot generation of rare pathologies may not transfer well (Wang et al., 9 Jul 2025).
These findings motivate the practical recommendations reported with the benchmark: leverage the open-access Hopkins RFOs Bench for fine-tuning specialized detectors, inject a moderate amount of physics-based synthetic RFOs to mitigate data scarcity, and use rigorous annotation protocols and cross-validation to monitor overfitting to synthetic artifacts. This suggests that reliable performance on rare-object medical detection depends less on generic detector scaling alone than on the interaction among curated positive cases, domain-informed simulation, and transfer from broader but less clinically specific corpora.
A separate nomenclature point is occasionally useful. Despite the shared institutional label "Hopkins," Hopkins RFOs Bench is unrelated to the "Hopkins 155" database used in motion segmentation research. The latter is a benchmark of 155 real video sequences with tracked feature trajectories for evaluating motion segmentation methods such as Spectral Curvature Clustering, not a chest radiography dataset for retained foreign object detection (0909.1608). The overlap is nominal rather than methodological.