Syn-Mediverse: Synthetic Medical Benchmark
- Syn-Mediverse is a synthetic benchmark dataset featuring hyper-realistic, multimodal medical scenes with dense annotations for tasks like object detection, segmentation, and depth estimation.
- It provides 48,000 high-resolution images captured from three calibrated cameras across 13 diverse medical room types, supporting five key scene-understanding tasks.
- The dataset leverages synthetic simulation and domain randomization to overcome privacy, sterility, and annotation challenges in hospitals, aiding robust development of medical robotic systems.
Syn-Mediverse is a publicly released synthetic benchmark dataset for intelligent scene understanding in healthcare facilities, presented as the first hyper-realistic, multimodal synthetic dataset targeted at diverse medical environments for autonomous or assistive healthcare robots (Mohan et al., 2023). It contains 16,000 simulated frames captured simultaneously from three cameras—left, center, and right—yielding 48,000 images at resolution, with aligned depth and dense labels supporting five scene-understanding tasks: depth estimation, object detection, semantic segmentation, instance segmentation, and panoptic segmentation. The benchmark is accompanied by an online evaluation server at http://syn-mediverse.cs.uni-freiburg.de, and is motivated by the difficulty of collecting large-scale annotated hospital imagery under constraints imposed by privacy, sterility protocols, and fast-paced clinical workflows (Mohan et al., 2023).
1. Motivation and application domain
Syn-Mediverse is designed for healthcare facilities as safety-critical, highly dynamic, and visually specialized environments in which perception failures can directly affect robotic assistance, navigation, and situational awareness (Mohan et al., 2023). The paper situates the dataset in the context of future medical robots, including surgical assistants, autonomous delivery robots, and systems for surgical auditing, all of which require robust scene understanding in cluttered, human-populated, equipment-heavy spaces.
The central motivation is the mismatch between the data requirements of modern deep learning systems and the practical constraints of real hospitals. The paper identifies several obstacles to real-data collection and release: hospital workflows are fast-paced, many spaces must obey strict sterility protocols, and patient and staff privacy regulations constrain both recording and public dissemination of imagery. Existing real-world datasets in this area are therefore described as typically small, often confined to a single room or procedure, focused on only one task, and sometimes not publicly accessible (Mohan et al., 2023).
Synthetic simulation is used here as a scalable substitute for expensive, privacy-sensitive real data. The intended benefit is simultaneous generation of large volumes of dense supervision across multiple tasks, including labels that would be prohibitively costly to annotate manually in real facilities. The paper emphasizes that hospital scenes combine medical equipment, furniture, clinicians, patients, occlusion, clutter, and variable lighting, making them a demanding target for scene-understanding models (Mohan et al., 2023).
2. Dataset construction and sensor model
The generation pipeline is built in NVIDIA Isaac Sim, with room assets sourced from the Unreal Marketplace (Mohan et al., 2023). To standardize semantics across heterogeneous scenes, every asset type is manually assigned a consistent semantic name. Scene variation is introduced through changes in interior surfaces, object placement, lighting, and medical staff presence.
The environments cover 13 distinct medical rooms and room types. The paper explicitly mentions surgical suites, dental offices, radiology labs, and general hospital rooms; elsewhere it also refers to operating rooms and consultation rooms. The scenes include surgical robots, tools, beds, imaging devices, pendants, life-support devices, and ordinary indoor objects, with the stated goal of spanning a broad set of realistic hospital situations (Mohan et al., 2023).
A distinctive design feature is the simulated multi-camera rig modeled after industry-standard cameras for surgical navigation, specifically NDI Polaris Vega and Stryker FP8000. The rig contains three cameras—left, center, and right. The left and right cameras are approximately $422$ cm apart, and each is oriented at relative to the center camera. The dataset includes the intrinsic and extrinsic calibration parameters for this setup. Frames are captured at 10 FPS along camera trajectories, although the dataset is primarily used as a frame-based benchmark rather than a temporal sequence benchmark (Mohan et al., 2023).
Two capture methodologies are used. First, for each room the authors manually designed more than 20 camera trajectories that move through the room in a human-like manner; during capture, the initial camera position is randomized and one trajectory is selected at random. Second, cameras are placed at random positions without parametric constraints and then manually curated to retain representative views. This combines structured room traversal with more unconstrained viewpoint diversity (Mohan et al., 2023).
The realism pipeline includes post-processing intended to reduce the sim-to-real gap. The authors add noise through histogram matching based on cumulative distribution functions so that the color distribution of synthetic images resembles private real-world operating-room images collected in a hospital. The paper states that this is meant to make the synthetic data “more realistic and comparable to real-world data,” but it does not provide physically based rendering parameters, ray-tracing settings, material-model details, or a quantitative realism-validation study (Mohan et al., 2023).
3. Annotation schema and scene-understanding tasks
Syn-Mediverse supports five tasks: object detection, semantic segmentation, instance segmentation, panoptic segmentation, and monocular depth estimation (Mohan et al., 2023). For object detection it provides 2D bounding boxes. For semantic segmentation it provides per-pixel labels. For instance segmentation it provides per-instance masks for the thing classes; the paper states that instance-level annotations are available for 27 of the 31 original classes. For panoptic segmentation it unifies semantic and instance information, although the paper does not specify the encoding format of panoptic IDs. For monocular depth estimation it provides metric depth ground truth for all three cameras, evaluated up to a maximum depth of 15 m (Mohan et al., 2023).
The label space contains 31 semantic classes, which are merged into 21 super-classes for benchmark evaluation. Detection and instance segmentation use 17 thing classes: chairs, medical tables, storage, beds, medical tools, medical equipment, imaging, life support, cabinet, stand, healthcare participants, curtain, furniture and decor, other electronics, fixtures, office supplies, and building elements. Semantic and panoptic segmentation use those 17 classes plus four stuff classes: wall, floor, ceiling, and miscellaneous (Mohan et al., 2023).
The paper reports substantial scene complexity. The instance-count distribution peaks between 10 and 50 instances per image and reaches as high as 126 instances. The semantic-class count per image peaks roughly between 8 and 22 classes. Most object pixels lie within 5 m, although depth labels are provided up to 15 m. This combination of many instances, many classes, and clinically specific object categories is one of the main sources of difficulty in the benchmark (Mohan et al., 2023).
A numerical inconsistency in the paper concerns annotation count. The abstract states that Syn-Mediverse provides “more than 1.5M annotations,” whereas the conclusion later says “over 580000 annotations.” The paper does not reconcile these figures. What is unambiguous is the scale in images, views, task coverage, and label taxonomy (Mohan et al., 2023).
4. Benchmark protocol and baseline evaluation
The dataset split is 9,000 frames for training, 2,000 for validation, and 5,000 for testing, shared across tasks (Mohan et al., 2023). Since each frame has three views, the dataset contains corresponding left, center, and right images, but the withheld test annotations used by the benchmark server are only for the center camera. All experiments reported in the paper use center-camera images only, except monocular depth training, where all three camera streams are treated as independent training examples because each has depth ground truth.
The training setup reported in the supplementary material uses PyTorch on a system with two AMD EPYC 7452 processors and 8 NVIDIA GeForce RTX 3090 GPUs with batch size 16 (Mohan et al., 2023). Detection and instance-segmentation methods generally use resize in the 0.5–2.0 scale range, random crop to , and random flipping. Semantic segmentation baselines also use resize 0.5–2.0, crop , random flip, and SGD with a poly schedule of power 0.9. Depth baselines use crop , horizontal flip, color augmentation, and random rotation within .
The principal baselines and best reported test results are summarized below.
| Task | Metrics | Best reported test result |
|---|---|---|
| Object detection | , , | EfficientDet: 53.7 / 75.2 / 59.1 |
| Semantic segmentation | mIoU | SegFormer: 76.4 |
| Instance segmentation | $422$0, $422$1, $422$2 | DetectorRS: 42.4 / 63.3 / 44.1 |
| Panoptic segmentation | PQ, SQ, RQ, $422$3, $422$4 | Mask2Former: 65.0 / 86.1 / 74.8 / 60.1 / 86.2 |
| Monocular depth | AbsRel, RMSE, SILog, $422$5, $422$6, $422$7 | DepthFormer: 39.4 / 1.390 m / 41.3 / 42.6 / 65.9 / 79.3 |
For object detection, the baselines are SSD, YOLO, Faster R-CNN, DETR, and EfficientDet, with EfficientDet performing best on both validation and test (Mohan et al., 2023). For semantic segmentation, the baselines are DeepLabV3+, HRNet, OCRNet, SETR, and SegFormer, with SegFormer achieving the top mIoU. For instance segmentation, the baselines are YOLACT, Mask R-CNN, SOLOv2, and DetectorRS, with DetectorRS ranked highest. For panoptic segmentation, the baselines are Seamless, Panoptic-DeepLab, EfficientPS, and Mask2Former, with Mask2Former best on PQ and related metrics. For monocular depth, the baselines are BinsFormer, DepthFormer, and SimIPU, with DepthFormer clearly outperforming the other two on the reported metrics (Mohan et al., 2023).
5. Empirical difficulty and relation to prior medical datasets
The benchmark is explicitly presented as nontrivial. In object detection, the paper interprets the relatively modest scores of strong baselines as evidence that the dataset is difficult and balanced across splits (Mohan et al., 2023). In semantic segmentation, large and visually consistent classes such as wall, floor, and ceiling achieve high scores, whereas medical tools, life-support devices, and storage are more challenging because of shape and appearance variation. On the test set, for example, SegFormer scores 94.7 on healthcare personnel, 76.3 on imaging, 63.4 on life support, 51.3 on medical tools, 89.9 on wall, 87.1 on floor, and 91.1 on ceiling (Mohan et al., 2023).
The panoptic results reinforce the same pattern. $422$8 is consistently higher than $422$9, indicating that stuff classes remain easier than individual object instances. This suggests that the principal challenge is not coarse room parsing alone, but fine-grained differentiation of cluttered medical objects and equipment (Mohan et al., 2023).
Depth estimation is also difficult. DepthFormer achieves the best depth results, but the paper notes a substantial drop from validation to test, especially for BinsFormer and SimIPU. The authors state that this suggests depth prediction in these medical scenes remains difficult, likely because of clutter, texture variation, reflective surfaces, and challenging room geometries (Mohan et al., 2023).
Relative to earlier medical-vision datasets, Syn-Mediverse is positioned as broader in room coverage, annotation richness, and public accessibility. The paper states that many prior datasets focus on minimally invasive surgery, open-surgery close-ups, or pose estimation in a single operating room. Syn-Mediverse instead targets wide-angle room-scale understanding across multiple room types and includes not only clinicians and patients but also beds, surgical robots, imaging devices, trolleys, cabinets, tools, life-support equipment, and building structure. In the comparison reported by the paper, it is the only dataset offering detection, instance segmentation, semantic segmentation, panoptic segmentation, and monocular depth together (Mohan et al., 2023).
6. Transfer, use cases, and limitations
The paper includes a qualitative sim-to-real study for semantic segmentation because no public real dataset with directly matching annotations is available (Mohan et al., 2023). The study combines SAM for class-agnostic object masks with a SegFormer model pretrained on COCO and fine-tuned on Syn-Mediverse, then transfers predictions to real images from MVOR and 4D-OR via majority voting. The qualitative examples show correct segmentation of medical equipment, beds, floors, walls, people, chairs, cabinets, storage, and, in some cases, scissors as medical tools. The paper also reports failures due to mask selection, over-segmentation, and view- or lighting-dependent confusion. This supports transfer potential, but it is not a quantitative domain-adaptation benchmark.
The intended uses of Syn-Mediverse are broad within healthcare robotics and medical scene understanding. The dataset is described as suitable for training and evaluating perception systems for healthcare robotics, pretraining and sim-to-real transfer, multitask learning across detection, segmentation, and depth, and perception modules for autonomous navigation, surgical assistance, situational awareness, and clinical scene analysis (Mohan et al., 2023). Because the dataset provides calibrated multi-view imagery, it is also potentially relevant to embodied AI and robotic perception, although the paper does not frame it as an embodied-action benchmark.
Several limitations are explicit. The most fundamental is the synthetic domain itself. The paper provides only qualitative rather than quantitative sim-to-real validation. The benchmark is frame-based despite 10 FPS capture, so it does not constitute a temporal reasoning or video benchmark. Test labels are withheld only for the center camera. The paper does not specify the depth convention, panoptic ID encoding, class-to-ID mapping, camera projection equations, or licensing terms in the provided text. It also does not present formal domain-randomization equations, physically based rendering details, or realism-ablation experiments (Mohan et al., 2023).
A further limitation is that some implementation details important for downstream users are underspecified. The paper does not define the bounding-box coordinate convention, the encoding of panoptic labels, or the exact annotation-file structure. These omissions do not alter the benchmark’s role as a large public dataset, but they mean that some exact implementation details must be recovered from the released code or benchmark server rather than from the paper alone (Mohan et al., 2023).
Taken together, Syn-Mediverse occupies a specific position in medical computer vision: it is a multi-view, multitask, synthetic benchmark for room-scale healthcare scene understanding, designed to fill a data gap created by privacy, sterility, and annotation-cost constraints. Its main strengths are medical-scene diversity, calibrated sensor geometry, dense task supervision, and public benchmarking; its main limitations are the synthetic domain, incomplete format specification in the paper, and qualitative rather than quantitative transfer evidence (Mohan et al., 2023).