---
title: 'Hybrid AMO Dataset: Multimodal & Amodal Insights'
url: https://www.emergentmind.com/topics/hybrid-amo-dataset
type: topic
---

# Hybrid AMO Dataset: Multimodal & Amodal Insights

A Hybrid AMO (Amodal-Multimodal-Organized) Dataset refers to a dataset designed to provide rich, multi-faceted ground truth for learning models that must reason about both modal (visible) and amodal (invisible) parts or properties of objects/scenes. Such datasets integrate information across modalities (e.g., imaging types, annotation types, or simulated/real sources) and domains (e.g., vision, robotics, medical imaging), structuring each data point to support the estimation or reconstruction of object attributes beyond the directly observable. Hybrid AMO datasets are typically constructed to facilitate robust, generalizable learning and evaluation for segmentation, completion, or whole-body control tasks where direct perception is incomplete or compromised.

## 1. Concept and Motivation

A hybrid AMO dataset explicitly fuses information from multiple annotation or sensor modalities, and—crucially—embodies a task structure involving both modal and amodal supervision. In amodal instance segmentation or completion, the goal is to estimate not only what is visible, but also to infer the occluded or unobservable regions and content. This is relevant in scenarios ranging from robotic manipulation (estimating the shape and pose of objects under occlusion) to medical imaging (segmenting organs across different imaging modalities) and humanoid robotics (whole-body motion where not all degrees of freedom are observable or directly controlled).

Motivation for these datasets arises from the observed empirical gap between standard datasets focusing on visible (modal) regions or standard kinematic tasks, and the real-world demands of generalization to occlusion, distribution shift, or high-level command decomposability [2507.00339, 2208.12041, 2505.03738, 2503.06261].

## 2. Dataset Construction Methodologies

Hybrid AMO datasets are constructed using workflows that integrate multimodal data sources and combine synthetic, real, or algorithmically augmented signals. Key examples include:

- **Simulation + Procedural Labeling:** Synthetic scenes with rendered ground-truth provide occlusion-free amodal labels, as in MOVi-MC-AC, which simulates cluttered household environments with rendered RGB, modal/amodal masks, per-object “x-ray” images, and consistent 3D IDs across multi-camera settings [2507.00339].
- **Real + Synthetic Augmentation:** Real hand-annotated object masks and categories (modal) are post-processed by systematically overlaying synthetic occluders to generate paired modal/amodal mask annotations (e.g., Amodal-LVIS) [2503.06261].
- **MoCap + Trajectory Optimization Fusion:** In whole-body humanoid control, upper-body motion data from motion capture is combined with algorithmically sampled high-dimensional torso and base commands, which are used in a dynamics-constrained optimal control loop to generate full-body (including amodal lower-body) joint states, forming a hybrid AMO dataset for reference motion [2505.03738].
- **Cross-Modality Medical Datasets:** Hybrid AMO datasets such as AMOS provide both CT and MRI volumes for the same segmentation benchmark, requiring modality-aware pre-processing, label harmonization, and domain-specific augmentation pipelines [2206.08023, 2208.12041].

Table: Key Examples and Their Modalities

| Dataset         | Modalities/Annotations Integrated                     | AMO Task Domain        |
|-----------------|------------------------------------------------------|------------------------|
| MOVi-MC-AC      | RGB, depth, modal mask, amodal mask, amodal “x-ray”  | Amodal segmentation/content completion |
| Amodal-LVIS     | Real modal mask, synthetic amodal mask               | Image amodal segmentation |
| Hybrid AMO (AMO, robotics) | MoCap (modal) + trajectory-optimized amodal poses | Humanoid whole-body reference completion |
| AMOS            | CT, MRI, voxel-level 15-organ masks                  | Hybrid medical segmentation |

## 3. Annotation and Ground-truth Generation

The annotation protocols in hybrid AMO datasets differ from single-modal datasets. For example:

- **Amodal Segmentation and Content Completion:** In MOVi-MC-AC, each object instance in a given camera/frame has both its modal mask (occluded by other objects) and amodal mask (complete silhouette as if unobstructed), as well as amodal RGB content generated by rendering objects in isolation. Similarly, Amodal-LVIS provides for each object both its visible mask (modal) and its ground-truth amodal mask, produced by compositing occluders following an explicit occlusion ratio protocol [2507.00339, 2503.06261].
- **Cross-modal Medical Labeling:** In AMOS, 15 abdominal structures are annotated voxelwise for both CT and MRI, using cross-institutional consensus protocols and normalization pipelines for annotation consistency [2206.08023].
- **Trajectory Fusion:** In humanoid robotics, the hybrid AMO dataset provides tuples of upper-body joint angles, sampled torso commands (yaw, pitch, roll, height), and trajectory-optimized reference lower-body joint angles, generated via a multi-contact optimal control problem that enforces dynamic feasibility [2505.03738].

## 4. Dataset Composition, Statistics, and Format

Hybrid AMO datasets are typically large-scale and cover a wide state/action or scene space. Examples:

- **MOVi-MC-AC:** Encompasses 2,041 simulated scenes, each with 6 cameras and 24 frames, yielding ∼5.9M object instances with both modal and amodal masks, and amodal content, of which ∼4.1M instances are partially or fully occluded in at least one view [2507.00339].
- **Amodal-LVIS:** Contains 301,493 images and 399,398 object instances, with occlusion statistics such as 50% Percentage of Instances Occluded (POI), and average Ratio of Occluded Regions (ROR) of 34.5% [2503.06261].
- **AMOS CT+MRI:** 600 volumes (500 CT, 100 MRI) with 15-organ voxelwise ground truth per modality, with training and test splits detailed by scanner vendor and domain [2206.08023].
- **Hybrid AMO, Humanoid Robotics:** Number of samples not stated, but collection involves dense sampling of upper-body trajectories and uniformly sampled torso commands, supporting robust coverage across SO(3) orientation and base height for lower-body reference estimation [2505.03738].

Data formats often follow established standards, e.g., COCO-style JSON for Amodal-LVIS, specialized per-scene/camera/object directories for video-based datasets, and matrix or tensorized segmentation volumes in medical datasets.

## 5. Task Definitions, Losses, and Evaluation Protocols

Primary tasks enabled by hybrid AMO datasets include:

- **Amodal Segmentation:** Predicting the full object mask, including hidden parts, given (potentially multi-view/multi-frame) observations. Losses: Binary cross-entropy and Dice loss on mask predictions, evaluated via mIoU and mIoUₒcc (restricted to occluded pixels) [2507.00339, 2503.06261].
- **Amodal Content Completion:** Inferring RGB or depth content of the occluded regions. Losses include $L_1$ error, PSNR, SSIM, LPIPS, computed globally and over occluded regions only [2507.00339].
- **Multi-view Consistency:** Reprojection error between camera pairs and 3D centroid consistency for inferred object representations [2507.00339].
- **Hybrid Multi-organ Segmentation:** Volumetric and surface agreement evaluated using Dice Similarity Coefficient (DSC), normalized surface Dice (NSD), and Jaccard/IoU. Task-specific benchmarks for both intra-modality and hybrid splits [2208.12041, 2206.08023].
- **Whole-body Reference Mapping (Robotics):** Supervised loss for lower-body reference prediction ($L_\mathrm{sup}$ as squared error w.r.t. trajectory optimization output), with subsequent penalty in RL for deviation from module outputs at deployment [2505.03738].

## 6. Representative Results and Benchmarking

Hybrid AMO datasets serve as the foundation for new modeling and benchmarking approaches:

- **MOVi-MC-AC:** Provides task guidance but does not report comprehensive baselines, recommending standard models (U-Net, Mask R-CNN, diffusion models) and explicit reporting metrics on the held-out test split [2507.00339].
- **AMOS (Medical):** Deep supervised 3D U-Net with hybrid CT/MRI input achieves mean DSC of 0.8504 (CT-only) and 0.8476 (CT+MRI) on the test set; normalized surface Dice follows a similar trend [2208.12041].
- **Amodal-LVIS:** In zero-shot evaluations, models trained on the hybrid dataset outperform previous state of the art by >10 AP on the COCOA-cls benchmark and >8 AP on D2SA, as measured by Average Precision and Average Recall under standard AP / AP50 / AP75 protocols [2503.06261].
- **Hybrid AMO (Robotics):** The AMO module trained on the hybrid dataset enables real-time, robust, and adaptive whole-body motion control, with OOD command generalization for the Unitree G1 robot [2505.03738].

## 7. Challenges, Limitations, and Recommendations

- **Synthetic vs. Real Discrepancy:** Synthetic occlusion and content creation may not fully capture the complexity of real-world occlusions, translucency, or shadowing, leading to domain adaptation challenges [2503.06261].
- **Long-tail and Diversity Gaps:** In datasets inheriting real-world distributions (e.g., LVIS), rare classes and extreme occlusions are underrepresented, impacting generalization [2503.06261].
- **Normalization and Modality Shift:** For hybrid modality tasks (e.g., CT+MRI), proper normalization and balanced sampling are essential to prevent model collapse to the dominant modality. Performance drops of 0.5–1.0% in DSC are observed when combining CT with MRI without sufficient normalization and sampling care [2208.12041].
- **Augmentation and Validation:** Modality-specific augmentations and validation splits are recommended to mitigate distribution gaps and support robust generalization [2208.12041, 2206.08023].
- **Hybrid Data Design:** Explicit protocols for inter-annotator agreement, modality-aware augmentation, and per-modality validation are encouraged when constructing new hybrid AMO datasets.

A plausible implication is that hybrid AMO datasets are becoming central to developing and evaluating models capable of generalizable, occlusion-robust, and multimodal perception, with methodological innovations focusing on dataset construction, cross-modal/occlusion augmentation, and uncertainty-aware model design.

**References**:  
[2507.00339], [2208.12041], [2206.08023], [2505.03738], [2503.06261]

Source: https://www.emergentmind.com/topics/hybrid-amo-dataset