LabelFusion: Scalable Annotation Fusion

• LabelFusion is a robust framework that combines multi-view RGB-D data, expert annotations, and automated models to generate scalable, high-quality labels.
• It employs techniques such as 3D volumetric reconstruction, ICP mesh registration, and uncertainty quantification to improve segmentation and pose estimation.
• The framework adapts label fusion strategies across domains—from medical imaging to text understanding—ensuring reliable multi-source integration.

LabelFusion encompasses both a family of frameworks for robust data labeling and an eponymous pipeline for scalable ground truth generation in real RGB-D scenes. The term covers methods for fusing label information from diverse sources—expert annotators, automated perception models, or LLMs—to generate high-quality, uncertainty-aware labels at scale. Across modalities, these approaches enable accelerated annotation, principled uncertainty quantification, reliable data curation, and improved model performance for downstream tasks in vision, robotics, medical imaging, and natural language processing.

1. Origins: LabelFusion for Real-World RGB-D Annotation

The original LabelFusion pipeline (Marion et al., 2017) was introduced to address the need for large, precisely labeled RGB-D datasets for object segmentation and pose estimation in cluttered scenes. Its process involves:

• Multi-View RGB-D Video Acquisition: Scenes are captured via hand-held or robot-arm-mounted depth sensors, generating synchronized video streams (≈120 s per log, ∼3,600 frames/scene), covering diverse lighting, backgrounds, and occlusions.
• 3D Dense Volumetric Reconstruction: ElasticFusion is used to build a TSDF-based mesh of the environment, tracking all camera poses.
• Human-Assisted Mesh Registration: For each object, a CAD mesh is aligned to the reconstruction using a 3-point initialization (Kabsch algorithm) and iterative point-to-point ICP refinement.
• Pixelwise Label Generation: The object meshes, registered in the global frame, are reprojected onto all captured camera views using stored poses and intrinsic parameters, yielding dense, per-pixel, per-frame semantic and 6-DoF pose labels.

LabelFusion scales this pipeline, generating over 1,000,000 labeled object instances from 352,000 images with annotation times of ≈30 s/object, orders of magnitude faster than polygonal mask labeling. Experiments demonstrate that training segmentation networks (e.g., DeepLabv2-ResNet101) on this data yields high mean IoU, with multi-object, occluded scenes and diverse backgrounds providing substantially higher generalization than large quantities of single-object data. Diminishing returns are noted beyond 0.3–3 Hz frame sampling for coverage (Marion et al., 2017).

2. Label Fusion Methodologies in Multi-Source and Multi-Annotator Settings

Label fusion extends beyond geometric pipelines, generalizing to the unification of inconsistent or heterogeneous labels from multiple sources:

A. Medical Imaging with Multiple Expert Raters

Standard fusion schemes include:

• STAPLE: Simultaneous estimation of latent truth and rater performance via EM, learning voxel‐wise sensitivities and specificities (Lemay et al., 2022).
• Averaging: Voxelwise mean of rater masks to generate “soft” labels reflecting inter-rater disagreement.
• Random Sampling: During training, randomly sample one annotator's mask per instance/batch, exposing the model to all expert perspectives over optimization.

The SoftSeg framework treats the segmentation task as regression to soft targets, addressing information loss from binarization. It employs normalized ReLU activations and the Adaptive Wing loss, resulting in systematically superior calibration and fidelity to inter-rater uncertainty versus classical Dice loss with sigmoid/softmax. Empirically, SoftSeg+STAPLE or random sampling best preserves uncertainty structure, while averaging can cause underconfidence. The best fusion strategy is task-dependent (Lemay et al., 2022).

B. Sample-wise Label Fusion for Noisy Annotators

Sample-wise label fusion, as formalized by Gao et al. (Gao et al., 2022), models both the per-sample confusion matrix and importance weights for each annotator. For a classification datum with $R$ noisy labels, the network:

• Predicts sample-specific confusion matrices $P_n^{(r)}$ for each annotator, composed as convex combinations of permutation matrices leveraging the Birkhoff–von Neumann theorem.
• Infers annotator weights $w_n$, indicating confidence per sample.
• Fuses “cleaned” individual labels into a soft target, optimizes a KL divergence to this target with regularization.

This approach produces improved robustness and convergence relative to global confusion/weight methods (majority voting, MBEM, TraceReg, WDN), with especially notable gains when annotator quality is highly variable across samples (e.g., +15 pp on ImageNet-100 vs. best baseline) (Gao et al., 2022).

3. Advanced Multi-View and Modality Fusion in Vision

Modern label fusion includes multi-view and multi-model approaches for both supervised and weakly supervised annotation:

A. 3D Mesh-Based Semantic Fusion

LabelFusion in semantic segmentation (Fervers et al., 2021) fuses per-frame segmentation predictions onto a 3D mesh representation:

• Triangular mesh creation: via BundleFusion or COLMAP.
• Texel subdivision: Each triangle is subdivided into texels, collecting all texels across the mesh.
• Projection and aggregation: Per-pixel network outputs (class probability vectors) from each frame are projected into the texel mesh and combined. Aggregators include unweighted sum-and-normalize, frame-weighted averages, or Bayesian multiplicative fusion (product of probabilities).
• Rendering: Improved semantic images are generated by rendering fused texels via original camera parameters.

This uncertainty-aware, CUDA-accelerated approach yields significant improvements (e.g., ESANet on ScanNet: 52.05% → 58.25% pixel accuracy with label fusion) (Fervers et al., 2021).

B. Fusion of Perception Models and Multi-View Consistency

The term "LabelFusion" also encompasses recent pipelines that fuse labels generated by multiple state-of-the-art perception models and resolve inconsistencies via multi-view voting (Li et al., 2023). Key components include:

• Single-View Fusion: Runs segmentation (MaskFormer), object detection (Detic), and segment-anything (SAM) in parallel; segments are assigned semantic labels and merged using weighted per-pixel voting.
• Multi-View Region Voting: Uses known poses and depth to project single-view segmentations into a common frame, then reassigns each "superpixel" label via majority voting from all views, correcting errors due to occlusion or viewpoint.
• Pseudo-Label Generation: Applies multiple thresholds, region size checks, and consistency filters.

This approach, validated in indoor scene annotation (ADE20K, Active Vision Dataset), produces pseudo-labels comparable to dense human annotation, supports downstream navigation and part-discovery, and improves manipulation task success rates (+10.5 pp over a zero-shot VLMap baseline) (Li et al., 2023).

4. LabelFusion in Text and Document Understanding

The LabelFusion ensemble (Schlee et al., 11 Dec 2025) extends the fusion paradigm to robust text classification by combining a strong, fine-tuned transformer backbone (e.g., RoBERTa) with per-class scores from LLMs:

• Architecture: Transformer embedding/logits $\mathbf{h}\in\mathbb{R}^d$ and LLM per-class scores $\mathbf{s} \in \mathbb{R}^K$ (via structured prompt engineering) are concatenated and passed to a compact fusion MLP, yielding the final probability vector.
• Prompting: Each candidate class is enumerated explicitly and probabilities are requested in a parseable format.
• Training: End-to-end optimization with cross-entropy (multi-class) or independent BCE (multi-label), default learning rates of $2\mathrm{e}^{-5}$ (transformer), $1\mathrm{e}^{-3}$ (MLP), and batch sizes 32 (transformer), 8 (LLM).

The learned fusion delivers state-of-the-art robustness, e.g., 92.4% accuracy on AG News and 92.3% on Reuters 21578, outperforming transformer or LLM alone especially in low-data regimes (92.2% at 20% data, whereas transformer alone requires all data) (Schlee et al., 11 Dec 2025). It enables fine-grained trade-offs between latency, cost, and accuracy by gating inference to invoke LLMs only when beneficial.

An analogous methodology enables label-efficient document layout analysis by fusing structured LLM priors with visual detector outputs via inverse-variance weighting and learned adaptive gates, yielding refined pseudo-labels for semi-supervised learning, outperforming the document-pretrained LayoutLMv3 and requiring only 5% annotations to match SOTA (Shihab et al., 12 Nov 2025).

5. Theoretical and Algorithmic Advancements

Label fusion methodologies have been mathematically formalized for distributed tracking, medical uncertainty, and multi-model inference:

• Distributed Sensor Fusion: The joint labeled covariance intersection (JL-GCI) approach lifts labels to a product space, combines per-agent LMB densities via geometric mean, and marginalizes to yield globally consistent labels, handling the label-association ambiguity inherent in multi-agent tracking. JL-GCI enables more accurate cardinality and labeling under challenging detection rates compared to conventional label-matching GCI (Jin et al., 2020).
• Uncertainty Calibration: SoftSeg and similar regression-style frameworks with soft fusion targets systematically yield well-calibrated models (ECE ~2–3%) that closely reflect inter-rater entropy, compared to overconfident Dice-loss models (~16–20%) (Lemay et al., 2022).
• Adaptivity: Instance-adaptive fusion and sample-specific confusion modeling (as in (Gao et al., 2022, Shihab et al., 12 Nov 2025)) allow label fusion systems to automatically discover when label sources disagree or are contextually unreliable, optimizing integration at per-instance granularity.

6. Limitations, Trade-offs, and Future Directions

Across domains, the following considerations are salient:

• Data and Computational Cost: While LabelFusion pipelines enable high-quality annotation at scale, methods involving LLM queries, multi-view fusion, or intensive pseudo-label regeneration still require careful management of API costs, storage, and compute.
• Optimality and Generalization: In extremely sparse data regimes, reliance on weakly trained backbones can degrade fusion performance (Schlee et al., 11 Dec 2025). Some pipelines depend on accurate depth, pose, or additional structural priors (e.g., registration errors in 3D meshes, pose errors in SLAM).
• Adaptability: The most effective fusion and uncertainty-calibration strategy is often dataset- and task-specific; tuning may be needed to balance accuracy, calibration, and preservation of real-world uncertainty or disagreement.
• Extension Paths: Adaptive fusion gating, multi-provider/model ensembles, and dynamic prompt engineering with soft or learned prompts are identified as promising directions for further robustness, efficiency, and generalization (Schlee et al., 11 Dec 2025).

LabelFusion, in all its variants, represents a broad paradigm shift toward leveraging the complementary strengths of diverse annotation sources—human, algorithmic, and generative—via principled, uncertainty-aware fusion for scalable annotation and robust model training.