---
title: Sample-wise Label Fusion
url: https://www.emergentmind.com/topics/sample-wise-label-fusion
type: topic
---

# Sample-wise Label Fusion

Sample-wise label fusion refers to the family of algorithmic strategies that combine multiple sources of label information—annotations, predictions, or per-class scores—for each data sample independently to produce a unified, sample-specific target or inference. Unlike global or static fusion, sample-wise approaches adapt fusion logic to the characteristics or uncertainties of each sample, and are highly relevant in multi-annotator supervision, ensemble classification, semi-supervised settings, medical imaging, and retrieval-based tasks.

## 1. Fundamental Architectures and Algorithms

Sample-wise label fusion can be instantiated through diverse methods depending on the problem domain and the sources of label signals:

- **Vector Concatenation Models:** LabelFusion’s AutoFusionClassifier fuses a traditional transformer embedding $h^{(b)}\in \mathbb{R}^d$ with a large language model–derived per-class score vector $s^{(\text{LLM})}\in\mathbb{R}^C$ for each text input, forming a sample-specific $h^{(\text{fusion})} = [ h^{(b)} ; s^{(\text{LLM})} ] \in \mathbb{R}^{d+C}$, which is then passed through a FusionMLP to output the final prediction [2512.10793].

- **Multi-annotator Confusion/Competence Approaches:** Methods such as STAPLE consensus, average fusion, and random sampling operate per sample (medical image, instance) to aggregate multiple rater masks into a fused segmentation, retaining inter-rater uncertainty at the sample level [2202.07550]. Probabilistic fusion with sample-wise confusion matrices is prominent in multi-annotator settings; each sample $n$ is associated with one confusion matrix per annotator $P_n^{(r)}$ and a learned fusion vector $w_n$, yielding a soft target $y_n^{\text{targ}}$ specific to that sample [2207.11327].

- **Non-local Atlas/Feature Fusion:** In anatomical segmentation, CompareNet combines a voxel-wise classification unary score with a deep feature-based similarity term computed in a non-local window around each target voxel. The fusion is carried out individually at the sample (volume) and voxel levels [1910.04797].

- **Rank-Based Retrieval Fusion:** In extreme multi-label text classification (XMTC), sample-wise fusion involves aggregating candidate label rankings from separate retrieval systems (sparse BM25, dense BERT) per sample, using schemes such as CombSUM, CombMNZ, or Reciprocal Rank Fusion, followed by per-sample normalization and thresholding [2507.03761].

- **Label-Wise Encoding and Fusion:** LW-PT designs a per-label encoder and for each document stacks the outputs across all labels (regardless of ground-truth presence) into a single sample-specific representation, which is passed as input to the multi-label classifier [2008.06695].

## 2. Mathematical Foundations of Sample-wise Fusion

Key mathematical structures underlying sample-wise label fusion include:

- **Concatenation and MLP Fusion:** Given $h^{(b)}$ and $s^{(\text{LLM})}$, the AutoFusionClassifier forms $h^{(\text{fusion})} = [ h^{(b)} ; s^{(\text{LLM})} ]$, followed by $h_1 = \text{ReLU}( W_1 h^{(\text{fusion})} + b_1 )$, $z = W_2 h_1 + b_2$, and probabilities via softmax or sigmoid [2512.10793].

- **Sample-wise Confusion Matrices:** In multi-annotator settings, for sample $n$, annotator $r$ has a confusion matrix $P_n^{(r)} \in \mathbb{R}^{K \times K}$; the cleaned label is $y_n^{(r,cln)} = P_n^{(r)} y_n^{(r)}$ (with $y_n^{(r)}$ one-hot), and the final soft target is $y_n^{\text{targ}} = \sum_{r=1}^{R} w_{n,r} y_n^{(r,cln)}$ [2207.11327].

- **STAPLE EM Consensus:** For segmentation, the posterior of the true label per voxel $i$ is $q_i$, computed via EM using all rater masks and rater-specific sensitivity/specificity, yielding $q_i = P(t_i=1|y_i^{(1:R)})$ [2202.07550].

- **Deep Non-local Fusion:** At each target voxel $p$, CompareNet computes $S_{p}(T) = S_{\text{unary},p}(T;\theta_{u}) + \alpha S_{\text{pairwise},p}(T;\theta_{p})$, with $S_{\text{pairwise},p}(T)$ a weighted sum over nearby atlas voxels using learned similarity [1910.04797].

- **Rank-based Fusion Algorithms:** Typical formulas are:

   - $F_{\text{CombSUM}}(s,\ell) = f'_s(s,\ell) + f'_d(s,\ell)$
   - $F_{\text{CombMNZ}}(s,\ell) = (f'_s(s,\ell) + f'_d(s,\ell)) \cdot c(s,\ell)$
   - $F_{\text{RRF}}(s,\ell) = 1/(k+r_s(s,\ell)) + 1/(k+r_d(s,\ell))$ [2507.03761].

## 3. Application Domains and Empirical Performance

Sample-wise label fusion is employed across varied domains:

- **Medical Imaging (Inter-Rater and Atlas Fusion):** STAPLE, average fusion, and random sampling all provide sample-specific fusion masks for segmentation; SoftSeg regression frameworks result in superior calibration and preservation of inter-rater uncertainty, with ECE reduced to ≈2–3% and MAE on predictive entropy cut by 45–50% [2202.07550]. CompareNet’s end-to-end deep sample-wise fusion achieves Dice scores surpassing classical methods (e.g., 84.5% vs. 80.2% on IBSR, 74.6% vs. 71.9% on MICCAI 2012) while offering enhanced robustness to pathologies [1910.04797].

- **Multi-annotator Classification:** Sample-wise confusion/fusion models outperform global approaches on MNIST, CIFAR-100, and ImageNet-100 (e.g., 92.49% vs. 87.72% accuracy on MNIST; up to 19.5 points improvement on ImageNet-100), with marked gains on samples subject to annotator-specific reliability shifts [2207.11327].

- **Text Classification Ensembles:** LabelFusion’s concatenation of transformer and LLM signals at the sample level yields 92.4% accuracy on AG News and enables cost-aware gating of LLM queries [2512.10793]. LW-PT’s sample-wise fusion via concatenation of label-specific encodings boosts Macro-F1 by up to +14 points over state-of-the-art baselines [2008.06695].

- **Extreme Multi-label Retrieval:** Sample-wise fusion of sparse and dense retrieval outputs improves nDCG@1, Precision@5 for both head and tail labels, with up to +2.5 absolute points gain on tail categories across benchmarks; CombMNZ over ZMUV-normalized scores consistently delivers highest coverage [2507.03761].

- **Pairwise Multi-label Classifiers:** Local fuzzy confusion-matrix–based sample-wise fusion corrects supports per test sample, with classifier weights based on per-sample mutual information; empirical results confirm enhanced macro-F1 and exact-match accuracy on imbalanced and dense multi-label datasets [1710.09710].

## 4. Calibration, Uncertainty, and Adaptivity

Sample-wise fusion methods are characterized by their ability to calibrate uncertainty and adapt fusion weights or mechanisms on a per-sample basis:

- **Preservation of Inter-Rater Uncertainty:** SoftSeg frameworks preserve the entropy of expert disagreement and output better-calibrated probabilities in medical segmentation. STAPLE and random-sampling fusion under SoftSeg minimize ECE and entropy-MAE, while average fusion yields greater underconfidence [2202.07550].

- **Per-sample Reliability Estimation:** Sample-wise confusion matrices ($P_n^{(r)}$) and fusion weights ($w_n$) enable per-instance reliability modeling, capturing annotator–sample dependencies unaddressed by global models. The weighting further addresses bias in samples with annotator-dependent error structure [2207.11327].

- **Information-theoretic Local Weighting:** Local fuzzy confusion matrix models assess classifier competence via sample-specific mutual information, enabling dynamic weighting at inference for improved coverage of rare or ambiguous labels [1710.09710].

- **Cost-aware Adaptivity:** LabelFusion employs confidence gating, adaptive LLM budgeting, and disk caching—all sample-dependent—to optimize accuracy–cost trade-offs [2512.10793].

## 5. Comparative Analysis with Global Fusion Schemes

Sample-wise fusion offers several distinct advantages over traditional global schemes:

- **Higher empirical accuracy and coverage:** Per-sample fusion adapts to sample-specific uncertainty, local structure, and annotator reliability, consistently outperforming majority voting, global confusion matrix, or single-model approaches [2207.11327], [2202.07550], [2008.06695].

- **Superior calibration characteristics:** Regression-style sample-wise fusion is systematically less overconfident, achieves lower ECE, and matches inter-rater entropy more closely [2202.07550].

- **Robustness to heterogeneity:** Sample-by-sample adjustment (as in local confusion matrix models or deep non-local fusion) allows the model to manage label/annotation density, dataset imbalance, and rare label performance without hand-tuned class-balancing [1710.09710], [1910.04797].

- **Efficient signal integration:** Architectural approaches such as direct concatenation of expert scores or label-wise encodings support optimization via single backpropagation passes and permit flexible inclusion of variable sources or retrieval signals [2512.10793], [2008.06695], [2507.03761].

## 6. Limitations, Extensions, and Future Directions

While sample-wise label fusion demonstrates broad empirical and algorithmic superiority, certain practical considerations and open research questions remain:

- **Computational cost:** Fine-grained sample-wise confusion modeling (e.g., with permutation matrix decompositions) incurs higher memory and computation cost per sample, although decomposition and batching strategies mitigate overhead [2207.11327].

- **Task dependence:** The optimal fusion strategy (STAPLE, average, random-sampling) can be task- and dataset-dependent; careful benchmarking is necessary [2202.07550].

- **Annotation structure scaling:** Extensions to large numbers of annotators ($R$), missing-label scenarios, or multi-source fusion are discussed as future work, with permutation-matrix decomposition and masking as promising directions [2207.11327].

- **Interpretability:** The adaptive and dynamic nature of sample-wise fusion models introduces complexity in interpretation of fusion weights/local confusion structure, motivating the need for diagnostic metrics and visualization tools.

- **Generalization:** While label-wise fusion architectures permit flexible stacking (e.g., in LW-PT), concatenation-based models may benefit from attention/gating extensions for large-scale, highly-correlated label spaces [2008.06695].

Sample-wise label fusion thus represents a versatile paradigm for integrating heterogeneous, uncertain, or multi-source label signals per sample, enabling enhanced accuracy, calibration, uncertainty quantification, and robustness relative to global or static fusion approaches. Its continued evolution addresses core challenges in multi-label learning, ensemble prediction, annotation reliability modeling, and cost-effectiveness of inference.

Source: https://www.emergentmind.com/topics/sample-wise-label-fusion