Multimodal Joint Label Correction (MJC)

• The paper introduces MJC as a mechanism that jointly refines labels using historical self-predictions, reducing biases from single-modality predictions.
• It utilizes methods like majority voting, cross-modal consensus, and per-class masking to correct modality-specific noise in tasks such as 2D–3D retrieval and audio-visual parsing.
• This approach improves classification and cross-modal alignment by addressing label noise, registration errors, and weak supervision across diverse datasets.

Searching arXiv for the named MJC paper and closely related multimodal label-correction work to ground the article in published sources. arXiv.search(query="all:\"Multimodal Joint label Correction\" OR ti:\"Multi-level cross-modal adaptive Correction and Alignment\" OR (Zou et al., 8 Aug 2025)", max_results=5, sort_by="submittedDate")

Multimodal Joint Label Correction (MJC) denotes a class of label-refinement procedures in which the trustworthiness of supervision is inferred jointly from multiple modalities rather than from a single branch in isolation. In the strictest usage presently available in the literature, MJC is the correction module introduced within the MCA framework for noisy 2D–3D cross-modal retrieval, where multimodal historical self-predictions are used to jointly model modality prediction consistency and to produce corrected hard labels together with a clean/noisy partition (Zou et al., 8 Aug 2025). More broadly, several earlier and parallel methods can be interpreted as MJC-like because they jointly reason over cross-modal inconsistency, modality-specific noise, or pseudo-label disagreement in order to suppress harmful supervision transfer or revise modality-level labels (Namin et al., 2017).

1. Definition and conceptual scope

In MCA, MJC is defined operationally as a mechanism that “leverages multimodal historical self-predictions to jointly model the modality prediction consistency, enabling reliable label refinement” for 2D–3D retrieval under noisy labels (Zou et al., 8 Aug 2025). The immediate target is category-annotation noise attached to 2D and 3D samples, where corrupted labels damage both classification supervision and cross-modal alignment. The method is motivated by the claim that correcting samples independently within each modality is vulnerable to confirmation bias, because a single branch can become overconfident in its own mistaken prediction (Zou et al., 8 Aug 2025).

A broader reading of the literature supports a more general encyclopedia definition: MJC comprises methods that use one modality as a consistency witness, denoising source, or conflict detector for another modality, and that modify supervision or message passing on the basis of joint multimodal evidence. This broader interpretation is explicitly suggested by “Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsing,” whose JoMoLD procedure starts from weak labels that are noisy for each modality, computes modality-specific evidence, compares evidence across modalities, decides which labels are likely noisy in each modality, and modifies supervision before optimization (Cheng et al., 2022).

Under that wider scope, MJC is not restricted to corrupted class annotations. It also includes cross-modal inconsistency caused by registration errors, missing observations, domain-specific label mismatch, weak supervision, and extreme label scarcity. “Soft Correspondences in Multimodal Scene Parsing” is particularly important in this regard, because it addresses 2D–3D multimodal semantic labeling when corresponding regions do not necessarily deserve the same label, and it introduces latent nodes to explicitly model such inconsistencies (Namin et al., 2017).

2. Problem settings and forms of label inconsistency

The most explicit MJC setting is noisy-label 2D–3D retrieval. MCA defines a multimodal dataset

$$\mathcal{D}=\left\{\mathcal{M}_j\right\}_{j=1}^M=\left\{\mathcal{X}_j,\mathcal{Y}_j\right\}_{j=1}^M,$$

with modality-specific samples $(\boldsymbol{x}_i^j,y_i^j)$ and noisy labels $y_i^j$. In this setting, wrong class labels harm not only intra-modal discrimination but also the semantic criteria that determine cross-modal alignment (Zou et al., 8 Aug 2025).

A distinct but closely related setting is weakly supervised audio-visual video parsing. JoMoLD assumes that only a video-level binary label $y \in \{0,1\}^C$ is available during training, whereas the desired outputs at evaluation time are segment-level audio, visual, and audio-visual labels. The paper defines modality-specific noisy labels as labels that are valid at the video level but invalid for one modality, such as a visible but silent dog or an off-screen ringing telephone. Its key assumption is that “a labeled event should appear in at least one modality,” which turns cross-modal loss asymmetry into a denoising signal (Cheng et al., 2022).

In multimodal scene parsing, the inconsistency problem is more geometric and ontological. “Soft Correspondences in Multimodal Scene Parsing” studies 2D images and 3D point clouds and lists several cross-modal inconsistencies: sensor misalignment and registration errors, different visibility or missing observations, domain-specific label mismatch, dynamic scene mismatch, and annotation or projection-induced errors. On the DATA61/2D3D dataset, the paper quantifies the phenomenon by stating that 17% of the 2D–3D connections correspond to inconsistent labels (Namin et al., 2017). This directly challenges the common assumption that corresponding regions in different modalities must share identical labels.

A further extension arises in multimodal sentiment analysis. MUG assumes that multimodal labels are available but unimodal labels are missing, and argues that simply reusing the multimodal label as a unimodal target creates noisy pseudo-labels because modalities can differ in polarity, intensity, or informativeness. The framework therefore treats multimodal labels as weak supervision for learning corrected modality-specific labels (Mai et al., 2024).

These settings suggest that “label correction” in multimodal learning is best understood as a family of operations over supervision mismatch, not only as recovery from random annotation corruption.

3. Core algorithmic mechanisms

The canonical MJC mechanism in MCA is based on temporal aggregation within each modality and agreement across modalities. For sample $i$ and modality $j$, the model stores recent predictions in a historical bank

$$\mathcal{H}_i^j = [p_{i,(m-h+1)}^j, p_{i,(m-h+2)}^j, \ldots, p_{i,m}^j]_i,$$

updated in a First-In First-Out manner. Intra-modal consistency is then computed by majority voting,

$C_i^j = unique(\mathcal{H}_i^j),$

and inter-modal consistency is enforced through the consensus rule

$H(x_i^j)=\cap\{C_i^j\}.$

The sample is placed in the clean set or noisy set according to

$$S_c=\{x_i \mid H(x_i^j)=y_i^j\}, \qquad S_n=\{x_i \mid H(x_i^j)\neq y_i^j\}.$$

The output is therefore a hard corrected label $(\boldsymbol{x}_i^j,y_i^j)$0 together with a clean/noisy partition, not a soft pseudo-label distribution or an explicit scalar confidence score (Zou et al., 8 Aug 2025).

JoMoLD uses a different mechanism. It relies on two observations: networks tend to learn clean samples first, and a labeled event should appear in at least one modality. The method sorts the losses of all instances within a mini-batch individually in each modality, selects noisy samples according to the relationships between intra-modal and inter-modal losses, and estimates the noise ratio by calculating the proportion of instances whose confidence is below a preset threshold. The correction action is explicit masking rather than class relabeling:

$(\boldsymbol{x}_i^j,y_i^j)$1

Thus, selected positive labels are removed for a specific modality while the video-level label remains unchanged (Cheng et al., 2022).

“Soft Correspondences” embodies an earlier graph-based formulation. Its general multimodal CRF for $(\boldsymbol{x}_i^j,y_i^j)$2 modalities includes unary, intra-domain pairwise, and inter-domain pairwise terms, but the crucial innovation is the addition of latent nodes between modalities. These latent nodes explicitly model inconsistencies between modalities, allow the model to leverage information from both domains when a correspondence is reliable, and cut edges when regions are inconsistent. The paper also replaces hand-tuned parameters with learned intra-domain and inter-domain potential functions (Namin et al., 2017).

MUG introduces yet another mechanism: modality-specific residual correction networks under weak global supervision. For modality $(\boldsymbol{x}_i^j,y_i^j)$3, the corrected unimodal label is produced by a Meta Uni-label Correction Network (MUCN), and the final output has residual form

$(\boldsymbol{x}_i^j,y_i^j)$4

Training uses a bi-level optimization strategy with a unimodal denoising task in the inner loop and a multimodal denoising task in the outer loop. This is joint in supervision and optimization, though not a single coupled probabilistic correction model over all labels (Mai et al., 2024).

4. Representative systems

The current literature contains several methods that instantiate different versions of MJC or MJC-like behavior.

Paper	Setting	Correction action
“MCA: 2D-3D Retrieval with Noisy Labels via Multi-level Adaptive Correction and Alignment” (Zou et al., 8 Aug 2025)	Noisy-label 2D–3D retrieval	Historical voting, cross-modal consensus, clean/noisy partition, hard label correction
“Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsing” (Cheng et al., 2022)	Weakly supervised AVVP with modality-specific noisy labels	Per-class, per-sample masking of modality-specific positives
“Soft Correspondences in Multimodal Scene Parsing” (Namin et al., 2017)	2D–3D scene parsing with inconsistent correspondences	Latent-node inconsistency modeling and edge cutting
“Meta-Learn Unimodal Signals with Weak Supervision for Multimodal Sentiment Analysis” (Mai et al., 2024)	Missing unimodal labels under multimodal supervision	Modality-specific residual label correction via MUCN
“Cross-Modality Clustering-based Self-Labeling for Multimodal Data Classification” (Zyblewski et al., 2024)	Semi-supervised multimodal classification under extreme label scarcity	Per-modality label proposals plus cross-modal conflict resolution

These systems differ materially in what they correct. MCA corrects hard class labels and partitions data into clean and noisy sets (Zou et al., 8 Aug 2025). JoMoLD performs denoising-by-removal rather than full relabeling (Cheng et al., 2022). “Soft Correspondences” corrects the effect of bad correspondences by suppressing harmful cross-modal propagation, which can be interpreted as joint label correction plus correspondence rejection (Namin et al., 2017). CMCSL is better described as joint missing-label completion or pseudo-label arbitration than as noisy-label learning in the conventional sense, because it clusters each modality independently, propagates labels from a small seed set, and resolves disagreements by comparing Euclidean distances to modality-specific centroids (Zyblewski et al., 2024).

This diversity suggests that MJC is a methodological family rather than a single algorithmic template.

5. Relation to adjacent paradigms and common misconceptions

A common misconception is that MJC necessarily means full relabeling of corrupted annotations. The literature does not support that restriction. JoMoLD removes suspicious modality-specific positives without reassigning them to another class (Cheng et al., 2022). “Soft Correspondences” may leave labels unchanged while cutting unreliable inter-modal edges (Namin et al., 2017). CMCSL produces a shared pseudo-label through cross-modal arbitration, but it is driven by sparse supervision rather than observed label corruption (Zyblewski et al., 2024).

A second misconception is that MJC always requires a single unified probabilistic model over all labels and modalities. Some methods fit that description only partially. MUG, for example, is coupled through shared multimodal representations, weak supervision, and joint downstream optimization, yet its corrected unimodal labels are generated by separate modality-specific correction networks rather than by one explicit joint distribution over all corrected labels (Mai et al., 2024).

A third misconception is that all label noise in multimodal learning is annotation noise in the ordinary sense. Several papers show otherwise. In AVVP, the noise arises because a valid video-level label may be invalid for one modality (Cheng et al., 2022). In scene parsing, the issue may be projection-induced boundary error, missing visibility such as sky in 2D but not 3D, or semantic mismatch such as grass in 2D versus horizontal plane in 3D (Namin et al., 2017). This suggests that MJC is as much about modeling cross-view non-equivalence as about correcting corrupted labels.

Related but distinct research lines further clarify the boundaries of the topic. “Multi-Level Label Correction by Distilling Proximate Patterns for Semi-supervised Semantic Segmentation” proposes MLLC, a unimodal method based on Semantic-Level Graphs and Class-Level Graphs for pseudo-label correction; it is relevant conceptually but does not study multimodal joint label correction directly (Xiao et al., 2024). “Reconstruction Regularized Deep Metric Learning for Multi-label Image Classification” uses a joint image–label embedding space and a reconstruction module “for recovering correct labels,” but it is best categorized as joint multimodal embedding with implicit label refinement rather than explicit MJC (Li et al., 2020).

6. Limitations, open issues, and research directions

The present literature leaves several aspects of MJC underspecified or method-dependent. In MCA, reliability is implicit in temporal majority plus cross-modal agreement; the paper does not define a separate continuous confidence score, a standalone MJC loss, a threshold for accepting correction, a momentum coefficient, or a temperature parameter specific to MJC (Zou et al., 8 Aug 2025). This suggests that current formulations often rely on discrete consensus criteria rather than calibrated uncertainty estimation.

Another unresolved issue is the level at which correction should occur. Some methods operate at the sample level, some at the class-within-sample level, and some at the correspondence level. JoMoLD is explicitly class-aware and sample-aware, performing per-class, per-sample masking within a mini-batch (Cheng et al., 2022). “Soft Correspondences” acts on inter-modal regions through latent nodes and CRF inference (Namin et al., 2017). MUG corrects scalar modality-specific sentiment labels, not the multimodal label itself (Mai et al., 2024).

The interaction between correction and representation learning is also central. In MCA, MJC precedes MAA, because alignment requires reliable supervision (Zou et al., 8 Aug 2025). In MUG, corrected unimodal labels are subsequently used in joint unimodal and multimodal training (Mai et al., 2024). In CMCSL, self-labeling depends on modality-specific feature spaces and on distance comparability after preprocessing, with the best reported choice being $(\boldsymbol{x}_i^j,y_i^j)$5 normalization plus standard scaling (Zyblewski et al., 2024). A plausible implication is that MJC is rarely separable from feature learning, even when its visible output is only a corrected label.

Finally, the field still spans heterogeneous assumptions about what may be trusted. JoMoLD assumes that a positive event should not be noise for both modalities simultaneously (Cheng et al., 2022). MUG assumes that multimodal labels are available and mostly trustworthy, even if unimodal labels are not (Mai et al., 2024). “Soft Correspondences” assumes that cross-modal agreement must be modeled softly rather than imposed categorically (Namin et al., 2017). These differing assumptions explain why the term “MJC” can refer either to a specific module in MCA or to a broader research program concerned with joint label correction, denoising, and correspondence validation across modalities.