Incomplete Multi-View Multi-Label Classification via Shared Codebook and Fused-Teacher Self-Distillation

Abstract: Although multi-view multi-label learning has been extensively studied, research on the dual-missing scenario, where both views and labels are incomplete, remains largely unexplored. Existing methods mainly rely on contrastive learning or information bottleneck theory to learn consistent representations under missing-view conditions, but loss-based alignment without explicit structural constraints limits the ability to capture stable and discriminative shared semantics. To address this issue, we introduce a more structured mechanism for consistent representation learning: we learn discrete consistent representations through a multi-view shared codebook and cross-view reconstruction, which naturally align different views within the limited shared codebook embeddings and reduce feature redundancy. At the decision level, we design a weight estimation method that evaluates the ability of each view to preserve label correlation structures, assigning weights accordingly to enhance the quality of the fused prediction. In addition, we introduce a fused-teacher self-distillation framework, where the fused prediction guides the training of view-specific classifiers and feeds the global knowledge back into the single-view branches, thereby enhancing the generalization ability of the model under missing-label conditions. The effectiveness of our proposed method is thoroughly demonstrated through extensive comparative experiments with advanced methods on five benchmark datasets. Code is available at https://github.com/xuy11/SCSD.

• The paper's main contribution is SCSD, which integrates shared codebook quantization and fused-teacher self-distillation to robustly address incomplete multi-view multi-label classification.
• The proposed approach uses cross-view reconstruction and label-correlation fusion to enhance semantic alignment and outperform baselines by significant margins in AP and related metrics.
• Experimental validations on five benchmark datasets confirm the framework's stability under increasing missing data and its efficiency in modeling complex label dependencies.

Incomplete Multi-View Multi-Label Classification via Shared Codebook and Fused-Teacher Self-Distillation

Problem Setting and Motivation

This paper addresses the multi-view multi-label classification under dual-missing conditions, i.e., where both views and labels are incomplete, termed IMVMLC. The dual-missing regime is realistic but underexplored due to practical issues such as sensor failure, annotation sparsity, and privacy-preserving data collection. Existing approaches rely heavily on loss-based alignment (e.g., contrastive loss, information bottleneck) for representation consistency or on fusion strategies that do not explicitly model label structure, resulting in limitations in cross-view semantic alignment and robustness to missingness.

Methodology

Shared Codebook Quantization and Consistent Representation Learning

The core methodological innovation is the use of a learnable multi-view shared codebook for discrete representation quantization, combined with cross-view reconstruction. Each view is encoded to a shared latent space, quantized via grouped vector quantization into codebook tokens, thereby enforcing stronger alignment across views within a discrete (limited) embedding set. This reduces view redundancy and encourages compact, semantically-consistent representation learning. The cross-view reconstruction mechanism further aligns modality-specific decoders by requiring each view’s quantized embeddings to reconstruct not only their own input but those of other views, thereby exploiting cross-modal semantic regularities beyond simple feature alignment.

Label-Correlation-Oriented Decision Fusion

At the output decision stage, a novel view-weighting fusion strategy is proposed. This strategy computes label correlation matrices from both ground-truth and each view’s predictions and weights predictions according to their ability to preserve label co-occurrence structures. The weights are computed adaptively with a temperature-controlled softmax on the Frobenius norm between predicted and ground-truth label relation matrices, prioritizing views whose predictions better match the semantic dependencies of the true label space. This approach explicitly leverages multi-label structure beyond naïve averaging or learnable weight fusion, enhancing discriminative fusion under variable view quality.

Fused-Teacher Self-Distillation

To address label missingness and further improve single-view branch generalization, the fused multi-view prediction is used as a teacher in a self-distillation framework. The teacher distills global knowledge to each view-specific classifier via a multi-label logit distillation (MLD) loss, allowing the student branches to absorb fused global semantics even in the absence of complete view or label data. The overall training objective combines binary cross-entropy loss, reconstruction and quantization penalties, and the distillation term.

Complexity

The framework exhibits linear computational complexity in the number of samples, and codebook utilization analysis demonstrates full engagement of the quantization dictionary, mitigating the risk of codebook collapse and preserving representation diversity.

Experimental Validation

Extensive experiments on five benchmark multi-view multi-label datasets (Corel5k, Pascal07, Espgame, Iaprtc12, Mirflickr) are conducted under both dual-missing and complete data regimes. SCSD yields leading performance across all metrics (AP, 1-HL, 1-RL, AUC, 1-OE, 1-Cov), substantiating its superiority.

Notably, SCSD outperforms the strongest baseline (DRLS) on Espgame and Iaprtc12 by 5.83% and 8.15% in AP, respectively, with even larger, consistent gains relative to contrastive (DICNet) and information bottleneck-based (SIP) approaches—average AP improvements of 14.94% and 8.65% across all datasets. The results are especially strong for datasets with high label cardinality, signaling the framework’s ability to model complex label dependencies and maintain multi-view consistency under severe missingness.

Ablation studies confirm the necessity of the codebook quantization, cross-view reconstruction, correlation-based fusion, and self-distillation components. Parameter sensitivity and missingness analysis show stable performance as missingness increases, with view-missing having larger impact due to the role of cross-view alignment.

Implications and Future Directions

This work makes significant advances in robust multi-view, multi-label learning under practical, challenging missingness settings. The discrete shared codebook mechanism enforces semantic regularity and reduces redundancy, while the fusion and self-distillation modules synergistically leverage label structure and fused knowledge to mitigate incomplete supervision.

Potential directions include further reducing the memory and computational overhead of maintaining and updating the shared codebook, developing adaptive mechanisms for extreme view-missing rates, and extending this framework to non-linear fusion strategies or incorporating prior domain-specific knowledge into the fusion process. The quantization paradigm opens avenues for application in resource-constrained environments, federated learning, and explainable multi-modal AI.

Conclusion

The SCSD framework establishes new performance standards for incomplete multi-view multi-label classification by combining shared-codebook quantization, structural label-aware fusion, and fused-teacher distillation. It demonstrates robust generalization across scenarios and provides a versatile foundation for future multi-modal semantic learning research (2604.04170).