Coupled Training with Privileged Information and Unlabeled Data

Abstract: In many prediction problems, we have extra information during training (for example, measurements that are expensive or slow to collect) that will not be available when the model is deployed. A common strategy is to first train a model that uses all training information, then use its predictions on unlabeled examples to train a second model that only uses the inputs available at test time. However, when the extra training-only information is weak or noisy, this Two-Stage approach can mislead the deployment model and even hurt accuracy. We propose a joint training method that learns the two models together, so the deployment model can benefit from the extra information only when it actually helps, instead of inheriting its mistakes. We provide guarantees that describe when joint training improves prediction accuracy and analyze a simple alternating training algorithm for large, high-dimensional models. Experiments on synthetic data and real-world prediction tasks show that our approach avoids these failures and robustly outperforms standard Two-Stage baselines.

• The paper introduces a joint optimization framework that couples deployment and privileged-view models to prevent negative transfer from noisy or weak privileged information.
• It leverages additional unlabeled paired data and adaptive interpolation to balance robust supervised learning with pseudo-labeling strategies.
• Empirical results across regression and classification tasks demonstrate superior performance over Two-Stage methods, especially in high-dimensional, noise-prone settings.

Coupled Training with Privileged Information and Unlabeled Data: Summary and Analysis

Problem Setting and Motivation

The paper "Coupled Training with Privileged Information and Unlabeled Data" (2605.23268) addresses a nuanced semi-supervised learning scenario: models may access privileged information $W$ solely during training, alongside primary features $X$ and labels $Y$. At test time, only $X$ is available. Furthermore, a substantial amount of training data includes $W$ without labels. This situation arises in medical and industrial prediction, transfer learning, and distribution shift contexts. The objective is to leverage $W$ during training to improve the final prediction function $f(X)$, robustly minimizing distributional risk while avoiding negative transfer when $W$ is weak or noisy.

Limitations of Two-Stage Approaches

The prevalent Two-Stage approach first trains a rich-view model $g$ on $(X,W)$, then uses $X$0 to generate pseudo-labels for unlabeled samples, which are subsequently used to train the deployment model $X$1 on $X$2. However, when $X$3 provides weak signal, errors from $X$4 are inherited by $X$5, resulting in performance degradation relative to supervised learning on $X$6 alone. Negative transfer arises particularly when the privileged view $X$7 introduces high-dimensional nuisance variation or is only marginally informative.

Figure 1: Strong privileged signal (large $X$8).

Figure 2: Privileged signal strength $X$9.

These figures demonstrate how aggressive knowledge transfer via $Y$0 benefits the deployment model $Y$1 when privileged signal is strong, but degrades when signal is weak—underscoring the necessity for adaptive methods.

Coupled Training Framework

The paper proposes a coupled training framework, which jointly optimizes $Y$2 and $Y$3, instead of treating $Y$4 as a fixed teacher. The algorithm alternates between updating $Y$5 using current pseudo-responses from $Y$6, and recalibrating $Y$7 to agree with $Y$8 over the unlabeled set, bounded by a constraint parameter $Y$9 (or equivalently, penalized via a Lagrangian $X$0). This coupling adaptively regulates the influence of privileged information, interpolating between supervised learning (large $X$1 or small $X$2) and Two-Stage pseudo-labeling (small $X$3 or large $X$4).

Theoretical Guarantees

The authors rigorously analyze risk bounds for the proposed method. They show the population minimizers $X$5 solve fixed-point equations: $X$6 projects onto the regression target averaged over the joint model, and $X$7 interpolates between the deployment target and rich-view regression, weighted by $X$8.

Correlation-controlled risk bounds are derived, demonstrating the excess prediction error depends multiplicatively on the statistical complexity of the joint function class and a correlation coefficient $X$9 that quantifies the alignment between the errors of $W$0 and $W$1. When privileged information is uncorrelated with deployment error, the bound tightens; when highly correlated, the gain diminishes, recovering the labeled-only scenario.

High-Dimensional and Dictionary-Based Extension

A practical alternating forward selection (AFS) algorithm is developed for high-dimensional settings, constructing $W$2 and $W$3 as sparse expansions over dictionary elements. The AFS procedure achieves provable sublinear objective decay, and time complexity $W$4, offering substantial savings over naive blockwise selection.

Empirical Results

Comprehensive experiments are conducted on synthetic controls, regression (Parkinson's Telemonitoring), and classification (Bank Marketing). The results validate several key claims:

• Coupled Training outperforms both Two-Stage and labeled-only baselines when privileged signal is noisy or weak.
• The method exhibits stability against increasing nuisance dimensions in $W$5 (which degrade Two-Stage).
• Performance improves with additional unlabeled paired $W$6 data.
• For real-world datasets, Coupled Training consistently achieves lower test error and Brier scores than Two-Stage pseudo-labeling, generalized distillation, and SVM+ baselines.

Figure 3: Parkinson's dataset. Test MSE versus $W$7.

Figure 4: Bank Marketing dataset. Holdout Brier score versus $W$8.

Figure 5: PneumoniaMNIST. Test AUROC versus $W$9 for Algorithm 1.

Figure 6: Synthetic binary classification diagnostic. Test 0–1 error for the cross-entropy analogue of Coupled Training as a function of $W$0, averaged over seeds $W$1.

Empirical U-shaped curves with respect to $W$2 are observed, as predicted by theory: neither extreme (pure supervised or pure pseudo-labeling) is universally optimal, and adaptive interpolation is required.

Practical and Theoretical Implications

Theoretical results and empirical findings imply that joint optimization of deployment and privileged-view models allows robust exploitation of privileged information, even when it is noisy or high-dimensional. This is critical in real-world high-stakes domains (medicine, finance) where privileged test-time access is infeasible, and labeled training data are scarce.

The coupled framework generalizes classical LUPI, agreeing-style SSL, and dictionary-based greedy optimization, providing a unifying methodology with statistical guarantees.

Future Directions

Key avenues for future methodological and theoretical work include:

• Extension to non-square loss functions and general classification regimes.
• Handling semi-paired data, misspecification, and distributional shift.
• Development of practical proxies for the correlation coefficient $W$3 to guide tuning of $W$4.
• Broader deployment in high-dimensional nonlinear and tree-based models, as well as cross-domain transfer learning and multi-modal settings.

Conclusion

This paper advances semi-supervised privileged learning by introducing a coupled training framework, theoretically characterizing its robustness to negative transfer, and empirically demonstrating its superiority in diverse settings. The dictionary-based high-dimensional extension makes it practically viable at scale. The methodology reconciles several disparate strands of SSL/LUPI research, providing a principled approach for leveraging privileged information when available, but avoiding overreliance when it is unreliable. Robust interpolation between labeled-only and pseudo-labeling regimes emerges both in theory and practice, with clear implications for future adaptive learning systems in AI.