Fused Multinomial Logistic Regression Utilizing Summary-Level External Machine-learning Information

Abstract: In many modern applications, a carefully designed primary study provides individual-level data for interpretable modeling, while summary-level external information is available through black-box, efficient, and nonparametric machine-learning predictions. Although summary-level external information has been studied in the data integration literature, there is limited methodology for leveraging external nonparametric machine-learning predictions to improve statistical inference in the primary study. We propose a general empirical-likelihood framework that incorporates external predictions through moment constraints. An advantage of nonparametric machine-learning prediction is that it induces a rich class of valid moment restrictions that remain robust to covariate shift under a mild overlap condition without requiring explicit density-ratio modeling. We focus on multinomial logistic regression as the primary model and address common data-quality issues in external sources, including coarsened outcomes, partially observed covariates, covariate shift, and heterogeneity in generating mechanisms known as concept shift. We establish large-sample properties of the resulting fused estimator, including consistency and asymptotic normality under regularity conditions. Moreover, we provide mild sufficient conditions under which incorporating external predictions delivers a strict efficiency gain relative to the primary-only estimator. Simulation studies and an application to the National Health and Nutrition Examination Survey on multiclass blood-pressure classification.

• The paper introduces a fusion framework that integrates individual-level primary data with summary-level ML predictions to improve multinomial logistic regression efficiency.
• It leverages empirical likelihood with moment constraints to robustly address covariate shift, concept shift, and data heterogeneity without relying on density-ratio modeling.
• Simulation and real-data studies illustrate up to a 25% reduction in standard deviation and tighter confidence intervals compared to conventional MLE, confirming practical robustness.

Fused Multinomial Logistic Regression with Summary-Level External ML Information

Introduction

The paper "Fused Multinomial Logistic Regression Utilizing Summary-Level External Machine-learning Information" (2604.03939) develops a data fusion methodology for integrating individual-level data from a primary study with summary-level, black-box ML predictions obtained from large external datasets. The method leverages empirical likelihood principles to improve multiclass classification tasks—specifically, multinomial logistic regression—by incorporating robust, nonparametric external predictions through flexible moment restrictions. Importantly, the proposed framework is robust to covariate and concept shift—a significant advancement over methods that rely on explicit density-ratio modeling or assume aligned populations.

Problem Setting and Motivation

Practical research settings increasingly leverage multiple disparate datasets: high-quality, limited-size primary studies and large, summary-only external sources, often equipped with advanced ML predictions but lacking individual-level data access. The main inferential aim is accurate and interpretable estimation of multinomial logistic regression parameters in the primary domain. This is challenging due to the following:

• External data is only available as summary-level ML predictors and may have partial feature coverage or coarsened class labels.
• Covariate shift (distributional change in feature space between primary and external sources) and concept shift (discrepancies in outcome-generation mechanisms) are the rule rather than the exception.
• Efficient and robust integration is hindered by the lack of scalable methodology that can accommodate: (i) partial/missing features, (ii) label coarsening, (iii) distributional shift, and (iv) ML predictions with opaque mechanisms.

Methodology

Empirical Likelihood Fusion via External ML-Predicted Moments

The core methodological contribution is a general empirical likelihood framework using constraint-enriched likelihoods:

• The primary model: standard multinomial logistic regression fitted to $n$ iid observations $\{(Y_i, X_i)\}$ from the target population.
• The external source: summary-level, black-box nonparametric predictions $q_k(W)$ for class probabilities, learned from a much larger external sample, with $W$ a subset of available features.
• External information is introduced as moment constraints: functions of the type

$$E\left[ \{p_k(X|\theta) - q_k(W)\} h(X) \mid S=1 \right] = 0,$$

for a class of weight functions $h$.

• Full empirical likelihood is optimized under these external moment constraints, allowing for estimation of both shared (transportable) and source-specific (free) parameters.

Handling Heterogeneities and Data Quality Issues

• Covariate Shift: Rather than explicit density ratio modeling, the approach assumes missing components in $X$ (not present in $W$) are missing at random (MAR), justifying the validity of constraining moments involving external predictions.
• Concept Shift: Outcome-generating parameters (logit coefficients) are decomposed into shared and free components (intercepts for class-proportion differences, slopes for shared effects).
• Coarsened Labels and Partial Covariates: Accommodated by matching sums over the appropriate class aggregates in constraints.
• Numerical Stability: L2-penalization of Lagrange multipliers is used to regularize likelihood maximization, avoiding degeneracy.

Theoretical Properties

The authors establish a full large-sample theory for the fused maximum likelihood estimator (FMLE):

• Consistency: FMLE achieves consistency for the target parameters under standard regularity assumptions and MAR for unmeasured covariates in the external source.
• Asymptotic Normality: FMLE enjoys an explicit asymptotic normal distribution, with the variance structure reflecting both within-study information and the gain from external constraints.
• Strict Efficiency Gain: Sufficient (and mostly mild) conditions for strict efficiency improvement relative to the primary-only MLE are provided. Under appropriate richness of the constraint set (number of functions $h$), FMLE can substantially improve the variance of transported parameter estimates.

Simulation Results

The methodology is evaluated on an extensive synthetic study with $K=3$ outcome classes and primary/external samples $\{(Y_i, X_i)\}$0, examining different covariate shift scenarios (none, mean shift, variance shift, both). Key observations:

• Substantial SD Reduction: FMLE offers a pronounced reduction in SD (up to ~25%) for parameters associated with outcome groups adequately represented in the external data, with uniform robustness to covariate shift scenarios.
• MSE of Probabilities: FMLE further yields improved MSE in class probability prediction, especially when external label structure matches primary targets.
• Coverage and Bias: Both MLE and FMLE are nearly unbiased. Coverage probabilities for FMLE are close to nominal except in a minority of cases with omitted external features, where external prediction error affects finite-sample properties.

Real Data Application: NHANES Blood Pressure Classification

The approach is applied to blood pressure category classification in NHANES (US health survey), fusing

• a primary dataset with 9,186 units (14 features, full labels), and
• an external dataset with 12,425 units (8 shared features, possibly coarsened labels).

Covariate and concept shift are clearly present (substantial differences in distributions and class prevalences).

• Standardized mean differences (Figure 1) highlight covariate shift between datasets.

  Figure 1: Standardized mean differences for shared covariates between the primary and external sources demonstrate notable covariate shift.

• Efficiency Gains: FMLE produces point estimates comparable to MLE but with narrower confidence intervals, particularly when the primary-to-external sample size ratio is small. Improvements are most evident for shared features, aligning with theoretical predictions.
• Point Estimates and CIs: FMLE yields tighter CIs, demonstrating tangible gains of external fusion in realistic heterogeneity settings.

  Figure 2: Point estimates (dots) and confidence intervals (vertical bars) for primary (MLE) and fused (FMLE) estimators: tighter CIs for FMLE, reflecting efficiency gain.

Discussion and Future Directions

This work delivers a theoretically principled and practically robust fusion framework for multiclass regression under real-world mismatches in variable and label structure and population distribution. The fusion via empirical likelihood constraints efficiently and robustly leverages external ML models without density-ratio modeling, extending to settings with partial features and coarsened outcomes.

Practical and theoretical implications include:

• Scalability: The framework is compatible with any externally trained black-box ML model that outputs class probabilities.
• Robustness: The MAR-type assumption for missing (external) covariates enables reliable moment construction under broad scenarios of covariate and concept shift.
• Generalization: While the present development is for multinomial logistic regression, generalization to survival, longitudinal, or high-dimensional models is feasible.

Open directions include:

• High-dimensional extension incorporating feature selection or penalties;
• Data-driven constraint filtering for cases where MAR does not hold or invalid moments may introduce bias;
• Extension to settings with multiple external summary-only sources, structured outcome types (e.g., ordinal, time-to-event), or non-iid sampling.

Conclusion

The fused empirical likelihood strategy for multinomial logistic regression outlined in this work provides a rigorous and practical template for combining interpretable, primary-study-based inference with the efficiency and scale of external ML model predictions. The approach addresses key limitations of prior synthetic data and GMM-type methods by operating solely with summary-level, black-box predictors and rigorously permitting heterogeneities encountered in modern data integration.

The resulting FMLE is consistent, asymptotically normal, and enjoys strict (and quantifiable) efficiency gain under practically attainable moment conditions, as supported by simulation and real-world evidence. This methodology forms a solid foundation for future research in robust, scalable fusion of structured statistical inference with external, nonparametric ML information.