Model-Independent Data Aggregation

• Model-independent data aggregation is a design principle that fuses predictions, parameters, distributions, or summary statistics without requiring joint retraining on raw data.
• It employs varied techniques—such as covariance-weighted black-box ensembles, unsupervised CDF transforms, and Bayesian parameter matching—to adapt aggregation methods to data-sharing constraints.
• This approach enhances robustness in risk evaluation and treatment-effect estimation while preserving privacy and reducing computational overhead.

Searching arXiv for the cited papers to ground the article in current arXiv records. Model-independent data aggregation denotes a class of aggregation procedures that combine multiple information sources without requiring intrusive access to a shared internal model form. In the recent arXiv literature, the aggregated objects include black-box predictors, unlabeled input variables, local parameter sets learned on sequestered datasets, families of loss distributions, and study-level summary statistics. The unifying feature is that aggregation is carried out at the level of predictions, ranks, parameters, distributions, or sufficient statistics rather than by retraining a single pooled model on raw data (Bourdais et al., 2024, Ulan et al., 2021, Yurochkin et al., 2019, Mao et al., 2022, Panigrahi et al., 2022).

1. Scope and formal viewpoints

The phrase model-independent is used in several closely related but technically distinct senses. In black-box ensemble construction, it means that the contributing models are treated as functions whose internal structure is irrelevant to aggregation. In unsupervised aggregation, it means that the aggregation rule is learned without access to ground-truth labels. In parameter-matching frameworks, it means that independently trained local models can be fused through their parameterizations rather than through a common training pipeline. In robust risk evaluation, it means that one aggregates an uncertainty set of distributions into a robust cdf that is independent of any specific risk measure. In privacy-sensitive treatment-effect estimation, it means that only summary statistics are shared, so raw records remain local (Bourdais et al., 2024, Ulan et al., 2021, Yurochkin et al., 2019, Mao et al., 2022, Panigrahi et al., 2022).

Setting	Aggregated object	Characteristic mechanism
Black-box prediction aggregation	$f_{\mathrm{agg}}(x;w)=\sum_{m=1}^M w_m f_m(x)$	Minimal empirical variance estimation
Unsupervised aggregation	$A:[0,1]^k\to[0,1]$ from transformed variables	Empirical CDF transforms with entropy and dependency weights
Statistical parameter aggregation	Local parameters $\{v_{j\ell}\}$	Beta-process/Bernoulli-process matching into global atoms $\{\phi_i\}$
Risk-model aggregation	Uncertainty set $\mathcal F$ of cdf’s	Robust cdf $F^*:=\bigvee \mathcal F$ under stochastic dominance
Efficient data aggregation	Per-study summaries $(\hat\xi_k,\hat\Xi_k)$	Data carving and pooled unbiased estimation of $\theta$

A plausible implication is that model-independent data aggregation is not a single algorithmic family but a design principle: the aggregation operator is chosen to match the object being fused and the information-sharing constraints under which fusion must occur.

2. Black-box predictor aggregation via minimal variance

The framework introduced in "Minimal Variance Model Aggregation: A principled, non-intrusive, and versatile integration of black box models" defines aggregation for $M$ predictors $f_1(x),\ldots,f_M(x)$ through a linear ensemble

$A:[0,1]^k\to[0,1]$0

with weights satisfying $A:[0,1]^k\to[0,1]$1 and often $A:[0,1]^k\to[0,1]$2. Two optimization principles are distinguished. Minimal Error Aggregation (MEA) minimizes empirical squared error on held-out data:

$A:[0,1]^k\to[0,1]$3

In the unconstrained case, if $A:[0,1]^k\to[0,1]$4 is the $A:[0,1]^k\to[0,1]$5 matrix with entries $A:[0,1]^k\to[0,1]$6, then

$A:[0,1]^k\to[0,1]$7

The paper notes that $A:[0,1]^k\to[0,1]$8 may be ill-conditioned when models are collinear or $A:[0,1]^k\to[0,1]$9, so direct inversion overfits and yields large, unstable weights (Bourdais et al., 2024).

Minimal Variance Aggregation (MVA) instead assumes unbiased models of the form

$\{v_{j\ell}\}$0

and minimizes the variance of an unbiased linear combination:

$\{v_{j\ell}\}$1

The Lagrangian calculation gives the closed form

$\{v_{j\ell}\}$2

No positivity constraint is needed; negative weights appear only if $\{v_{j\ell}\}$3 has strong negative correlations. The core claim of Minimal Empirical Variance Aggregation (MEVA) is that estimating $\{v_{j\ell}\}$4 and then inverting it is statistically more robust than estimating $\{v_{j\ell}\}$5 directly by least squares. Under bounded moments and a spectral gap in $\{v_{j\ell}\}$6, the empirical MEA weight has $\{v_{j\ell}\}$7 fluctuations, whereas the empirical MVA weight derived from $\{v_{j\ell}\}$8 has $\{v_{j\ell}\}$9 fluctuations around $\{\phi_i\}$0 (Bourdais et al., 2024).

Practical estimation proceeds by estimating $\{\phi_i\}$1 either diagonally or as a full matrix. In the diagonal approximation,

$\{\phi_i\}$2

so the weights satisfy $\{\phi_i\}$3. In the full-matrix approach, one computes residuals $\{\phi_i\}$4, forms the empirical covariance

$\{\phi_i\}$5

diagonalizes $\{\phi_i\}$6, and fits smooth log-eigen-value functions through kernel/RKHS regression. For a new $\{\phi_i\}$7, the inverse covariance estimate is reconstructed by

$\{\phi_i\}$8

followed by

$\{\phi_i\}$9

The paper explicitly recommends shrinkage or Tikhonov regularization by adding $\mathcal F$0 when $\mathcal F$1 is nearly singular (Bourdais et al., 2024).

Empirical evaluation spans both data science and PDE settings. On Boston Housing $\mathcal F$2, aggregating six off-the-shelf regressors, MEVA via GP-shrinkage reduced MAE by $\mathcal F$3–$\mathcal F$4 over the best single model, whereas MEEA overfit and performed worse. For the Laplace equation, six classical PDE solvers plus one GP-PDE solver were aggregated; with FNO used to learn $\mathcal F$5, MEVA achieved a geometric mean test error of $\mathcal F$6 in $\mathcal F$7 versus $\mathcal F$8 for the best FEM. For Burgers’ equation, with seven solvers, the geometric mean $\mathcal F$9 improved from $F^*:=\bigvee \mathcal F$0 for the best single solver to $F^*:=\bigvee \mathcal F$1 for the aggregate, and the method robustly switched away from diverging solvers near shocks. The framework’s stated limitations are that it requires held-out data to estimate $F^*:=\bigvee \mathcal F$2, assumes unbiasedness, and offers no guarantee that the aggregate will dominate the single best model when models have identical errors (Bourdais et al., 2024).

3. Unsupervised aggregation from unlabeled data

"Aggregation as Unsupervised Learning and its Evaluation" formulates aggregation as learning a mapping

$F^*:=\bigvee \mathcal F$3

from unlabeled data, with the requirements that $F^*:=\bigvee \mathcal F$4 preserve monotonicity and boundary conditions and be derived without access to any ground-truth $F^*:=\bigvee \mathcal F$5. The method first transforms each variable through an empirical cumulative probability map and then computes weights that trade off entropy against redundancy. For variable $F^*:=\bigvee \mathcal F$6 and instance $F^*:=\bigvee \mathcal F$7,

$F^*:=\bigvee \mathcal F$8

Thus $F^*:=\bigvee \mathcal F$9 is the empirical CDF of variable $(\hat\xi_k,\hat\Xi_k)$0, reversed when its direction is negative. By construction, $(\hat\xi_k,\hat\Xi_k)$1 (Ulan et al., 2021).

Two partial weights are then computed. The entropy-based weight uses

$(\hat\xi_k,\hat\Xi_k)$2

where $(\hat\xi_k,\hat\Xi_k)$3 is the empirical frequency of rank value $(\hat\xi_k,\hat\Xi_k)$4. The dependency-based weight compares the variable-specific rank $(\hat\xi_k,\hat\Xi_k)$5 to a joint rank $(\hat\xi_k,\hat\Xi_k)$6 derived from the joint CDF

$(\hat\xi_k,\hat\Xi_k)$7

With $(\hat\xi_k,\hat\Xi_k)$8,

$(\hat\xi_k,\hat\Xi_k)$9

The final unnormalized weight is $\theta$0, then normalized so that $\theta$1. Aggregation is performed either by a weighted sum,

$\theta$2

or by a weighted product,

$\theta$3

Because $\theta$4, $\theta$5, and each $\theta$6 is monotone into $\theta$7, the construction yields a monotone aggregator with $\theta$8 and $\theta$9 (Ulan et al., 2021).

The procedure has no iterative optimization; it is closed-form apart from sorting and frequency counting. Its complexity is dominated by sorting at $M$0 per variable and by Spearman rank-correlation computation at $M$1. Evaluation was performed on 169 regression datasets from the UCI Machine Learning Repository after filtering out non-numeric datasets, tasks with too few instances, and duplicates. Dataset sizes ranged from 60 to over 4 million instances, with 2 to 373 inputs. Ground truth was used only at evaluation time. External measures were Spearman’s $M$2 and Kendall’s $M$3 distance to the true ranking; internal measures were Kemeny distance to the inputs and Sensitivity, defined as the number of unique aggregate values divided by the number of unique input tuples (Ulan et al., 2021).

Median performance over the 169 datasets showed a marked separation between data-agnostic and data-driven methods. PROD and MIN each had median Spearman $M$4, MAX had $M$5, SUM had $M$6, WSM had $M$7, WPM had $M$8, and the supervised reference REG had $M$9. WPM also achieved Kendall $f_1(x),\ldots,f_M(x)$0 distance $f_1(x),\ldots,f_M(x)$1, Kemeny distance $f_1(x),\ldots,f_M(x)$2, and Sensitivity $f_1(x),\ldots,f_M(x)$3. The paper’s interpretation is that WPM and WSM outperform basic methods because they adapt to each variable’s scale and redundancy, the CDF transform spreads out congested value ranges, and the dependency weight down-weights variables whose information is already represented by others. The stated limitations are that the method only handles numeric, continuous features, assumes a global monotonic relationship among inputs and the latent target, and does not capture complex non-monotone interactions (Ulan et al., 2021).

4. Parameter matching across sequestered local models

"Statistical Model Aggregation via Parameter Matching" addresses aggregation when each site trains its own local model on a private or heterogeneous dataset and only local parameter estimates are available. Let $f_1(x),\ldots,f_M(x)$4 be the site-specific datasets, and let site $f_1(x),\ldots,f_M(x)$5 produce local parameters

$f_1(x),\ldots,f_M(x)$6

The goal is to infer a shared global latent representation $f_1(x),\ldots,f_M(x)$7 of unknown size $f_1(x),\ldots,f_M(x)$8 and a binary assignment matrix $f_1(x),\ldots,f_M(x)$9 with $A:[0,1]^k\to[0,1]$00 iff $A:[0,1]^k\to[0,1]$01 is a noisy realization of global atom $A:[0,1]^k\to[0,1]$02, so that the posterior $A:[0,1]^k\to[0,1]$03 is maximized (Yurochkin et al., 2019).

The meta-model is Bayesian nonparametric. Global atoms are drawn from a Beta process prior,

$A:[0,1]^k\to[0,1]$04

each site selects a finite subset through a Bernoulli process,

$A:[0,1]^k\to[0,1]$05

and each local parameter is modeled as a noisy draw from its assigned global atom,

$A:[0,1]^k\to[0,1]$06

with $A:[0,1]^k\to[0,1]$07 and $A:[0,1]^k\to[0,1]$08 chosen from the exponential family. Marginalizing out $A:[0,1]^k\to[0,1]$09 yields an Indian-Buffet-Process prior over the assignment structure. The inference scheme pursues a MAP estimate by alternating assignment updates and hyperparameter updates. For each site, holding the other assignments fixed, the local parameters are reassigned by solving a linear-sum-assignment problem with a cost matrix whose entries are differences in log-marginal-likelihood contributions for joining an existing atom versus creating a new one. The paper uses the Hungarian algorithm for this step and states that each assignment move monotonically increases the joint log-posterior. Hyperparameters $A:[0,1]^k\to[0,1]$10 are then updated by maximizing a smooth objective, and global atoms are estimated from their assigned local parameters (Yurochkin et al., 2019).

The framework is specialized to three model families. For Gaussian topic models, the local parameters are topic centroids in embedding space, with Gaussian $A:[0,1]^k\to[0,1]$11 and Gaussian $A:[0,1]^k\to[0,1]$12. For HDP-HMMs, the local parameters are state-emission sufficient statistics in a Normal-Wishart family, and SPAHM matches local states across subjects while allowing the number of discovered states to vary by site. For sparse Gaussian processes, the local parameters are inducing-input/weight pairs, and matching yields a global set of shared inducing inputs and weights that can be dispatched to each local GP. The algorithmic complexity of each Hungarian step is $A:[0,1]^k\to[0,1]$13, with total cost approximately $A:[0,1]^k\to[0,1]$14, memory $A:[0,1]^k\to[0,1]$15, and a local-model-training phase that is trivially parallelizable across sites (Yurochkin et al., 2019).

Empirical results are reported for simulated Gaussian mixtures, Gaussian topic models, sparse GPs, and motion capture. On the Gutenberg topic-model experiment, 40 books were summarized into 155 global topics; compared with Gaussian LDA with 150 topics, SPAHM achieved UCI coherence $A:[0,1]^k\to[0,1]$16 versus $A:[0,1]^k\to[0,1]$17, with runtime $A:[0,1]^k\to[0,1]$18 versus approximately $A:[0,1]^k\to[0,1]$19 for 100 MCMC iterations. In US temperature forecasting with 50 states and 50 inducing points per local sparse GP, SPAHM had slightly higher RMSE on self-prediction, $A:[0,1]^k\to[0,1]$20 versus $A:[0,1]^k\to[0,1]$21, but much better cross-state generalization, $A:[0,1]^k\to[0,1]$22 versus $A:[0,1]^k\to[0,1]$23. On MoCap sequences, local sticky HDP-HMMs with 9 and 13 states were matched into 14 global states, and SPAHM reached near-joint-HDP-HMM accuracy at less than half the compute time. The stated limitations are the need to choose exponential-family priors, sensitivity to hyperparameters controlling the degree of sharing, and the restriction that SPAHM does not split or merge within a site, so poor local partitions can propagate to the global matching (Yurochkin et al., 2019).

5. Aggregation of distributions for risk evaluation

Mao–Wang–Wu define model aggregation for prudent risk evaluation by aggregating an uncertainty set of univariate loss distributions into a single robust cdf. Given an uncertainty set

$A:[0,1]^k\to[0,1]$24

a partial order $A:[0,1]^k\to[0,1]$25 such as first-order stochastic dominance $A:[0,1]^k\to[0,1]$26 or increasing-convex order $A:[0,1]^k\to[0,1]$27, and optionally a law-invariant $A:[0,1]^k\to[0,1]$28-monotone risk functional $A:[0,1]^k\to[0,1]$29, the Model-Aggregation (MA) approach first computes

$A:[0,1]^k\to[0,1]$30

the supremum of $A:[0,1]^k\to[0,1]$31 under the chosen order, and then reports either $A:[0,1]^k\to[0,1]$32 itself or $A:[0,1]^k\to[0,1]$33 as the robust risk evaluation (Mao et al., 2022).

The paper gives explicit lattice-theoretic formulas. Under first-order stochastic dominance,

$A:[0,1]^k\to[0,1]$34

Under increasing-convex order,

$A:[0,1]^k\to[0,1]$35

where

$A:[0,1]^k\to[0,1]$36

and hence

$A:[0,1]^k\to[0,1]$37

This construction differs from classic worst-case risk evaluation because it produces not only a robust value of risk evaluation but also a robust distributional model, and does so independently of any specific risk measure (Mao et al., 2022).

For robust optimization, the objective becomes

$A:[0,1]^k\to[0,1]$38

where $A:[0,1]^k\to[0,1]$39 is convex and $A:[0,1]^k\to[0,1]$40 is convex in $A:[0,1]^k\to[0,1]$41. Under standard assumptions, if $A:[0,1]^k\to[0,1]$42 admits a finite Kusuoka form, then the MA robust optimization problem admits an extended linear or conic reformulation through the minimax representation of Expected Shortfall. The paper states that when $A:[0,1]^k\to[0,1]$43 is generated by finitely many scenarios or by simple moment, box, or ellipsoid constraints, the resulting sup-expectations reduce to LP or SOCP subproblems (Mao et al., 2022).

Two uncertainty families admit explicit MA solutions. For a Wasserstein ball

$A:[0,1]^k\to[0,1]$44

the FSD supremum quantile $A:[0,1]^k\to[0,1]$45 solves

$A:[0,1]^k\to[0,1]$46

while under ICX with $A:[0,1]^k\to[0,1]$47 the paper gives

$A:[0,1]^k\to[0,1]$48

For the mean-variance uncertainty set

$A:[0,1]^k\to[0,1]$49

the MA approach yields explicit cdf forms $A:[0,1]^k\to[0,1]$50 under FSD and $A:[0,1]^k\to[0,1]$51 under ICX, from which

$A:[0,1]^k\to[0,1]$52

and

$A:[0,1]^k\to[0,1]$53

follow directly (Mao et al., 2022).

A further contribution is the equivalence analysis between worst-case risk and MA risk. Defining

$A:[0,1]^k\to[0,1]$54

the paper proves $A:[0,1]^k\to[0,1]$55 by monotonicity and introduces convex-polytope MA-equivalence. Theorem 8 characterizes $A:[0,1]^k\to[0,1]$56 as the unique risk functional on $A:[0,1]^k\to[0,1]$57 that is translation-invariant, positively homogeneous, lower semicontinuous, and $A:[0,1]^k\to[0,1]$58-cEMA. Theorem 9 analogously characterizes $A:[0,1]^k\to[0,1]$59 under $A:[0,1]^k\to[0,1]$60-cEMA. In rolling-window backtests on 20 stocks, WR and MA produced similar robust portfolios, and both outperformed SAA and Markowitz in Sharpe (Mao et al., 2022).

6. Summary-statistic aggregation and data carving for treatment effects

Panigrahi, Wang and He study data aggregation in a meta-analytic setting where the shared target is a treatment effect $A:[0,1]^k\to[0,1]$61. There are $A:[0,1]^k\to[0,1]$62 independent existing studies and a new validation study $A:[0,1]^k\to[0,1]$63. In study $A:[0,1]^k\to[0,1]$64, one observes outcomes $A:[0,1]^k\to[0,1]$65, treatments $A:[0,1]^k\to[0,1]$66, and high-dimensional covariates $A:[0,1]^k\to[0,1]$67 under the linear model

$A:[0,1]^k\to[0,1]$68

where $A:[0,1]^k\to[0,1]$69 is the true support with $A:[0,1]^k\to[0,1]$70 and $A:[0,1]^k\to[0,1]$71 is independent of $A:[0,1]^k\to[0,1]$72. The validation study collects $A:[0,1]^k\to[0,1]$73 on the union of selected variables $A:[0,1]^k\to[0,1]$74. The goal is to construct an unbiased, efficient estimator of $A:[0,1]^k\to[0,1]$75 by transmitting only summary statistics from each existing study to a central analyst (Panigrahi et al., 2022).

Each study first runs a LASSO regression,

$A:[0,1]^k\to[0,1]$76

with $A:[0,1]^k\to[0,1]$77 and $A:[0,1]^k\to[0,1]$78. The active set is $A:[0,1]^k\to[0,1]$79, with $A:[0,1]^k\to[0,1]$80. Rather than sharing raw data, study $A:[0,1]^k\to[0,1]$81 reports the active set $A:[0,1]^k\to[0,1]$82, the penalty weights $A:[0,1]^k\to[0,1]$83, the sign vector $A:[0,1]^k\to[0,1]$84, and the first- and second-moment summaries

$A:[0,1]^k\to[0,1]$85

The paper explicitly states that these are exactly the sufficient statistics for refitting the selected submodel by least squares (Panigrahi et al., 2022).

After collecting summaries, the central analyst computes validation-study summaries

$A:[0,1]^k\to[0,1]$86

and for each study forms the pooled covariance and pooled MLE

$A:[0,1]^k\to[0,1]$87

$A:[0,1]^k\to[0,1]$88

Without correction, $A:[0,1]^k\to[0,1]$89 is biased because the same data were used for selection. The paper therefore uses data carving. It shows that the Karush–Kuhn–Tucker randomization map can be written so that $A:[0,1]^k\to[0,1]$90 is asymptotically jointly Gaussian, and that the observed selection event

$A:[0,1]^k\to[0,1]$91

is equivalent to a set of affine inequalities in $A:[0,1]^k\to[0,1]$92. By an exact-carving argument, the uniformly minimum variance unbiased estimator of $A:[0,1]^k\to[0,1]$93, conditional on $A:[0,1]^k\to[0,1]$94, is a Rao–Blackwellized conditional expectation. In practice, the paper uses a Laplace-type approximation and a small convex program in $A:[0,1]^k\to[0,1]$95 to compute the carved estimator $A:[0,1]^k\to[0,1]$96 (Panigrahi et al., 2022).

The final aggregated estimator is

$A:[0,1]^k\to[0,1]$97

or, as the paper notes, one may use weights proportional to $A:[0,1]^k\to[0,1]$98. Under screening, exponential-moment, polyhedral-selection probability, and operator-norm control conditions, Theorem 3.1 gives

$A:[0,1]^k\to[0,1]$99

so $\{v_{j\ell}\}$00 is asymptotically unbiased. Lemma 3.2 further quantifies a uniform variance gain over the split estimator that uses only the validation study. The privacy claim is explicit: each existing study shares only $\{v_{j\ell}\}$01 numbers, raw records remain local, and the main extra computation is solving a convex program of complexity $\{v_{j\ell}\}$02 (Panigrahi et al., 2022).

7. Cross-cutting assumptions, misconceptions, and limitations

Across these formulations, model-independent aggregation does not mean assumption-free aggregation. MEVA is non-intrusive and model-agnostic, but it requires held-out data and assumes unbiasedness of the constituent models. The unsupervised CDF-based method uses no ground truth, but it is limited to numeric, continuous features and assumes a global monotonic relationship. SPAHM is fully model-independent at the level of local models, yet it depends on exponential-family priors and on the quality of local partitions. The MA approach for risk evaluation is independent of any specific risk measure, but it is defined relative to a chosen stochastic order. The data-carving estimator preserves privacy by exchanging only summaries, but its guarantees require screening, selection-probability approximation, and operator-norm control (Bourdais et al., 2024, Ulan et al., 2021, Yurochkin et al., 2019, Mao et al., 2022, Panigrahi et al., 2022).

A common misconception is that aggregation is synonymous with averaging. The literature instead includes point-wise linear ensembles with covariance-derived weights, weighted products after empirical CDF transforms, Beta-Bernoulli parameter matching, stochastic-dominance suprema of cdf’s, and Rao–Blackwellized pooling of carved estimators. This suggests that model-independent data aggregation is better viewed as a family of interface-level combination rules whose mathematical form is dictated by the object being aggregated.

Another common misconception is that model-independent aggregation is necessarily unsupervised. One strand is explicitly unsupervised and label-free (Ulan et al., 2021), but other strands use held-out targets to estimate error covariance (Bourdais et al., 2024), use posterior matching over local parameter sets (Yurochkin et al., 2019), use uncertainty sets of distributions rather than labels (Mao et al., 2022), or use a validation study together with summary statistics to estimate a shared treatment effect (Panigrahi et al., 2022). The available evidence therefore supports a broader interpretation: the defining property is not the absence of supervision in every case, but the avoidance of intrusive joint retraining on raw, fully pooled data.