Multi-group Uncertainty Quantification

• Multi-group Uncertainty Quantification is a framework that estimates uncertainties by explicitly accounting for structured group heterogeneity in data and parameters.
• It employs diverse methodologies including multi-group CIRCE, group-specific conformal prediction, and nonparametric multi-output Gaussian processes to achieve tailored uncertainty estimates.
• By enforcing group-level calibration and fairness, it advances reliable decision-making in applications ranging from engineering systems to medical diagnostics.

Multi-group Uncertainty Quantification addresses the problem of modeling, propagating, and analyzing uncertainty in systems where the data, parameters, or underlying structures are inherently partitioned into groups—by experiment type, demographic attributes, network subcomponents, or structural subpopulations. The central objective is to deliver uncertainty estimates that are both accurate within each group and coherent across groups, frequently motivated by heterogeneity in model performance, systematic variability across conditions, or the need for group-level fairness and interpretability. Methodologies span Gaussian and log-Gaussian parametric models, conformal and calibration-based prediction, nonparametric multi-output Gaussian processes, block-sparse regression, scalable network decompositions, and fairness-constrained uncertainty sets.

1. Fundamental Principles and Problem Definition

Multi-group uncertainty quantification operationalizes the estimation of model or prediction uncertainty by explicitly accounting for a group structure in data or parameters. The formalization is guided by the partitioning of a dataset or parameter vector into $G$ groups (indexed by $s=1,\ldots,G$), each possibly associated with different input ranges, experimental conditions, geometries, demographic characteristics, or physical subnetworks. Typical objectives include:

• Estimation of group-specific variance (and sometimes mean) parameters for modeling epistemic uncertainty, as exemplified by the multi-group CIRCE approach, which seeks to determine whether uncertainty is homogeneous or distinct across groups;
• Construction of prediction intervals or uncertainty sets with guaranteed coverage not just on aggregate, but uniformly across all groups, crucial for reliable deployment in applications with fairness or subgroup safety constraints.

The distinction between group-conditional and marginal approaches is a foundational aspect. Marginal UQ methods provide global guarantees that can be violated in specific subgroups, whereas multi-group approaches enforce or estimate per-group properties tailored to observed heterogeneity (Damblin et al., 2023, Li et al., 8 May 2025, Liu et al., 2024).

2. Representative Methodologies

2.1 Multi-group CIRCE

The multi-group generalization of the CIRCE method quantifies input model uncertainty in thermal-hydraulic system codes by introducing group-specific variance parameters $\Sigma_s$ for $s=1,...,G$, while maintaining a common mean $m$ for the multiplicative random factor $\Lambda$ affecting closure relations. The statistical model for experiments in group $s$ is:

• $\lambda_i - m \sim N(0, \Sigma_s)$,
• $Y_i = H_i (m + (\lambda_i - m)) + \epsilon_i$, with $\epsilon_i \sim N(0, R_i)$.

A joint likelihood is maximized via an ECME algorithm with the following steps:

• E-step: compute conditional expectations/variances of group-specific latent variables,
• CM1: update group variances,
• CM2: update the shared mean,
• Convergence via tolerance on parameter changes.

The model supports both synthetic cases (with known ground truth group variances) and real data (e.g., BETHSY critical mass flow with multiple geometries). Group structure is interrogated via statistical hypothesis testing (e.g., Wald test on variance differences) to decide on modeling granularity (Damblin et al., 2023).

2.2 Group-wise Conformal and Calibration Approaches

Group-conditional conformal prediction constructs prediction intervals $s=1,\ldots,G$0 for each group $s=1,\ldots,G$1, ensuring coverage $s=1,\ldots,G$2 for all $s=1,\ldots,G$3. The FUQ framework for depression prediction enforces both reliability and group-level fairness via per-group calibration and width optimization subject to coverage and gap constraints:

• For each group, compute a conformity (or residual) score on calibration data;
• Set group-specific quantile thresholds controlling interval width;
• Minimize average interval width while maintaining strict inter-group coverage gap $s=1,\ldots,G$4 (Equal Opportunity Coverage criterion).

This paradigm extends readily to any setting with group-labeled data and distribution-free valid uncertainty intervals (Li et al., 8 May 2025, Liu et al., 2024).

2.3 Multicalibration and Multivalid Conformal Prediction

For tasks such as LLM long-form text generation, canonical calibration and split conformal prediction provide only marginal guarantees, with empirical evidence demonstrating breakdown in specific subgroups. Multi-group methods such as:

• Multicalibration: iterative group-wise adjustment of scores/calibrations (e.g., Iterative Grouped Histogram Binning) to drive group-specific calibration error below predefined thresholds,
• Multivalid conformal prediction: iterative quantile patching on calibration subsets per group (e.g., Multivalid Split Conformal procedure) to enforce coverage for each group, achieve subgroup-uniform calibration and coverage, with provable iteration complexity and empirical superiority on metrics such as average squared calibration error (ASCE) and Brier score (Liu et al., 2024).

2.4 Nonparametric Multi-output GP Approaches

For functional data such as multiple closed curves, multi-output Gaussian process models incorporate group structure through coregionalization kernels. Each group (e.g., curve, subpopulation) is assigned a covariance structure via a group-level kernel $s=1,\ldots,G$5, which, together with within-group structure and coordinate-level kernel, enables nonparametric UQ that "borrows strength" across curves. Posterior uncertainties reflect both within- and between-group variability, with applications to shape reconstruction and population-level uncertainty attribution (Luo et al., 2022).

2.5 Group-based Bootstrap in High-dimensional Regression

For penalized regression under group-sparsity (e.g., group lasso), a modified parametric bootstrap simulates the sampling distribution of group-level coefficients. The process,

• bootstraps data according to pilot group-lasso estimates,
• re-fits under the same grouping structure,
• produces simultaneous group-level $s=1,\ldots,G$6 confidence sets and $s=1,\ldots,G$7-values, demonstrates controlled familywise error rates and adaptivity to complex group dependencies (Zhou et al., 2015).

2.6 Decomposition-based Approaches in Networks

For large dynamic networks, group decomposition (via spectral clustering) into weakly coupled subnetworks enables localized uncertainty quantification, using techniques such as Probabilistic Waveform Relaxation (PWR). Each subnetwork is treated as a group; uncertainty propagation leverages intrusive (Galerkin-based) or non-intrusive (collocation-based) UQ, with parallel waveform relaxation iterations, achieving scalability and convergence even for high-dimensional (many-group) problems (Surana et al., 2011).

3. Statistical Properties and Theoretical Guarantees

Multi-group uncertainty quantification methods are typically constructed to ensure:

• Consistency and finite-sample validity of coverage and calibration for each group;
• Maximum likelihood estimation in parametric models, involving explicit calculation of the Fisher information for identifiability diagnostics (e.g., normalized error coefficients in CIRCE);
• Simultaneous familywise error control in hypothesis testing across groups in sparse regression;
• Iteration complexity bounds for convergence of multicalibrated predictors (polylogarithmic in group count and accuracy);
• Scalability via model decomposition, with error bounds on waveform relaxation convergence dictated by inter-group coupling strength (Damblin et al., 2023, Liu et al., 2024, Zhou et al., 2015, Surana et al., 2011).

4. Empirical Evaluation and Case Studies

Empirical studies across the literature demonstrate the necessity and practical impact of multi-group UQ:

• In synthetic simulation, multi-group CIRCE accurately recovers group-level variances, with improved interval coverage relative to pooled estimates (Damblin et al., 2023);
• In real-world depression prediction (video/audio/EEG), FUQ achieves parity in groupwise coverage (PICP-gap $s=1,\ldots,G$80.5%) under demographic grouping, outperforming vanilla conformal methods (Li et al., 8 May 2025);
• In LLM claim verification, multicalibrated and multivalid methods reduce maximum subgroup calibration errors and miscoverage rates to one-fourth or less versus single-calibration baselines (Liu et al., 2024);
• Performance on benchmarks such as MPEG-7 shapes, gene expression data, and engineered networks validates the superiority of joint-group modeling and decomposition for complex, high-dimensional uncertainty tasks (Luo et al., 2022, Zhou et al., 2015, Surana et al., 2011).

5. Model Selection, Tuning, and Practical Guidance

Practitioners must address key challenges:

• Pool or split? Statistical testing (Wald, AIC) and diagnostics (NEC, Q–Q plots) assess whether group-level heterogeneity is significant enough to warrant multi-group modeling; otherwise, pooling can benefit estimation precision.
• Data sufficiency: Calibration and conformal guarantees per group require adequate sample sizes to avoid overfit or degenerate intervals.
• Optimization strategy: For conformal approaches, constrained post-hoc optimization or explicit Lagrangian penalties ensure groupwise validity and minimal average width.
• Interpretability: Reporting both group-wise and global uncertainty estimates, along with the rationale for grouping, is essential for downstream decision-making (Damblin et al., 2023, Li et al., 8 May 2025, Liu et al., 2024).

6. Broader Applicability and Current Limitations

Multi-group uncertainty quantification methodologies extend broadly in domains such as credit scoring, medical risk estimation, text generation, environmental modeling, and shape analysis. The core requirements are group-level structure, exchangeable conformity/nonconformity scores, and tractable estimation per group. However, limitations remain:

• Empirical group definitions may be incomplete or require richer, possibly intersectional/learned groupings.
• Finite-sample guarantees degrade with very small group sizes.
• The need for ground-truth or accurate calibration sets per group is a persistent bottleneck.
• Negative variance estimates in low-information groups (as noted in multi-group CIRCE) necessitate ad hoc corrections (e.g., truncation to zero).
• Fairness and adaptivity across dynamically evolving group structures (e.g., text domains or network topologies) remains an area of active research (Damblin et al., 2023, Li et al., 8 May 2025, Liu et al., 2024, Luo et al., 2022).

7. Summary Table: Selected Multi-group UQ Paradigms

Method	Group Structure	Uncertainty Metric
Multi-group CIRCE (Damblin et al., 2023)	Experiment/geometric	Groupwise variance/mean
FUQ (Li et al., 8 May 2025)	Demographic	Coverage, interval width
Multicalibration/MVSC (Liu et al., 2024)	Flexible, overlapping	Calibration, conformal coverage
Nonparametric multi-GP (Luo et al., 2022)	Functional (shapes)	Posterior variance
Group-lasso bootstrap (Zhou et al., 2015)	Parameter blocks	Confidence regions, p-values
PWR (Surana et al., 2011)	Network subgroups	Mean/variance, time series

These methods collectively enable rigorous, granular, and application-adapted uncertainty analysis for grouped data, models, and networks.