Covariate-dependent Nonparametric $g$-modeling for regression via infinite Mixture-of-Expertizing class

Abstract: Empirical Bayes $g$-modeling captures unit-level heterogeneity by estimating a latent prior distribution from observed data. In the existing formulations, however, the prior is shared by all units. In this paper, we develop a covariate-dependent g-modeling framework for regression in which the entire prior distribution of the regression coefficients is allowed to depend on covariates. We formulate the estimation of the prior as nonparametric maximum likelihood estimation (NPMLE) of the covariate-dependent prior, and show that the unrestricted problem is ill-posed. To resolve this, we introduce the infinite Mixture-of-Expertizing class of conditional priors, under which the NPMLE is precisely a softmax-gated Mixture of Experts (MoE) whose number of experts is not fixed in advance but is determined by the data. Building on a first-order optimality condition, we propose two exemplar-based estimation algorithms that select experts automatically, together with a post-hoc aggregation of experts for interpretation. On the theoretical side, we show that every conditional prior in the class is Lipschitz continuous in the covariates, and that aggregated softmax gates can approximate any continuous gate function. The effectiveness of the proposed NPMLE is shown through application to synthetic datasets and real datasets.

• --The paper introduces a Covariate-dependent Nonparametric g-modeling method for regression via Infinite Mixture-of-Expertizing class (iMoE). This model replaces a common prior for regression coefficients with a covariate-dependent prior that captures heterogeneity across units. --The method resolves ill-posed issues in unrestricted nonparametric maximum likelihood estimation and introduces algorithms selecting an optimal number of active experts. -- Simulations and applications illustrate better prediction accuracy: compared to standard methods, up to 25.5% increase in $R^2$.

Problem formulation and contribution

Okazaki and Yano address a structural limitation of empirical-Bayes $g$-modeling for unit-level regression. In the conventional formulation, latent regression coefficients are independently drawn from a common prior $G(\beta)$, so all units are assumed to share the same distribution of regression relationships. The paper replaces this assumption with a covariate-dependent prior $G(\beta \mid z)$, where $z$ may differ from the explanatory covariates $x$ entering the regression predictor. The resulting model is

$$y_i \mid x_i,\beta_i \sim \mathrm{EF}, \qquad \eta_i=x_i^\top\beta_i, \qquad \beta_i\mid z_i\sim G^\ast(\cdot\mid z_i).$$

This formulation separates two sources of heterogeneity: systematic variation in the distribution of regression coefficients as $z$ changes, and residual unit-level variation conditional on a fixed value of $z$. The distinction is substantive. For example, geographic coordinates can determine how the relationship between housing characteristics and prices varies across submarkets, while the conditional prior retains unexplained heterogeneity among geographically similar units.

The central contribution is “Covariate-dependent Nonparametric $g$-modeling for regression via infinite Mixture-of-Expertizing class” (2610.08321). The paper establishes that unrestricted nonparametric maximum likelihood estimation over all conditional priors is ill-posed, introduces the infinite Mixture-of-Expertizing (iMoE) class as a regularized conditional-prior class, proves that its NPMLE has an exact softmax-gated Mixture-of-Experts representation, and develops two exemplar-based algorithms that select the number of active experts automatically.

The resulting model is not merely a conventional MoE with a prespecified number of components. Its target is the conditional prior measure itself, and the number of experts is induced by the NPMLE rather than fixed before estimation.

Ill-posed unrestricted estimation

For a conditional prior $G(\cdot\mid z)$, the marginal likelihood is

$G(\beta)$0

A direct extension of classical $G(\beta)$1-modeling would maximize the likelihood over all conditional probability measures. The paper shows that this problem is ill-posed whenever the observed $G(\beta)$2 values are distinct. For each observation, one can choose a degenerate conditional prior concentrated at a coefficient value maximizing that observation’s likelihood contribution:

$G(\beta)$3

Because the conditional prior can be chosen independently at each observed $G(\beta)$4, the construction interpolates the observations without imposing any meaningful relationship between nearby covariate values. The likelihood is therefore maximized by a collection of unrelated point masses, while the prior remains arbitrary away from the observed covariates.

This diagnosis motivates a restriction on the entire conditional-prior function rather than merely on a low-dimensional summary such as its mean, variance, or a single mixture probability. The proposed restriction is designed to preserve nonparametric flexibility in $G(\beta)$5 while enforcing smooth, structured dependence on $G(\beta)$6.

The iMoE conditional-prior class

Let $G(\beta)$7 and $G(\beta)$8 be compact. For a probability measure $G(\beta)$9 on $G(\beta \mid z)$0, define

$G(\beta \mid z)$1

The iMoE class is the collection of all conditional priors generated in this way. If $G(\beta \mid z)$2 is discrete,

$G(\beta \mid z)$3

then

$G(\beta \mid z)$4

Thus, the support points $G(\beta \mid z)$5 are regression experts and the functions $G(\beta \mid z)$6 are softmax gates. The corresponding marginal response density is

$G(\beta \mid z)$7

which is exactly a softmax-gated MoE.

The important distinction is that the iMoE class is initially defined through a general mixing measure $G(\beta \mid z)$8, potentially with infinitely many support points. The finite MoE form is derived from the NPMLE rather than assumed as the primary model. This construction allows a finite fitted model to approximate continuous conditional priors or nonlinear gate behavior through multiple nearby experts.

The parameterization also admits the usual affine-gate form by absorbing $G(\beta \mid z)$9 into an intercept. However, the original $z$0 parameterization is inherited naturally from the conditional-prior construction and is useful for support selection.

Existence, discreteness, and first-order structure

The likelihood depends on $z$1 through $z$2 quantities: the numerator terms

$z$3

and the normalizing terms

$z$4

The log-likelihood is

$z$5

Since the vector formed by $z$6 lies in a convex hull generated by $z$7, Carathéodory’s theorem yields a discrete maximizer with at most $z$8 support points. Consequently, the iMoE-NPMLE has the representation

$z$9

This is an existence and representation result, not a claim that the fitted model will typically use $x$0 experts. In the simulations, the estimated support sizes range from $x$1 to $x$2, substantially below the theoretical bound.

The directional derivative in the direction of a point mass at $x$3 is

$x$4

At an NPMLE $x$5, every candidate point must satisfy

$x$6

and equality must hold on the support of $x$7. This provides a generalized KKT condition for the infinite-dimensional optimization problem and a practical posterior diagnostic: active support points should have approximately zero directional derivative, while inactive candidates should not have a positive derivative.

For interior support points, differentiating the condition yields two coupled equations. The $x$8 equation is a weighted GLM score equation, with weights proportional to the posterior responsibility of the support point for each observation. The $x$9 equation is a gate-balance condition involving all units. A key asymmetry follows: by Carathéodory’s theorem, each $$y_i \mid x_i,\beta_i \sim \mathrm{EF}, \qquad \eta_i=x_i^\top\beta_i, \qquad \beta_i\mid z_i\sim G^\ast(\cdot\mid z_i).$$0 support point can be characterized using at most $$y_i \mid x_i,\beta_i \sim \mathrm{EF}, \qquad \eta_i=x_i^\top\beta_i, \qquad \beta_i\mid z_i\sim G^\ast(\cdot\mid z_i).$$1 observations, whereas no corresponding sparse representation is established for $$y_i \mid x_i,\beta_i \sim \mathrm{EF}, \qquad \eta_i=x_i^\top\beta_i, \qquad \beta_i\mid z_i\sim G^\ast(\cdot\mid z_i).$$2. The regression coefficient can therefore be generated from local exemplars, but gate parameters must be estimated using the full dataset.

Estimation algorithms

The computational problem is nonconvex because of the normalizing terms $$y_i \mid x_i,\beta_i \sim \mathrm{EF}, \qquad \eta_i=x_i^\top\beta_i, \qquad \beta_i\mid z_i\sim G^\ast(\cdot\mid z_i).$$3. The authors propose two algorithms collectively called EXMoE.

Free-EXMoE initializes candidate regression coefficients using exemplars and then updates mixture weights $$y_i \mid x_i,\beta_i \sim \mathrm{EF}, \qquad \eta_i=x_i^\top\beta_i, \qquad \beta_i\mid z_i\sim G^\ast(\cdot\mid z_i).$$4, gate parameters $$y_i \mid x_i,\beta_i \sim \mathrm{EF}, \qquad \eta_i=x_i^\top\beta_i, \qquad \beta_i\mid z_i\sim G^\ast(\cdot\mid z_i).$$5, and regression coefficients $$y_i \mid x_i,\beta_i \sim \mathrm{EF}, \qquad \eta_i=x_i^\top\beta_i, \qquad \beta_i\mid z_i\sim G^\ast(\cdot\mid z_i).$$6 by block coordinate descent. The $$y_i \mid x_i,\beta_i \sim \mathrm{EF}, \qquad \eta_i=x_i^\top\beta_i, \qquad \beta_i\mid z_i\sim G^\ast(\cdot\mid z_i).$$7 block uses pairwise Frank–Wolfe updates on the simplex. Away steps can set candidate weights exactly to zero, so support selection occurs during optimization rather than through a sweep over the number of experts. The $$y_i \mid x_i,\beta_i \sim \mathrm{EF}, \qquad \eta_i=x_i^\top\beta_i, \qquad \beta_i\mid z_i\sim G^\ast(\cdot\mid z_i).$$8 block is optimized by L-BFGS-B with ridge regularization, and the $$y_i \mid x_i,\beta_i \sim \mathrm{EF}, \qquad \eta_i=x_i^\top\beta_i, \qquad \beta_i\mid z_i\sim G^\ast(\cdot\mid z_i).$$9 block is updated by weighted GLM fits using IRLS. Multiple random restarts address dependence on initialization.

Grid-EXMoE fixes the exemplars and constructs a dictionary from their products with a grid of gate parameters. It then optimizes only the mixture weights. An MM update supplies a warm start, followed by active-set refinement using pairwise Frank–Wolfe steps. This variant is computationally attractive when $z$0 is small, because it can handle a large candidate dictionary without jointly optimizing every expert parameter.

The exemplar construction follows the sparse representation of the $z$1 score equation. An anchor unit is selected, nearby observations in $z$2-space are collected, and a small subset is used to solve a local GLM score equation. This procedure is intended to generate candidate regression relationships that are locally compatible with the covariate-dependent prior.

The output may contain redundant experts, particularly when the true prior is continuous or when a nonlinear gate must be approximated by a sum of linear softmax gates. The paper therefore proposes post-hoc aggregation. Support points are clustered according to $z$3, using average gate weights in the clustering criterion. For a cluster $z$4, the aggregated coefficient and gate are

$z$5

The aggregated prior remains in the iMoE class because aggregation preserves the underlying mixing-measure representation. However, aggregation changes the fitted likelihood and therefore creates an explicit prediction–interpretability trade-off.

Theoretical interpretation and approximation properties

The iMoE form has a hierarchical Bayesian interpretation. Suppose $z$6 has a joint prior and the covariate distribution belongs to an exponential family with canonical parameter $z$7:

$z$8

Then the posterior distribution of $z$9 conditional on $z$0 has precisely the iMoE form. Thus, $z$1 can be interpreted as a posterior distribution induced by observing the covariate $z$2, even though the likelihood used for estimation does not model the marginal distribution of $z$3.

The boundedness of $z$4 imposes a quantitative smoothness condition. For any $z$5 in the iMoE class,

$z$6

where $z$7. The total variation distance also satisfies

$z$8

The implication is direct: $z$9 controls the spatial scale at which the conditional prior can change. Large gate slopes permit sharper transitions, while a small bounded $g$0 enforces smoother variation. This explains why a discontinuous conditional prior cannot belong to the class for any fixed bounded $g$1, although it may be approximated by increasingly steep gates.

The paper also proves a universal approximation result for aggregated gates. On a compact covariate domain, any continuous gate function taking values in the interior of a finite-dimensional simplex can be approximated in $g$2 by aggregated softmax gates. The construction uses groups of gates sharing slopes. In the low-temperature limit, these groups approximate affine Voronoi partitions, while within-group weights encode arbitrary simplex values. Hence a collection of linear softmax gates can represent nonlinear gate functions after aggregation.

This result concerns gate functions, not unrestricted conditional-prior convergence. It establishes approximation capacity but does not establish statistical consistency of the estimated prior.

Simulation evidence

The simulations use Gaussian unit-level regression with $g$3, $g$4, and three conditional-prior mechanisms.

The first, “sin,” has two fixed coefficient vectors whose mixing probability oscillates as $g$5. The true gate is nonlinear and cannot be represented by a single linear softmax gate. The second, “smooth Voronoi,” has six coefficient vectors and smoothly varying region weights that lie within the iMoE class. The third, “switching circle,” has a continuous prior over a circle in coefficient space, with the circle’s center and radius discontinuously changing at $g$6; this mechanism lies outside the iMoE class for bounded $g$7.

Grid-EXMoE uses $g$8 regression exemplars and $g$9 gate values, producing a dictionary of $G(\cdot\mid z)$0 candidate experts. The selected noise standard deviation is $G(\cdot\mid z)$1 for the sin data and the true value $G(\cdot\mid z)$2 for the other two settings. The fitted support sizes are:

Data-generating process	Selected $G(\cdot\mid z)$3	Active experts	Maximum active KKT error per unit	Maximum inactive derivative per unit
Sin	0.4	43	$G(\cdot\mid z)$4	$G(\cdot\mid z)$5
Smooth Voronoi	0.3	41	$G(\cdot\mid z)$6	$G(\cdot\mid z)$7
Switching circle	0.3	67	$G(\cdot\mid z)$8	$G(\cdot\mid z)$9

The fitted models satisfy the candidate-point first-order conditions to the reported tolerance. The support sizes are much smaller than both $G(\beta)$00 and the candidate dictionary, demonstrating substantial practical sparsity.

The estimated pushforward priors on the predictor scale reproduce the principal regression structures in all three settings. In the sin experiment, multiple support points cluster around the two true coefficients, and their aggregated gates recover the oscillatory mixing pattern. This is an important and somewhat counterintuitive result: individual gates remain linear and unimodal, but their sum over coefficient clusters approximates the nonlinear target gate.

Figure 1: Individual and aggregated gates in the sinusoidal simulation, showing how multiple linear gates combine to approximate an oscillatory mixing function.

For the smooth Voronoi data, the individual experts do not reproduce the six true gate functions cleanly, but aggregation recovers their broad region-specific structure. The elbow procedure selects four clusters rather than the true six, while BIC selects ten. Thus, the number of aggregated experts is sensitive to the complexity criterion and should not be interpreted as a consistent estimator of the number of latent relationships.

The switching-circle experiment tests both continuous support in $G(\beta)$01 and an abrupt transition in $G(\beta)$02. The method approximates the continuous coefficient distribution with many discrete support points and uses gates with opposite slopes to transition near $G(\beta)$03. At $G(\beta)$04 and $G(\beta)$05, approximately $G(\beta)$06 and $G(\beta)$07 of the fitted mass, respectively, is assigned to the appropriate side of the discontinuity. This result is necessarily approximation-based: the fitted prior is Lipschitz in $G(\beta)$08, so it cannot reproduce a genuine jump exactly.

Free-EXMoE, which updates $G(\beta)$09 and $G(\beta)$10 rather than relying on a fixed grid, often achieves higher likelihood with fewer experts. On the sin data, for example, it improves the independent test log-likelihood from $G(\beta)$11 to $G(\beta)$12 and reduces the conditional-mean RMSE from $G(\beta)$13 to %%%%114$x$3115$G(\beta)$5116%%%%-1477.3$G(\beta)$17-1456.0$G(\beta)$180.112$G(\beta)$190.101$$. These comparisons show that the two algorithms involve a genuine search-space and computational trade-off rather than a uniformly superior ordering.</p> <h2 class='paper-heading' id='california-housing-application'>California housing application</h2> <p>The California housing analysis uses$$G(\beta)$20 census blocks after excluding observations at the upper response bound. Latitude and longitude are used as gate covariates, while six housing attributes enter the Gaussian regression experts. The fitted EXMoE contains $G(\beta)$21 active support points.

On the held-out sample of $G(\beta)$22 blocks, EXMoE achieves

$G(\beta)$23

compared with $G(\beta)$24 for the covariate-independent NPMLE and $G(\beta)$25 for OLS. The improvement over the common-prior NPMLE is substantial and supports the paper’s central empirical claim that the distribution of regression coefficients varies geographically.

Method	Test $G(\beta)$26
EXMoE	0.719
EXMoE, elbow aggregation	0.598
EXMoE, BIC aggregation	0.634
Covariate-independent NPMLE	0.595
OLS	0.575

The unaggregated model is predictive but difficult to summarize: only $G(\beta)$27 of the $G(\beta)$28 support points are the largest gate for at least one block, and the resulting spatial assignment is fragmented. The elbow aggregation produces three clusters. The largest cluster covers approximately $G(\beta)$29 of blocks, the second approximately $G(\beta)$30, and the third approximately $G(\beta)$$31. The third cluster has the largest intercept and the most negative coefficient for average occupancy, while the second has the smallest intercept and is concentrated in inland regions.</p> <p><img src="https://images.emergentmind.com/paper-images/2610-08321/housing_atom_map.png" alt="Figure 2" title="" class="markdown-image" loading="lazy"></p> <p><img src="https://images.emergentmind.com/paper-images/2610-08321/housing_cluster_map.png" alt="Figure 2" title="" class="markdown-image" loading="lazy"></p> <p><p class="figure-caption">Figure 2: Geographic assignment of California housing blocks before aggregation, illustrating the fragmented territories associated with individual support points.</p></p> <p><img src="https://images.emergentmind.com/paper-images/2610-08321/housing_path_map.png" alt="Figure 3" title="" class="markdown-image" loading="lazy"></p> <p><img src="https://images.emergentmind.com/paper-images/2610-08321/housing_path_heat_MedInc.png" alt="Figure 3" title="" class="markdown-image" loading="lazy"></p> <p><img src="https://images.emergentmind.com/paper-images/2610-08321/housing_path_heat_HouseAge.png" alt="Figure 3" title="" class="markdown-image" loading="lazy"></p> <p><p class="figure-caption">Figure 3: The estimated conditional prior along a path through six California cities, showing spatial movement in the distributions of regression coefficients.</p></p> <p>The aggregated models lose predictive accuracy:$$G(\beta)$32 falls to $G(\beta)$33 with the three-cluster elbow solution and $G(\beta)$34 with the twelve-cluster BIC solution. This result makes the interpretation–prediction trade-off concrete. Aggregation is useful for identifying broad submarket relationships, but it is not likelihood-preserving and can discard structure needed for prediction.

NHANES depression-score application

The second application analyzes $G(\beta)$35 adults from NHANES using PHQ-9 scores. The response is overdispersed and contains many zeros, so the primary analysis uses negative-binomial experts with a log link. Age, sex, income-to-poverty ratio, sleep hours, and BMI are used both in the experts and in the gates.

NB-EXMoE selects dispersion $G(\beta)$36, regularization $G(\beta)$37, and $G(\beta)$38 active support points. On $G(\beta)$39 test subjects, it obtains a mean log-likelihood of $G(\beta)$40 and AUC $G(\beta)$41 for classifying PHQ-9 scores of at least $10. This is better than the negative-binomial GLM, ZINB, and the covariate-independent Poisson EXMoE in both reported metrics.

Method	Test log-likelihood	AUC
NB-EXMoE	$G(\beta)$42	0.711
NB-EXMoE, BIC aggregation	$G(\beta)$43	0.704
Poisson EXMoE, $G(\beta)$44	$G(\beta)$45	0.709
ZINB	$G(\beta)$46	0.699
NB-GLM	$G(\beta)$47	0.700
Poisson GLM	$G(\beta)$48	0.700

The estimated conditional prior varies relatively little across covariates because the selected regularization is strong. Sleep hours nevertheless produce visible heterogeneity: as sleep increases, prior mass shifts from high-intercept experts toward lower-intercept experts. The BMI coefficient is near zero for shorter sleep durations but becomes positive for longer sleep durations in some high-weight components.

Figure 4: Estimated movement of the intercept and BMI coefficient distributions as sleep hours vary in the NHANES analysis.

The predictive mean reproduces the observed U-shaped relationship between sleep duration and depression score, with the lowest predicted scores around seven to eight hours. This is obtained despite linear experts and linear gates, because the gate-weighted mixture induces nonlinear marginal behavior. The result illustrates how covariate-dependent mixing can represent population-level nonlinearities without inserting explicit nonlinear interaction terms into each expert.

Limitations and open questions

The principal theoretical limitation is that the paper does not establish consistency or convergence rates for the iMoE-NPMLE toward a true conditional prior. The $G(\beta)$49 support bound is a finite-sample representation result, and the universal approximation theorem concerns aggregated gate functions rather than the statistical behavior of the estimator.

Identifiability is also unresolved for continuous conditional priors. Classical identifiability results apply to several discrete-mixture settings, but it remains open which aspects of a continuous $G(\beta)$50 are identifiable from one response observation per unit and varying $G(\beta)$51. This matters directly for interpreting support points as latent regression relationships rather than computational approximations.

The iMoE restriction imposes Lipschitz continuity in $G(\beta)$52 through bounded $G(\beta)$53. Consequently, abrupt changes in the true conditional prior can only be approximated, with the approximation controlled by the allowable gate slopes. The switching-circle simulation demonstrates empirical approximation but does not remove this structural mismatch.

The optimization procedures target local stationary points. Free-EXMoE uses nonconvex block updates and random restarts, while Grid-EXMoE depends on the quality of the exemplar and gate dictionaries. Neither algorithm provides a global optimization guarantee. The regularization parameter, support bounds, grid resolution, exemplar construction, and aggregation rule can all affect the fitted representation.

Finally, interpretation becomes difficult when the dimension of $G(\beta)$54 is large. In the California analysis, two-dimensional geographic gates can be visualized directly. For $G(\beta)$55, locating and summarizing regions of covariate-dependent heterogeneity is substantially less straightforward. The paper leaves open how to obtain statistically justified, high-dimensional summaries of the gate structure.

Conclusion

The paper develops a coherent extension of empirical-Bayes $G(\beta)$56-modeling from a shared prior to a covariate-dependent prior over unit-level regression coefficients. Its central device, the iMoE class, resolves the ill-posedness of unrestricted conditional-prior NPMLE while retaining a nonparametric target. The resulting estimator is exactly a softmax-gated MoE with data-selected support, admits directional-derivative optimality conditions, and can approximate continuous priors and nonlinear gate functions through redundant expert representations.

The simulations show that EXMoE can recover covariate-dependent regression structure, approximate continuous and discontinuous conditional priors, and select sparse active representations. The California housing and NHANES analyses provide complementary evidence: geographic covariate dependence yields a large predictive improvement over a common prior, while a comparatively simple eleven-expert model improves count prediction and depression classification. The unresolved questions concern statistical convergence, continuous-prior identifiability, global optimization, principled aggregation, and high-dimensional interpretation.