Neural Additive Experts (NAEs)

• Neural Additive Experts (NAEs) are neural architectures that relax the strict additivity of GAMs by using context-gated mixtures of per-feature experts for enhanced flexibility.
• They employ feature-specific encoders and dynamic gating to capture context-dependent interactions, bridging the gap between interpretable GAMs and more expressive models.
• Additivity control through regularization penalties enables a trade-off between model expressivity and the clear, feature-level attribution essential for interpretability.

Neural Additive Experts (NAEs) are a neural architecture that preserves an additive decomposition at the level of feature contributions while relaxing the strict additivity of generalized additive models through context-gated mixtures of per-feature experts. In the formulation introduced in "Neural Additive Experts: Context-Gated Experts for Controllable Model Additivity" (Xiong et al., 11 Feb 2026), the prediction remains a sum of feature-wise terms, but each term is produced by multiple specialized experts whose mixture weights can depend on all input features. This construction is intended to retain feature-level attribution while capturing interaction structure that standard GAMs and NAMs cannot represent directly.

1. Definition and conceptual position

The motivating comparison is with the classical GAM form

$f(x) = \omega_0 + \sum_{i=1}^n f_i(x_i),$

which yields clear feature-level interpretability but cannot represent genuinely context-dependent effects. A standard extension is the GA$^2$M form

$$f(x) = \omega_0 + \sum_{i=1}^n f_i(x_i) + \sum_{1\le i < j \le n} f_{ij}(x_i,x_j),$$

which improves expressivity at the cost of explanation complexity growing to $O(n^2)$ components. NAEs are designed to occupy the intermediate regime: they keep a top-level sum over features, but replace each single-feature function by a context-gated mixture of experts, thereby relaxing rigid additivity without abandoning a per-feature decomposition (Xiong et al., 11 Feb 2026).

This design differs from a conventional mixture-of-experts architecture in a precise way. In NAEs, experts are organized per feature rather than as global subnetworks over the whole input, and the final prediction is still written as a sum of feature contributions. The resulting model is additive in its outputs $o_i(x)$, but those outputs need not be additive in the raw inputs because the gating for feature $i$ can depend on $x_1,\dots,x_n$ (Xiong et al., 11 Feb 2026).

A broader technical reading of the term has emerged around related additive decompositions. Several works interpret concept-level, feature-level, cluster-level, or neuron-level decompositions as NAE-like even when they do not use the term explicitly; examples include CAT’s concept encoders plus TaylorNet, CRISP-NAM’s feature-wise subnetworks for competing risks, NeurCAM’s additive clustering heads with selection gates, and MoNE’s decomposition of FFN experts into additive neuron experts (Duong et al., 2024, Ramachandram et al., 27 May 2025, Upadhya et al., 2024, Cheng et al., 7 Oct 2025).

2. Architectural formulation

NAEs always predict through a feature-additive outer form: $\hat{y}(x) = \omega_0 + \sum_{i=1}^n o_i(x).$ The difference from a NAM lies in how each $o_i(x)$ is computed. First, each feature is encoded by a feature-specific encoder

$$\mathcal{E}_i : \mathcal{X}_i \to \mathbb{R}^d, \quad z_i = \mathcal{E}_i(x_i).$$

In the reported experiments, $^2$0 is a small MLP. For feature $^2$1, the model instantiates $^2$2 experts

$^2$3

with $^2$4 typically implemented as a single linear layer (Xiong et al., 11 Feb 2026).

The gating mechanism is the non-additive part. For feature $^2$5, NAE computes expert scores

$^2$6

where $^2$7 is a bias vector and $^2$8 is a routing matrix. After adding a sparsity-inducing mask $^2$9, a softmax produces mixture weights

$$f(x) = \omega_0 + \sum_{i=1}^n f_i(x_i) + \sum_{1\le i < j \le n} f_{ij}(x_i,x_j),$$0

The contribution of feature $$f(x) = \omega_0 + \sum_{i=1}^n f_i(x_i) + \sum_{1\le i < j \le n} f_{ij}(x_i,x_j),$$1 is then

$$f(x) = \omega_0 + \sum_{i=1}^n f_i(x_i) + \sum_{1\le i < j \le n} f_{ij}(x_i,x_j),$$2

and the full predictor becomes

$$f(x) = \omega_0 + \sum_{i=1}^n f_i(x_i) + \sum_{1\le i < j \le n} f_{ij}(x_i,x_j),$$3

Thus, each expert remains feature-local in content, but expert selection is context-sensitive (Xiong et al., 11 Feb 2026).

Two variants are described. NAE-D uses diagonal routing, setting $$f(x) = \omega_0 + \sum_{i=1}^n f_i(x_i) + \sum_{1\le i < j \le n} f_{ij}(x_i,x_j),$$4 for $$f(x) = \omega_0 + \sum_{i=1}^n f_i(x_i) + \sum_{1\le i < j \le n} f_{ij}(x_i,x_j),$$5, so each feature’s expert relevance depends only on its own encoder; this recovers a strictly additive regime. NAE-E alters the activation pattern so that activated experts are weighted evenly, producing a more discrete combinatorial gating behavior (Xiong et al., 11 Feb 2026).

3. Additivity control, regularization, and explanation semantics

A central contribution of NAEs is explicit control over the degree of additivity. The paper defines the additivity metric

$$f(x) = \omega_0 + \sum_{i=1}^n f_i(x_i) + \sum_{1\le i < j \le n} f_{ij}(x_i,x_j),$$6

with $$f(x) = \omega_0 + \sum_{i=1}^n f_i(x_i) + \sum_{1\le i < j \le n} f_{ij}(x_i,x_j),$$7 for numerical stability. If $$f(x) = \omega_0 + \sum_{i=1}^n f_i(x_i) + \sum_{1\le i < j \le n} f_{ij}(x_i,x_j),$$8 is purely a function of $$f(x) = \omega_0 + \sum_{i=1}^n f_i(x_i) + \sum_{1\le i < j \le n} f_{ij}(x_i,x_j),$$9, the ratio is $O(n^2)$0; if $O(n^2)$1 varies strongly with other features, the ratio falls below $O(n^2)$2 (Xiong et al., 11 Feb 2026).

Training minimizes the task loss plus an expert-variation penalty: $O(n^2)$3 This second term penalizes the within-feature variance of expert outputs. When $O(n^2)$4 is large, all experts for a feature are pushed toward the same output, making the gate irrelevant and driving the model toward a GAM-like limit (Xiong et al., 11 Feb 2026).

The interpretability mechanism remains feature-level. Because the predictor is always written as $O(n^2)$5, each feature has a scalar contribution on every example. In addition, the model defines intrinsic bounds

$O(n^2)$6

which provide a context-agnostic envelope of possible contributions for a given feature value. This differs from GA$O(n^2)$7M-style interaction explanations, where a single feature’s effect is dispersed across many pairwise surfaces (Xiong et al., 11 Feb 2026).

The empirical effect of $O(n^2)$8 is illustrated on the Housing dataset. With $O(n^2)$9, Additivity is approximately $o_i(x)$0 and RMSE is approximately $o_i(x)$1; with $o_i(x)$2, Additivity is approximately $o_i(x)$3 and RMSE remains approximately $o_i(x)$4; with $o_i(x)$5, Additivity is approximately $o_i(x)$6 and RMSE rises to approximately $o_i(x)$7; with $o_i(x)$8, Additivity is approximately $o_i(x)$9 and RMSE is approximately $i$0; with $i$1, Additivity reaches $i$2 and RMSE is approximately $i$3. The reported behavior is therefore a continuous trade-off between flexibility and strict additivity (Xiong et al., 11 Feb 2026).

4. Expressivity and theoretical properties

The theoretical analysis places NAEs strictly beyond standard GAMs. First, any $i$4 can be represented exactly by an NAE with $i$5: one sets $i$6 and chooses $i$7. This establishes the containment

$i$8

Second, the paper proves approximate containment of GA$i$9M. The argument uses finite separable approximations

$x_1,\dots,x_n$0

followed by a two-expert construction that represents each separable term through context-dependent gating. In this sense, NAEs can approximate additive-plus-pairwise-interaction models to arbitrary precision (Xiong et al., 11 Feb 2026).

A further theorem formalizes the additive limit induced by the variation penalty. Let

$x_1,\dots,x_n$1

As $x_1,\dots,x_n$2, $x_1,\dots,x_n$3, expert outputs within each feature collapse, the gate ceases to matter, and any limit point of minimizers is a GAM. The additivity metric is correspondingly nondecreasing in $x_1,\dots,x_n$4 and converges to $x_1,\dots,x_n$5 (Xiong et al., 11 Feb 2026).

This theoretical boundary has consequences for formal explanation. "Provably Explaining Neural Additive Models" shows that for strict NAMs one can obtain provably cardinally-minimal sufficient explanations with $x_1,\dots,x_n$6 verification queries to the full model after preprocessing of univariate components. The same work also states that this machinery does not apply directly to additive-expert architectures with multivariate global gating, because the effective contribution of a feature is no longer a purely univariate function of that feature alone (Bassan et al., 19 Feb 2026). A plausible implication is that NAEs gain expressivity partly by moving outside the structural regime that currently supports the strongest formal minimality guarantees.

5. Empirical behavior

On synthetic data, NAEs are reported to match NAMs when the ground truth is purely additive and to exceed them when the target is multimodal or context-dependent. In a unimodal additive case, NAM and NAE both recover the correct shape for $x_1,\dots,x_n$7. In a multimodal case where the conditional relation between $x_1,\dots,x_n$8 and $x_1,\dots,x_n$9 flips sign according to a binary $\hat{y}(x) = \omega_0 + \sum_{i=1}^n o_i(x).$0, NAM collapses to a near-linear shape while NAE reconstructs the oscillatory pattern through multiple experts and context gating. On this multimodal task, the reported additivity scores are $\hat{y}(x) = \omega_0 + \sum_{i=1}^n o_i(x).$1, $\hat{y}(x) = \omega_0 + \sum_{i=1}^n o_i(x).$2, and $\hat{y}(x) = \omega_0 + \sum_{i=1}^n o_i(x).$3 for $\hat{y}(x) = \omega_0 + \sum_{i=1}^n o_i(x).$4, $\hat{y}(x) = \omega_0 + \sum_{i=1}^n o_i(x).$5, and $\hat{y}(x) = \omega_0 + \sum_{i=1}^n o_i(x).$6, respectively (Xiong et al., 11 Feb 2026).

A second synthetic interaction study uses

$\hat{y}(x) = \omega_0 + \sum_{i=1}^n o_i(x).$7

For this target, NAE achieves RMSE approximately $\hat{y}(x) = \omega_0 + \sum_{i=1}^n o_i(x).$8 whereas NAM achieves RMSE approximately $\hat{y}(x) = \omega_0 + \sum_{i=1}^n o_i(x).$9, illustrating the effect of context-dependent expert selection on explicitly non-additive structure (Xiong et al., 11 Feb 2026).

On real-world tabular datasets, the reported results compare NAE primarily against feature-explaining baselines such as NAM, EBM, and NODE-GAM.

Dataset	Metric	Selected results
Housing	RMSE $o_i(x)$0	NAE 0.451; NAM 0.572; EBM 0.559; NODE-GAM 0.558
MIMIC-II	AUC $o_i(x)$1	NAE 0.847; NAM 0.834; EBM 0.835; NODE-GAM 0.832
MIMIC-III	AUC $o_i(x)$2	NAE 0.825; NAM 0.813; EBM 0.809; NODE-GAM 0.814
Income	AUC $o_i(x)$3	NAE 0.927; NAM 0.910; EBM 0.927; NODE-GAM 0.927
Credit	AUC $o_i(x)$4	NAE 0.982; NAM 0.977; EBM 0.974; NODE-GAM 0.981
Year	MSE $o_i(x)$5	NAE 78.66; NAM 85.25; EBM 85.81; NODE-GAM 85.09

These results support the paper’s claim that NAEs substantially outperform standard GAMs and NAMs and are often competitive with interaction-explaining GA$o_i(x)$6M variants and black-box baselines, while retaining feature-level explanation complexity of $o_i(x)$7 rather than $o_i(x)$8 (Xiong et al., 11 Feb 2026).

The implementation used AdamW, cosine annealing, feature encoders with 3–4 MLP layers, hidden dimensions typically $o_i(x)$9 or $$\mathcal{E}_i : \mathcal{X}_i \to \mathbb{R}^d, \quad z_i = \mathcal{E}_i(x_i).$$0, normalization via LayerNorm or BatchNorm, and dropout in the range $$\mathcal{E}_i : \mathcal{X}_i \to \mathbb{R}^d, \quad z_i = \mathcal{E}_i(x_i).$$1. Main experiments used $$\mathcal{E}_i : \mathcal{X}_i \to \mathbb{R}^d, \quad z_i = \mathcal{E}_i(x_i).$$2 with all experts active; ablations extended to $$\mathcal{E}_i : \mathcal{X}_i \to \mathbb{R}^d, \quad z_i = \mathcal{E}_i(x_i).$$3 for NAE and $$\mathcal{E}_i : \mathcal{X}_i \to \mathbb{R}^d, \quad z_i = \mathcal{E}_i(x_i).$$4 for NAE-E (Xiong et al., 11 Feb 2026).

6. Relation to neighboring additive-expert models

The explicit NAE framework sits within a wider family of additive neural decompositions. CAT groups raw features into high-level concepts, encodes each group into a scalar concept value, and feeds the resulting concept vector into a Tucker-decomposed TaylorNet. A common interpretation is that each concept encoder acts as an expert over a feature group, while TaylorNet supplies additive and low-order interaction terms over concepts; unlike NAEs, CAT has no gating network and keeps all concept-level terms active (Duong et al., 2024).

CRISP-NAM extends neural additive models to competing-risks survival analysis by assigning a dedicated FeatureNet to each feature and linear cause-specific readouts to each feature representation. The cause-specific log-hazard remains additive over feature contributions, making it a multi-task additive-expert model with shared feature experts and task-specific heads, but again without context-dependent gating across features (Ramachandram et al., 27 May 2025).

NeurCAM introduces interpretable clustering through additive models with selection gates. Its cluster logits are sums of univariate and pairwise shape functions, and entmax-based gates choose which features or interactions populate a fixed number of slots. This suggests an alternative NAE design in which sparsity is controlled by slot budgets rather than by mixtures of experts per feature (Upadhya et al., 2024).

AxNN organizes additive experts through ensembles of GAMnet and xNN subnetworks. GAMnet supplies per-feature experts, xNN supplies additive index experts, and the overall ensemble is decomposed into main effects and higher-order interaction components through thresholded projection patterns. In that sense, AxNN is an additive-expert system built from boosted or stacked subnetworks rather than from context-gated per-feature mixtures (Chen et al., 2020).

Monotonic Neural Additive Models specialize the per-feature additive regime to regulated credit scoring. Each feature has its own one-dimensional neural subnetwork, and derivative penalties enforce individual monotonicity and pairwise monotonicity relations. The resulting architecture shows how expert-factorized additive models make shape constraints computationally cheap, because monotonicity can be enforced on univariate grids rather than through global verification (Chen et al., 2022).

Hybrid deep additive neural networks replace some affine layers by additive basis-expansion layers and combine them with standard fully connected layers. Their analysis shows that additive/basis-function “expert layers” can retain universal approximation while improving parameter efficiency, suggesting a different route to additive experts in which specialization is implemented by basis blocks rather than by gating (Kim et al., 2024).

A distinct but relevant direction appears in Mixture of Neuron Experts. There, a GLU expert is decomposed into a sum of linear neuron-experts

$$\mathcal{E}_i : \mathcal{X}_i \to \mathbb{R}^d, \quad z_i = \mathcal{E}_i(x_i).$$5

and only the top-$$\mathcal{E}_i : \mathcal{X}_i \to \mathbb{R}^d, \quad z_i = \mathcal{E}_i(x_i).$$6 neurons are computed inside each selected MoE expert. This is an additive-expert view below the feature level: expert selection happens at the neuron granularity, not at the level of input features or concepts (Cheng et al., 7 Oct 2025).

7. Limitations and open directions

The principal limitation reported for NAEs is computational. Full routing uses $$\mathcal{E}_i : \mathcal{X}_i \to \mathbb{R}^d, \quad z_i = \mathcal{E}_i(x_i).$$7 for all feature pairs $$\mathcal{E}_i : \mathcal{X}_i \to \mathbb{R}^d, \quad z_i = \mathcal{E}_i(x_i).$$8, yielding $$\mathcal{E}_i : \mathcal{X}_i \to \mathbb{R}^d, \quad z_i = \mathcal{E}_i(x_i).$$9 parameters in the gating matrices. This can become expensive on very high-dimensional tabular problems. NAE-D reduces this cost but also removes the mechanism by which cross-feature interactions are represented (Xiong et al., 11 Feb 2026).

Interpretation also becomes less direct as the number of experts grows. In practice, the recommended interpretation is not to inspect every $^2$00 separately but to analyze aggregate feature curves and their lower/upper envelopes. The paper further notes sensitivity to the hyperparameters $^2$01, $^2$02, and $^2$03; larger $^2$04 often requires stronger regularization to avoid overfitting, and no automatic selection rule is provided (Xiong et al., 11 Feb 2026).

A further issue is visualization of higher-order interaction structure. NAEs can represent multi-way effects through gating, and the paper presents three-way interaction examples, but it also notes the absence of a standardized visualization toolkit for such behavior. Suggested future directions include sparsifying the routing matrix $^2$05, improving procedures for selecting the number of experts and the variation penalty, and developing better tools for multi-way interaction visualization (Xiong et al., 11 Feb 2026).

From the perspective of formal interpretability, the broader literature suggests a structural trade-off. Strictly additive NAMs admit provable cardinally-minimal sufficient explanations under efficient verification routines, whereas additive-expert models with global context gating do not directly satisfy those assumptions. This suggests that NAEs exchange some of the formal tractability of pure additive models for a wider expressive envelope that includes pairwise and higher-order context effects (Bassan et al., 19 Feb 2026).