GA²M: Additive Models with Pairwise Interactions

• GA²M is a statistical model that extends traditional GAMs by incorporating pairwise interactions, enabling the capture of nonlinear, nonadditive feature effects.
• Its structure combines univariate main effects with bivariate interaction terms, allowing clear visualization and interpretation through 1D and 2D plots.
• Regularization techniques like boosting, Bayesian selection, and group-lasso are used to balance model complexity while maintaining interpretability.

A Generalized Additive Model with Pairwise Interactions (GA²M) is a model class that extends classical Generalized Additive Models (GAMs) to include explicit, interpretable modeling of second-order (pairwise) feature interactions. This allows the model to capture nonlinear, nonadditive relationships while maintaining interpretability suitable for domains demanding transparency such as healthcare, finance, and scientific applications.

1. Model Specification and Mathematical Formulation

The canonical GA²M is defined as follows: $$f(x) = c + \sum_{s=1}^D f_s(x_s) + \sum_{s_1 \neq s_2} f_{s_1s_2}(x_{s_1}, x_{s_2})$$ where:

• $f_s(x_s)$ are univariate functions (main effects) capturing nonlinear contributions of individual features.
• $f_{s_1s_2}(x_{s_1}, x_{s_2})$ are bivariate functions (pairwise interactions) encoding interactions between distinct features.
• $c$ is a global intercept.

For generalized linear models, the GA²M structure can be embedded in the link function: $$g(\mathbb{E}[y|x]) = f_0 + \sum_{j=1}^D f_j(x_j) + \sum_{j=1}^D \sum_{j'>j} f_{jj'}(x_j, x_{j'})$$ with $g(\cdot)$ denoting the canonical link (e.g., logit for logistic regression).

The pairwise terms enable the model to directly learn and visualize two-dimensional patterns in feature pairs, addressing contexts where simple additivity fails to capture the underlying mechanism.

2. Interpretability and Motivation

The primary motivation for GA²Ms is to balance model expressiveness with interpretability. Each univariate and bivariate component can be visualized and examined directly, facilitating both global (overall) and local (case-level) explanations of predictions. This structure mitigates key interpretability challenges of both linear models (no nonlinearity) and black-box machine learning methods (uninterpretable interaction complexity).

However, the number of pairwise terms scales quadratically with the number of features, leading to a tradeoff:

• With few features or selected interactions, interpretability remains tractable, and insights from two-dimensional plots are actionable.
• In high-dimensional settings, careful interaction selection or regularization is essential to maintain interpretability and avoid overwhelming the analyst (Gkolemis et al., 2023, Scheipl, 2011).

3. Estimation and Regularization Techniques

Numerous approaches exist for efficiently estimating and regularizing GA²Ms:

• Backfitting and Boosting: Component functions can be fitted via backfitting (cyclically updating $f_s$ and $f_{s_1s_2}$) or gradient boosting with shallow trees/splines (Yang et al., 2020, Chang et al., 2021).
• Hierarchical and Bayesian Selection: Bayesian stochastic search using spike-and-slab priors allows automatic inclusion/exclusion of main and interaction terms, both linear and nonlinear, with explicit posterior inclusion probabilities (Scheipl, 2011).
• Group-Lasso Regularization: Hierarchical group-lasso (e.g., glinternet) enforces strong or weak hierarchy constraints to ensure that pairwise terms are only included if their main effects are present, allowing scaling to very high dimensions while preserving identifiability (Lim et al., 2013).
• Reluctant Interaction Selection: Algorithms based on conditional marginal likelihood improvements (Sprinter) prefer main effects and only add interactions if they offer unique predictive signal beyond additive terms, enabling scalable screening among millions of interactions without assuming hierarchy (Lu et al., 16 Jan 2024).
• Neural Approaches: Neural analogues such as NODE-GA²M and GAMI-Net enforce the GA²M inductive bias via neural subnetworks, gates, or feature selection constraints, ensuring that the output can still be decomposed into main and pairwise interaction effects (Chang et al., 2021, Yang et al., 2020).

Hierarchy Principles

• Strong hierarchy requires that both corresponding main effects be included before an interaction is allowed.
• Weak hierarchy requires at least one main effect.
• Approaches without any hierarchy can detect anti-hierarchical or “pure” interactions (Lu et al., 16 Jan 2024).

4. Extensions and Variants of GA²M

Significant research explores modifications to traditional GA²M frameworks:

• Monotonicity Constraints (CGA²M+): Enforcing monotonicity on selected shape functions improves interpretability and generalization, particularly where domain knowledge dictates the expected direction of effects. Constraints are implemented, for example, via monotonic boosting or tree construction (Watanabe et al., 2021).
• Higher-Order Interactions: CGA²M+ further augments the GA²M structure with an explicit “higher-order” term, fit to the residuals after main and pairwise effects, enabling improved accuracy in the presence of higher-order dependencies, while quantifying the interpretability loss through the term’s importance (Watanabe et al., 2021).
• Missing Data (M-GAM): Models incorporating missingness indicators and their pairwise interactions, with $\ell_0$ regularization, maintain interpretability and predictive power when data are not fully observed. Proposition 1 in (McTavish et al., 3 Dec 2024) establishes that direct modeling of missingness strictly generalizes impute-then-predict paradigms without unnecessary loss of signal.
• Moderated Interactions: Moderated Network Models (MNM) introduce context-dependent pairwise interactions, where each edge can be linearly modulated by the value of other variables. Estimation uses $\ell_1$-regularized nodewise regressions, supporting explicit modeling of effect moderation (Haslbeck et al., 2018).
• Regionally Additive Models (RAM): These address interaction limitations by partitioning the feature space into subregions with minimized interactions, fitting additive models locally. This enables modeling context-specific effects without resorting to an exponential number of interaction terms (Gkolemis et al., 2023).

5. Neural and Hybrid Implementations

Recent work synthesizes GA²M structure with neural and hybrid architectures to overcome scalability and flexibility limitations:

• Neural GA²Ms (NODE-GA²M, GAMI-Net): These models construct main and pairwise terms via neural subnetworks, enforcing de-coupling via architectural constraints (e.g., feature gates, subnetworks per interaction), and leverage GPU-accelerated optimization, scaling to millions of samples/features (Chang et al., 2021, Yang et al., 2020). Marginal clarity, sparsity, and heredity can be explicitly regularized.
• Hybrid Energy-Based Models: Combining an explicit pairwise energy with a neural energy (MLP) allows faithful recovery of interpretable pairwise parameters in the presence of higher-order dependencies, outperforming both pairwise-only and “black-box” neural models in cases where higher-order structure is present but not dominating (Feinauer et al., 2020).

Model Class	Interactions Modeled	Main Regularization Principle	Scalability
Classical GA²M (trees/splines)	Pairwise (arbitrary)	Additive/Select via boosting/backfit	Limited high-d
spikeSlabGAM	Pairwise, Higher-order	Spike-and-slab Bayesian selection	Moderate
glinternet (Group-Lasso)	Pairwise, strong/weak H	Hierarchical group-lasso, screening	High
Sprinter (Reluctant)	Pairwise, no hierarchy	Conditional marginal likelihood	Very high
NODE-GA²M/GAMI-Net	Pairwise (arbitrary)	Neural, feature gating, regularization	Very high
CGA²M+	Pairwise + higher order	Monotonicity, hierarchical selection	Moderate-high
M-GAM	Pairwise + missingness	$\ell_0$ sparsity/selection	High

6. Empirical and Theoretical Properties

Empirical results consistently show that GA²Ms, whether implemented via boosting, Bayesian selection, group-lasso, or neural subnetworks, generally outperform purely additive models when interactions are present, without forfeiting global interpretability so long as the number of active interaction terms remains tractable (Chang et al., 2021, Yang et al., 2020, Watanabe et al., 2021, Scheipl, 2011, Lim et al., 2013). Key points include:

• Predictive Performance: Performance often approaches that of full black-box models (e.g., Random Forest, XGBoost, neural networks), particularly on structured data.
• Interpretability: 1D and 2D effect plots yield actionable insights; sparsity via regularization is critical for scalability and clarity.
• Regularization Impact: Strong or reluctant regularization helps maintain interpretability in high dimensions while recovering “pure” or anti-hierarchical interactions as needed (Lu et al., 16 Jan 2024).
• Higher-Order Effects: Explicit modeling of higher-order terms can further improve accuracy where complex dependencies exist, but their contribution should be carefully monitored to avoid opacity (Watanabe et al., 2021).
• Missing Data: Direct modeling of missingness leads to superior sparsity and interpretability outcomes compared to impute-then-predict approaches, with theoretical guarantees (McTavish et al., 3 Dec 2024).
• Empirical Benchmarks: On canonical datasets (e.g., Bike Sharing, California Housing), regionally additive models, neural GA²Ms, and CGA²M+ variants achieve accuracy competitive with DNNs while improving interpretability and robustness (Gkolemis et al., 2023, Watanabe et al., 2021, Chang et al., 2021).

7. Practical Considerations and Future Directions

Deployment of GA²Ms involves decisions regarding:

• Interaction selection (automated or user-directed),
• Choice of regularization (sparsity vs. hierarchy tradeoff),
• Model complexity vs. interpretability,
• Integration of domain knowledge (e.g., monotonicity, subregions, missing data handling).

Recent trends emphasize:

• Use of neural parameterizations for massive-scale tabular prediction while enforcing the GA²M structure,
• Inference of interaction structures under complex, high-dimensional settings via scalable, hierarchy-free algorithms,
• Expanding the interpretable frontier of additive models to encompass higher-order, conditional, and context-dependent effects,
• Hybridization with deep models for flexible, robust, yet interpretable models (Feinauer et al., 2020, Chang et al., 2021, Yang et al., 2020, McTavish et al., 3 Dec 2024).

GA²Ms are now central in both research and applied settings requiring legibility and accountability, providing a versatile and extensible architecture for interpretable statistical and machine learning modeling in the presence of interaction effects.