---
title: 'SPAM: Scalable Polynomial Additive Models'
url: https://www.emergentmind.com/topics/scalable-polynomial-additive-models-spam
type: topic
---

# SPAM: Scalable Polynomial Additive Models

Scalable Polynomial Additive Models (SPAM) represent a fundamental advance in interpretable machine learning, bridging the expressive limitations of traditional Generalized Additive Models (GAMs) and the scalability challenges posed by high-dimensional data and higher-order interactions. SPAMs employ low-rank tensor polynomial structures to model all orders of feature interactions with a parameter count and computational demand that scale linearly in the ambient feature dimension, thus matching or exceeding the performance of state-of-the-art black-box models while retaining inherent interpretability [2205.14108].

## 1. Formal Definition and Model Structure

A Scalable Polynomial Additive Model seeks to approximate an unknown target function over $d$-dimensional input features $x = (x_1,\dots,x_d)\in \mathbb{R}^d$ as a polynomial of degree $k$:
\[
P(x) = b + \sum_{i=1}^d w_i^{(1)}x_i + \sum_{i<j} w_{ij}^{(2)} x_i x_j + \cdots + \sum_{i_1<\dots<i_k} w_{i_1,\dots,i_k}^{(k)} x_{i_1}\cdots x_{i_k}
\]
where each order-$l$ term captures all $l$-way feature interactions [2205.14108].

A naïve expansion of this polynomial faces a combinatorial explosion: the number of parameters at order $l$ is $\binom{d}{l}$, rendering direct implementation infeasible for large $d$ or $k$. SPAM circumvents this with two observations:
- Each order-$l$ weight tensor $W^{(l)} \in \mathbb{R}^{d\times\cdots\times d}$ (with $l$ factors) can be assumed symmetric without loss of generality.
- Any symmetric $l$-th order tensor admits a symmetric low-rank CP decomposition:
  \[
  W^{(l)} = \sum_{i=1}^{r_l} \lambda_{li} u_{li}^{\otimes l}, \qquad u_{li} \in \mathbb{R}^d
  \]
Plugging these decompositions into the polynomial yields:
\[
P(x) = b + \sum_{l=1}^k \sum_{i=1}^{r_l} \lambda_{li} \langle u_{li}, x \rangle^l
\]
with total parameter count $O(d \sum_{l=1}^k r_l)$. Each rank $r_l$ is typically much smaller than $d^l$, ensuring scalability [2205.14108].

## 2. Optimization and Algorithmic Implementation

SPAM parameters $\{u_{li}, \lambda_{li}\}$ are learned through empirical risk minimization:
\[
\min_{\{u_{li},\lambda_{li}\}} \ \frac{1}{n}\sum_{j=1}^n \ell(P(x_j),y_j) + \beta\, \Omega(\{u,\lambda\})
\]
where $\ell$ is the loss (e.g., cross-entropy for classification, squared loss for regression), and $\Omega$ encodes $L_1$ or $L_2$ penalties for regularization [2205.14108].

The entire parameterization supports mini-batch SGD or AdamW with full GPU acceleration. Key implementation innovations include:
- Input rescaling for stabilization: each feature is transformed via $x_j \mapsto \mathrm{sign}(x_j)\,|x_j|^{1/l}$ for order-$l$ terms, harmonizing the scales across interaction orders.
- “Basis dropout” on rank-1 components—randomly zeroing out $\lambda_{li}$ during training—to regularize and improve robustness.
- For multi-class extensions, basis vectors $u_{li}$ can be shared across classes, reducing the parameter count for $C$ classes from $O(r d C)$ to $O(r d + C r)$.

The computational cost of evaluation and training scales as $O\left(d \sum_l r_l\right)$ per sample, avoiding the $O(d^k)$ regime of naïve polynomial expansions.

## 3. Theoretical Risk Guarantees and Expressivity

SPAM provably matches the risk convergence rates of full-rank polynomials under mild “spectral-decay” assumptions (i.e., rapid decay of the singular values $\lambda_{li}$ of the true polynomial). Specifically, for $L_2$-regularization and 1-Lipschitz losses, the excess risk satisfies:
\[
\mathcal{R}(\widehat P) - \mathcal{R}(P^*) \le 2B_\lambda \left(\sum_{l=1}^k B_u^l \sqrt{r_l}\right) \sqrt{\frac{d}{n} + \frac{C_1}{C_2}\sum_{l=2}^k e^{-r_l^\gamma}} + O\left(\sqrt{\frac{\log(1/\delta)}{n}}\right)
\]
The remainder vanishes as $r_l$ increases. This ensures that statistical efficiency is not sacrificed for scalability or interpretability [2205.14108].

## 4. Interpretability and Feature Attribution

SPAMs remain inherently interpretable despite modeling arbitrary higher-order feature interactions. Each polynomial term $\langle u_{li}, x \rangle^l$ can be directly inspected, and the model’s output decomposes additively across these terms. In practice, feature importances and attributions can be computed in closed form or by differentiation, providing clarity on how individual features and interaction groups influence predictions [2205.14108].

A human-subject evaluation (Prediction Task on Amazon Mechanical Turk) empirically confirmed improved interpretability over linear (0.67 mean user accuracy) and LIME explanations (0.65), with SPAM achieving 0.71 (one-sided $p<10^{-3}$) using the top $E=7$ most salient features [2205.14108].

## 5. Benchmark Performance and Empirical Scalability

On a wide range of tasks—including regression, binary and multi-class classification, and object detection—SPAM matches or outperforms prior interpretable models (linear, EBMs, NAMs) and aligns closely with deep neural networks and boosted ensembles (e.g., XGBoost). For example:
- SPAM-Linear (order 2) surpasses all interpretable baselines and matches XGBoost on tasks such as News20 and object detection.
- SPAM-Neural (order 2) matches or exceeds XGBoost for regression and binary classification.
- SPAM (order 3) closes the remaining gap to MLPs on tasks like CoverType [2205.14108].

Throughput measurements demonstrate that SPAM sustains tens of millions of examples/sec for datasets with up to 146,000 features, outperforming MLPs and exceeding Neural Additive Models (NAMs) by orders of magnitude in speed [2205.14108].

## 6. Extensions: Tensorization and Higher-Order Data

The concept of scalable polynomial additive modeling generalizes to tensor-valued inputs, as formalized in the Tensor Polynomial Additive Model (TPAM). Instead of flattening tensors and losing structural information, TPAMs employ hierarchical low-order symmetric tensor decompositions—first representing high-order parameter tensors via a sum of symmetric rank-1 tensors, then factorizing each core via a CP decomposition across modes [2406.02980]:

- T-TPAM maintains the full tensor basis, reducing parameter count to $O(R_k \prod_j I_j)$ for order $k$, where $I_j$ are input dimensions.
- V-TPAM further reduces complexity to $O(R_k \sum_n I_n)$ using vectorized CP decompositions.

This approach delivers up to $100\times$ parameter compression and enables application to image and multi-way data, with empirical improvements in both predictive accuracy (up to $+30\%$) and localization metrics (PI-CAM yields >80% saliency overlap with true object boxes) relative to traditional SPAMs and state-of-the-art CNN attribution methods [2406.02980].

## 7. Limitations, Open Problems, and Future Directions

While SPAMs achieve compelling empirical and theoretical properties, certain limitations persist:
- The worst-case tensor rank may still be $O(d^l)$, though empirical spectral decay mitigates this in practical datasets [2205.14108].
- SPAM models yield dense basis vectors $u_{li}$, complicating direct interpretation in high dimensions; group sparsity regularizers may address this at the cost of certain fidelity–sparsity trade-offs.
- Current SPAM and TPAM frameworks handle real-valued data; applications to discrete, structured, or graph domains remain active topics.
- Extensions beyond interpretability use cases include incorporating SPAM polynomial layers into language models, privacy-aware architectures, and other domains demanding high expressivity with transparent attribution.

Further research continues on both architectural innovations (e.g., structured sparsity, tensor networks), hardware optimization (distributed solvers), and expanding self-explanatory mechanisms (e.g., new activation mapping strategies for deep tensorized features) [2205.14108, 2406.02980].

Source: https://www.emergentmind.com/topics/scalable-polynomial-additive-models-spam