LIT-LVM: Structured Regularization for Linear Interactions

• LIT-LVM is a structured regularization method that leverages low-dimensional latent vectors to model high-dimensional pairwise feature interactions.
• It mitigates overfitting by imposing an approximate low-rank structure on the interaction coefficients, outperforming methods like elastic net and factorization machines.
• The interpretable latent embeddings facilitate exploratory data analysis and visualization in applications such as genomics and biomedical prediction.

LIT-LVM (Linear Interaction Terms with Latent Variable Model) is a structured regularization method for linear predictors containing interaction terms. This framework advances the modeling of pairwise feature interactions—crucial for capturing non-linearities in statistical and machine learning problems—by employing latent variable models to impose approximate low-dimensional structure on the interaction coefficient matrix. LIT-LVM thereby mitigates overfitting in settings where the number of interaction terms is large relative to the sample size, often outperforming established methods such as elastic net regularization and factorization machines, and providing interpretable low-dimensional representations for feature analysis.

1. Motivation and Conceptual Overview

LIT-LVM is motivated by the challenge of estimating interaction coefficients in linear predictors when feature dimensionality is high. Given $p$ input features, the inclusion of all possible pairwise interactions leads to a parameter space of size $\binom{p}{2}$ for the interaction coefficients $\theta_{jk}$, which quickly becomes much larger than the number of observations $n$ in standard data settings. Conventional regularization approaches, such as the lasso or elastic net, shrink or sparsify the parameter space but do not exploit any underlying structure among the $\theta_{jk}$.

The core hypothesis of LIT-LVM is that interaction coefficients possess an approximate low-dimensional structure: specifically, they can be described by assigning to each feature a $d$-dimensional latent vector (with $d \ll p$), so that $$\theta_{jk} \approx f(\boldsymbol{z}_j, \boldsymbol{z}_k)$$ for some symmetric function $f$. Model flexibility is retained by not enforcing strictly low-rank structure (as in matrix factorization approaches), but regularizing towards it, allowing for deviations attributable to noise, sparsity, or model mismatch.

This structured regularization provides enhanced generalization in high-dimensional interaction models and yields feature embeddings that can aid in interpretability and exploratory data analysis.

2. Structured Regularization Implementation

LIT-LVM explicitly encodes low-dimensional structure into the interaction coefficient matrix $\Theta = [\theta_{jk}]$ by two principal models:

• Low-rank latent variable model:

$$\theta_{jk} = \boldsymbol{z}_j^T \boldsymbol{z}_k + \epsilon_{jk}$$

where $\boldsymbol{z}_j \in \mathbb{R}^d$ is the latent vector for feature $j$, and $\epsilon_{jk}$ captures residuals not explained by the latent structure.

• Latent distance model:

$$\theta_{jk} = \alpha_0 - \|\boldsymbol{z}_j - \boldsymbol{z}_k\|^2 + \epsilon_{jk}$$

with an intercept $\alpha_0$ and pairwise distances providing a flexible parameterization of interactions.

The predictor for an input $\mathbf{x} \in \mathbb{R}^p$ is then: $$\hat{y} = f(\boldsymbol{\beta}^T \mathbf{x} + \mathbf{x}^T \Theta \mathbf{x})$$ where $f$ is a link function (identity for least squares, logit for logistic, etc.). The vectorized version $\tilde{\beta}$ includes all main effects and interaction parameters, and the augmented input $\tilde{x}$ includes original features and all unique interaction terms.

The learning objective for parameters $(\tilde{\beta}, \Theta, Z)$ is: $$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{pred}} + \lambda_r \mathcal{L}_{\text{reg}} + \lambda_l \mathcal{L}_{\text{lvm}}$$ where:

• $\mathcal{L}_{\text{pred}}$ is the standard prediction loss (e.g., squared loss, cross-entropy).
• $\mathcal{L}_{\text{reg}}$ is an elastic net penalty (sum of $\ell_1$ and $\ell_2$ norms on $\tilde{\beta}$).
• $\mathcal{L}_{\text{lvm}} = \|\Theta - F(Z)\|^2_F$ penalizes deviations of $\Theta$ from the latent variable model $F(Z)$.
• $\lambda_r, \lambda_l$ are tuning parameters.

Optimization is performed using Adam, with non-smooth penalties handled via proximal gradients.

The approach is compatible with both full and targeted interaction structures—for example, restricting $\Theta$ to biologically plausible interactions in genomics or clinical datasets.

3. Comparison with Elastic Net and Factorization Machines

LIT-LVM generalizes and surpasses existing interaction modeling strategies:

• Elastic Net with Interaction Terms: Shrinks and sparsifies parameters individually, but does not capture dependencies or redundancies among interaction coefficients and deteriorates when $p^2 \gg n$.
• Factorization Machines (FM): Restrict $\Theta$ to exactly low-rank ($\Theta = Z Z^T$). FM can underfit when the true interaction matrix deviates from strict low rank, and lacks flexibility for incorporating structured deviations or sparsity.
• Nuclear Norm Regularization: Imposes convex low-rank relaxations but with heavy computational burden ($O(p^3)$ per iteration).
• Structured Sparsity: Combines sparsity with low-rank assumptions, but still less adaptable if $\Theta$ only approximately exhibits low-rank structure.

Empirical results consistently show that LIT-LVM yields lower prediction error than both elastic net and FM, especially in regimes where the number of interaction terms approaches or exceeds the sample size.

4. Applications and Empirical Results

Simulation Experiments

• Linear and logistic regression simulations confirm that when the true interaction matrix is approximately low-dimensional (but not exactly low-rank), LIT-LVM achieves superior mean squared error or AUC compared to elastic net interactions and FM, particularly as $p^2/n$ grows.
• Phase transitions in simulation (where performance of unstructured models rapidly degrades) are effectively mitigated by LIT-LVM.

Real Data – OpenML Benchmarks

• Across 12 regression and 10 classification datasets, LIT-LVM is best or statistically equivalent to the best method in nearly all cases with no datasets where it is substantially subpar. Gains are pronounced in high-dimensional and moderate-sample regimes.

High-Dimensional Biomedical Prediction

• In survival analysis of kidney transplantation (Scientific Registry of Transplant Recipients), modeling donor-recipient HLA type interactions:
  • LIT-LVM provides the lowest integrated Brier score (risk prediction calibration) and highest discrimination (C-index), outperforming standard Cox elastic net, FM, PCA-based models, and Random Survival Forests.
  • LIT-LVM's latent vectors yield interpretable feature embeddings, where compatible HLA types are mapped to nearby locations—enabling post hoc visualization of compatibility relationships.

5. Interpretability and Visualization

The learned latent vectors ($\boldsymbol{z}_j$) from LIT-LVM succinctly encode the interaction landscape among features, facilitating:

• Visualization: Mapping features (e.g., HLA alleles) in low-dimensional space, illuminating biological or domain-specific compatibility structures.
• Exploratory Analysis: Cluster analysis, relationship inspection, or further downstream use.
• This approximates or extends conceptually the low-dimensional structure sought by matrix factorization but adds interpretability even when the ideal low-rank constraint does not hold.

6. Practical Implications and Future Directions

• Model Flexibility and Scalability: LIT-LVM bridges sparse (lasso) and low-rank (FM) regularizations, handling regimes where interaction structure is only approximate and sample sizes are limited.
• Extensibility: The framework is adaptable to higher-order tensors for modeling interactions beyond pairs, targeted/masked interactions, and a wide range of GLMs.
• Interpretability: The low-dimensional embeddings serve both to regularize estimation and to advance scientific understanding, especially in biological, medical, and social science contexts.
• Potential Extensions: Future directions include tensor factorization for third- or higher-order interactions, alternative nonlinear latent models, deeper integration with domain-specific structural constraints, and large-scale distributed implementations.

Summary Table: LIT-LVM and Comparator Methods

Aspect	LIT-LVM	Elastic Net	Factorization Machines (FM)
Structure/Regularizer	Approx. low-rank via latent vectors + penalty	Unstructured, sparsity	Exact low-rank (strict)
Flexibility	High; admits structured deviations	Moderate	Low (strict low-rank only)
Performance (hi-dim)	Superior $(p^2 \sim n)$	Degrades	Degrades
Interpretability	Latent feature embeddings	None	Feature embeddings
Computational Cost	$O(np^2)$/epoch	$O(np^2)$/epoch	$O(np^2)$/epoch

LIT-LVM advances linear modeling with interactions by combining structured regularization via latent variable modeling, robust performance in high dimensions, and interpretable feature embeddings, making it a versatile tool for modern statistical learning where complex feature relationships are central.