Interaction-Based Feature Learning

• Interaction-based feature learning is a set of methods that explicitly capture combinatorial relationships among variables, improving model insight and performance in complex datasets.
• These approaches integrate tensor operations, attention mechanisms, and statistical criteria to uncover non-linear, context-dependent interactions vital for tasks like CTR prediction and multi-modal analysis.
• Practical implementations balance efficiency and interpretability through techniques such as safe feature pruning and adaptive gating, ensuring scalable deployment in diverse domains.

Interaction-based feature learning encompasses a class of techniques designed to explicitly model, discover, and utilize feature interactions—combinatorial relationships between variables or feature sets—in statistical learning. Such methods aim to surpass the limitations of additive or independent-feature models by capturing dependencies, synergy, and context-dependent effects among features. They have become essential in domains with high-dimensional, sparse, or multi-modal data, particularly as deep learning and automated feature engineering (AutoFE) paradigms have matured. Modern interaction-based approaches span hand-crafted statistical tools, interpretable neural architectures, information-theoretic AutoFE systems, and interaction-oriented components for vision, language, and multi-modal data.

1. Core Principles of Interaction-Based Feature Learning

Feature interaction learning systematically incorporates relational patterns—second-order (pairwise), higher-order (multi-way), and cross-modal—into representational models. Classical approaches such as factorization machines (FM) apply simple inner or element-wise products to model pairwise feature effects, but these approaches are restricted to a single “semantic space” and miss context- or mode-specific relationships (Wu et al., 2020).

Contemporary deep models extend this principle by:

• Introducing parameterized geometric or bilinear operators (e.g., learnable tensors, attention, or multi-head transformations) to capture multiple semantic subspaces and interaction orders (Wu et al., 2020, Song et al., 2018, Zou et al., 2020).
• Employing explicit interaction mechanisms—such as tensor contractions, cross-attention, or statistical gating—to encode diverse, potentially non-linear dependencies between features or modalities.
• Integrating feature interaction selection, pruning, or gating for efficiency and interpretability, as in ℓ₁-regularized gating (Wu et al., 2020) or tree-structured screening (Nakagawa et al., 2015).

A defining attribute of interaction-based learning is that the interactions themselves—rather than only individual features—are modeled as first-class representational entities and can drive learning, inference, and downstream task performance.

2. Architectural Paradigms and Methodologies

2.1 Tensor and Attention-Based Neural Models

• Tensor-Based Interaction: Models such as TFNet introduce a rank-three “operating tensor” $T\in\mathbb{R}^{d\times m\times d}$, where each slice $T[k]$ captures a semantic mode; per-pair embeddings are dynamically gated via attention, yielding $s_{ij}=v_i^\top T v_j\in\mathbb{R}^m$ (Wu et al., 2020). Instead of a shared bilinear product, this architecture supports heterogeneous types of interaction, automatically selected and sparsified via learnable gates.
• Multi-Head Self-Attention: AutoInt employs multi-layer, multi-head self-attention on field-wise embeddings, such that higher-order feature combinations are formed as information propagates across layers; residual connections ensure lower-order and main-effect signals are retained (Song et al., 2018).
• 3-Dimensional Relation Tensor: FINN generalizes these principles with a learnable $W\in\mathbb{R}^{k\times k\times l}$, mapping the outer product of field embeddings onto an $l$-dimensional vector for each pair, then stacking non-linear layers to capture further cross-feature interactions (Zou et al., 2020).

2.2 Information-Theoretic and Statistical Criteria in AutoFE

• Interaction Information (II) Criterion: Methods such as IIFE identify pairs and operations that maximize interaction information $\tau_{ij} = I(F_i;F_j;Y)$—the three-way mutual information between feature $i$, feature $j$, and target $Y$—guiding both feature selection and operator construction in iterative search (Overman et al., 2024).
• Statistical Synergy Rewards: InHRecon embeds the Friedman–Popescu $H$-statistic, quantifying two-way interaction effect strength, directly into a hierarchical RL agent’s reward schedule, ensuring that newly constructed features are statistically synergistic with respect to the task (Azim et al., 2023).

2.3 Pruning and Interaction Selection

• Safe Feature Pruning (SFP): For high-order polynomial models (e.g., LASSO with all up-to-$k$-way interactions), tree-based SFP rules certify entire subtrees of the interaction lattice as inactive if an analytically computed upper bound is below threshold, drastically reducing computational cost by screening out $O(p^k)$ interactions before optimization (Nakagawa et al., 2015).

2.4 Interaction-Layered and Multi-Branch Architectures

• Cross-Generation and Cross-Modal Interactions: CFIL for kinship verification unifies local and non-local similarity-based weighting inside the model graph itself, embedding all similarity computations as differentiable operators within CNN branches, yielding representations directly optimized for interaction-specific signal (Dong et al., 2021).
• Multi-Modal and Cross-Branch Fusion: CFCI-Net utilizes Selective Complementary Feature Fusion (SCFF) modules and Modal Feature Compression Interaction (MFCI) transformers to adaptively fuse and compress redundant features across MRI modalities, using multi-head attention and soft complementary weighting (Chen et al., 20 Mar 2025).
• Vision-Language Interaction: CMI-MTL for medical visual QA interleaves fine-grained feature queries and text via Cross-Mamba blocks, ensuring that cross-modal interactions are dynamically constructed and utilized in answer generation (Jin et al., 3 Nov 2025).

3. Information-Theoretic and Statistical Viewpoints

Interaction-based feature learning is grounded in rigorous statistical interaction concepts:

• Interaction Information ($I(F_i;F_j;Y)$) as an objective—measuring synergistic predictive power (Overman et al., 2024).
• H-Statistic—quantifying the degree to which prediction variance is explained by interactions rather than additive effects (Azim et al., 2023).
• Conditional Mutual Information (CMI)—for robust evaluation with both discrete and continuous features, ensuring statistical rigor in the identification of non-additive dependencies (Overman et al., 2024).
• Feature creation, selection, and crossing operators are explicitly driven by these interaction-awareness metrics rather than merely overall accuracy or main effects.

This ensures interpretability (feature selections can be traced to strong statistical interactions), convergence rate (guided exploration), and robust out-of-distribution generalization (via selection of domain-invariant synergistic patterns (Wang et al., 2022)).

4. Applications and Empirical Impact

Interaction-based feature learning exhibits state-of-the-art performance—and empirical robustness—across diverse data modalities and task settings:

• CTR Prediction: Tensor-based and attention-based interaction models (TFNet, FINN, EulerNet, AutoInt) achieve up to +0.5 points in AUC over FM and DeepFM on Criteo, Avazu, and large-scale online A/B tests, demonstrating measurable lift in production KPIs such as click-through and ARPU (Wu et al., 2020, Tian et al., 2023, Zou et al., 2020, Song et al., 2018).
• Automated Feature Engineering: IIFE and InHRecon deliver +5–20% improvements in average downstream classification/regression metrics over expand-all and random-crossing baselines, and attain faster convergence through interaction-driven reward shaping (Azim et al., 2023, Overman et al., 2024).
• Kinship Verification and Cross-Modal Tasks: CFIL and CFCI-Net outperform non-interaction models in face kinship verification and multi-modal MRI segmentation tasks by 1–3 percentage points in standard metrics (Dice, accuracy) (Dong et al., 2021, Chen et al., 20 Mar 2025).
• Domain Generalization: Spatial interaction modules for fusing frequency-decomposed features (interaction gating) yield new state-of-the-art results on Digit-DG, Office-Home, and PACS (Wang et al., 2022).
• Robotic Manipulation: Interaction-aware contrastive, predictive, and detection objectives (e.g., frame-prediction plus object-box in unseen state) improve real-world and simulated manipulation performance by 6–26 points across standard suites (Zeng et al., 2024).
• Graph Representations: FI-GNN integrates personalized attention-weighted pairwise interactions to surpass non-interaction GNNs by 6–9 percentage points on tasks involving high-dimensional sparse node attributes (Ding et al., 2019).
• Theoretical Analysis: The interaction tensor formalism provides the basis for closed-form accuracy and agreement predictions, enabling the identification of when ensemble disagreement matches generalization error (GDE) and its failure modes (Jiang et al., 2023).

5. Interpretability, Efficiency, and Practical Considerations

Interaction-based models provide explicit mechanisms for interpretability and computational tractability:

• Interpretability: Attention and gating weights, H-statistics, and interaction information all serve as direct explanations of which feature relationships drive predictions or new feature creation (Song et al., 2018, Azim et al., 2023, Overman et al., 2024).
• Efficiency: Techniques such as adaptive gating (TFNet), tree-based SFP (safe feature pruning) (Nakagawa et al., 2015), and operator search guided by statistical interaction avoid exponential blowup in feature set size, enabling practical deployment on massive datasets and high-order interaction spaces (Wu et al., 2020, Azim et al., 2023).
• Fusion and Regularization: Various approaches explicitly regularize or compress interaction representations—dropout on interaction layers, ℓ₁-regularized control gates, MFCI transformer channel compression, and explicit selection stages to maintain feature set parsimony (Chen et al., 20 Mar 2025, Wu et al., 2020, Azim et al., 2023).
• Generalization: Interaction-centric procedures are robust across model classes and downstream learners, as performance gains persist when the target model is changed (RF, SVM, Ridge, LightGBM, neural net) (Azim et al., 2023, Overman et al., 2024).

6. Extensions, Open Challenges, and Theoretical Insights

While interaction-based feature learning has achieved demonstrable success across settings, open questions remain:

• Optimal Selection and Order Discovery: How to automatically determine maximal useful order without combinatorial expansion, especially in domains with latent or multi-modal structure (Tian et al., 2023, Azim et al., 2023).
• Heavy-Tailed and Rare-Feature Regimes: Theoretical analysis via the interaction tensor shows that calibration and generalization properties (e.g., the Generalization Disagreement Equality) depend critically on the heavy-tailed structure of feature frequencies and alignment of model capacity with data distribution (Jiang et al., 2023).
• Efficiency–Expressiveness Tradeoff: Model designs such as low-rank interaction tensors, sparse attention, and gating allow tuning resource usage vs expressiveness, but real-world deployment still mandates careful resource–accuracy balancing (Song et al., 2018, Wu et al., 2020, Tian et al., 2023).
• Generality across Modalities: The architecture patterns—statistical selection, tensor contraction, attention fusion—generalize from tabular to vision, language, multi-modal, and even reinforcement learning domains; interaction-centric perspectives are increasingly prevalent in graph, biomedicine, robotics, and beyond (Jin et al., 3 Nov 2025, Zeng et al., 2024).

Interaction-based feature learning is thus emerging as a unifying construct in modern machine learning, with systematic modeling of feature dependencies now driving not only empirical advances across traditional metrics, but also providing deeper understanding of the interplay between data, models, and generalization properties.