Modeling Nonlinear Feature Interactions with Product-Unit Residual Networks

Abstract: Understanding nonlinear feature interactions is crucial in science and engineering, yet standard multilayer perceptrons (MLPs) often capture such interactions only implicitly, leading to entangled representations that can impair robustness and interpretability. We investigate product-unit residual networks (PURe) that integrate multiplicative product units with residual connections to explicitly model cross-feature couplings while stabilizing optimization. We conduct a systematic evaluation on an interaction-driven synthetic benchmark and two real-world datasets, assessing predictive accuracy, robustness to Gaussian feature noise, and performance under limited training data, and we compare real- and complex-valued variants under a matched parameter budget. Beyond accuracy, SHapley Additive exPlanations (SHAP)-based interaction analyses show that PURe learns more concentrated and structurally coherent interaction patterns than MLP baselines. Overall, PURe achieves competitive or improved performance, better robustness and sample efficiency in low-data regimes, and enhanced interaction-level interpretability.

• The paper introduces PURe, integrating explicit multiplicative product units with residual connections to directly model nonlinear feature interactions.
• It demonstrates lower asymptotic MSE and higher robustness than traditional MLPs, particularly excelling in low-data and high-noise regimes.
• The architecture produces more interpretable, concentrated interaction maps via SHAP analysis, benefiting applications in material science and socio-economic forecasting.

Explicit Modeling of Nonlinear Feature Interactions with Product-Unit Residual Networks

Background and Motivation

The challenge of accurately modeling nonlinear interactions between features is fundamental in scientific and engineering regression applications. Classical multilayer perceptrons (MLPs), though versatile and theoretically capable of representing complex functions, tend to encode feature interactions only implicitly through additive compositionality, which can entangle dependencies and hamper both generalization and interpretability—particularly in low-data or noisy regimes. In response, this work evaluates product-unit residual networks (PURe): neural architectures integrating explicit multiplicative product units with residual connections to enable direct modeling of nonlinear feature couplings while enhancing optimization stability.

Theoretical and empirical evidence has suggested that product units naturally parameterize high-order multiplicative interactions. However, their deployment in deeper architectures has traditionally been impeded by numerical instability and optimization difficulties. The introduction of residual connections is hypothesized to mitigate these issues and provide more reliable convergence, especially when combined with product-unit nonlinearity in both real and complex parameterizations.

PURe Architecture

PURe blocks are based on the compositional stacking of product units, which compute each hidden dimension as an exponentiated log-linear combination of non-negative intermediates:

$$\mathbf{h}_\text{out} = \mathbf{h} + \mathrm{FC}_2\big(\mathrm{PU}( \mathrm{FC}_1(\mathbf{h}) )\big)$$

Here, non-negativity is enforced before applying the product-unit transformation, precluding log-domain instabilities. For signed or complex inputs, the PU transformation is extended to the complex domain, separately rectifying the real and imaginary components before applying the complex logarithm and exponent. Residual skip connections provide an identity path that regularizes gradient flow and stabilizes learning of the multiplicative branches.

For each benchmark, architectures are strictly capacity-matched: real-valued models use wider layers, and complex-valued models halve the width (since every complex parameter represents two degrees of freedom), ensuring that comparisons reflect modeling capability rather than raw parameter count.

Benchmark Evaluation and Training Dynamics

PURe is systematically evaluated on three datasets: the interaction-driven synthetic Friedman 1, the materials-focused Concrete Compressive Strength, and the large California Housing socio-economic dataset. Models are compared in both real- and complex-valued forms, with MLP and PURe as primary contenders.

Test loss trajectories across datasets reveal that PURe architectures achieve lower asymptotic mean squared error (MSE) than MLP baselines. While MLPs tend to converge faster in the earliest epochs, PURe—especially its complex-valued variant (CV-PURe)—yields superior final error, often after a more gradual, stable descent. This effect is most pronounced on datasets driven by strong interactions:

Figure 1: Training and validation loss curves (log$_{10}$ MSE) for Friedman 1 (top), Concrete (middle), and California Housing (bottom), demonstrating PURe variants’ superior convergence properties.

Under Gaussian input corruption, PURe variants, and particularly CV-PURe, evidence consistently higher robustness compared to standard MLPs. The improvement is observed as both a reduction in the absolute increase in MSE and a lower variance across training runs, indicating a stabilizing effect from both explicit multiplicative modeling and residualization. Notably, residual connections in a purely additive MLP (Res-MLP) do not improve performance and may slightly degrade it, highlighting that the principal benefit arises from the product-unit bias itself, with residuals serving primarily as a stabilizing adjunct.

Interpretability and Feature Interaction Structure

A central hypothesis driving PURe’s design is that explicit multiplicative parameterization should induce more concise and interpretable interaction maps. SHAP analysis on pairwise feature combinations reveals that PURe models, especially in the complex domain, recover interaction patterns more aligned with known underlying data-generating processes or domain-relevant couplings.

Figure 2: SHAP-based feature interaction maps across models and datasets; complex-valued product-unit residual networks yield the most concentrated and structurally coherent interactions.

On Friedman 1, PURe constrains most interaction strength to the dominant true pairwise coupling (e.g., $x_0 \times x_1$), whereas MLP and even real-valued PURe variants are more diffusive. This tendency generalizes to real datasets, where CV-PURe emphatically isolates a smaller set of prominent interactions, congruent with scientific understanding of underlying processes (e.g., certain concrete mixture components) and yielding lower entropy and higher concentration in interaction quantification metrics.

Sample Efficiency

The sample efficiency of PURe is evaluated via subsampling experiments on the Concrete dataset. Across all data regime granularities (from 5% to 80% of available samples), PURe variants demonstrate consistently lower MSE compared to their MLP counterparts. The performance gap is most significant in low-data settings, indicating that the explicit interaction-aligned inductive bias provides superior generalization when supervision is sparse.

Figure 3: Test MSE versus training fraction on Concrete; PURe models, especially CV-PURe, maintain significant advantages in low-data regimes.

The complex-valued PURe architecture not only achieves the best mean test error at every training fraction but also shows reduced variability across runs in the smallest-data settings, reflecting both more stable optimization and a tighter hypothesis class aligned with the true generative structure.

Ablation and Architectural Insights

Ablation experiments clarify the individual roles of multiplicative product units and residual connections. Pure product-unit architectures (without residual connections) provide the major performance gains, while residualization further reduces training variability and stabilizes convergence. Conversely, introducing residuals into a standard MLP is ineffectual or mildly detrimental, reaffirming that the inductive bias aligning with cross-feature interactions is the underlying lever for generalization and interpretability improvements.

Theoretical and Practical Implications

These findings solidify several claims:

• PURe achieves lower test MSE and improved robustness to input noise compared to MLP baselines, with the greatest benefits in settings where nonlinear interactions drive the target variable.
• Complex-valued PURe models further improve expressiveness and sample efficiency, albeit with marginally increased variability due to a richer hypothesis space.
• Explicit multiplicative architectures yield more interpretable, compact, and functionally coherent interaction maps, as measured by SHAP-based concentration metrics.

On a practical level, PURe is a compelling candidate for regression problems characterized by strong domain-driven feature interactions—examples include physical system modeling, material science, and spatial socio-economic forecasting. Its ability to stabilize learning and enhance interpretability under sample and noise constraints extends its utility to real-world scientific and engineering deployments.

Future Directions

Logical avenues for future research include: extending interaction analysis with higher-order metrics for more granular validation; addressing numerical stabilization challenges for deep complex product units; evaluating robustness under distribution shifts and missing features; and exploring hybrid models integrating product-unit modules within broader architectures such as transformers for scientific tabular data or multi-modal regression.

Conclusion

Product-unit residual networks provide an explicit, interaction-aligned inductive bias that improves generalization, robustness, interpretability, and sample efficiency in regression problems governed by nonlinear feature dependencies. The complex-valued variant, in particular, enables richer modeling of dependencies with magnitude and phase, and demonstrates further gains in both accuracy and concentration of learned interactions. These results support broader adoption of explicit multiplicative structures in both scientific machine learning and applied data-driven modeling, especially in low-data or high-noise environments.