Law of Neural Interaction: Depth-Width Shape, Interaction Efficiency, and Generalization

Abstract: The guidance of scaling laws has increased the resource demands of modern LLMs, yet it remains questionable whether these models utilize resources effectively under a fixed budget. Previous research has proved superposition as a key contributor to loss. By leveraging the Neural Feature Ansatz, we extend superposition from parameter space to gradient space and define it as neural interaction. We find that under a fixed budget, good generalization is usually accompanied by efficient neural interactions, and the model can be placed in an efficient interaction interval by adjusting its depth-width ratio ($R_{D/W}$). In addition, as the budget scales up, the efficient interaction interval of the model remains relatively stable. By comparing existing small scale dense LLMs, we observe that models operating near this interval tend to perform better on the MMLU-Pro benchmark. Our findings reveal that the $R_{D/W}$ influences resource utilization efficiency and thereby affects generalization, providing insights into model shape initialization and the understanding of model generalization mechanisms. Code for Neural Interaction Law is available at: https://anonymous.4open.science/r/Neural_Interaction_Law-D788

• The paper shows that efficient gradient-mediated neural interaction, quantified by AOFE and AOFE-ratio, directly correlates with improved generalization performance.
• Systematic depth-width sweeps reveal that optimal architectural shape enhances reusable feature extraction while minimizing energetic costs.
• Empirical results across CNNs, GRUs, ViTs, and Tiny Transformers underscore the importance of interaction efficiency in effective model scaling.

Law of Neural Interaction: Shape, Efficiency, and Generalization in Neural Networks

Background and Motivation

Prevailing scaling laws have fundamentally shaped the construction of large neural models, positing that predictable improvements in generalization stem from increases in model size, data, and compute. However, the effective use of resources under fixed budgets has remained underexplored. Classical analyses (asymptotic statistics, geometry, optimization, feature learning) primarily focus on unbounded scaling, overlooking how neural architectures organize representational capacity when constrained. Building on the superposition perspective—which connects loss with representational feature interference—the paper "Law of Neural Interaction: Depth–Width Shape, Interaction Efficiency, and Generalization" (2605.27989) generalizes superposition beyond parameter space, introducing a formal framework for gradient-mediated neural interaction and its role in generalization efficiency.

Neural Interaction and Quantification

The paper leverages the Neural Feature Ansatz (NFA), which relates weight-space feature geometry to the Average Gradient Outer Product (AGOP). Through NFA, superposition is reframed as "neural interaction," characterized by off-diagonal structure in the AGOP. Two metrics quantify this:

• AGOP Off-diagonal Frobenius Energy (AOFE): Absolute strength of cross-directional gradient coupling.
• AOFE-ratio: Fraction of total sensitivity contributed by neural interaction, normalized against overall AGOP energy.

These metrics together define interaction efficiency: AOFE tracks the absolute energetic cost, and AOFE-ratio identifies the proportion of sensitivity that is interaction-driven.

Figure 1: Definition and mathematical structure of AGOP and neural interaction metrics.

Double Descent and Emergence of Benign Superposition

The double descent scenario demonstrates the nuanced interplay between memorization and feature learning. In highly overparameterized regimes, networks achieve low test loss absent efficient interaction (low AOFE-ratio, high diagonal AGOP structure). As sample size increases beyond interpolation threshold, the AOFE-ratio saturates and AOFE reduces, signifying efficient, reusable feature extraction—a regime denoted as benign superposition. Statistical correlation between AOFE and test loss confirms that gradient-mediated interaction, as conceptualized via NFA, directly measures generalization-relevant feature geometry.

Figure 2: AGOP structure, loss, and AOFE metrics across training regimes in double descent.

Depth-Width Shape Sweeps Across Architectures

Systematic sweeps over depth-to-width ratio ($R_{D/W}$) at fixed parameter budgets reveal that generalization is not solely dictated by scale, but by the architecture’s efficiency in organizing interactions. CNNs, GRUs, and ViTs empirically show strong negative correlation between AOFE-ratio and test loss; optimal configurations maximize AOFE-ratio while minimizing AOFE, indicating the model is efficiently converting parameters into reusable structure shared across inputs. Depth promotes compositionality; excessive depth incurs elevated AOFE without adequate interaction contribution, while excessive width yields weakly coupled features, limiting cross-input feature reuse.

Figure 3: Cross-network shape sweeps: test losses and AOFE-ratio across depth-width ratios.

Language Modeling and Stable Interaction Interval

Extending findings to LLMs, controlled sweeps of Tiny Transformer architectures across parameter budgets establish that optimal generalization emerges within a remarkably stable depth-width interval ($R_{D/W} \approx 0.023$–$0.047$), consistent across scales (excluding extreme underparameterization). Test loss correlates negatively with AOFE-ratio, and AOFE tracks absolute scale. This interval persists as budgets grow, indicating architectural stability in the efficient interaction regime.

External comparison across small dense LLMs (0.5B–9B parameters) demonstrates that models closer to this interaction-efficient interval achieve higher MMLU-Pro scores, particularly in mid-scale regimes (1–2.5B). The correlation weakens at very large scales, where post-training and data/control dominate.

Figure 4: Budget-wise shape sweeps in Tiny Transformers; optimal $R_{D/W}$ interval and AOFE-ratio correlations.

Theoretical and Practical Implications

The Law of Neural Interaction reframes fixed budget generalization as a resource organization problem: efficient models maximize the fraction of sensitivity carried by neural interaction (AOFE-ratio) while minimizing energetic cost (AOFE). This insight has architectural, practical, and theoretical significance:

• Model Design: Architectural hyperparameters (depth, width) should be initialized not based on empirical or hardware constraints but on interaction efficiency, organizing fixed resources into reusable feature structures.
• Scaling Laws: The classical scaling paradigm omits architectural shape; incorporating interaction efficiency enables more effective model scaling at fixed budgets.
• Transferability: The interaction-efficient interval is stable across tasks and architectures, but its numerical value depends on architecture class, overparameterization degree, and domain/task specifics.
• LLM Efficiency: Many current LLMs underutilize their parameter and data budgets; optimizing depth-width shape for interaction efficiency could yield significantly better performance per unit resource.

Limitations and Future Directions

The interval defining efficient $R_{D/W}$ was derived from byte-level TinyGPTs up to 10M parameters; direct transfer to multi-billion-parameter models requires further empirical validation. Benchmark correlations are confounded by training protocol differences, and efficient shape should be treated as an architectural covariate, not a performance predictor. Future research should track AOFE and AOFE-ratio during full-scale LLM training, refine theoretical optimality conditions, and investigate resource allocation strategies beyond parameter count.

Conclusion

The paper establishes neural interaction as a fundamental principle mediating generalization under fixed resource budgets. Efficient allocation of sensitivity to interaction-driven structure, regulated by depth-width shape, correlates strongly with generalization across architectures and tasks. This framework provides actionable guidance for model shape initialization and scaling strategies, suggesting a critical refinement to the conventional resource allocation paradigm in deep learning. Future studies will extend this law to larger models and diverse domains.