The Rules-and-Facts Model for Simultaneous Generalization and Memorization in Neural Networks

Abstract: A key capability of modern neural networks is their capacity to simultaneously learn underlying rules and memorize specific facts or exceptions. Yet, theoretical understanding of this dual capability remains limited. We introduce the Rules-and-Facts (RAF) model, a minimal solvable setting that enables precise characterization of this phenomenon by bridging two classical lines of work in the statistical physics of learning: the teacher-student framework for generalization and Gardner-style capacity analysis for memorization. In the RAF model, a fraction $1 - \varepsilon$ of training labels is generated by a structured teacher rule, while a fraction $\varepsilon$ consists of unstructured facts with random labels. We characterize when the learner can simultaneously recover the underlying rule - allowing generalization to new data - and memorize the unstructured examples. Our results quantify how overparameterization enables the simultaneous realization of these two objectives: sufficient excess capacity supports memorization, while regularization and the choice of kernel or nonlinearity control the allocation of capacity between rule learning and memorization. The RAF model provides a theoretical foundation for understanding how modern neural networks can infer structure while storing rare or non-compressible information.

• The paper demonstrates that the RAF model unifies rule learning and exception memorization by analyzing kernel methods and capacity allocation.
• It reveals a trade-off in classical linear models and shows how overparameterized kernels can achieve low generalization and memorization errors simultaneously.
• The study quantifies the impact of kernel geometry and regularization, offering actionable insights for designing neural architectures that balance abstraction and factual recall.

Detailed Analysis of the Rules-and-Facts Model for Generalization and Memorization in Neural Networks

Theoretical Foundations

The "Rules-and-Facts (RAF) Model" (2603.25579) provides a rigorous framework that unifies two classical paradigms in statistical physics of learning: the teacher-student model (generalization) and Gardner-type capacity analysis (memorization). The RAF model considers a training data set consisting of two types of samples: those generated by a structured rule (teacher) and those with random, unstructured labels (facts/exceptions). The proportion of randomly labeled facts is controlled by the parameter $\varepsilon$, while the sample complexity $\alpha = n/d$ captures the ratio of examples to input dimension.

The learner is expected to achieve two goals concurrently: infer the underlying teacher rule (generalize) and recall the random labels (memorize) present in the training set. The model is analyzed in the high-dimensional limit where both input dimension and sample count diverge with fixed $\alpha$ and $\varepsilon$, leveraging replica techniques for analytical tractability.

The paper systematically characterizes the conditions under which simultaneous generalization and memorization are possible, focusing particularly on kernel regression, random features regression, and baseline linear models. Model performance is quantified via generalization error (test error on rule data) and memorization error (misclassification rate on fact data). The theory isolates how excess capacity, regularization, and kernel geometry allocate resources between rule learning and fact memorization.

Memorization–Generalization Trade-off in Classical and Overparameterized Regimes

A core result is that classical linear models (perceptron) exhibit a fundamental trade-off between generalization and memorization: at any regularization strength, improved memorization comes at the expense of generalization, and vice versa. This aligns with classical statistical learning theory and interpolation thresholds. No setting achieves both low generalization and memorization errors simultaneously.

Figure 1: Generalization–memorization trade-off curves induced by regularization for the RAF model, highlighting the incompatibility in a linear model.

In contrast, highly overparameterized models (random features, kernel methods) fundamentally alter this landscape. Overparameterized kernel models can allocate orthogonal components to rule learning and memorization, enabling simultaneous attainment of low errors. Excess capacity is not simply tolerated—it is essential for achieving both objectives. The memorization–generalization trade-off is orchestrated by regularization strength $\lambda$ and kernel geometry.

Figure 2: Random feature models show convergence to kernel limit as model width increases, illustrating enhanced simultaneous generalization and memorization.

Kernel Geometry and Capacity Allocation

The paper demonstrates that kernel geometry governs the allocation between rule and fact via two parameters, $\mu_1$ and $\mu_\star$, extracted from the Hermite expansion of the nonlinear activation or dot-product kernel. $\mu_1$ quantifies rule alignment (generalization), while $\mu_\star$ captures nonlinear capacity (memorization). The trade-off is succinctly parameterized by the angle $\gamma = \arctan(\mu_1/\mu_\star)$, which determines the optimal kernel for the RAF model.

Figure 3: Generalization and memorization dependence on kernel angle $\gamma$, showing the minimal generalization error is achieved at an angle dependent solely on $\varepsilon$.

Notably, for kernel ridge regression with square loss, there exists a unique kernel angle that achieves both perfect memorization (zero error) and minimal generalization error, analytically characterized as $\gamma^{\text{opt}}_{mem}(\varepsilon)$. For hinge loss, numerical optimization shows similar but not identical behavior. Extensive cross-validation confirms the angular dependence, and the theoretical predictions are supported by direct kernel experiments.

Figure 4: Memorization–generalization trade-off curves for kernels spanning representative angles, highlighting compatibility regimes and optimality.

Extensions to Real Data and Universality

The model is benchmarked against real data (CIFAR10-RAF), demonstrating that the qualitative phenomena found in the RAF model persist in structured, non-Gaussian inputs. Bandwidth tuning of RBF kernels on CIFAR10-RAF reveals distinct optimal points for factual recall versus rule generalization, mirroring theoretical expectations.

Figure 5: Theory-experiment comparison for RBF kernel ridge regression on CIFAR10-RAF, with generalization minima aligning at different bandwidths for memorization and rule extraction tasks.

Large Sample Complexity and Generalization Rates

The analysis exposes the limits of kernel methods: for fixed $\varepsilon > 0$, kernel ridge regression cannot achieve the Bayes-optimal $\alpha^{-1}$ generalization rate, instead decaying as $\alpha^{-1/2}$. This underscores an intrinsic limitation in the trade-off: simultaneous memorization and optimal generalization cannot be realized by kernel-based learners; they are suboptimal relative to the information-theoretic bound.

Figure 6: Generalization errors plotted as a function of $\alpha$, showing $\alpha^{-1/2}$ decay for kernel methods contrasted with $\alpha^{-1}$ for Bayes-optimal.

Numerical evidence for the hinge loss (SVM) further corroborates the universality of this suboptimal rate, regardless of kernel geometry.

Phase Diagram and Practical Implications

The RAF model enables the derivation of phase diagrams delineating regions where simultaneous generalization and memorization are feasible as a function of sample complexity and fraction of facts. This analytic tractability allows precise assessment of network capacity allocation.

Figure 7: Heatmap of Bayes-optimal test error across $(\alpha, \varepsilon)$, visualizing the phase space for generalization in the RAF model.

The model’s minimal structure abstracts the dual demands on modern neural networks—the necessity to infer broad abstractions and recall rare, non-compressible exceptions. Crucially, the framework gives a rigorous explanation for benign overfitting: overparameterization is a positive mechanism, not simply a byproduct of contemporary practice.

Implications and Future Directions

The theoretical framework provided by the RAF model fundamentally clarifies why overparameterized neural architectures such as LLMs can generalize and memorize concurrently. It rationalizes the roles of kernel geometry, regularization, and explicit capacity allocation in modern AI systems. Practically, it informs design choices for learning algorithms in settings where factual recall is not only compatible but essential with abstract rule learning, such as in biomedical knowledge bases, code synthesis, or language tasks.

The paper raises open questions regarding the limitations of kernel methods. Achieving information-theoretic generalization rates while retaining full memorization may require architectures with adaptive feature learning, such as trainable two-layer networks, transformers, or models with explicit modularization between rule and fact components.

The RAF model also provides fertile ground for connections to neuroscience (complementary learning systems), cognitive theories, and controlled studies of privacy, memorization localization, and benign overfitting in modern neural networks.

Conclusion

The RAF model establishes a rigorous, analytically tractable foundation for studying the interplay of generalization and memorization in high-dimensional learning. It quantifies the precise mechanisms—kernel geometry, regularization, overparameterization—that enable modern neural networks to perform both abstract inference and factual recall. This work advances both theoretical understanding and practical guidance for constructing AI systems that robustly balance the dual demands of abstraction and memory. Future research should explore richer architectures and adaptive feature learning to bridge the gap toward optimality, and further leverage the RAF framework to investigate localization, privacy, and modular specialization in memory systems.