Analytical characterization of optimal regularization for hinge-loss SVM on RAF data

Derive a closed-form analytical characterization of the optimal regularization parameter λ_opt that minimizes the generalization error for support vector machines with hinge loss trained on the Rules-and-Facts (RAF) data model, as a function of the sample complexity α, the fraction of facts ε, and the kernel geometry (μ1, μ⋆).

Background

For kernel ridge regression on RAF data, the paper derives an explicit formula for the optimal regularization λ_opt that minimizes the generalization error and characterizes when perfect memorization is compatible with optimal generalization. For hinge-loss SVMs, the authors rely on numerical cross-validation to select λ and observe improved generalization at the expense of memorization but lack an analytical solution.

An analytical λ_opt for the hinge loss would parallel the square-loss result, enabling precise trade-off characterizations between memorization and generalization for SVMs on RAF data.

References

For the hinge loss, we do not have analytical results when it comes to the optimal regularization, yet we can perform the cross-validation numerically.

— The Rules-and-Facts Model for Simultaneous Generalization and Memorization in Neural Networks  (2603.25579 - Farné et al., 26 Mar 2026) in Section 3.3 (Kernel geometry controls the rule–fact allocation), paragraph “Optimal generalization”

Several directions remain open. First, the present theory is restricted to balanced, linearly separable data; extending it to class-imbalanced designs \citep{Loffredo2024RestoringBalance} or to the nonseparable perceptron regime \citep{Franz2019CriticalJammed} is a natural next step. Second, the theoretical framework is based on a replica-symmetric characterization and does not provide a rigorous proof of the RS ansatz or its stability; establishing such results, or identifying conditions under which the RS solution can be justified rigorously, is an important theoretical problem. Third, the framework assumes a Gaussian-mixture model for the class-conditional distributions, and extending the theory to broader distributional classes would increase its applicability. Fourth, the current analysis is restricted to the $L_1$ penalty; extending the proportional-asymptotic theory to nonconvex penalties such as SCAD \citep{fan2001variable} or MCP could combine reduced regularization bias with formal inferential guarantees. Fifth, a theoretically justified, data-driven procedure for selecting the tuning parameter $\lambda$ remains an important problem.

— High-Dimensional Statistical Inference for Sparse Support Vector Machines  (2610.08345 - Zeng et al., 6 Oct 2026) in Conclusion, Section 5