Semantic Initialization & Planning Losses

• The paper establishes a theoretical framework that decomposes network outputs to quantify how nonzero initialization bias increases generalization error.
• It shows that in the NTK regime, the choice of loss function affects early training dynamics but converges to a solution determined by kernel-norm proximity to initialization.
• Antisymmetrical initialization (ASI) is introduced to eliminate residual bias, accelerate convergence, and enhance performance on both synthetic and real-world datasets.

Semantic initialization and planning losses constitute a fundamental theme in modern machine learning, especially in the context of deep neural network generalization, structured learning, neurosymbolic systems, and robotics. The concepts bridge initialization strategies—how models are seeded in function or parameter space—and loss functions designed to encourage semantically meaningful predictions or plans. These techniques are rigorously explored in settings ranging from neural tangent kernel (NTK) analysis to path planning in robotics and structured symbolic prediction, shedding light on how initialization and the design of auxiliary/planning losses interact to shape both optimization trajectories and generalization behavior.

1. Initialization-Induced Generalization Error in Deep Neural Networks

In the NTK regime, the impact of initialization on the generalization properties of deep neural networks is quantitatively characterized. For a regression problem with training set (X, Y), the output after training a network with initial output function $h_{ini}$ satisfies the identity:

$$h_k(\cdot; h_{ini}, X, Y) = h_k(\cdot; 0, X, Y) + h_{ini} - h_k(\cdot; 0, X, h_{ini}(X))$$

where $h_k(\cdot; 0, X, Y)$ is the solution obtained when initializing the network output at zero. This decomposition makes explicit that if the initialization $h_{ini} \ne 0$, the network output inherits a linear residual term derived from the unconstrained components of $h_{ini}$, i.e., those orthogonal to the subspace constrained by the training data. As a result, the expected generalization error for random, unbiased initialization is given by:

$$\mathbb{E}[R_S(h_k(\cdot; h_{ini}, X, f(X)), f)] = R_S(h_k(\cdot; 0, X, f(X)), f) + \mathbb{E}[R_S(h_k(\cdot; 0, X, h_{ini}(X)), h_{ini})]$$

The additional term represents an artificial prior imposed by the initialization which, unless fully constrained by data (i.e., in the infinite data regime), leads to increased test error—particularly from high-frequency components misaligned with the underlying function.

2. Role of Loss Functions in Optimization and Kernel Norm Minimization

The optimization in the NTK regime, for a wide class of smooth and strongly convex loss functions, converges to the same global minimum—specifically, the function closest in the RKHS (reproducing kernel Hilbert space) norm to the initialization that exactly fits the training data:

$$\underset{h: h(X)=Y}{\text{argmin}}\,\, \|h - h_{ini}\|_{K}$$

Here, $K$ denotes the neural tangent kernel of the network. While different loss functions dictate the path or speed of convergence (e.g., $L_2$ vs $L_4$ norm), the kernel-norm proximity to the initial function fully determines the final solution. Loss selection in this regime therefore has no asymptotic influence on expressive capacity or generalization, but may affect early optimization behavior or ease of training.

3. Antisymmetrical Initialization (ASI): Eliminating Initialization-Induced Error

To eliminate the additional generalization error term from nonzero $h_{ini}$, the antisymmetrical initialization (ASI) trick is introduced. ASI constructs the initial output to be zero everywhere, while preserving the network’s tangent kernel:

$$h_{\text{ASI}}(x, t) = \frac{\sqrt{2}}{2} h^{(L)}(x, t) - \frac{\sqrt{2}}{2} h^{(L)}(x, t)'$$

where $h^{(L)}(x, t)'$ is a neuronwise twin with identical input weights and anti-correlated output weights, and identical initialization $h^{(L)}(x, 0)' = h^{(L)}(x, 0)$. This enforces $h_{\text{ASI}}(x, 0) = 0$ for all $x$, yet the kernel $K$ remains identical to the original. Unlike naïve doubling, which can distort the NTK, this mechanism leaves the function space unchanged but advances training convergence and reduces residual bias.

4. Experimental Findings and Theoretical Validation

The theoretical predictions are substantiated by experiments on synthetic (e.g., learning $\sin(4x)$) and real datasets (e.g., Boston Housing regression, MNIST classification). Key observations include:

• With random $h_{ini}$, test-time error decomposes as predicted, quantifying the influence of the inherited bias.
• Alternative loss functions in the global convex class converge to the same final solution, confirming the minimization is kernel-norm dictated, not loss dictated.
• Applying the ASI trick reliably suppresses oscillatory residual error and accelerates early training, raising initial accuracy (e.g., on MNIST above 90% after one epoch) even when the NTK regime assumptions do not strictly hold.

5. Quantification and Decomposition of Generalization Error

In the kernel-norm analytical framework, the total mean squared error is expressed as:

$$R_S = \|h_k(\cdot; 0, X, f(X)) - f\|_{L^2(\Omega)}^2 + \mathbb{E}[\|\Delta\|_{L^2(\Omega)}^2]$$

with the residual $\Delta(x) = h_{ini}(x) - h_k(x; 0, X, h_{ini}(X))$. This identifies a distinct, quantifiable source of generalization error—separate from variance or bias traditionally considered—arising solely from the unconstrained component of $h_{ini}$ orthogonal to the training data. Increasing dataset size incrementally reduces this error, but ASI can eliminate it entirely by construction.

6. Implications for Robustness, Regularization, and Prior Design

These findings emphasize the criticality of carefully managing the initialization in overparameterized and kernel-regime neural networks. Arbitrary or careless seeding acts as a strong prior that may degrade performance, especially when training data are limited or when high-frequency components are unconstrained. This framework highlights that “good” prior selection, including the possibility of deliberately imposing structured initializations (through ASI or analogous mechanisms), is as central to deep learning generalization as loss choice or architecture tuning. The demonstrated invariance of global optimization to loss selection also encourages exploration of training-speed-optimized losses—so long as kernel-norm minimization is preserved.

Aspect	Standard Initialization	ASI Trick
Initial bias term	Nonzero (random $h_{ini}$)	Zero everywhere
Final output bias	Residual inherited from $h_{ini}$	None
NTK preservation	Yes	Yes
Generalization improvement	No	Yes (especially with limited data)

7. Perspective and Broader Relevance

The explicit, quantitative connection between semantic initialization, planning losses, and generalization provides a principled lens through which to design and debug deep learning pipelines. By delineating how initial output structure persists through training and how the selection of loss function in the NTK regime shapes (or fails to shape) the final function, this body of work clarifies the conditions under which initialization-induced priors must be neutralized or may be purposefully exploited. These insights underlie advances not only in regression and classification, but may generalize to structured prediction, transfer learning, and deep kernel methods, framing a robust research direction toward systematic prior and loss design in data-limited regimes.