Ghosts of Softmax: Complex Singularities That Limit Safe Step Sizes in Cross-Entropy

Abstract: Optimization analyses for cross-entropy training rely on local Taylor models of the loss to predict whether a proposed step will decrease the objective. These surrogates are reliable only inside the Taylor convergence radius of the true loss along the update direction. That radius is set not by real-line curvature alone but by the nearest complex singularity. For cross-entropy, the softmax partition function $F=\sum_j \exp(z_j)$ has complex zeros -- ``ghosts of softmax'' -- that induce logarithmic singularities in the loss and cap this radius. To make this geometry usable, we derive closed-form expressions under logit linearization along the proposed update direction. In the binary case, the exact radius is $ρ^*=\sqrt{δ^2+ π^2}/Δ_a$. In the multiclass case, we obtain the lower bound $ρ_a=π/Δ_a$, where $Δ_a=\max_k a_k-\min_k a_k$ is the spread of directional logit derivatives $a_k=\nabla z_k\cdot v$. This bound costs one Jacobian-vector product and reveals what makes a step fragile: samples that are both near a decision flip and highly sensitive to the proposed direction tighten the radius. The normalized step size $r=τ/ρ_a$ separates safe from dangerous updates. Across six tested architectures and multiple step directions, no model fails for $r<1$, yet collapse appears once $r\ge 1$. Temperature scaling confirms the mechanism: normalizing by $ρ_a$ shrinks the onset-threshold spread from standard deviation $0.992$ to $0.164$. A controller that enforces $τ\leρ_a$ survives learning-rate spikes up to $10{,} 000\times$ in our tests, where gradient clipping still collapses. Together, these results identify a geometric constraint on cross-entropy optimization that operates through Taylor convergence rather than Hessian curvature.

• The paper identifies complex singularities in the softmax function (ghosts) that fundamentally limit safe step sizes in cross-entropy optimization.
• It demonstrates that Taylor expansions are valid only within a convergence radius determined by the nearest complex singularity, not by real curvature.
• Empirical results show that normalizing step sizes by a bound (ρₐ) enhances training stability across architectures and mitigates abrupt failures.

Geometric Constraints on Cross-Entropy Optimization: Complex Singularities and Taylor Convergence Radius

Introduction

The paper "Ghosts of Softmax: Complex Singularities That Limit Safe Step Sizes in Cross-Entropy" (2603.13552) investigates a fundamental constraint governing step size selection for neural network training with the cross-entropy loss. The analysis centers on the analyticity of the loss function relative to parameter updates and demonstrates that local Taylor expansions—used to reason about step safety—are valid only within the radius determined by the nearest complex singularity, not by curvature on the real line. For cross-entropy, these singularities arise from zeros in the complex plane of the softmax partition function, termed "ghosts of softmax", which create logarithmic branch points and sharply delineate the domain in which derivative-based surrogate models accurately track the true loss.

Analytic Structure and Taylor Convergence Radius

Traditional optimization theory relies on $L$-smoothness (curvature-based) bounds for learning rate safety. However, the paper establishes that local Taylor surrogate models are fundamentally limited by the convergence radius determined by the proximity of complex singularities, not real curvature. In cross-entropy, the critical analytic structure emerges from the softmax partition function $F(\tau) = \sum_k \exp(z_k(\tau))$. This function inevitably possesses complex zeros, creating branch points (ghosts) for $\log F$, and thus restricts Taylor-series validity to within a radius $\rho^*$ dictated by these zeros. Beyond $\rho^*$, polynomial approximations may diverge or exhibit unphysical behavior, rendering descent guarantees based on local models unreliable.

For binary classification under logit linearization, the exact radius is

$\rho^* = \frac{\sqrt{\delta^2 + \pi^2}}{\Delta_a}$

where $\delta$ is the logit gap, and $\Delta_a$ is the spread of directional logit derivatives. In practice, for multiclass and batch settings, a conservative and tractable lower bound is given by

$\rho_a = \frac{\pi}{\Delta_a}$

where $\Delta_a = \max_k a_k - \min_k a_k$ and $a_k$ is the directional derivative, computable via a single Jacobian-vector product (JVP).

Implications for Learning Rate and Optimization

The normalized step size $r = \tau/\rho_a$ provides a principled, geometry-aware scale for distinguishing safe from hazardous updates. Empirical results show that, across various architectures and optimizer directions, $r < 1$ reliably ensures stability, whereas $r \geq 1$ is associated with collapse. Notably, conventional methods like gradient clipping and learning rate schedules do not enforce this geometric constraint and may fail abruptly when $\rho_a$ contracts during training. The authors demonstrate that a controller enforcing $\tau \leq \rho_a$ is robust to extreme learning-rate spikes (up to $10^4\times$ tested) and in practical settings (e.g., ResNet-18/CIFAR-10) attains high accuracy without hand-designed learning-rate schedules ($85.3\%$, compared to $82.6\%$ for best fixed rate).

Temperature scaling experiments confirm the theoretical mechanism: normalizing step sizes by $\rho_a$ reduces the spread in collapse-onset thresholds by a factor of $6\times$, supporting $r$ as an architecture- and schedule-independent coordinate.

Theoretical and Practical Impact

The geometric constraint described operates independently from real-line curvature, challenging the sufficiency of Hessian-based or $L$-smoothness descent lemmas. As training progresses and predictions sharpen, $\Delta_a$ increases, causing $\rho_a$ to decrease and reducing permissible step sizes even when curvature-based limits suggest larger steps. This manifests in late-training fragility and loss spikes observed in large-scale models. The analytic viewpoint also elucidates why curvature vanishes exponentially for confident predictions, while the convergence radius contracts only algebraically.

The paper provides a framework for:

• Diagnosing instability: computing $r = \tau/\rho_a$ at divergence events reveals step-size violations dictated by complex analytic structure.
• Monitoring training: tracking $\rho_a$ signals looming hazards, enabling preemptive learning-rate adaptation.
• Adaptive control: implementing a controller based on $\rho_a$ reduces step-size sensitivity and outperforms gradient clipping under extreme perturbations.

Extensions and Activation Singularities

The methodology extends to activation functions, showing that analytic structure can impose additional constraints through their own complex singularities. Activation functions like ReLU introduce real-axis kinks (not analytic), potentially tightening the convergence radius further. The paper suggests radius-friendly activation designs—such as RIA (Rectified Integral Activation) and GaussGLU—that avoid finite singularities. Analytic normalization layers derived from the Weierstrass transform are also proposed.

Empirical Validation

Controlled experiments across small and moderate neural architectures validate the theoretical findings:

• Learning rate spike tests confirm $\rho_a$-based controller survival compared to Adam and gradient clipping under adversarial conditions.
• Cross-architecture sweeps (MLP, CNN, Transformer variants) show no instability for $r < 1$, with failures tightly correlated to step size exceeding $\rho_a$.
• Random direction tests underline the direction-independence of $r$.
• Temperature scaling collapses instability thresholds into a universal scale via $\rho_a$ normalization.
• Realistic training on ResNet-18/CIFAR-10 demonstrates organic tracking of $r$ with natural instability corresponding to violations of the bound.

Limitations

While successful in controlled and medium-scale settings, the approach remains conservative—models can sometimes survive $r > 1$ due to confidence-margin slack (exact radius $\rho^*$ exceeding $\rho_a$). The bound is based on logit linearization; real networks may have higher-order nonlinearities, though empirical validation shows reliability. Large-scale and production-grade testing is an essential next step. Overhead for computing the bound is moderate (up to $129\%$ on ResNet-18/CIFAR-10), but scalable estimation is feasible.

Conclusion

The central contribution is the identification of a geometric, optimizer-agnostic constraint on step sizes in cross-entropy optimization, rooted in the analyticity and complex singularity structure of the softmax partition function. The tractable bound $\rho_a = \pi / \Delta_a$ allows prediction and prevention of instabilities that evade curvature-based analysis, yielding both diagnostic and actionable control over training dynamics. The framework opens avenues for future work in large-scale validation, integration with adaptive optimizers, and radius-conscious architectural design.