Sampling behavior for smooth activations

Characterize the sampling behavior governing disagreement or transition-layer events for smooth neural-network activations in the analogue of the two-unit plasticity model.

Background

For ReLU activations, the distinction between two nearly aligned units is localized to a wedge where their gates disagree, and the paper bounds the cumulative probability of sampling that wedge. Smooth activations replace the sharp wedge by a transition layer rather than an exact gate boundary.

The authors explicitly state that the relevant sampling behavior for this transition layer is unresolved, leaving open whether an analogous finite-batch SGD obstruction and disagreement-budget analysis can be established for smooth activations.

References

For smooth activations the wedge becomes a transition layer whose sampling is open.

— Rare Gate Disagreements Can Limit Plasticity: When Gradient Flow Mispredicts Finite-Batch SGD  (2610.11475 - Zhao et al., 8 Oct 2026) in Section 6, Conclusion and Future Work, paragraph “Scope and extensions”