Risk control by input mass in wider networks

Characterize when the input mass on which units in a wider bias-free ReLU network respond differently controls the network’s risk, despite the presence of other units that may fit the target and prevent a clone pair from blocking learning.

Background

The paper proves the separation mechanism for a two-unit, bias-free ReLU network: source training makes the two units nearly proportional, while target adaptation requires sampling inputs on which their ReLU gates disagree. As the angle between the units shrinks, the probability of such inputs decreases, causing finite-batch SGD to remain near the proportional configuration for very long horizons.

The authors note that some ingredients extend to wider networks, but additional units may independently fit the target. Consequently, it is unresolved when the mass of inputs distinguishing a nearly cloned pair remains relevant for the overall risk in wider architectures.

References

In a wider network other units may fit the target, so a clone pair need not block learning; the open question is when the input mass on which units respond differently controls the risk.

— Rare Gate Disagreements Can Limit Plasticity: When Gradient Flow Mispredicts Finite-Batch SGD  (2610.11475 - Zhao et al., 8 Oct 2026) in Section 6, Conclusion and Future Work, paragraph “Scope and extensions”