Extending NESS to Bias and BatchNorm Layers

Extend NESS (Null-space Estimated from Small Singular values) to incorporate bias parameters and BatchNorm layers by designing appropriate parameterizations and constraints that preserve the null-space update property and maintain model capacity, enabling the method to address components beyond linear and convolutional layers.

Background

The current implementation of NESS applies only to linear and convolutional layers because these components dominate network behavior. Biases and BatchNorm layers were left unchanged, which may reduce model capacity and cause slight performance losses.

The authors suggest that adapting NESS to these components likely requires different techniques, motivating the need for a tailored extension that preserves the method’s stability–plasticity trade-off while covering all trainable parts of modern architectures.