Necessity of Width-Growing Constructions for Fixed Activations

Determine whether the constructive widths N+r for RMSNorm-based activation-normalized and preactivation-normalized networks, and N+2r for the corresponding LayerNorm-based networks, are necessary when the activation function is a standard fixed activation.

Background

The paper establishes approximation constructions whose widths grow from a target width N to N+r for RMSNorm and to N+2r for LayerNorm in the general continuous-activation settings. These are upper bounds that guarantee the approximation error tends to zero, but the paper does not prove matching lower bounds for these width-growing regimes. Its worst-case sharpness results apply instead to the fixed overheads N+1 for RMSNorm and N+2 for LayerNorm under the settings analyzed.

The unresolved issue is whether the additional replicated coordinates required by the general constructions are an artifact of the proof technique or are genuinely unavoidable for standard fixed activations such as commonly used neural-network activations. Establishing necessity would require deriving lower bounds that match the N+r and N+2r constructive widths, or otherwise characterizing the minimum width needed for approximation in these fixed-activation settings.