Scaling of the filtering benefit beyond 350M parameters

Determine whether the validation-loss benefit of filtering through value gates continues to grow beyond the 350-million-parameter model scale.

Background

The paper finds that the benefit of noise filtering increases across the tested model sizes, from 10M through 350M parameters, while the benefit of abstention declines. The authors note that the observed trend may continue at larger scales, but their experiments do not test models beyond 350M parameters. This question concerns whether the empirically observed scaling behavior of value-path filtering persists in substantially larger LLMs.

References

The first is whether the growth of the filtering benefit continues past 350M, as the survey of pretrained models in \cref{sec:habitat} suggests.

— Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention  (2609.22005 - Wang, 18 Sep 2026) in Section 6, paragraph “Open questions” (Section 6.3, Discussion and Limitations)