Explain the unresolved performance gap of NormFormer on Tiny-ImageNet

Explain why NormFormer exhibits substantially lower clean top-1 accuracy than the other evaluated vision architectures on Tiny-ImageNet-200 under the shared training recipe.

Background

The Tiny-ImageNet-200 experiment evaluates several Transformer normalization and scale-control variants under a common recipe. NormFormer reaches markedly lower clean accuracy than most other architectures, while the paper does not determine whether the gap arises from optimization, hyperparameter transfer, implementation details, or an interaction between NormFormer and the shared recipe.

Resolving this issue is important for interpreting the cross-architecture comparison, because the experiment uses a matched recipe rather than architecture-specific retuning.

References

NormFormer reaches $43.2\%$ clean top-1 accuracy, about $12$ percentage points below most other baselines under this shared recipe. We report this result under the shared recipe, without architecture-specific retuning. The cause of the gap remains unresolved.

— Neural Dimensional Analysis: Foundations of Dimensionless and Scale-Consistent Computation  (2610.17969 - Kim et al., 13 Oct 2026) in Appendix, Section 'Full results', subsection 'Vision: robustness to photometric and input-unit shifts', paragraph 'A second dataset at larger scale' (App.\ref{sec:exp-vit})