Scaling architecture ordering and ternary penalties beyond 60K parameters

Determine whether the architecture ordering and architecture-dependent ternary penalties observed at 60K parameters change at 1M or 10M parameters, thereby assessing whether the reported 944K-parameter reversal generalizes beyond the experimentally accessible scale.

Background

All experiments in the paper use models with approximately 60K parameters. The authors explicitly state that they cannot determine whether the relative ordering of the routed, transformer, and recurrent architectures—or the differences in their ternary penalties—persists at substantially larger scales such as 1M or 10M parameters.

This question is especially relevant because the audited prior work reported a reversal at 944K parameters, but reproducing that comparison with a comparable bytes-per-parameter budget would require roughly two orders of magnitude more compute than available on the authors’ laptop.

References

We cannot say whether the architecture ordering or the ternary penalties change at 1M or 10M, and we therefore do not address the 944K reversal that motivated the audit.

— Baseline Shape Decides the Verdict: A Controlled Re-Examination of Ternary Language Models at 60K Parameters  (2609.29397 - Veldanda, 24 Sep 2026) in Section 7, Limitations, subsection “One parameter count”