Scaling Behavior of HGF at Billion-Parameter Scale

Determine the scaling behavior of the Hybrid Gated Flow (HGF) architecture when applied to billion-parameter language models, assessing whether performance and stability characteristics observed at small scale persist at the 1B–7B+ parameter regime.

Background

The paper introduces Hybrid Gated Flow (HGF), which combines a 1.58-bit ternary backbone with a gated low-rank FP16 correction path to recover quality lost to extreme quantization. Results are demonstrated on TinyStories with a ~25M parameter model.

While preliminary larger-scale experiments are mentioned (1.2B, 3B, and 7B) with promising but non-final indications, the authors explicitly flag scaling to billion-parameter models as an open question, emphasizing the need to validate whether the observed quality recovery and stability hold at larger scales.