Identify the cause of HyGPT’s fluency–knowledge trade-off

Identify the cause of the simultaneous fluency improvement and knowledge degradation observed in HyGPT-10b after continued pretraining on Armenian, given that its training recipe and corpus are undisclosed.

Background

The paper observes that HyGPT-10b becomes more fluent while losing performance on knowledge-oriented Armenian evaluation tasks, reproducing the fluency–knowledge trade-off found in the controlled experiments with the released adaptation recipe.

Because HyGPT’s approximately 10-billion-token Armenian training corpus and training procedure have not been disclosed, the paper cannot determine whether the trade-off results from learning rate, data composition, repetition, replay, tokenizer choices, or another aspect of its adaptation process.

References

Its recipe is undisclosed, so we cannot isolate the cause, but the pattern of rising fluency and falling knowledge is exactly the signature above, and the unauditability is itself part of our argument for open recipes.

— From Zero to Hero: An Open LLM Ecosystem for Armenian  (2609.03350 - Arakelyan et al., 3 Sep 2026) in Section ‘Results’, paragraph ‘Existing Armenian models show the same trade at scale.’