Competition between task diversity and other data structures

Investigate the competition between task diversity and other structures in non-ergodic data, particularly generic non-ergodic hidden Markov models and heterogeneous corpora, to determine how heterogeneity affects data statistics, scaling laws, and the interaction between improvements to individual mixture components.

Background

The random topic model is presented as the simplest example of a non-ergodic hidden Markov model: documents are generated from a finite mixture of latent topics, and learning must balance adaptation to unseen topics against specialisation to the finite training mixture.

The paper leaves open whether the theoretical tools developed for this model extend to more general hidden Markov mixtures and how task diversity interacts with other forms of data structure. The authors specifically motivate solvable extensions as a way to understand heterogeneous corpora, their statistical properties and scaling laws, and the effects of improving individual components within a mixture.

References

The second is the competition between task diversity and other structures of the data. The tools developed here can address this problem in general non-ergodic hidden Markov models~\citep{riechers2025next}, of which the random topic model is the simplest instance. Such mixtures are natural models of large heterogeneous corpora. Solvable models like the one proposed here could help understand how heterogeneity contributes to data statistics and scaling laws~\citep{cagnetta2026deriving}, and how improvements on the individual components of a mixture interact.

— In-Context Learning Before Specialisation: Theory of Learning in a Solvable Topic Model  (2610.21901 - Cagnetta et al., 15 Oct 2026) in Section Conclusions