Random Hierarchy Model

• Random Hierarchy Model is a synthetic hierarchical process defined by recursive grammar rules that expand class labels into multi-level feature compositions.
• It provides a controlled testbed for deep network learnability, showing that deep architectures overcome the curse of dimensionality through invariant internal representations.
• The model extends to sparse variants and other domains, linking applications in grammatical composition, network inference, and quantum walk dynamics.

The term Random Hierarchy Model is not field-invariant. In recent machine-learning literature, it denotes a synthetic hierarchical data-generating process designed to study how deep networks learn compositional structure (Cagnetta et al., 2023). In other literatures, closely related usages denote recursive random graph hierarchies, exchangeable random hierarchies represented by sampling from real trees, hierarchical dependence structures on dyads, or random hierarchies of barriers in quantum walks (Paluch et al., 2015, Forman et al., 2011, Sadeghi et al., 2016, Sharma et al., 2022). The most developed contemporary usage is the grammar-based model for hierarchical classification, where a class label generates higher-level features, each feature generates lower-level features, and the recursive process continues until the leaves form the input (Cagnetta et al., 2023).

1. Grammar-based Random Hierarchy Model

In the machine-learning usage, the Random Hierarchy Model (RHM) is a family of synthetic tasks inspired by the hierarchical structure of language and images (Cagnetta et al., 2023). The model is a classification task where each class corresponds to a group of high-level features, chosen among several equivalent groups associated with the same class. In turn, each feature corresponds to a group of sub-features chosen among several equivalent ones and so on, following a hierarchy of composition rules.

At the basic level, there is a set of class labels

$\mathcal{C}=\{1,\dots,n_c\},$

and there are $L$ levels of feature vocabularies,

$$\mathcal{V}_\ell \equiv \{a_1^\ell,\dots,a_{v_\ell}^\ell\}.$$

A label $\alpha\in\mathcal{C}$ generates a set of higher-level features, each higher-level feature generates several lower-level features, and this recursive process continues until one reaches the input level. The recursive rules are written as

$$\alpha \mapsto \mu^{(L)}_1,\dots,\mu^{(L)}_s,\qquad \mu^{(L)}_i\in \mathcal{V}_L,$$

and

$$\mu^{(\ell)} \mapsto \mu^{(\ell-1)}_{1},\dots,\mu^{(\ell-1)}_{s}, \qquad \mu^{(\ell)}\in\mathcal{V}_\ell,\ \mu^{(\ell-1)}_i\in\mathcal{V}_{\ell-1},$$

for $\ell=L,\dots,1$ (Cagnetta et al., 2023).

Because each feature expands into $s$ subfeatures at each level, the effective input dimension is

$d=s^L.$

The number of distinct data points per class grows exponentially in depth, and the model therefore serves as a controlled testbed for the curse of dimensionality (Cagnetta et al., 2023). A key feature is the presence of synonyms: the same higher-level feature can be realized by multiple lower-level compositions. At each level, a single feature has $m$ synonymous children. The model therefore does not define one unique decomposition from label to input, but rather a family of equivalent decompositions (Cagnetta et al., 2023).

A useful equivalent description is as an $L$0-level context-free grammar, or as a rooted $L$1-ary tree. In the 2026 formalization, an $L$2-level RHM instance with branching factor $L$3 is

$L$4

where $L$5 is the vocabulary at level $L$6, $L$7 is the label distribution over $L$8, and

$L$9

gives the conditional distribution of a level-$$\mathcal{V}_\ell \equiv \{a_1^\ell,\dots,a_{v_\ell}^\ell\}.$$0 patch given a level-$$\mathcal{V}_\ell \equiv \{a_1^\ell,\dots,a_{v_\ell}^\ell\}.$$1 token (Ren et al., 27 Jan 2026). In the CFG-induced version, each token has a finite set of production rules and the paper focuses on $$\mathcal{V}_\ell \equiv \{a_1^\ell,\dots,a_{v_\ell}^\ell\}.$$2-uniform RHMs, where all levels have vocabulary size $$\mathcal{V}_\ell \equiv \{a_1^\ell,\dots,a_{v_\ell}^\ell\}.$$3 and each symbol has exactly $$\mathcal{V}_\ell \equiv \{a_1^\ell,\dots,a_{v_\ell}^\ell\}.$$4 production rules (Ren et al., 27 Jan 2026).

2. Correlations, invariances, and empirical learnability

The random composition rules are chosen uniformly at random among valid assignments of $$\mathcal{V}_\ell \equiv \{a_1^\ell,\dots,a_{v_\ell}^\ell\}.$$5 $$\mathcal{V}_\ell \equiv \{a_1^\ell,\dots,a_{v_\ell}^\ell\}.$$6-tuples of lower-level features to each higher-level feature (Cagnetta et al., 2023). This randomness induces nontrivial correlations between low-level features and class labels. Although the rules are random, they are not arbitrary noise: certain $$\mathcal{V}_\ell \equiv \{a_1^\ell,\dots,a_{v_\ell}^\ell\}.$$7-tuples are statistically predictive of particular classes. The 2023 study identifies these correlations as the crucial source of learnability (Cagnetta et al., 2023).

The central empirical finding is that shallow networks are cursed by dimensionality, while deep networks can learn with polynomial sample complexity (Cagnetta et al., 2023). For deep networks with depth larger than the hierarchy depth $$\mathcal{V}_\ell \equiv \{a_1^\ell,\dots,a_{v_\ell}^\ell\}.$$8, the test error shows a sharp transition as the training set size $$\mathcal{V}_\ell \equiv \{a_1^\ell,\dots,a_{v_\ell}^\ell\}.$$9 increases, and the paper defines the sample complexity $\alpha\in\mathcal{C}$0 as the smallest $\alpha\in\mathcal{C}$1 such that the test error falls below $\alpha\in\mathcal{C}$2. Empirically,

$\alpha\in\mathcal{C}$3

Since $\alpha\in\mathcal{C}$4, this is polynomial in the input dimension rather than exponential (Cagnetta et al., 2023).

The same work reports that deep networks learn the task by developing internal representations invariant to exchanging equivalent groups (Cagnetta et al., 2023). To quantify this, it introduces the synonymic sensitivity measure $\alpha\in\mathcal{C}$5, where $\alpha\in\mathcal{C}$6 replaces a level-$\alpha\in\mathcal{C}$7 tuple by one of its synonymous alternatives and $\alpha\in\mathcal{C}$8 is the representation at layer $\alpha\in\mathcal{C}$9. The main empirical result is that the hidden representations become invariant at around the same training set size $$\alpha \mapsto \mu^{(L)}_1,\dots,\mu^{(L)}_s,\qquad \mu^{(L)}_i\in \mathcal{V}_L,$$0 at which test error drops, and invariance to level-$$\alpha \mapsto \mu^{(L)}_1,\dots,\mu^{(L)}_s,\qquad \mu^{(L)}_i\in \mathcal{V}_L,$$1 synonym exchange appears at layer $$\alpha \mapsto \mu^{(L)}_1,\dots,\mu^{(L)}_s,\qquad \mu^{(L)}_i\in \mathcal{V}_L,$$2 (Cagnetta et al., 2023). This suggests a layerwise progressive collapse of synonymous low-level descriptions into invariant higher-level representations.

The same paper also explains why the scale $$\alpha \mapsto \mu^{(L)}_1,\dots,\mu^{(L)}_s,\qquad \mu^{(L)}_i\in \mathcal{V}_L,$$3 appears. For a patch $$\alpha \mapsto \mu^{(L)}_1,\dots,\mu^{(L)}_s,\qquad \mu^{(L)}_i\in \mathcal{V}_L,$$4 in position $$\alpha \mapsto \mu^{(L)}_1,\dots,\mu^{(L)}_s,\qquad \mu^{(L)}_i\in \mathcal{V}_L,$$5, the conditional class probability is

$$\alpha \mapsto \mu^{(L)}_1,\dots,\mu^{(L)}_s,\qquad \mu^{(L)}_i\in \mathcal{V}_L,$$6

The signal scales as

$$\alpha \mapsto \mu^{(L)}_1,\dots,\mu^{(L)}_s,\qquad \mu^{(L)}_i\in \mathcal{V}_L,$$7

while the finite-sample noise scales like

$$\alpha \mapsto \mu^{(L)}_1,\dots,\mu^{(L)}_s,\qquad \mu^{(L)}_i\in \mathcal{V}_L,$$8

Balancing signal and noise gives

$$\alpha \mapsto \mu^{(L)}_1,\dots,\mu^{(L)}_s,\qquad \mu^{(L)}_i\in \mathcal{V}_L,$$9

which matches the observed $$\mu^{(\ell)} \mapsto \mu^{(\ell-1)}_{1},\dots,\mu^{(\ell-1)}_{s}, \qquad \mu^{(\ell)}\in\mathcal{V}_\ell,\ \mu^{(\ell-1)}_i\in\mathcal{V}_{\ell-1},$$0 (Cagnetta et al., 2023).

3. Provable learning and hierarchical shallow-to-deep chaining

The 2026 analysis proves that, under mild and explicit assumptions, a deep convolutional network can be efficiently trained to learn RHMs (Ren et al., 27 Jan 2026). Its central statement is that if intermediate layers can receive clean signal from the labels and the relevant features are weakly identifiable, then layerwise training each individual layer suffices to hierarchically learn the target function. The proof formalizes a general shallow-to-deep chaining principle (Ren et al., 27 Jan 2026).

The architecture mirrors the hierarchy. Since the branching factor $$\mu^{(\ell)} \mapsto \mu^{(\ell-1)}_{1},\dots,\mu^{(\ell-1)}_{s}, \qquad \mu^{(\ell)}\in\mathcal{V}_\ell,\ \mu^{(\ell-1)}_i\in\mathcal{V}_{\ell-1},$$1 is known, the network groups the input into length-$$\mu^{(\ell)} \mapsto \mu^{(\ell-1)}_{1},\dots,\mu^{(\ell-1)}_{s}, \qquad \mu^{(\ell)}\in\mathcal{V}_\ell,\ \mu^{(\ell-1)}_i\in\mathcal{V}_{\ell-1},$$2 patches and uses an $$\mu^{(\ell)} \mapsto \mu^{(\ell-1)}_{1},\dots,\mu^{(\ell-1)}_{s}, \qquad \mu^{(\ell)}\in\mathcal{V}_\ell,\ \mu^{(\ell-1)}_i\in\mathcal{V}_{\ell-1},$$3-layer convolutional architecture, one layer per hierarchy level. At level $$\mu^{(\ell)} \mapsto \mu^{(\ell-1)}_{1},\dots,\mu^{(\ell-1)}_{s}, \qquad \mu^{(\ell)}\in\mathcal{V}_\ell,\ \mu^{(\ell-1)}_i\in\mathcal{V}_{\ell-1},$$4, the network forms

$$\mu^{(\ell)} \mapsto \mu^{(\ell-1)}_{1},\dots,\mu^{(\ell-1)}_{s}, \qquad \mu^{(\ell)}\in\mathcal{V}_\ell,\ \mu^{(\ell-1)}_i\in\mathcal{V}_{\ell-1},$$5

then applies a trainable linear map $$\mu^{(\ell)} \mapsto \mu^{(\ell-1)}_{1},\dots,\mu^{(\ell-1)}_{s}, \qquad \mu^{(\ell)}\in\mathcal{V}_\ell,\ \mu^{(\ell-1)}_i\in\mathcal{V}_{\ell-1},$$6 followed by normalization,

$$\mu^{(\ell)} \mapsto \mu^{(\ell-1)}_{1},\dots,\mu^{(\ell-1)}_{s}, \qquad \mu^{(\ell)}\in\mathcal{V}_\ell,\ \mu^{(\ell-1)}_i\in\mathcal{V}_{\ell-1},$$7

The favored nonlinearity is a random Fourier feature map for the RBF kernel,

$$\mu^{(\ell)} \mapsto \mu^{(\ell-1)}_{1},\dots,\mu^{(\ell-1)}_{s}, \qquad \mu^{(\ell)}\in\mathcal{V}_\ell,\ \mu^{(\ell-1)}_i\in\mathcal{V}_{\ell-1},$$8

which has unit norm and approximately preserves near-orthogonality for sufficiently separated inputs (Ren et al., 27 Jan 2026).

Training is layerwise, from top to bottom. At stage $$\mu^{(\ell)} \mapsto \mu^{(\ell-1)}_{1},\dots,\mu^{(\ell-1)}_{s}, \qquad \mu^{(\ell)}\in\mathcal{V}_\ell,\ \mu^{(\ell-1)}_i\in\mathcal{V}_{\ell-1},$$9, the paper trains only $\ell=L,\dots,1$0 and uses the ridge-regression loss

$\ell=L,\dots,1$1

It chooses $\ell=L,\dots,1$2, uses zero initialization, and a step size $\ell=L,\dots,1$3 (Ren et al., 27 Jan 2026).

The analysis does not require exact recovery of the hidden symbols. Instead, it uses a surrogate representation

$\ell=L,\dots,1$4

If two patches are synonyms, they induce the same $\ell=L,\dots,1$5; if they are not synonyms, Assumption 2 requires

$\ell=L,\dots,1$6

The paper also assumes non-degeneracy:

$\ell=L,\dots,1$7

Under these assumptions, the sample complexity is

$\ell=L,\dots,1$8

with width and iteration bounds polynomial in the same parameters and $\ell=L,\dots,1$9 (Ren et al., 27 Jan 2026).

For random production rules, the paper proves a lower bound on signal separation:

$s$0

for non-synonyms, with high probability (Ren et al., 27 Jan 2026). By contrast, the shallow-network heuristic inherited from the original RHM paper is

$s$1

which is exponential in the input length $s$2 (Ren et al., 27 Jan 2026). This suggests a rigorous optimization-based separation between deep and shallow learning on a hierarchical task.

4. Sparse Random Hierarchy Model

The Sparse Random Hierarchy Model (SRHM) extends the RHM by adding sparsity to the generative hierarchy (Tomasini et al., 2024). The paper’s stated aim is to unify two ideas that are often studied separately: hierarchical compositional structure in the data, and insensitivity to spatial transformations such as small shifts or diffeomorphisms. It does this by adding an uninformative feature $s$3 to each vocabulary and imposing that each production rule contains exactly $s$4 uninformative features (Tomasini et al., 2024).

In the paper’s model A, called the SRHM, each of the $s$5 informative features is embedded in a sub-patch of size $s$6, with exactly $s$7 empty elements. One production step therefore creates a patch of size

$s$8

At each level, every uninformative feature generates a patch of $s$9 uninformative features at the next level. Hence the full input contains only $d=s^L.$0 informative features, embedded in an ambient dimension

$d=s^L.$1

(Tomasini et al., 2024).

The central conceptual point is that sparsity induces insensitivity to discrete spatial transformations. Because informative features are surrounded by empty positions, small shifts in where the informative features sit often do not change the class. The paper distinguishes two task symmetries: synonym exchange and discrete diffeomorphism or shift (Tomasini et al., 2024).

To measure these effects, it defines sensitivities for internal representations. For a hidden layer $d=s^L.$2, the synonym sensitivity is

$d=s^L.$3

where $d=s^L.$4 replaces each informative $d=s^L.$5-patch by one of its synonyms. The discrete-diffeomorphism sensitivity is

$d=s^L.$6

where $d=s^L.$7 shifts informative features within the allowed sparse positions (Tomasini et al., 2024).

The paper reports that the sample complexity for learning the task, the sample complexity for learning invariance to synonyms, and the sample complexity for learning invariance to spatial shifts all occur at essentially the same training set size. In the notation of the paper,

$d=s^L.$8

For locally connected networks,

$d=s^L.$9

with observations consistent with $m$0. For CNNs,

$m$1

The paper interprets this as evidence that CNNs exploit the repeated local structure much more efficiently than LCNs because weight sharing removes the exponential dependence on depth $m$2 coming from the sparsity factor (Tomasini et al., 2024).

5. Other meanings in network science and stochastic systems

A common source of confusion is that Random Hierarchy Model and closely related phrases denote several non-equivalent constructions outside machine learning.

In network inference, "Structural Inference of Hierarchies in Networks" gives a precise definition of hierarchical structure, gives a generic model for generating arbitrary hierarchical structure in a random graph, and describes a statistically principled way to learn the set of hierarchical features that most plausibly explain a particular real-world network [0610051]. The same abstract states applications to the interpretation of network data, the annotation of graphs with edge, vertex and community properties, and the generation of generic null models for further hypothesis testing [0610051].

In random graph theory, "Models of random graph hierarchies" introduces Random Graph Hierarchy (RGH) and Limited Random Graph Hierarchy (LRGH) (Paluch et al., 2015). In both models a set of nodes at a given hierarchy level is connected randomly, as in the Erdős–Rényi random graph, with a fixed average degree equal to a system parameter $m$3. Clusters of the resulting network are treated as nodes at the next hierarchy level and they are connected again at this level and so on, until the process cannot continue (Paluch et al., 2015). In both models the number of nodes at a given hierarchy level $m$4 decreases approximately exponentially with $m$5, and the height of the hierarchy $m$6 increases logarithmically with the system size $m$7. In the LRGH model, clusters of size $m$8 stop participating in further steps, $m$9 reaches a maximum for a certain $L$00, and the distribution of separate cluster sizes is a power law with an exponent about $L$01 (Paluch et al., 2015).

A related financial-market construction appears in the "Hierarchical Cont-Bouchaud model", which extends the original Cont-Bouchaud herding model by introducing a multi-level nested Erdős–Rényi graph and a limited hierarchical Erdős–Rényi graph (Paluch et al., 2015). The first construction, HERG + Potts dynamics, does not lead to a broad return distribution outside a parameter regime close to the original Cont-Bouchaud model. The second, LHERG + original CB dynamics, leads to a heavy-tail distribution of cluster sizes and relative price changes in a wide range of connection densities, not only close to the percolation threshold (Paluch et al., 2015).

The phrase also appears in network dependence modeling, but with a different meaning. "Hierarchical Models for Independence Structures of Networks" explicitly states that it does not introduce a “random hierarchy model” in the sense of a recursive random tree or latent nested partition model (Sadeghi et al., 2016). Instead, it combines a baseline dyadic-independence network model with a hierarchical log-linear graphical model on the dyads, with the hierarchy encoded by a dependency graph on dyads (Sadeghi et al., 2016). This is hierarchy as structured conditional dependence, not as recursive random aggregation.

Other field-specific meanings include flow hierarchy in directed random networks, quantified by the global reaching centrality

$L$02

with non-monotonic dependence on correlations (Mones, 2012); a random-walk-based hierarchy measure on directed networks, where the hierarchy score is

$L$03

and regular trees outrank chains and stars in the thermodynamic limit (Czégel et al., 2015); and random block-hierarchical directed networks with level-dependent Bernoulli probabilities

$L$04

whose directed triad significance profile falls into the same superfamily as neuron networks in the classification of U. Alon et al. (Avetisov et al., 2010).

In quantum transport, the phrase refers to a different object again: a one-dimensional discrete-time quantum walk on a line whose coin parameters are arranged in a hierarchy of barriers and then perturbed by sparse hierarchical randomness (Sharma et al., 2022). In that model, the regular hierarchy alone slows transport but does not localize, whereas adding randomness only at hierarchy levels can be enough to induce localization (Sharma et al., 2022).

6. Probabilistic foundations, adjacent constructions, and scope

From a probabilistic standpoint, exchangeable random hierarchies admit a representation theorem analogous to de Finetti and Kingman. "A representation of exchangeable hierarchies by sampling from real trees" defines a hierarchy on a set $L$05 as a collection $L$06 of subsets of $L$07 such that $L$08, each singleton subset belongs to $L$09, and if $L$10 then $L$11 equals either $L$12 or $L$13 or $L$14 (Forman et al., 2011). Every exchangeable random hierarchy of positive integers has the same distribution as one obtained by sampling i.i.d. points $L$15 from a random rooted weighted real tree $L$16 and taking all subsets of the form

$L$17

where $L$18 is the fringe subtree rooted at $L$19 (Forman et al., 2011). The paper also gives the alternative characterization through a random hierarchy on $L$20 and i.i.d. Uniform$L$21 variables (Forman et al., 2011). This is not the same object as the machine-learning RHM, but it provides a foundational latent-tree representation for exchangeable hierarchies.

Adjacent work in Bayesian nonparametrics treats hierarchy through random measures rather than recursive symbolic composition. "Hierarchical random measures without tables" shows that the hierarchical Dirichlet process can be viewed, and generalized, through hierarchical completely random measures and vectors, yielding quasi-conjugate posteriors and exact or faster sampling algorithms without the usual latent table representation (Catalano et al., 5 May 2025). This is a hierarchy of dependent random measures, not an RHM in the grammar-based sense.

A nearby grammar-based construction is the Random LLM, an ensemble of context-free grammars with random rule weights (Giorlandino et al., 26 Jun 2026). In a double-scaling limit,

$L$22

the model exhibits a hierarchy of phase transitions: child-child correlations emerge at $L$23, single-symbol marginals become nonuniform at $L$24, and freezing occurs at $L$25 (Giorlandino et al., 26 Jun 2026). This is best regarded as an adjacent random grammar model rather than as the same Random Hierarchy Model studied in deep-learning theory.

A plausible implication of this literature is that the phrase Random Hierarchy Model is best treated as a family resemblance term rather than a canonical model class. In the machine-learning sense, it denotes a random compositional grammar with synonymous rules and label-correlated low-level statistics (Cagnetta et al., 2023, Ren et al., 27 Jan 2026). In network science and probability, it may denote recursive Erdős–Rényi inclusion hierarchies, exchangeable laminar structures, or dyadic dependency hierarchies (Paluch et al., 2015, Forman et al., 2011, Sadeghi et al., 2016). The common denominator is hierarchical organization generated by a stochastic mechanism, but the latent objects, observables, and learning or inference problems differ substantially across domains.