A Layered Simplex Architecture for Large Alphabets

Abstract: Probability estimation over large alphabets under log loss is a well-studied problem, with celebrated methods such as the Good-Turing estimator. We introduce and study a new Bayesian estimator with four notable properties. First, its construction is exceptionally simple: multiply independent uniform draws from the probability simplex coordinate-wise and renormalize. Depth is the only structural parameter, and averaging over depths eliminates the need to tune it. Second, the regret of the resulting mixture, the excess code length it pays relative to a code that knows the source, admits an explicit and efficiently computable expression. Third, despite its simplicity and lack of tuned constants, the estimator is competitive across a diverse set of synthetic and real-text benchmarks with substantially more specialized methods, including Good-Turing. Fourth, the tractability of its regret allows us to identify scaling laws in data, alphabet size, and depth. For Zipf targets with exponent above one, the regret has a simple reading as long as the sample reveals only a small fraction of the alphabet. It closely matches the description length of the set of discovered symbols, at one bit of code per bit of description, plus a further cost per symbol. The data exponent is therefore the rate at which new symbols are discovered.

• The paper introduces a layered simplex prior that multiplies and renormalizes independent uniform simplex draws, producing a coherent Bayesian predictor whose depth controls sparsity without adding fitted parameters.
• The exact regret framework and experiments show that shallow depths suit uniform targets, deeper logarithmic depths suit heavy-tailed targets, and depth averaging stays within (log₂ Lmax)/N bits per symbol of the best depth while scaling to alphabets of 10⁶ symbols.
• The symbol-discovery analysis links regret to the description length of observed symbols, predicts the Zipf data exponent 1−1/α, and supports competitive results including 0.095 bits per token on the King James Bible and a 49-fold envelope improvement over depth one in a concentrated setting.

The construction

Feder, Fogel, and Urbanke study probability estimation over a large alphabet $\mathcal{X}_d = \{1,\dots,d\}$ under log loss using a Bayesian prior built from a layered simplex architecture (LSA). At depth $L$, one draws $L$ independent uniform points on the probability simplex $\Delta_d$ (equivalently, normalized vectors of iid $\operatorname{Exp}(1)$ variables), multiplies them coordinate-wise, and renormalizes. This "product of experts" construction has a single structural parameter, the depth $L$, and induces a non-uniform prior $\mathcal{M}_{d,L}$ on $\Delta_d$. The Bayesian mixture over multinomial likelihoods under this prior defines the LSA predictor.

Two regimes matter. At fixed $L$, the prior is dense: at $L=1$ it is exactly the uniform Dirichlet prior, and the resulting mixture is Laplace's add-one rule, with $L$0 (2608.19908). In the logarithmic-depth regime $L$1, the log-weights of coordinates are sums of $L$2 iid terms with spread $L$3, so typical prior draws become sparse and heavy-tailed: depth does not add parameters to fit but reshapes where prior mass sits on the simplex.

Exact regret and the depth-averaged predictor

Because the prior is exchangeable, the mixture probability of a sequence depends only on the count profile — the multiset of nonzero counts. The per-symbol regret admits the exact identity

$L$4

which separates target dependence ($L$5, $L$6) from architecture dependence ($L$7) (2608.19908). All probabilities are computed from explicit formulas (a layer recursion, integral representations, and Mellin–Barnes contour evaluations for exact kernel rows), with Monte Carlo used only to average over count profiles; the implementation is validated against closed forms and quadrature to tolerances around $L$8 nats, and the machinery scales to $L$9 and depths up to $L$0.

The central empirical finding is that depth tilts the complexity spectrum. Evaluating every integer depth up to $L$1 at $L$2, shallow depths minimize regret on near-uniform targets while deeper depths minimize it on concentrated ones; at $L$3, $L$4, the family envelope improves on $L$5 by a factor of about 49. Crucially, the depth-averaged mixture over $L$6 tracks the best single depth to within $L$7 bits per symbol on every target — a bound attained exactly on flat targets, where the posterior collapses onto $L$8. Since the layer recursion computes depth $L$9 from depth $\Delta_d$0, the entire family costs no more than its deepest member, so depth selection is unnecessary.

Scaling laws from symbol discovery

The paper's most substantive analytical contribution connects the regret to symbol discovery. For Zipf targets $\Delta_d$1, a sample of size $\Delta_d$2 reveals $\Delta_d$3 distinct symbols, which grows as $\Delta_d$4. The finite-size prediction is

$\Delta_d$5

i.e., the regret is the description length of the discovered symbol set, charged at one bit of code per bit of description, plus a per-symbol surcharge $\Delta_d$6. Three numerical tests support this account:

• Alphabet scaling: for $\Delta_d$7, the measured slope of $\Delta_d$8 against the description length per symbol ranges from $\Delta_d$9 to $\operatorname{Exp}(1)$0, close to the predicted value of one. At $\operatorname{Exp}(1)$1 the slope degrades to $\operatorname{Exp}(1)$2 at $\operatorname{Exp}(1)$3 because the sample reveals a substantial fraction of smaller alphabets — a regime the law explicitly does not cover.
• Data scaling: the measured exponent of $\operatorname{Exp}(1)$4 versus $\operatorname{Exp}(1)$5 matches the refined finite-size prediction to within $\operatorname{Exp}(1)$6 at $\operatorname{Exp}(1)$7 and $\operatorname{Exp}(1)$8 at $\operatorname{Exp}(1)$9, with the limit $L$0 identified as the symbol-discovery rate.
• Depth scaling: the price per bit of description is derived as $L$1 below the freezing threshold and exactly $L$2 beyond it, where $L$3 is the Legendre transform of $L$4. The price reaches its floor flatly at $L$5, which explains both the broad minima in the measurements and the absence of a kink in the regret at the freezing phase transition — a genuine phase transition of the prior that is operationally invisible.

The offset $L$6 grows with $L$7 in the direction predicted by a lift argument ($L$8 bits per discovered symbol asymptotically) but drifts with $L$9; the authors report it rather than model it, and the derivation of $\mathcal{M}_{d,L}$0 is approximate for $\mathcal{M}_{d,L}$1. The scaling law itself is supported over finite ranges ($\mathcal{M}_{d,L}$2, $\mathcal{M}_{d,L}$3) rather than proved.

Competitive performance

On an eleven-target extension of the Orlitsky–Suresh benchmark ($\mathcal{M}_{d,L}$4), the depth-averaged LSA predictor — one fixed prior with no tuned constants — beats or ties the best classical method within one standard error on eight of eleven targets (2608.19908). On Zipf $\mathcal{M}_{d,L}$5 it matches Good–Turing ($\mathcal{M}_{d,L}$6 bits at $\mathcal{M}_{d,L}$7, within $\mathcal{M}_{d,L}$8 of the natural oracle); on Zipf $\mathcal{M}_{d,L}$9 through $\Delta_d$0 it overtakes Good–Turing, e.g., $\Delta_d$1 vs. $\Delta_d$2 bits at $\Delta_d$3. The clear exceptions are the flattest targets (uniform, step, Zipf $\Delta_d$4), where Good–Turing exploits count-frequency statistics — reaching $\Delta_d$5 bits on the uniform target versus $\Delta_d$6 for the best LSA member — information that no exchangeable mixture in this family uses. The posterior over depths behaves as the theory predicts: it collapses to $\Delta_d$7 on flat targets, selects $\Delta_d$8–$\Delta_d$9 on Zipf $L$0 (just beyond $L$1), and spreads flatly over the deep end for $L$2, confirming that depth beyond roughly $L$3 is inert.

On the King James Bible ($L$4 tokens, fixed $L$5 vocabulary), the depth-averaged predictor is the best method tested at every prefix length, with $L$6 bits per token redundancy versus $L$7 for Good–Turing and $L$8 for Ristad's natural law of succession. The depth posterior is essentially a point mass drifting from $L$9 to $L=1$0 as the sample grows — squarely in the logarithmic regime, and exhibiting the same downward adaptation seen on synthetic targets. The paper also stresses coherence: unlike Good–Turing and Ristad's law (whose one-step rules do not multiply to an exchangeable sequence probability), the LSA predictor is a single coherent Bayesian code with an exact regret identity, and this coherence costs essentially nothing except on flat targets.

Adding memory

Treating memory as a second architectural axis, an order-one model with per-state depth-averaged LSA conditionals drops the Bible code from $L=1$1 to $L=1$2 bits per token at $L=1$3 states, ahead of per-state Good–Turing ($L=1$4). A notable reversal emerges: per-state KT codes at $L=1$5 bits are worse than the memoryless layered model, because each state is another $L=1$6-symbol alphabet on which an add-constant prior wastes its mass — at this alphabet size, the marginal prior matters more than the memory it feeds. The number of split contexts has an interior optimum (the code rises again past $L=1$7), and a fully sequential variant that learns the context set online costs $L=1$8 bits, $L=1$9 above the semi-adaptive version.

Limitations and open questions

The paper is candid about its boundaries. The analysis beyond $L$00 is numerical; the scaling law is empirically supported over finite ranges rather than proved. Targets are iid with known alphabet size, and the comparison inherits the specific benchmark choices of the competitive-estimation literature. The derivation of the price function $L$01 is reliable only near and above $L$02 and overshoots for $L$03; a full derivation through the induced density on shapes, which would also explain the $L$04-dependence of $L$05, is left open and requires the phase-structure analysis of a companion work. The most pointed open question is whether the depth-averaged LSA prior admits a worst-case competitive guarantee of the Good–Turing type against the natural oracle — the uniform-target results show the present family does not achieve this.

Conclusion

The paper shows that a minimal layered prior — multiply independent uniform simplex draws and renormalize — yields a coherent Bayesian estimator that matches or beats specialized large-alphabet estimators on heavy-tailed targets and real text, with an exactly computable regret. Its scaling law gives a first-principles account in which the data exponent $L$06 is the symbol-discovery rate and logarithmic depth moves only constants, while the freezing transition at $L$07 enters the regret only through a price function that is flat at the transition. As a solvable instance of the broader thesis that layered architectures induce priors with a broad complexity range, it demonstrates how much of both the scaling-law and competitive-estimation stories follows from a single structural principle, without invoking training dynamics.