---
title: Stochastic Tokenisation Methods
url: https://www.emergentmind.com/topics/stochastic-tokenisation
type: topic
---

# Stochastic Tokenisation Methods

Stochastic tokenisation is the class of tokenisation schemes in which the same input string can be encoded into multiple valid token sequences rather than a single canonical segmentation. In contemporary NLP and LLM practice, the term covers at least three distinct regimes: stochastic subword segmentation over a fixed vocabulary, such as SentencePiece subword regularisation and BPE-dropout; post-tokenisation expansion schemes that split existing tokens while preserving the original vocabulary, such as StochasTok; and end-to-end architectures in which token boundaries themselves are sampled and learned inside the model by optimization of the downstream language-model objective [2605.03799] [2506.01687] [2602.13940]. Across these settings, the shared principle is that segmentation is treated as a random variable, typically during training, so that models are exposed to multiple legal decompositions of the same surface form. This has been framed as data augmentation, robustness training, and, in the most formal treatments, as inference over stochastic maps between character strings and token strings [2407.11606].

## 1. Formal definition and statistical foundations

A precise abstract formulation models tokenisation as a pair of stochastic maps between free monoids: an encoder $\tau: A^{*} \rightsquigarrow V^{*}$ from character strings to token strings, and a decoder $\kappa: V^{*} \rightsquigarrow A^{*}$ in the reverse direction. In this view, stochastic tokenisation means that $\tau(v \mid a)$ places mass on multiple tokenisations $v$ for a given text $a$, rather than collapsing all mass onto a single output [2407.11606]. This formulation subsumes deterministic tokenisers as a special case.

The central statistical condition is the “Fundamental Principle of Tokenization”: if $q^{*} = \tau p^{*}$ is the token-level reference distribution and $\{q_n\}$ is a consistent estimator of $q^{*}$, then the decoded sequence $\{\kappa q_n\}$ is a consistent estimator of the original character-level distribution $p^{*}$ if and only if $\kappa \tau p^{*} = p^{*}$ [2407.11606]. Exact tokenisers satisfy the stronger condition $\kappa \tau = id$ on $A^{*}$, and are therefore consistent for every distribution, not only for a particular $p^{*}$.

This framework clarifies several recurrent issues. First, stochasticity does not by itself imply inconsistency: consistency depends on how decoding composes with encoding. Second, ambiguity is intrinsic. Even when a tokenizer is exact, a token-level model may assign nonzero probability to alternative segmentations that decode to the same text, so evaluation based on one-best decoding can differ from evaluation based on proper marginalisation [2407.11606] [2306.17757]. Third, multiplicative decoders with trivial kernel ensure prefix preservation and finite preimages, which is important for autoregressive modeling and for making marginalisation at least conceptually tractable [2407.11606].

The broader theoretical literature also treats tokenisation as a transformation of the statistical problem faced by the model. For ergodic $k$-th order Markov sources with $k>1$, tokenisation can convert a sequence with strong character-level dependencies into one that is close to i.i.d. at the token level, so that even unigram token models become near-optimal under suitable conditions on the emitted token distribution [2404.08335]. That result is derived for deterministic tokenisers, but the same paper explicitly argues that the key sufficient condition is not determinism as such, but whether the emitted token distribution preserves “heavy-hitter” behavior through long, low-probability tokens [2404.08335].

## 2. Principal algorithmic families

Stochastic tokenisation is not a single algorithmic technique. The current literature contains several families with different inductive biases, computational costs, and deployment assumptions.

| Family | Mechanism | Characteristic property |
|---|---|---|
| Unigram subword regularisation | Sample segmentation from $P_\alpha(S \mid w)$ over a segmentation lattice | Vocabulary fixed; stochasticity at encode time |
| BPE-dropout | Skip applicable merges with probability $p$ | Falls back to finer-grained decomposition |
| StochasTok | Randomly expand tokens into valid binary splits | Preserves original vocabulary |
| StochasTok-uni / Uniform-k | Uniform sampling over broader segmentation spaces | Reduces bias and enlarges support |
| End-to-end RL tokenisation | Sample boundary actions inside the model | Tokenisation optimized for LM loss |

SentencePiece-style Unigram tokenisation defines a distribution over valid segmentations of a word $w$. If $S(w)$ is the set of valid segmentations and $p(t)$ is the learned token probability, then $P(S \mid w) = \prod_{t \in S} p(t)$, and the corpus objective marginalises over all segmentations:
$$
L(V) = \sum_{w \in C} \log \left( \sum_{S \in S(w)} \prod_{t \in S} p(t) \right).
$$
Subword regularisation then samples from a tempered posterior
$$
P_\alpha(S \mid w) \propto [P(S \mid w)]^\alpha,
$$
with `nbest_size` controlling how many lattice paths are considered and $\alpha$ controlling the exploration–exploitation trade-off [2605.03799]. In practice, the same trained tokenizer can be used deterministically for validation and inference and stochastically during training.

BPE-dropout is the BPE analogue. Deterministic BPE applies a ranked merge list until no merge applies; BPE-dropout instead skips an applicable merge with probability $p$, yielding a random segmentation whose expectation approaches deterministic BPE as $p \to 0$ [2605.03799]. The practical ranges reported in the practicum are $p \in [0.05, 0.2]$ for moderate morphology and $p \in [0.1, 0.3]$ for highly inflected languages such as Russian and Tatar, with lower values advised for code- or URL-heavy domains [2605.03799].

StochasTok shifts the locus of randomness. Rather than resampling from a tokenizer-specific segmentation lattice, it starts from the deterministic base tokenisation $T(x) = (t_1,\dots,t_n)$ and repeatedly expands existing tokens into valid binary splits drawn from
$$
Splits(t_i) = \{(u,v) \mid decode(u)+decode(v)=s_i,\ u,v \in V\}.
$$
With $K=\lfloor p \cdot n \rfloor$ expansion steps, training minimizes the usual language-model objective in expectation over the stochastic expansion operator $S_p$:
$$
L(\theta)=E_{x\sim D}E_{S_p}\left[\sum_j \ell(model_\theta(S_p(T(x)))_j,y_j)\right].
$$
Because the vocabulary is unchanged, StochasTok can be introduced during pretraining, continued pretraining, or retrofitting of pretrained models without changing embeddings or the training loop [2506.01687].

The robustness literature emphasizes that not all stochastic samplers over segmentations are equally well behaved. StochasTok’s iterative binary splitting is biased and has incomplete support over tokenisations at a given edit distance. To address this, “StochasTok-uni” samples per-token split counts from a Dirichlet–multinomial law and then samples uniformly over per-token segmentation trees, while “Uniform-k” samples uniformly from the set $T_V^k(x,v^c)$ of segmentations at token-level edit distance $k$ from the canonical tokenisation using Multi-rooted Multi-valued Decision Diagrams [2604.16037]. These variants are designed to reduce sampling bias and improve worst-case robustness.

A separate line of work makes tokenisation itself part of the model. In “You Can Learn Tokenization End-to-End with Reinforcement Learning” [2602.13940], the input is a byte sequence $x=(x_1,\dots,x_N)$ and the tokenizer is a causal Bernoulli policy over boundary actions $a_t \in \{0,1\}$. Sampled boundaries downsample the sequence to token-level states, which are processed and then upsampled for next-byte prediction. Here, stochastic tokenisation is not augmentation around a pre-existing segmentation rule; it is the model’s learned policy for placing token boundaries.

## 3. Objectives, optimisation, and computational realization

Across stochastic tokenisation methods, the most common training objective is expected loss over segmentations. In the practicum formulation, if $f$ is trained on token sequences and $L$ is the per-example loss, then the objective is
$$
E_{S \sim P_\alpha(S \mid w)}[L(f(tokens(S)))].
$$
In practice, this expectation is approximated by Monte Carlo resampling per occurrence or per epoch [2605.03799]. StochasTok gives an equivalent “mixture-of-tokenisations” view in which the same architecture and loss are retained, but the data pipeline samples alternative valid segmentations [2506.01687].

The end-to-end RL formulation replaces tokenizer-side sampling with policy optimization over discrete boundary decisions. The objective is
$$
L(\theta)=\mathbb{E}_{z \sim p_\theta(z \mid x)}[L_{LM}(x,z)] + \lambda Reg(\theta),
$$
and the paper uses a score-function estimator rather than a straight-through estimator:
$$
\nabla_\theta \mathbb{E}_{z\sim p_\theta(z \mid x)}[J(x,z)]
=
\mathbb{E}_{z\sim p_\theta}\left[(J(x,z)-b(x))\nabla_\theta \log p_\theta(z\mid x)\right].
$$
Its practical viability depends on variance reduction: an early-exit baseline, discounted returns with $\gamma=0.99$, and batch-relative centering of advantages [2602.13940]. The paper’s reported settings use $\lambda_\pi=\lambda_{target}=1e{-2}$ and $\lambda_{early}=1e{-1}$, with a boundary-policy overhead of less than $0.1\%$ training compute [2602.13940].

Computationally, the methods differ sharply. SentencePiece subword regularisation requires segmentation lattices and $n$-best or full-lattice sampling; the practicum notes lattice construction complexity $O(|w| \cdot A)$, with $A$ the average branching factor, and practical `nbest_size` choices $\{10,64,-1\}$ with $\alpha \in [0.1,0.5]$ for mid-sized corpora [2605.03799]. BPE-dropout remains close to deterministic BPE but may require more passes as $p$ increases. StochasTok precomputes a splits dictionary in $O(|V| \cdot L_{avg})$ and incurs per-sequence cost $O(n+p\cdot n)$ [2506.01687]. Uniform-k requires MDD or MRMDD construction, with $O(|x| \cdot |V|)$ for the MDD and $O(|x|^2 \cdot |V|)$ for the MRMDD [2604.16037].

Finite-state transduction provides a unifying representation for the segmentation space itself. “Tokenization as Finite-State Transduction” [2410.15696] constructs transducers that encode all possible tokenizations of a regular language, and shows that BPE and MaxMatch fit this framework. Character-level constraints can be promoted to subword level by constructions such as $Min(Proj(A \circ T))$ for lexicon transducers, $Min(Proj(A \circ T_{Aho}))$ for MaxMatch, and $Min(Proj(A \circ (\circ_{m \in \mu} G_m)))$ for BPE [2410.15696]. The same framework can be weighted, yielding a partition function over paths and exact path sampling, which makes stochastic tokenisation naturally expressible as sampling from a weighted tokenization lattice [2410.15696].

A recurring operational pattern is two-mode usage. Multiple sources recommend enabling stochasticity during training and disabling it during validation, test, and deployment. The practicum prescribes `nbest_size=1` or `dropout=None` at evaluation time [2605.03799]; the RL paper recommends greedy boundary selection at inference for speed and reproducibility [2602.13940]; StochasTok similarly reverts to the original deterministic tokenizer for finetuning and inference [2506.01687].

## 4. Empirical behaviour: subword understanding, morphology, and robustness

The empirical motivations for stochastic tokenisation differ across domains, but three patterns recur: improved exposure to internal subword structure, improved robustness to non-canonical segmentations, and improved handling of morphologically rich or orthographically unstable data.

In low-resource and morphologically rich settings, stochastic segmentation is treated as controlled augmentation over valid morpheme-level decompositions. The practicum places particular emphasis on Russian, Tajik, and Tatar, arguing that inflectional richness and cross-script issues generate large families of rare surface forms. Within that framing, stochastic Unigram sampling is described as a natural extension for Tajik because it exposes models to multiple valid morpheme splits for the same lemma without inflating the vocabulary, while BPE-dropout with $p \approx 0.1$–$0.2$ is reported to help Tatar embeddings and classifiers by distributing information across stems and affixes and mitigating extreme sparsity [2605.03799].

StochasTok targets a different failure mode: opaque subword compression in LLMs. Its reported experiments show that models pretrained with StochasTok “finetune to near-perfect accuracy across all six” LangGame tasks, that CUTE scores are markedly higher than deterministic-BPE baselines, that out-of-distribution generalisation to longer unseen substring regimes is nearly perfect, and that multi-digit addition can “rapidly grok” under StochasTok even when evaluation uses tokenisation schemes never seen in training [2506.01687]. The same paper also reports that continued pretraining can retrofit this behavior into existing models: a 50M baseline improved after roughly $2k$–$3k$ StochasTok CPT iterations, and GPT-2 improved after $7k$ CPT steps at learning rate $1e{-4}$ followed by LangGame finetuning [2506.01687].

Robustness to tokenisation perturbations is the focus of “Stochasticity in Tokenisation Improves Robustness” [2604.16037]. That paper shows that evaluating a canonically trained Llama-1B model on uniformly sampled non-canonical tokenisations reduces accuracy by $29.8\%$, and reports an illustrative Language Game result in which canonical fine-tuning achieves $0.940$ on canonical queries but drops by $-0.298$ under uniform queries [2604.16037]. Stochastic fine-tuning largely removes this brittleness while preserving clean performance. On scBlue, examples include StochasTok $\alpha=0.5$ with canonical $0.948$ and uniform $\Delta=-0.005$, StochasTok-uni $\alpha=1.0$ with canonical $0.950$ and uniform $\Delta=+0.014$, Uniform-k $\alpha=1.0$ with canonical $0.948$ and uniform $\Delta=+0.028$, and Uniform with canonical $0.928$ and uniform $\Delta=+0.060$ [2604.16037]. Under adversarial tokenisation, canonical fine-tuning drops from $0.94$ to $0.061$, whereas Uniform-k $\alpha=1.0$ reaches $0.696$ [2604.16037]. The same study reports reduced canonical–noncanonical representation distances across layers after stochastic fine-tuning, consistent with more stable internal representations.

End-to-end RL tokenisation yields a different empirical profile. At approximately 100M scale, the score-function model improves validation bits-per-byte over uniform and straight-through baselines on FineWeb and CodeParrot. On FineWeb at about $1.95 \times 10^{18}$ FLOPs, the reported score-function model reaches $1.279$ bpb versus uniform $1.355$, dynamic STE around $1.350$–$1.360$, and H-Net $1.386$; on CodeParrot at about $0.95 \times 10^{18}$ FLOPs, it reaches $0.568$ versus H-Net $0.769$ [2602.13940]. The same paper reports zero-shot results of PIQA $0.565$, HellaSwag $0.271$, ARC-Easy $0.308$, and LAMBADA $0.086$, and qualitative boundary patterns that align with semantic separators such as spaces and newline characters [2602.13940].

These findings do not imply that stochasticity is universally beneficial. The practicum reports that too much randomness, such as large $\alpha$ or $p>0.3$, increases variance and slows convergence [2605.03799]. The robustness study similarly observes that very large stochasticity or unrestricted Uniform sampling can induce too many splits and slightly reduce clean accuracy [2604.16037]. The RL paper emphasizes sensitivity to $\gamma$, $\lambda_\pi$, $\lambda_{early}$, $\lambda_{target}$, scaling constant $D$, window size $w$, and target downsampling rate, with failure modes that include collapse to all-boundaries or no-boundaries [2602.13940].

## 5. Evaluation, marginalisation, and reproducibility

Evaluation of stochastic tokenisation has two distinct layers: tokenizer-centric properties and downstream behavior. The practicum formalizes tokenizer-centric evaluation through OOV rate on a held-out set, word fragmentation measured as mean number of subwords per original word, mean token length, compression ratio, and reversibility, with explicit unit tests for round-trip integrity [2605.03799]. Downstream tasks then use the standard metrics of the task family: Accuracy, macro-F1, and PR-AUC for classification; corpus likelihood or perplexity for language modeling; BLEU and ROUGE for MT or generation; and explicit stress tests with noisy text and misspellings for robustness [2605.03799].

A key methodological rule is that stochasticity is normally confined to training. The practicum requires tokenizers to be trained only on the training split, and downstream comparisons to keep evaluation tokenisation deterministic while varying only the training-time segmentation regime [2605.03799]. Repeated runs with different seeds and reporting of mean $\pm$ std are recommended for variance reporting, and tokenizer state, normalization rules, and dependency versions are to be serialized and pinned [2605.03799]. The robustness paper operationalizes a comparable protocol by averaging over $M=10$ random segmentations per input and by separating stochastic perturbation of inputs from canonical treatment of answer options during scoring [2604.16037].

Theoretically, the correct probability of a character string is the marginal over all valid tokenizations:
$$
P(x)=\sum_{t \in T(x)} P(t),
$$
not the score of a single default tokenization [2306.17757]. “Should you marginalize over possible tokenizations?” [2306.17757] develops an importance-sampling estimator for this quantity and finds that the gap in log-likelihood is no larger than $0.5\%$ in most cases, but becomes more pronounced for data with long complex words. The paper reports, for example, GPT-2 on Wikipedia with BPC $1.1076$ under default tokenization versus $1.1026$ under marginalisation, a $0.45\%$ relative gap, while GPT-2 on Twitter shows $1.9610$ versus $1.9303$, a $1.56\%$ relative gap [2306.17757]. For BLOOM, larger gaps occur in settings such as Basque, where $1.2432$ versus $1.2269$ yields a $1.31\%$ relative gap, and C++, where $0.6053$ versus $0.5993$ yields $0.98\%$ [2306.17757].

This result constrains a common misconception. It is formally correct that one should marginalize over tokenisations to obtain a string probability, and one-best scoring ignores spurious ambiguity; however, the empirical impact is often small for standard in-domain prose, and the computational cost is high [2306.17757] [2407.11606]. The tension between theoretical correctness and practical tractability is therefore central to stochastic tokenisation research.

## 6. Theory, design trade-offs, and open directions

The most developed general theory of tokenisation argues that the usefulness of tokenisation lies in altering the effective distributional structure seen by the model. For ergodic Markov sources, appropriate tokenization allows even token-level unigram models to approach optimal cross-entropy, whereas untokenized transformers may converge to a character-level stationary-unigram solution [2404.08335]. The relevant quantity is $H(\mu,P)=E_{t\sim \mu}[\log(1/P(t))]$, where $\mu$ is the emitted token distribution; if this is large enough, then the best token-unigram model is near-optimal [2404.08335]. The same paper explicitly notes that its sufficient condition is agnostic to whether segmentation is deterministic or random. This suggests that stochastic tokenisation can be theoretically justified when its randomness preserves the heavy-hitter structure rather than fragmenting tokens into short, high-probability pieces [2404.08335].

Design trade-offs follow directly from this. Stochastic tokenisation is most attractive when datasets are small to medium, when the language is morphologically rich or cross-script, or when robustness to noisy user text is important [2605.03799]. It is less attractive for very large datasets, strictly templated inputs such as code and URLs, or retrieval indices that require stable nearest neighbors; the practicum explicitly advises deterministic encoding for RAG indices and deterministic alignment phases for RLHF auditing [2605.03799]. The robustness study likewise emphasizes that benefits are largest on subword-sensitive tasks, and that systems relying on canonical span indices must keep references canonical even if model-side tokenisation varies [2604.16037].

Several limitations remain unresolved. StochasTok depends on the base vocabulary’s coverage of valid substrings and becomes a no-op when tokens have no valid binary splits; it may also cross morpheme boundaries in ways that are decode-preserving but linguistically crude [2506.01687]. Uniform samplers improve robustness but can be more expensive, since per-token tree construction is exponential in token length and MRMDD construction is $O(|x|^2 \cdot |V|)$ [2604.16037]. End-to-end RL tokenisation preserves theoretical fidelity to discrete boundary decisions, but still faces variance, sample-efficiency, and scaling questions beyond the reported $90$–$147$M parameter regime [2602.13940]. The formal theory of stochastic tokenisers also remains incomplete: the category-theoretic framework identifies exactness and consistency conditions, but leaves open broader questions about vocabulary choice, interpretability, and linguistic structure [2407.11606].

The main open directions in the current literature are explicit. They include optimal non-uniform or morphology-aware split policies, interactions with newer tokenizer designs, cross-lingual behavior across scripts and segmentation regimes, theoretical characterization of segmentation-invariant representations, adaptive downsampling targets in end-to-end models, and large-scale validation of learned boundary policies [2506.01687] [2602.13940]. Another unresolved question is how to combine principled stochastic training with principled evaluation: the theory of consistency and the practice of one-best deployment are still only partially reconciled [2407.11606] [2306.17757].

In aggregate, stochastic tokenisation is best understood not as a marginal variation on BPE, but as a family of methods that elevate segmentation from a fixed preprocessing step to an object of modeling, regularization, or optimization. Its concrete realizations range from Unigram posterior sampling and Bernoulli merge-skipping, through vocabulary-preserving post-tokenisation expansion, to fully learned boundary policies. The common thread is that segmentation variability is used to expose structure that deterministic tokenisation conceals, while preserving the requirement that all sampled token sequences decode to the same underlying text [2605.03799] [2506.01687] [2602.13940].

Source: https://www.emergentmind.com/topics/stochastic-tokenisation