Stochastic Tokenisation Methods

• Stochastic tokenisation is a technique that treats token boundaries as random variables, enabling multiple valid decompositions of the same text for robust learning.
• Methodologically, it encompasses stochastic subword segmentation, BPE-dropout, vocabulary-preserving splits (StochasTok), and end-to-end RL tokenisation, each balancing computational cost with performance.
• Empirical results show improved handling of morphologically rich and noisy data, demonstrating enhanced robustness and generalization in modern NLP applications.

Stochastic tokenisation is the class of tokenisation schemes in which the same input string can be encoded into multiple valid token sequences rather than a single canonical segmentation. In contemporary NLP and LLM practice, the term covers at least three distinct regimes: stochastic subword segmentation over a fixed vocabulary, such as SentencePiece subword regularisation and BPE-dropout; post-tokenisation expansion schemes that split existing tokens while preserving the original vocabulary, such as StochasTok; and end-to-end architectures in which token boundaries themselves are sampled and learned inside the model by optimization of the downstream language-model objective (Arabov, 5 May 2026, Sims et al., 2 Jun 2025, Dauncey et al., 15 Feb 2026). Across these settings, the shared principle is that segmentation is treated as a random variable, typically during training, so that models are exposed to multiple legal decompositions of the same surface form. This has been framed as data augmentation, robustness training, and, in the most formal treatments, as inference over stochastic maps between character strings and token strings (Gastaldi et al., 2024).

1. Formal definition and statistical foundations

A precise abstract formulation models tokenisation as a pair of stochastic maps between free monoids: an encoder $\tau: A^{*} \rightsquigarrow V^{*}$ from character strings to token strings, and a decoder $\kappa: V^{*} \rightsquigarrow A^{*}$ in the reverse direction. In this view, stochastic tokenisation means that $\tau(v \mid a)$ places mass on multiple tokenisations $v$ for a given text $a$, rather than collapsing all mass onto a single output (Gastaldi et al., 2024). This formulation subsumes deterministic tokenisers as a special case.

The central statistical condition is the “Fundamental Principle of Tokenization”: if $q^{*} = \tau p^{*}$ is the token-level reference distribution and $\{q_n\}$ is a consistent estimator of $q^{*}$, then the decoded sequence $\{\kappa q_n\}$ is a consistent estimator of the original character-level distribution $p^{*}$ if and only if $\kappa: V^{*} \rightsquigarrow A^{*}$0 (Gastaldi et al., 2024). Exact tokenisers satisfy the stronger condition $\kappa: V^{*} \rightsquigarrow A^{*}$1 on $\kappa: V^{*} \rightsquigarrow A^{*}$2, and are therefore consistent for every distribution, not only for a particular $\kappa: V^{*} \rightsquigarrow A^{*}$3.

This framework clarifies several recurrent issues. First, stochasticity does not by itself imply inconsistency: consistency depends on how decoding composes with encoding. Second, ambiguity is intrinsic. Even when a tokenizer is exact, a token-level model may assign nonzero probability to alternative segmentations that decode to the same text, so evaluation based on one-best decoding can differ from evaluation based on proper marginalisation (Gastaldi et al., 2024, Chirkova et al., 2023). Third, multiplicative decoders with trivial kernel ensure prefix preservation and finite preimages, which is important for autoregressive modeling and for making marginalisation at least conceptually tractable (Gastaldi et al., 2024).

The broader theoretical literature also treats tokenisation as a transformation of the statistical problem faced by the model. For ergodic $\kappa: V^{*} \rightsquigarrow A^{*}$4-th order Markov sources with $\kappa: V^{*} \rightsquigarrow A^{*}$5, tokenisation can convert a sequence with strong character-level dependencies into one that is close to i.i.d. at the token level, so that even unigram token models become near-optimal under suitable conditions on the emitted token distribution (Rajaraman et al., 2024). That result is derived for deterministic tokenisers, but the same paper explicitly argues that the key sufficient condition is not determinism as such, but whether the emitted token distribution preserves “heavy-hitter” behavior through long, low-probability tokens (Rajaraman et al., 2024).

2. Principal algorithmic families

Stochastic tokenisation is not a single algorithmic technique. The current literature contains several families with different inductive biases, computational costs, and deployment assumptions.

Family	Mechanism	Characteristic property
Unigram subword regularisation	Sample segmentation from $\kappa: V^{*} \rightsquigarrow A^{*}$6 over a segmentation lattice	Vocabulary fixed; stochasticity at encode time
BPE-dropout	Skip applicable merges with probability $\kappa: V^{*} \rightsquigarrow A^{*}$7	Falls back to finer-grained decomposition
StochasTok	Randomly expand tokens into valid binary splits	Preserves original vocabulary
StochasTok-uni / Uniform-k	Uniform sampling over broader segmentation spaces	Reduces bias and enlarges support
End-to-end RL tokenisation	Sample boundary actions inside the model	Tokenisation optimized for LM loss

SentencePiece-style Unigram tokenisation defines a distribution over valid segmentations of a word $\kappa: V^{*} \rightsquigarrow A^{*}$8. If $\kappa: V^{*} \rightsquigarrow A^{*}$9 is the set of valid segmentations and $\tau(v \mid a)$0 is the learned token probability, then $\tau(v \mid a)$1, and the corpus objective marginalises over all segmentations:

$\tau(v \mid a)$2

Subword regularisation then samples from a tempered posterior

$\tau(v \mid a)$3

with nbest_size controlling how many lattice paths are considered and $\tau(v \mid a)$4 controlling the exploration–exploitation trade-off (Arabov, 5 May 2026). In practice, the same trained tokenizer can be used deterministically for validation and inference and stochastically during training.

BPE-dropout is the BPE analogue. Deterministic BPE applies a ranked merge list until no merge applies; BPE-dropout instead skips an applicable merge with probability $\tau(v \mid a)$5, yielding a random segmentation whose expectation approaches deterministic BPE as $\tau(v \mid a)$6 (Arabov, 5 May 2026). The practical ranges reported in the practicum are $\tau(v \mid a)$7 for moderate morphology and $\tau(v \mid a)$8 for highly inflected languages such as Russian and Tatar, with lower values advised for code- or URL-heavy domains (Arabov, 5 May 2026).

StochasTok shifts the locus of randomness. Rather than resampling from a tokenizer-specific segmentation lattice, it starts from the deterministic base tokenisation $\tau(v \mid a)$9 and repeatedly expands existing tokens into valid binary splits drawn from

$v$0

With $v$1 expansion steps, training minimizes the usual language-model objective in expectation over the stochastic expansion operator $v$2:

$v$3

Because the vocabulary is unchanged, StochasTok can be introduced during pretraining, continued pretraining, or retrofitting of pretrained models without changing embeddings or the training loop (Sims et al., 2 Jun 2025).

The robustness literature emphasizes that not all stochastic samplers over segmentations are equally well behaved. StochasTok’s iterative binary splitting is biased and has incomplete support over tokenisations at a given edit distance. To address this, “StochasTok-uni” samples per-token split counts from a Dirichlet–multinomial law and then samples uniformly over per-token segmentation trees, while “Uniform-k” samples uniformly from the set $v$4 of segmentations at token-level edit distance $v$5 from the canonical tokenisation using Multi-rooted Multi-valued Decision Diagrams (Steger et al., 17 Apr 2026). These variants are designed to reduce sampling bias and improve worst-case robustness.

A separate line of work makes tokenisation itself part of the model. In “You Can Learn Tokenization End-to-End with Reinforcement Learning” (Dauncey et al., 15 Feb 2026), the input is a byte sequence $v$6 and the tokenizer is a causal Bernoulli policy over boundary actions $v$7. Sampled boundaries downsample the sequence to token-level states, which are processed and then upsampled for next-byte prediction. Here, stochastic tokenisation is not augmentation around a pre-existing segmentation rule; it is the model’s learned policy for placing token boundaries.

3. Objectives, optimisation, and computational realization

Across stochastic tokenisation methods, the most common training objective is expected loss over segmentations. In the practicum formulation, if $v$8 is trained on token sequences and $v$9 is the per-example loss, then the objective is

$a$0

In practice, this expectation is approximated by Monte Carlo resampling per occurrence or per epoch (Arabov, 5 May 2026). StochasTok gives an equivalent “mixture-of-tokenisations” view in which the same architecture and loss are retained, but the data pipeline samples alternative valid segmentations (Sims et al., 2 Jun 2025).

The end-to-end RL formulation replaces tokenizer-side sampling with policy optimization over discrete boundary decisions. The objective is

$a$1

and the paper uses a score-function estimator rather than a straight-through estimator:

$a$2

Its practical viability depends on variance reduction: an early-exit baseline, discounted returns with $a$3, and batch-relative centering of advantages (Dauncey et al., 15 Feb 2026). The paper’s reported settings use $a$4 and $a$5, with a boundary-policy overhead of less than $a$6 training compute (Dauncey et al., 15 Feb 2026).

Computationally, the methods differ sharply. SentencePiece subword regularisation requires segmentation lattices and $a$7-best or full-lattice sampling; the practicum notes lattice construction complexity $a$8, with $a$9 the average branching factor, and practical nbest_size choices $q^{*} = \tau p^{*}$0 with $q^{*} = \tau p^{*}$1 for mid-sized corpora (Arabov, 5 May 2026). BPE-dropout remains close to deterministic BPE but may require more passes as $q^{*} = \tau p^{*}$2 increases. StochasTok precomputes a splits dictionary in $q^{*} = \tau p^{*}$3 and incurs per-sequence cost $q^{*} = \tau p^{*}$4 (Sims et al., 2 Jun 2025). Uniform-k requires MDD or MRMDD construction, with $q^{*} = \tau p^{*}$5 for the MDD and $q^{*} = \tau p^{*}$6 for the MRMDD (Steger et al., 17 Apr 2026).

Finite-state transduction provides a unifying representation for the segmentation space itself. “Tokenization as Finite-State Transduction” (Cognetta et al., 2024) constructs transducers that encode all possible tokenizations of a regular language, and shows that BPE and MaxMatch fit this framework. Character-level constraints can be promoted to subword level by constructions such as $q^{*} = \tau p^{*}$7 for lexicon transducers, $q^{*} = \tau p^{*}$8 for MaxMatch, and $q^{*} = \tau p^{*}$9 for BPE (Cognetta et al., 2024). The same framework can be weighted, yielding a partition function over paths and exact path sampling, which makes stochastic tokenisation naturally expressible as sampling from a weighted tokenization lattice (Cognetta et al., 2024).

A recurring operational pattern is two-mode usage. Multiple sources recommend enabling stochasticity during training and disabling it during validation, test, and deployment. The practicum prescribes nbest_size=1 or dropout=None at evaluation time (Arabov, 5 May 2026); the RL paper recommends greedy boundary selection at inference for speed and reproducibility (Dauncey et al., 15 Feb 2026); StochasTok similarly reverts to the original deterministic tokenizer for finetuning and inference (Sims et al., 2 Jun 2025).

4. Empirical behaviour: subword understanding, morphology, and robustness

The empirical motivations for stochastic tokenisation differ across domains, but three patterns recur: improved exposure to internal subword structure, improved robustness to non-canonical segmentations, and improved handling of morphologically rich or orthographically unstable data.

In low-resource and morphologically rich settings, stochastic segmentation is treated as controlled augmentation over valid morpheme-level decompositions. The practicum places particular emphasis on Russian, Tajik, and Tatar, arguing that inflectional richness and cross-script issues generate large families of rare surface forms. Within that framing, stochastic Unigram sampling is described as a natural extension for Tajik because it exposes models to multiple valid morpheme splits for the same lemma without inflating the vocabulary, while BPE-dropout with $\{q_n\}$0–$\{q_n\}$1 is reported to help Tatar embeddings and classifiers by distributing information across stems and affixes and mitigating extreme sparsity (Arabov, 5 May 2026).

StochasTok targets a different failure mode: opaque subword compression in LLMs. Its reported experiments show that models pretrained with StochasTok “finetune to near-perfect accuracy across all six” LangGame tasks, that CUTE scores are markedly higher than deterministic-BPE baselines, that out-of-distribution generalisation to longer unseen substring regimes is nearly perfect, and that multi-digit addition can “rapidly grok” under StochasTok even when evaluation uses tokenisation schemes never seen in training (Sims et al., 2 Jun 2025). The same paper also reports that continued pretraining can retrofit this behavior into existing models: a 50M baseline improved after roughly $\{q_n\}$2–$\{q_n\}$3 StochasTok CPT iterations, and GPT-2 improved after $\{q_n\}$4 CPT steps at learning rate $\{q_n\}$5 followed by LangGame finetuning (Sims et al., 2 Jun 2025).

Robustness to tokenisation perturbations is the focus of “Stochasticity in Tokenisation Improves Robustness” (Steger et al., 17 Apr 2026). That paper shows that evaluating a canonically trained Llama-1B model on uniformly sampled non-canonical tokenisations reduces accuracy by $\{q_n\}$6, and reports an illustrative Language Game result in which canonical fine-tuning achieves $\{q_n\}$7 on canonical queries but drops by $\{q_n\}$8 under uniform queries (Steger et al., 17 Apr 2026). Stochastic fine-tuning largely removes this brittleness while preserving clean performance. On scBlue, examples include StochasTok $\{q_n\}$9 with canonical $q^{*}$0 and uniform $q^{*}$1, StochasTok-uni $q^{*}$2 with canonical $q^{*}$3 and uniform $q^{*}$4, Uniform-k $q^{*}$5 with canonical $q^{*}$6 and uniform $q^{*}$7, and Uniform with canonical $q^{*}$8 and uniform $q^{*}$9 (Steger et al., 17 Apr 2026). Under adversarial tokenisation, canonical fine-tuning drops from $\{\kappa q_n\}$0 to $\{\kappa q_n\}$1, whereas Uniform-k $\{\kappa q_n\}$2 reaches $\{\kappa q_n\}$3 (Steger et al., 17 Apr 2026). The same study reports reduced canonical–noncanonical representation distances across layers after stochastic fine-tuning, consistent with more stable internal representations.

End-to-end RL tokenisation yields a different empirical profile. At approximately 100M scale, the score-function model improves validation bits-per-byte over uniform and straight-through baselines on FineWeb and CodeParrot. On FineWeb at about $\{\kappa q_n\}$4 FLOPs, the reported score-function model reaches $\{\kappa q_n\}$5 bpb versus uniform $\{\kappa q_n\}$6, dynamic STE around $\{\kappa q_n\}$7–$\{\kappa q_n\}$8, and H-Net $\{\kappa q_n\}$9; on CodeParrot at about $p^{*}$0 FLOPs, it reaches $p^{*}$1 versus H-Net $p^{*}$2 (Dauncey et al., 15 Feb 2026). The same paper reports zero-shot results of PIQA $p^{*}$3, HellaSwag $p^{*}$4, ARC-Easy $p^{*}$5, and LAMBADA $p^{*}$6, and qualitative boundary patterns that align with semantic separators such as spaces and newline characters (Dauncey et al., 15 Feb 2026).

These findings do not imply that stochasticity is universally beneficial. The practicum reports that too much randomness, such as large $p^{*}$7 or $p^{*}$8, increases variance and slows convergence (Arabov, 5 May 2026). The robustness study similarly observes that very large stochasticity or unrestricted Uniform sampling can induce too many splits and slightly reduce clean accuracy (Steger et al., 17 Apr 2026). The RL paper emphasizes sensitivity to $p^{*}$9, $\kappa: V^{*} \rightsquigarrow A^{*}$00, $\kappa: V^{*} \rightsquigarrow A^{*}$01, $\kappa: V^{*} \rightsquigarrow A^{*}$02, scaling constant $\kappa: V^{*} \rightsquigarrow A^{*}$03, window size $\kappa: V^{*} \rightsquigarrow A^{*}$04, and target downsampling rate, with failure modes that include collapse to all-boundaries or no-boundaries (Dauncey et al., 15 Feb 2026).

5. Evaluation, marginalisation, and reproducibility

Evaluation of stochastic tokenisation has two distinct layers: tokenizer-centric properties and downstream behavior. The practicum formalizes tokenizer-centric evaluation through OOV rate on a held-out set, word fragmentation measured as mean number of subwords per original word, mean token length, compression ratio, and reversibility, with explicit unit tests for round-trip integrity (Arabov, 5 May 2026). Downstream tasks then use the standard metrics of the task family: Accuracy, macro-F1, and PR-AUC for classification; corpus likelihood or perplexity for language modeling; BLEU and ROUGE for MT or generation; and explicit stress tests with noisy text and misspellings for robustness (Arabov, 5 May 2026).

A key methodological rule is that stochasticity is normally confined to training. The practicum requires tokenizers to be trained only on the training split, and downstream comparisons to keep evaluation tokenisation deterministic while varying only the training-time segmentation regime (Arabov, 5 May 2026). Repeated runs with different seeds and reporting of mean $\kappa: V^{*} \rightsquigarrow A^{*}$05 std are recommended for variance reporting, and tokenizer state, normalization rules, and dependency versions are to be serialized and pinned (Arabov, 5 May 2026). The robustness paper operationalizes a comparable protocol by averaging over $\kappa: V^{*} \rightsquigarrow A^{*}$06 random segmentations per input and by separating stochastic perturbation of inputs from canonical treatment of answer options during scoring (Steger et al., 17 Apr 2026).

Theoretically, the correct probability of a character string is the marginal over all valid tokenizations:

$\kappa: V^{*} \rightsquigarrow A^{*}$07

not the score of a single default tokenization (Chirkova et al., 2023). “Should you marginalize over possible tokenizations?” (Chirkova et al., 2023) develops an importance-sampling estimator for this quantity and finds that the gap in log-likelihood is no larger than $\kappa: V^{*} \rightsquigarrow A^{*}$08 in most cases, but becomes more pronounced for data with long complex words. The paper reports, for example, GPT-2 on Wikipedia with BPC $\kappa: V^{*} \rightsquigarrow A^{*}$09 under default tokenization versus $\kappa: V^{*} \rightsquigarrow A^{*}$10 under marginalisation, a $\kappa: V^{*} \rightsquigarrow A^{*}$11 relative gap, while GPT-2 on Twitter shows $\kappa: V^{*} \rightsquigarrow A^{*}$12 versus $\kappa: V^{*} \rightsquigarrow A^{*}$13, a $\kappa: V^{*} \rightsquigarrow A^{*}$14 relative gap (Chirkova et al., 2023). For BLOOM, larger gaps occur in settings such as Basque, where $\kappa: V^{*} \rightsquigarrow A^{*}$15 versus $\kappa: V^{*} \rightsquigarrow A^{*}$16 yields a $\kappa: V^{*} \rightsquigarrow A^{*}$17 relative gap, and C++, where $\kappa: V^{*} \rightsquigarrow A^{*}$18 versus $\kappa: V^{*} \rightsquigarrow A^{*}$19 yields $\kappa: V^{*} \rightsquigarrow A^{*}$20 (Chirkova et al., 2023).

This result constrains a common misconception. It is formally correct that one should marginalize over tokenisations to obtain a string probability, and one-best scoring ignores spurious ambiguity; however, the empirical impact is often small for standard in-domain prose, and the computational cost is high (Chirkova et al., 2023, Gastaldi et al., 2024). The tension between theoretical correctness and practical tractability is therefore central to stochastic tokenisation research.

6. Theory, design trade-offs, and open directions

The most developed general theory of tokenisation argues that the usefulness of tokenisation lies in altering the effective distributional structure seen by the model. For ergodic Markov sources, appropriate tokenization allows even token-level unigram models to approach optimal cross-entropy, whereas untokenized transformers may converge to a character-level stationary-unigram solution (Rajaraman et al., 2024). The relevant quantity is $\kappa: V^{*} \rightsquigarrow A^{*}$21, where $\kappa: V^{*} \rightsquigarrow A^{*}$22 is the emitted token distribution; if this is large enough, then the best token-unigram model is near-optimal (Rajaraman et al., 2024). The same paper explicitly notes that its sufficient condition is agnostic to whether segmentation is deterministic or random. This suggests that stochastic tokenisation can be theoretically justified when its randomness preserves the heavy-hitter structure rather than fragmenting tokens into short, high-probability pieces (Rajaraman et al., 2024).

Design trade-offs follow directly from this. Stochastic tokenisation is most attractive when datasets are small to medium, when the language is morphologically rich or cross-script, or when robustness to noisy user text is important (Arabov, 5 May 2026). It is less attractive for very large datasets, strictly templated inputs such as code and URLs, or retrieval indices that require stable nearest neighbors; the practicum explicitly advises deterministic encoding for RAG indices and deterministic alignment phases for RLHF auditing (Arabov, 5 May 2026). The robustness study likewise emphasizes that benefits are largest on subword-sensitive tasks, and that systems relying on canonical span indices must keep references canonical even if model-side tokenisation varies (Steger et al., 17 Apr 2026).

Several limitations remain unresolved. StochasTok depends on the base vocabulary’s coverage of valid substrings and becomes a no-op when tokens have no valid binary splits; it may also cross morpheme boundaries in ways that are decode-preserving but linguistically crude (Sims et al., 2 Jun 2025). Uniform samplers improve robustness but can be more expensive, since per-token tree construction is exponential in token length and MRMDD construction is $\kappa: V^{*} \rightsquigarrow A^{*}$23 (Steger et al., 17 Apr 2026). End-to-end RL tokenisation preserves theoretical fidelity to discrete boundary decisions, but still faces variance, sample-efficiency, and scaling questions beyond the reported $\kappa: V^{*} \rightsquigarrow A^{*}$24–$\kappa: V^{*} \rightsquigarrow A^{*}$25M parameter regime (Dauncey et al., 15 Feb 2026). The formal theory of stochastic tokenisers also remains incomplete: the category-theoretic framework identifies exactness and consistency conditions, but leaves open broader questions about vocabulary choice, interpretability, and linguistic structure (Gastaldi et al., 2024).

The main open directions in the current literature are explicit. They include optimal non-uniform or morphology-aware split policies, interactions with newer tokenizer designs, cross-lingual behavior across scripts and segmentation regimes, theoretical characterization of segmentation-invariant representations, adaptive downsampling targets in end-to-end models, and large-scale validation of learned boundary policies (Sims et al., 2 Jun 2025, Dauncey et al., 15 Feb 2026). Another unresolved question is how to combine principled stochastic training with principled evaluation: the theory of consistency and the practice of one-best deployment are still only partially reconciled (Gastaldi et al., 2024, Chirkova et al., 2023).

In aggregate, stochastic tokenisation is best understood not as a marginal variation on BPE, but as a family of methods that elevate segmentation from a fixed preprocessing step to an object of modeling, regularization, or optimization. Its concrete realizations range from Unigram posterior sampling and Bernoulli merge-skipping, through vocabulary-preserving post-tokenisation expansion, to fully learned boundary policies. The common thread is that segmentation variability is used to expose structure that deterministic tokenisation conceals, while preserving the requirement that all sampled token sequences decode to the same underlying text (Arabov, 5 May 2026, Sims et al., 2 Jun 2025, Dauncey et al., 15 Feb 2026).