Optimal Token Baseline (OTB) in NLP

• OTB is a formal benchmark defining the minimal tokenization scheme by using dynamic programming and convex relaxations for precise lower bounds.
• It quantifies compression-optimal tokenization, providing measurable improvements over greedy methods and guiding compute-efficient scaling laws.
• OTB also establishes a per-token baseline that reduces variance in reinforcement learning, cutting gradient variance and token usage significantly.

The Optimal Token Baseline (OTB) is a formal, data-driven benchmark for assessing the minimal achievable tokenization—subject to specified constraints on the vocabulary—for language modeling and downstream tasks. OTB appears in multiple domains, encompassing compression-optimal tokenization, principled lower bounds for tokenization algorithms, and algorithms for variance reduction in reinforcement learning with LLMs. Across these contexts, OTB characterizes either the optimal segmentation (fewest tokens), the LP-based lower bound for tokenization cost, or the per-token baseline yielding provable variance reduction. OTB is consequential for both practical efficiency and the formulation of scaling laws in language modeling.

1. OTB in Compression-Optimal Tokenization

The archetypal instantiation of OTB is the global minimum-token segmentation for a fixed vocabulary, typically for subword schemes such as BPE. Given input byte-sequence $d = B_0B_1\dots B_{n-1}$ and vocabulary $V$, a segmentation $S$ is a sequence of tokens $t_1,\dots,t_K \in V$ with $t_1||\dots||t_K = d$. The OTB segmentation $S^*$ solves: $S^* = \arg\min_{S \in S(d)} |S|,$ subject to each $t_i \in V$. This provides a hard lower bound on token count achievable by any model using $V$.

Algorithmically, OTB is computed via dynamic programming:

• Let $dp[i]$ be the minimal number of tokens needed for $V$0.
• Recurrence: $V$1, with $V$2.
• Trie data structures facilitate $V$3 complexity (where $V$4 is input length, $V$5 is maximum token length) (Raj et al., 2024).

This approach yields a tokenization matching the minimizer of token sequence length, in direct contrast with greedy BPE which selects the longest valid token at each prefix, ignoring global optimality.

2. OTB as an Algorithmic Lower Bound via Convex Relaxations

Beyond dynamic programming, OTB also appears as a provable lower bound on the minimal tokenization cost, formulated as a convex relaxation of an integer linear program:

• Construct a tokenization graph (vertices: string positions; edges: potential tokens).
• Define binary variables for token usage ($V$6), inclusion ($V$7), and auxiliary edge variables.
• Objective: minimize total segments (compression cost) across the dataset.

The integer program ensures:

• Flow conservation (tokenization covers the input).
• A token is used only if present in the vocabulary (cardinality constraint).
• Vocabulary budget $V$8 (upper bound on $V$9).

Relaxing integrality to $S$0 yields a polytope; solving the LP provides an OTB certificate (lower bound on minimal achievable cost). Empirically, ConvexTok rounding matches the LP solution within 1% for practical vocabulary sizes (Tempus et al., 21 May 2026).

Tokenizer	LP Lower Bound	Achieved Cost	Integrality Gap
BPE (32k vocab)	$S$1	$S$2	1.29%
ConvexTok	$S$3	$S$4	0.07%

OTB thus quantifies the compressibility floor when optimizing both segmentation and vocabulary under resource constraints.

3. OTB for Variance Reduction in LLM Reinforcement Learning

In reinforcement learning (RL) with LLMs, OTB defines a per-token baseline that provably minimizes policy gradient variance for long-horizon tasks. The variance of the REINFORCE estimator explodes with trajectory length and reward sparsity: $S$5 Classic group-based baselines (mean or value function) are insufficiently adaptive to token/sequence heterogeneity.

The OTB for RL is the unique baseline $S$6 minimizing variance at each token: $S$7 with $S$8 the sum of future rewards (return) and $S$9 the cumulative token-level gradient norm squared. Using a logit-gradient proxy (computed from forward-pass probabilities), OTB is efficiently estimated without backward passes (Li et al., 6 Feb 2026).

Practically, OTB reduces requisite group (batch) size (e.g., from $t_1,\dots,t_K \in V$0 to $t_1,\dots,t_K \in V$1) while maintaining stability and performance, producing over 65% reduction in token usage during policy gradient RL.

4. Empirical Findings and Comparative Evaluation

Experiments validate OTB's superiority and practical value:

• OTB yields 3–5% token count reduction compared to greedy BPE for low-resource and morphologically-rich languages (e.g., Oromo, Quechua, Finnish, Turkish) (Raj et al., 2024).
• Longer and agglutinative words see the steepest compression gains.
• Downstream, OTB confers up to 10% accuracy improvement on tokenization-sensitive tasks, most pronounced for small capacity models (e.g., GPT-2 120M) and languages where standard tokenization is especially lossy.

In RL, OTB achieves a >50% variance reduction in gradient estimators and enables efficient long-horizon sequence modeling with dramatically fewer samples (Li et al., 6 Feb 2026).

In convex relaxation-based tokenization, ConvexTok attains tokenization costs within 1% of the theoretical OTB certificate and consistently improves bits-per-byte (BpB) metrics, with marginal improvements in deep LMs’ downstream performance (Tempus et al., 21 May 2026).

5. Relationship to Scaling Laws and Compute-Optimal Allocation

Scaling law studies have extended OTB's role to quantification of tokenization impact on compute efficiency. Instead of measuring data in tokens, recent work demonstrates the criticality of the byte/parameter ($t_1,\dots,t_K \in V$2) ratio; the OTB corresponds to settings that minimize loss for fixed bytes and model parameters. The optimum compression rate $t_1,\dots,t_K \in V$3 (bytes per token) is non-monotonic and decreases slowly with compute: $t_1,\dots,t_K \in V$4 with constants fitted from empirical scaling (e.g., $t_1,\dots,t_K \in V$5 for English) (Limisiewicz et al., 2 May 2026). This supports persistent adaptation of tokenization schemes as models and data scale, with OTB acting as a critical benchmarking and guiding tool.

6. Practical Guidelines and Implications

OTB motivates several recommendations:

• Evaluate tokenization systems against the OTB to quantify wasted tokens and guide optimization.
• Target byte/parameter ratios ($t_1,\dots,t_K \in V$6 for English) and optimal compression rates that emerge from OTB-guided scaling law analyses.
• When possible, derive or approximate the OTB via dynamic programming, LP relaxations, or per-token baselines for RL, to maximize compression or minimize variance, depending on task.
• For multilingual or morphologically complex languages, use OTB to reveal and correct inefficiencies inherent in standard greedy tokenization pipelines.

A plausible implication is that OTB will remain a central tool both for refining tokenization and for evaluating downstream impacts of segmentation algorithms, particularly as foundation models expand to more diverse tasks, languages, and compute regimes.