GreedTok & Partition Cover: Tokenization Optimization

• GreedTok is a polynomial-time greedy algorithm that formulates tokenization as a partition cover optimization problem linked to weighted maximum coverage.
• It leverages NP-hard reductions and approximation guarantees to improve tokenization efficiency compared to relaxed greedy methods like GreedWMC and BPE.
• Empirical evaluations show GreedTok reduces average tokens per word by 3–5% over BPE, highlighting its practical performance and optimization advantages.

GreedTok is a polynomial-time greedy algorithm for the partition cover approach to tokenization, formulated as an optimization problem that generalizes traditional NLP tokenization objectives. The partition cover perspective connects tokenization to the weighted maximum coverage problem and, in a broader context, links with the Partition Set Cover problem and its approximation-theoretic landscape. Both areas share foundations in combinatorial optimization and admit rigorous analysis in terms of computational hardness and approximation guarantees.

1. Partition Cover Formulation of Tokenization

Tokenization can be framed as selecting a set of tokens from a vocabulary to minimize the total number of segments (tokens) required to represent a corpus. Let $\Sigma$ be the finite alphabet and $W \subseteq \Sigma^+$ the set of words in the corpus, each with multiplicity $count(w)$. Singleton tokens $B = \{(a) : a \in \Sigma\}$ are always available. The designer may select up to $k$ additional tokens $S \subseteq T$, where $T$ is a candidate set of substrings (typically, length at least 2).

For each word $w$ and token-set $S \cup B$, define: $$partition(w, S \cup B) = \min\{\ell : \exists t_1,\dots, t_\ell \in S \cup B \text{ with } t_1\|\cdots\|t_\ell = w\}$$ where “$W \subseteq \Sigma^+$0” denotes concatenation.

Equivalently, one may define a covering score: $W \subseteq \Sigma^+$1 with the identity: $W \subseteq \Sigma^+$2.

Objective formulations:

• Compression (minimize tokens):

$W \subseteq \Sigma^+$3

• Cover maximization: (equivalent after adjusting by a constant)

$W \subseteq \Sigma^+$4

A mixed-integer programming (MIP) formulation for the cover version introduces binaries for token selection ($W \subseteq \Sigma^+$5) and pairwise coverage ($W \subseteq \Sigma^+$6 and $W \subseteq \Sigma^+$7), along with “no-overlap” constraints to enforce valid token segmentations (Lim et al., 8 Jan 2025).

2. Computational Hardness and Relation to Classical Cover Problems

The decision version of the partition cover tokenization problem is NP-hard. The hardness proof uses a reduction from Vertex Cover: an instance $W \subseteq \Sigma^+$8 of vertex cover is encoded into a tokenization instance where each graph edge becomes a word, and tokens correspond to covering graph vertices. Reaching a partition size threshold corresponds exactly to covering all edges with at most $W \subseteq \Sigma^+$9 vertices, establishing NP-hardness (Lim et al., 8 Jan 2025).

This formulation admits a direct relaxation to the weighted maximum coverage (WMC) problem: given elements (here, positions in words) with weights and eligible covering sets (candidate tokens), select up to $count(w)$0 sets to maximize covered weight. In this relaxation, coverage overlaps (i.e., invalid token segmentations) are allowed, and the standard greedy algorithm achieves a $count(w)$1-approximation (Lim et al., 8 Jan 2025).

3. GreedTok Algorithm and GreedWMC

GreedTok sequentially selects tokens to greedily maximize the increment in the cover objective, entirely respecting overlap constraints necessary for valid tokenizations. For each round up to $count(w)$2, it computes the marginal gain $count(w)$3—the summed counts of newly covered, so-far-uncovered segments—for each candidate $count(w)$4. The maximally beneficial token is appended to the token set, and the cover state is updated. The process is efficient: with precomputation of token occurrences, the total time is $count(w)$5 and space is $count(w)$6 (Lim et al., 8 Jan 2025).

GreedWMC applies standard greedy weighted maximum coverage—dropping “no-overlap” constraints—which guarantees a $count(w)$7-approximation for the relaxed covering problem. Implementation and complexity are analogous to GreedTok. GreedWMC may, however, generate invalid tokenizations due to overlapping token selection (Lim et al., 8 Jan 2025).

4. Partition Set Cover: Connections and Generalized Covering

The Partition Set Cover problem generalizes standard set cover by partitioning the ground set $count(w)$8 into $count(w)$9 color classes $B = \{(a) : a \in \Sigma\}$0 (possibly overlapping), with demand $B = \{(a) : a \in \Sigma\}$1 for each class. The goal is to select a minimum-weighted collection of sets so that for each $B = \{(a) : a \in \Sigma\}$2, at least $B = \{(a) : a \in \Sigma\}$3 elements from $B = \{(a) : a \in \Sigma\}$4 are covered.

The canonical integer program introduces indicator variables $B = \{(a) : a \in \Sigma\}$5 (for sets) and $B = \{(a) : a \in \Sigma\}$6 (for elements); its “natural” LP relaxation can exhibit a $B = \{(a) : a \in \Sigma\}$7 gap, thus a stronger LP plus knapsack-cover inequalities is employed. The approximation guarantee is stated in terms of $B = \{(a) : a \in \Sigma\}$8, the best-known rounding-factor for induced set cover projections. For instance, geometric settings (disks/halfspaces in $B = \{(a) : a \in \Sigma\}$9) yield $k$0, while other low-VC-dimension systems also achieve sublogarithmic $k$1 (Inamdar et al., 2018).

The main result is a randomized two-phase LP-rounding algorithm yielding an $k$2-approximation: heavy elements are covered by a $k$3-approximate algorithm; light elements are covered through $k$4 randomized rounds, ensuring each class’ demand is met with probability at least $k$5, and the total cost is $k$6 times the LP optimum (Inamdar et al., 2018). This bound is essentially tight: $k$7-hardness holds even for simple special cases.

5. Empirical Evaluation and Comparative Performance

Empirical studies compare GreedTok, GreedWMC, and Byte-Pair Encoding (BPE) on four English corpora (UN General Debate, arXiv abstracts, Wikipedia, PubMed) under equal vocabulary size constraints. Metrics include:

• Average tokens per word (TPW):

$k$8

• Covering score:

$k$9

Notable findings include:

• To reach the same TPW target, GreedTok uses on average 13% fewer tokens than BPE.
• At fixed vocabulary size $S \subseteq T$0, GreedTok achieves 3–5% lower TPW versus BPE; its covering scores are within 90% of GreedWMC (for large $S \subseteq T$1), consistent with near-optimality to the relaxed problem.

Vocabulary $S \subseteq T$2	BPE TPW	GreedTok TPW	GreedWMC TPW
3500	2.189	2.120 (–3.2%)	2.105 (–3.8%)
5000	1.992	1.905 (–4.4%)	1.892 (–5.0%)

Similar reductions in TPW and covering efficiency hold across other datasets (Lim et al., 8 Jan 2025).

6. Comparative Analysis and Algorithmic Insights

BPE is efficient ($S \subseteq T$3) but limited to pairwise merges, often selecting suboptimal tokens for global coverage. GreedWMC achieves the $S \subseteq T$4 theoretical guarantee for the relaxed problem but may produce segmentations violating the non-overlap constraint. GreedTok enforces exact partition constraints and, while only modestly slower, consistently outperforms BPE by 3–5% and closely tracks the optimal relaxed cover. The partition-cover MIP from which GreedTok derives allows for injective customization (e.g., domain-specific tokens or corpus-parallelization), which is unattainable for bottom-up merge algorithms (Lim et al., 8 Jan 2025).

From the set covering perspective, greedy algorithms for Partition Set Cover (e.g., extensions of Slavík’s PSC-greedy) achieve $S \subseteq T$5, but the LP-rounding yields tighter $S \subseteq T$6 bounds, especially when $S \subseteq T$7. Naive greedy for $S \subseteq T$8-frequency combinatorial systems gives $S \subseteq T$9, yet LP-rounding achieves $T$0. Ignoring cross-class coverage synergies leads to an $T$1 factor loss, whereas the advanced covering methods handle all class demands collectively (Inamdar et al., 2018).

7. Extensions, Limitations, and Outlook

The partition cover framework extends to related facility location objectives and outlier-minimum covering, maintaining $T$2-approximation via analogous decomposition into “heavy” and “light” client subsystems. Hardness persists: even geometric and metric variants disallow $T$3-approximations unless P=NP (Inamdar et al., 2018).

A plausible implication is that the merge-free, coverage-driven approach exemplified by GreedTok and its theoretical kin provides both increased flexibility and superior practical compression—subject to computational budget and candidate token enumeration feasibility. The ability to inject custom subtoken vocabularies and adapt to partitioned data suggests that these methodologies can subsume prior heuristic tokenization methods, offering a unified optimization-centric view.