---
title: 'GreedTok & Partition Cover: Tokenization Optimization'
url: https://www.emergentmind.com/topics/greedtok-and-partition-cover
type: topic
---

# GreedTok & Partition Cover: Tokenization Optimization

GreedTok is a polynomial-time greedy algorithm for the partition cover approach to tokenization, formulated as an optimization problem that generalizes traditional NLP tokenization objectives. The partition cover perspective connects tokenization to the weighted maximum coverage problem and, in a broader context, links with the Partition Set Cover problem and its approximation-theoretic landscape. Both areas share foundations in combinatorial optimization and admit rigorous analysis in terms of computational hardness and approximation guarantees.

## 1. Partition Cover Formulation of Tokenization

Tokenization can be framed as selecting a set of tokens from a vocabulary to minimize the total number of segments (tokens) required to represent a corpus. Let $\Sigma$ be the finite alphabet and $W \subseteq \Sigma^+$ the set of words in the corpus, each with multiplicity $count(w)$. Singleton tokens $B = \{(a) : a \in \Sigma\}$ are always available. The designer may select up to $k$ additional tokens $S \subseteq T$, where $T$ is a candidate set of substrings (typically, length at least 2).

For each word $w$ and token-set $S \cup B$, define:
\[
partition(w, S \cup B) = \min\{\ell : \exists t_1,\dots, t_\ell \in S \cup B \text{ with } t_1\|\cdots\|t_\ell = w\}
\]
where “$\|$” denotes concatenation.

Equivalently, one may define a covering score:
\[
cover(w, S) = \max\{\#\text{adjacent pairs of characters in } w \text{ merged by tokens in } S \}
\]
with the identity: $|w|+1 = partition(w, S \cup B) + cover(w, S)$.

Objective formulations:
- **Compression (minimize tokens):**
  \[
  \min_{S \subseteq T} \sum_{w \in W} count(w) \cdot partition(w, S \cup B) \quad \text{s.t.} \quad |S| \le k.
  \]
- **Cover maximization:** (equivalent after adjusting by a constant)
  \[
  \max_{S \subseteq T} \sum_{w \in W} count(w) \cdot cover(w, S) \quad \text{s.t.} \quad |S| \le k.
  \]

A mixed-integer programming (MIP) formulation for the cover version introduces binaries for token selection ($x_t$) and pairwise coverage ($m^w_{i,i+1}$ and $m^{w,t}_{i,i+1}$), along with “no-overlap” constraints to enforce valid token segmentations [2501.06246].

## 2. Computational Hardness and Relation to Classical Cover Problems

The decision version of the partition cover tokenization problem is NP-hard. The hardness proof uses a reduction from Vertex Cover: an instance $(G, k)$ of vertex cover is encoded into a tokenization instance where each graph edge becomes a word, and tokens correspond to covering graph vertices. Reaching a partition size threshold corresponds exactly to covering all edges with at most $k$ vertices, establishing NP-hardness [2501.06246].

This formulation admits a direct relaxation to the weighted maximum coverage (WMC) problem: given elements (here, positions in words) with weights and eligible covering sets (candidate tokens), select up to $k$ sets to maximize covered weight. In this relaxation, coverage overlaps (i.e., invalid token segmentations) are allowed, and the standard greedy algorithm achieves a $(1-1/e)$-approximation [2501.06246].

## 3. GreedTok Algorithm and GreedWMC

GreedTok sequentially selects tokens to greedily maximize the increment in the cover objective, entirely respecting overlap constraints necessary for valid tokenizations. For each round up to $k$, it computes the marginal gain $\Delta(t)$—the summed counts of newly covered, so-far-uncovered segments—for each candidate $t \in T \setminus S$. The maximally beneficial token is appended to the token set, and the cover state is updated. The process is efficient: with precomputation of token occurrences, the total time is $O(|T| \cdot k \cdot \sum_{w} |w|)$ and space is $O(|T| + \sum_w |w|)$ [2501.06246].

GreedWMC applies standard greedy weighted maximum coverage—dropping “no-overlap” constraints—which guarantees a $(1-1/e)$-approximation for the relaxed covering problem. Implementation and complexity are analogous to GreedTok. GreedWMC may, however, generate invalid tokenizations due to overlapping token selection [2501.06246].

## 4. Partition Set Cover: Connections and Generalized Covering

The Partition Set Cover problem generalizes standard set cover by partitioning the ground set $U$ into $r$ color classes $C_1, ..., C_r$ (possibly overlapping), with demand $k_t$ for each class. The goal is to select a minimum-weighted collection of sets so that for each $t$, at least $k_t$ elements from $C_t$ are covered.

The canonical integer program introduces indicator variables $x_i$ (for sets) and $z_j$ (for elements); its “natural” LP relaxation can exhibit a $\Theta(\sqrt{n})$ gap, thus a stronger LP plus knapsack-cover inequalities is employed. The approximation guarantee is stated in terms of $\beta$, the best-known rounding-factor for induced set cover projections. For instance, geometric settings (disks/halfspaces in $\mathbb{R}^2$) yield $\beta=O(1)$, while other low-VC-dimension systems also achieve sublogarithmic $\beta$ [1809.06506].

The main result is a randomized two-phase LP-rounding algorithm yielding an $O(\beta + \log r)$-approximation: heavy elements are covered by a $\beta$-approximate algorithm; light elements are covered through $O(\log r)$ randomized rounds, ensuring each class’ demand is met with probability at least $1/2$, and the total cost is $O(\beta + \log r)$ times the LP optimum [1809.06506]. This bound is essentially tight: $\Omega(\log r)$-hardness holds even for simple special cases.

## 5. Empirical Evaluation and Comparative Performance

Empirical studies compare GreedTok, GreedWMC, and Byte-Pair Encoding (BPE) on four English corpora (UN General Debate, arXiv abstracts, Wikipedia, PubMed) under equal vocabulary size constraints. Metrics include:
- **Average tokens per word (TPW):**
  \[
  \mathrm{TPW}(S) = \frac{1}{N}\sum_{w} count(w) \, partition(w,S\cup B)
  \]
- **Covering score:**
  \[
  \mathrm{Cover}(S) = \sum_{w} count(w) \, cover(w,S)
  \]

Notable findings include:
- To reach the same TPW target, GreedTok uses on average 13% fewer tokens than BPE.
- At fixed vocabulary size $k$, GreedTok achieves 3–5% lower TPW versus BPE; its covering scores are within 90% of GreedWMC (for large $k$), consistent with near-optimality to the relaxed problem.

| Vocabulary $k$ | BPE TPW | GreedTok TPW | GreedWMC TPW |
|:---------------|--------:|-------------:|-------------:|
| 3500           |   2.189 | 2.120 (–3.2%)| 2.105 (–3.8%)|
| 5000           |   1.992 | 1.905 (–4.4%)| 1.892 (–5.0%)|

Similar reductions in TPW and covering efficiency hold across other datasets [2501.06246].

## 6. Comparative Analysis and Algorithmic Insights

BPE is efficient ($O(k\sum|w|)$) but limited to pairwise merges, often selecting suboptimal tokens for global coverage. GreedWMC achieves the $(1-1/e)$ theoretical guarantee for the relaxed problem but may produce segmentations violating the non-overlap constraint. GreedTok enforces exact partition constraints and, while only modestly slower, consistently outperforms BPE by 3–5% and closely tracks the optimal relaxed cover. The partition-cover MIP from which GreedTok derives allows for injective customization (e.g., domain-specific tokens or corpus-parallelization), which is unattainable for bottom-up merge algorithms [2501.06246].

From the set covering perspective, greedy algorithms for Partition Set Cover (e.g., extensions of Slavík’s PSC-greedy) achieve $O(\log(\sum k_t)) \approx O(\log n + \log r)$, but the LP-rounding yields tighter $O(\beta+\log r)$ bounds, especially when $\beta \ll \log n$. Naive greedy for $f$-frequency combinatorial systems gives $O(f\log r)$, yet LP-rounding achieves $O(f+\log r)$. Ignoring cross-class coverage synergies leads to an $\Omega(r)$ factor loss, whereas the advanced covering methods handle all class demands collectively [1809.06506].

## 7. Extensions, Limitations, and Outlook

The partition cover framework extends to related facility location objectives and outlier-minimum covering, maintaining $O(\log r)$-approximation via analogous decomposition into “heavy” and “light” client subsystems. Hardness persists: even geometric and metric variants disallow $o(\log r)$-approximations unless P=NP [1809.06506].

A plausible implication is that the merge-free, coverage-driven approach exemplified by GreedTok and its theoretical kin provides both increased flexibility and superior practical compression—subject to computational budget and candidate token enumeration feasibility. The ability to inject custom subtoken vocabularies and adapt to partitioned data suggests that these methodologies can subsume prior heuristic tokenization methods, offering a unified optimization-centric view.

Source: https://www.emergentmind.com/topics/greedtok-and-partition-cover