Perceptual Group Tokenizer (PGT)

• Perceptual Group Tokenizer (PGT) is a vocabulary-free embedding approach that leverages character trigram patterns to produce sparse, high-dimensional representations.
• It employs hash-based sparse activations to directly embed words, dramatically reducing embedding layer sizes while matching or outperforming traditional tokenizers like BPE.
• Empirical findings indicate over 85% parameter reduction, enhanced cross-lingual robustness, and lower GPU memory footprints, leading to more stable training curves.

Sparse Trigram Activations (T-FREE) are a tokenizer-free representation for LLMs that replaces subword vocabularies with high-dimensional, extremely sparse codes defined over character trigrams. T-FREE directly embeds words via sparsely activated vectors without referring to a predefined vocabulary or corpus. By exploiting character-level morphological similarities, T-FREE achieves strong compression of embedding layers, with empirical results demonstrating competitive performance relative to traditional Unigram and BPE tokenization while reducing embedding-layer parameter count by over 85%. Further advantages include improved cross-lingual transfer and uniformly distributed gradients during training, resulting in stable loss curves and lower GPU memory footprints (Deiseroth et al., 27 Jun 2024).

1. Motivation and Problem Context

Developments in tokenization for LLMs have stagnated, with existing methods like Byte Pair Encoding (BPE) and Unigram encountering several limitations. These include computational overhead, inefficient vocabulary utilization, corpus bias that reduces efficacy for underrepresented languages, and unnecessarily large embedding and prediction head layers. By construction, fixed vocabularies are less adaptive to morphological variation and multi-lingual settings. T-FREE addresses these issues by removing vocabulary dependence entirely, embedding words as sparse codes derived from sequences of character trigrams (Deiseroth et al., 27 Jun 2024).

2. Mathematical Formulation and Encoding Pipeline

Let $w$ denote a word or symbol, including leading and trailing whitespace markers for boundary representation. A sliding window extracts an ordered sequence of $n=|w|+1$ overlapping character trigrams $[t_1, t_2, ..., t_n]$. For each trigram $t_\ell$, $m$ independent hashes are computed and reduced modulo a fixed vocabulary size $v$ (e.g., $v=8\,000$):

$$\bigl(h_{\ell,1}, ..., h_{\ell,m}\bigr) = \bigl(\mathrm{hash}(t_\ell \Vert 1), ..., \mathrm{hash}(t_\ell \Vert m)\bigr)$$

$$\mathcal I(w) = \{ h_{\ell,i} \bmod v : \ell=1,...,n, i=1,...,m \}$$

The sparse binary activation vector $z(w) \in \{0,1\}^v$ is defined element-wise as $z_j = 1$ if $j \in \mathcal I(w)$, otherwise $z_j = 0$. The resulting sparsity constraint is $\|z\|_0 \leq n m$. No $\ell_1$ or $\ell_2$ normalization is performed during encoding. Let $G \in \mathbb R^{h \times v}$ be the shared embedding matrix ($h$, e.g., $3\,072$). The final word embedding is given by summing the active columns:

$$E(w) = G z = \sum_{j=1}^v z_j G_{:,j} \in \mathbb R^h$$

Empirical configuration uses $v = 8\,000$, $h = 3\,072$, and typical $m$ values (not specified), yielding a compact representation.

Encoding proceeds as follows:

1. Raw text is split on non-alphanumeric boundaries, treating each word or symbol as a token.
2. Tokens are padded at both ends with a whitespace marker, then decomposed into overlapping trigrams.
3. For each trigram and for each of $m$ hash slots, a hash value is computed, reduced modulo $v$, and the corresponding index in a length-$v$ binary vector is set to 1.
4. The $m n$ columns in $G$ corresponding to nonzero entries in $z$ are summed to yield $E(w)$.

3. Integration with LLMs

T-FREE modifies only the input embedding layer and the language modeling (LM) head in a Transformer architecture. All self-attention and MLP blocks remain identical. The embedding and head layers' shapes transition from $\{\text{vocab} \times h\}$ (where vocabularies for BPE/Unigram are typically $64\,000$ or more) to $\{v \times h\}$, with $v$ much smaller for T-FREE.

During pre-training, the next-word prediction head outputs a logit vector $\widehat y \in \mathbb R^v$. Rather than a single-label softmax, multi-label binary cross-entropy (MLBCE) is used:

$$\mathcal{L}_{\mathrm{MLBCE}} = -\sum_{j=1}^v\Bigl[z_j \log\sigma(\widehat y_j) + (1-z_j)\log(1-\sigma(\widehat y_j))\Bigr]$$

No explicit regularizer on the $\ell_0$ norm is necessary due to fixed hash bucketed sparsity.

Decoding involves a pre-compiled dictionary matrix $D \in \{0,1\}^{d \times v}$, where each row is the sparse pattern $z(w)$ for the $d$ most frequent words. Given predicted logits, the dictionary is multiplied by $\sigma(\widehat y)$, and the softmax is computed over the resulting scores to select the output. This is efficiently implemented using sparse-dense kernels because $n m \ll h$.

4. Parameter and Memory Efficiency

The parameter count in both embedding and LM head layers is drastically reduced. With $h=3\,072$ and $v=8\,000$ for T-FREE—versus $64\,000$ for BPE—memory requirements are reduced to $12.5\%$, i.e., $87.5\%$ fewer parameters:

$\frac{8\,000}{64\,000} = 12.5\%$

This translates to savings of roughly $$(64\,\text{k} - 8\,\text{k}) \times 3\,\text{k} \approx 172$$ million parameters per layer. Peak GPU memory footprint is correspondingly reduced (e.g., $38$ GB vs $68$ GB for a $1$B-parameter model), with more stable training curves and fewer spikes in loss due to fixed hashing and uniform gradient updates.

5. Empirical Performance and Cross-Lingual Robustness

On 18 zero- and few-shot downstream benchmarks, $1$B-parameter models with T-FREE ($v = 8$k) match or surpass dense $64$k-token Unigram baselines, despite having $20\%$ fewer parameters overall. Fertility—average tokens per word—declines from $1.28 \rightarrow 1.16$ in English with T-FREE, indicating more efficient segmentation. Robustness is observed across German, Russian, Vietnamese, and Arabic, where classical tokenizers experience degraded performance. In continual pre-training for English-to-German transfer with a $3$B-parameter model, T-FREE narrows the German performance gap by $5\%$ after 20k steps, in contrast to minimal improvement with the standard tokenizer (Deiseroth et al., 27 Jun 2024).

6. Limitations and Future Directions

T-FREE's reliance on pooled sparse sums makes encoding very long words susceptible to underweighting, potentially diminishing representation quality. Decoding necessitates a pre-compiled dictionary for the $d$ most frequent words, which may limit arbitrarily open-vocabulary generation. Proposed future research directions include learning hash functions, introducing byte-fallback for rare Unicode blocks, and extending T-FREE to domains such as programming languages or morphologically rich scripts. These areas may address current challenges and extend the applicability of sparse trigram activations beyond natural language (Deiseroth et al., 27 Jun 2024).