Inside the LLM Word Factory

Abstract: Transformer LLMs process input provided as subword fragments, but natural language semantics usually rely on word-level concepts. Detokenization is the process where models reconcile these two facts, aggregating subwords into word-level representations through their computation. Prior work has found that this takes place mostly in early-to-middle layers, but so far the exact mechanics of the process have not been pinned down. We venture deep into detokenization using activation patching in controlled paired experiments that isolate the contribution of different model components, localizing English detokenization in Llama2-7B to a two-stage process at Layer 1. Attention transmits a token-specific signal from nonfinal subwords, using sequential relays if necessary, while the MLP composes it with the local embedding. This two-stage structure generalizes to twelve models from eight families, but the depth over which it takes place depends on the flavor of positional encoding: RoPE-based models detokenize over 1 to 5 layers, while learned-absolute models take 5 to 10. Finally, we provide a probe for determining the success of the detokenization process based on early-layer activations alone, performing at 0.94-0.97 AUROC depending on the amount of context.

• The paper demonstrates a two-stage mechanism where attention and MLP modules sequentially combine subword tokens into unified word representations.
• It uses activation patching and cosine similarity to quantify detokenization success, showing up to a 53% reduction in the canonicity gap.
• The study reveals that detokenization depth varies with positional encoding schemes across architectures, informing future model and tokenizer designs.

Mechanistic Analysis of Word-Level Detokenization in Transformer LLMs

Introduction

This work presents a comprehensive mechanistic analysis of how transformer-based LLMs, across diverse architectures, reconstruct word-level semantic representations from fragmented subword token inputs. The study defines detokenization as the aggregation of subword representations into unified word-level embeddings, a process that is essential given the prevalence of subword tokenization yet underexplored in terms of precise circuit-level mechanism. Previous literature has described detokenization as primarily an early-layer operation involving both attention and MLP modules, but lacked causal localization and clarity regarding architecture-specific depth or cross-model consistency.

Experimental Design and Metric

Detokenization is evaluated via controlled paired experiments, utilizing English words that can be encoded in both canonical (single-token) and artificially split (multi-token) forms. By comparing the representation produced at the last subword position of a split word to its canonical (single-token) counterpart using cosine similarity (“canonicity”) in layer $n-2$, the study obtains a robust, behaviorally-valid metric for assessing detokenization success. The analysis is restricted to words present as single tokens in the vocabulary, enabling accurate ground-truth references but excluding inherently multi-token or out-of-vocabulary words.

The setup involves constructing contrastive pairs of split words, controlling for either the first or last subword token. In the Last-Shared-Token (LST) setting, the second token is matched across the pair, isolating the influence of the first token through the attention pathway. In the First-Shared-Token (FST) setting, the first token is matched, isolating the downstream composition effect mediated by the MLP.

Figure 1: Activation patching experiments localize primary detokenization contributions to Layer 1 attention in LST (a), Layer 1 MLP in FST (b), and show a smooth effect of scaling Layer 1 MLP output at position 2 on canonicity (c).

Causal Localization: The Two-Stage Mechanism

Through systematic activation patching, the study causally localizes detokenization in Llama2-7B to a two-stage process concentrated at Layer 1. In LST, patching Layer 1 attention output at the second position closes 53% of the canonicity gap, implying that attention mediates a critical directional signal from the preceding token. In FST, this intervention is uninformative; instead, patching the Layer 1 MLP output yields a 53% gap closure, establishing that MLP composition at the last position is decisive when attention-transmittable context is held constant.

Decomposition of attention reveals that a subset of Layer 1 heads (notably heads 24, 27, 28) funnel a weak but crucial, token-dependent signal derived from the value at position 1, oriented towards the canonical word embedding. This effect is confirmed to be specific and not a generic “composition success” flag, since performance diverges in FST despite aligned attention outputs. The MLP, rather than acting as a discrete key-value switch, performs a smooth, continuous transformation, as shown by the monotonic increase in canonicity when scaling its output at position 2.

Detokenization Depth and Sequential Relay

Expanding to words with more than two subwords, the required depth for successful detokenization increases with word length but sublinearly. The layerwise patching approach demonstrates that each added intermediate token increases depth by less than a full layer after the first extension (2→3 tokens adds ∼3 layers; additional tokens add <1 layer each). This suggests parallel relay via intermediate positions, as corrupting intermediate residuals causes additional canonicity drops, especially when multiple positions are disrupted jointly, demonstrating redundancy and partial compensation in the relay dynamics.

Figure 2: Impact of intermediate position corruption on final canonicity. Single-position corruption results in localized drops, while joint corruption deepens the effect and reveals redundancy.

Cross-Architectural Replicability and Regimes

The two-stage mechanism (early attention relay and subsequent MLP composition) generalizes across twelve models from eight distinct architectures, spanning RoPE, ALiBi, and learned-absolute positional encoding schemes. However, the depth over which detokenization unfolds is architecture-dependent. RoPE-based models concentrate the process within 1–5 layers, while learned-absolute models distribute it over 5–10 layers. ALiBi-based models (e.g., BLOOM) display intermediate or scaling-dependent behaviors. The position encoding architecture is the primary determinant of detokenization depth; width, parameter count, tokenizer type, and corpus are not explanatory.

Figure 3: Residual patching shows detokenization is completed within 1–5 layers for RoPE models (blue), 5–10 layers for learned-absolute (green), and intermediate for ALiBi (orange).

Early-Layer Predictivity and Probing

Given the causal evidence for early-layer localization, the study investigates whether detokenization success can be predicted directly from early activations. Using a class-mean-difference probe trained on resid_post at the determined “mechanistic depth,” the authors achieve AUROC scores of 0.94 (isolated, contextless input) and 0.97 (in-sentence context) on Llama2-7B, with similar outcomes across other architectures. The probe direction learned from isolated controlled data generalizes to context-rich activations with minimal AUROC degradation, reinforcing the transferability and robustness of early-layer signals.

Figure 4: Layerwise AUROC visualization of the linear probe for distinguishing detokenization success, indicating discriminability peaks at mechanistic depths.

Implications and Future Directions

The universality and linear predictability of the mechanistic circuit for detokenization suggest both architectural invariants and practical applications. Architecturally, detokenization is realized as an early, compositional computation—rapid in models with relative position access (RoPE/ALiBi), slower in those using learned position embeddings. Practically, this enables early detection of tokenization-induced representation failures, which can inform tokenizer design, model introspection, and dynamic inference-time interventions. Future work may address naturally multi-token words and extend mechanistic analysis to non-English tokenization spaces or more deeply contextual architectures.

Conclusion

Through rigorous, controlled causal analysis, this study delineates a universal, early-layer two-stage mechanism by which transformers internally compose subword fragments into canonical word representations. This process, mediated first by attention and then by an MLP-driven transformation, is consistent across architectures and adapts its circuit depth based on positional encoding regime, not model scale. The outcome of detokenization is linearly accessible in early activations, offering novel opportunities for mechanistically grounded probing, failure detection, and architectural understanding. Limitations include the restriction to words present as single tokens and inability to generalize to inherently multi-token lexical items, motivating future extensions in unsupervised or alternative reference frameworks.