Character-level Tokenization

Updated 26 May 2026

Character-level tokenization is a method that maps each character in a text to a unique token, enabling fine-grained, position-aware processing essential for applications like spelling correction and code manipulation.
Recent adaptations such as character-level BPE and dynamic boundary predictors balance the trade-offs between detailed tokenization and computational efficiency, preserving local token boundaries.
Empirical studies demonstrate that while character-level tokenization enhances accuracy in alignment-sensitive tasks, it also introduces computational challenges due to increased sequence lengths.

Character-level tokenization refers to the process of decomposing textual (or, in related modalities, audio) input into individual characters, treating each character as a distinct atomic token, rather than grouping characters into words or subword units. This approach preserves a one-to-one correspondence between input characters and tokens, enabling precise string position tracking and manipulation, and is contrasted with standard subword tokenization methods such as Byte-Pair Encoding (BPE) or WordPiece, which merge multiple characters into a single token to maximize sequence compression (Xu et al., 2024, Uzan et al., 4 Aug 2025, Li et al., 2024, Fleshman et al., 2023, Bommarito et al., 21 Mar 2025, Sun et al., 2023, Libera et al., 30 Jan 2026). Character-level tokenization is critical for tasks demanding fine-grained, position-aware reasoning, but comes with significant computational and modeling tradeoffs. Recent research has systematically examined its limitations, architectural adaptations, and the mechanisms by which LLMs acquire or reconstruct character information under various tokenization regimes.

1. Formalization and Motivations

Character-level tokenization can be defined as a mapping from a string $S = c_1c_2 \ldots c_n$ over a character vocabulary $\Sigma$ to a length- $n$ token sequence $T = [t_1, \ldots, t_n]$ where $t_i \equiv c_i$ (Li et al., 2024, Bommarito et al., 21 Mar 2025, Sun et al., 2023, Ribeiro et al., 2018). This is in contrast to subword tokenization methods which, given the same string, would produce a shorter sequence of token IDs encoding multi-character segments (e.g., morphemes, words).

Key motivations for character-level tokenization include:

Alignment Precision: Essential for tasks such as spelling correction in Chinese, where character-level constraints (equal length and phonetic similarity) must be maintained (Li et al., 2024, Xu et al., 2024).
Manipulability: Direct addressability for operations such as insertion, deletion, or substitution at arbitrary character positions facilitates robust text and code manipulation (Xiong et al., 12 Feb 2025, Bommarito et al., 21 Mar 2025).
Robustness to Tokenization Errors: In adversarial or obfuscated settings, character-level tokenization ensures every input character remains accessible to the model (Zhuo et al., 16 Oct 2025, Yang et al., 11 Mar 2026).
Linguistic Alignment: In logographic languages (Chinese, Japanese), characters are natural semantic units, and treating them as tokens minimizes ambiguities caused by subword merges (Li et al., 2024, Sun et al., 2023, Si et al., 2021).

2. Character-level Tokenization in Practice

Several implementation regimes and specialized adaptations have emerged:

Character Tokenizers: Each unique character, including punctuation, whitespace, and control symbols, is assigned a dedicated token type (Li et al., 2024, Bommarito et al., 21 Mar 2025, Sun et al., 2023, Ribeiro et al., 2018). In Chinese, this yields practical vocabularies sized 7–8k.
Character-level BPE Tokenizers: KL3M tokenizers constrain merge operations so that no token exceeds $L_{\text{max}}$ characters, with typical settings $L_{\text{max}}=3$ or $4$ (Bommarito et al., 21 Mar 2025). This preserves nearly local token boundaries while allowing limited multi-character compression, crucial for alignment in OCR or noisy-text correction.
Character-aligned Speech Tokenizers: DyCAST maps audio frames directly to variably-chunked, character-aligned tokens, ensuring each speech segment corresponds closely to a character-level linguistic unit, which supports efficient ASR and TTS (Libera et al., 30 Jan 2026).
Architectural Modifications: Block-causal cross-attention mechanisms furnish a pathway from subword token embeddings to explicit character-level representations, vastly boosting character-level task performance (Cosma et al., 20 May 2025). Bridge architectures such as the Backpack (Sun et al., 2023) natively accommodate per-character sense representations.
Hybrid and Dynamic Models: Toucan employs a boundary predictor that learns, end-to-end, when spans of characters should be grouped into token-level vectors, combining efficiency with full character-level fidelity (Fleshman et al., 2023).

3. Empirical Findings: Character-level Task Performance

Comprehensive benchmarks such as CharBench (Uzan et al., 4 Aug 2025) and synthetic studies (Cosma et al., 20 May 2025) reveal:

Subword tokenization sharply impairs performance on character-indexing tasks, especially when the target character lies within a long multi-character token. CharBench shows state-of-the-art LLMs attain only 43–77% on counting and as low as 32% on positional tasks.
Token Length Effects: For positional tasks, accuracy is negatively correlated with the length of the subword token containing the queried character (mean $r\approx -0.2$ to $-0.23$ ). Counting tasks are much less sensitive to tokenization but degrade as the size of the count grows (Uzan et al., 4 Aug 2025, Sato et al., 5 Feb 2026).
Task Simplification: For strictly character-level tasks, character tokenization enables direct one-to-one mapping, making sequence-to-sequence objectives simple replications or substitutions, dramatically simplifying the learning problem (Li et al., 2024, Xiong et al., 12 Feb 2025).
Robustness to Character-fractured Input: Even when input is forcibly fragmented (e.g., by injecting invisible characters between every typed character), many contemporary LLMs maintain high accuracy through word recovery mechanisms, as shown in both empirical and mechanistic studies (Zhuo et al., 16 Oct 2025, Yang et al., 11 Mar 2026).

4. Mechanisms for Acquiring Character-level Information

Despite not being explicitly trained at the character level, modern LLMs demonstrate the ability to reconstruct character information under certain conditions:

Word Recovery via In-group Attention: When supplied character-fragmented input, early transformer layers aggregate characters belonging to the same canonical token (as would be produced by the default tokenizer) through in-group self-attention, enabling downstream layers to recover word/subword identities nearly perfectly. Causal removal of token direction subspaces or masking in-group attention leads to drastic drops in task performance under character-level tokenization (Yang et al., 11 Mar 2026).
Subword Embedding Probes: Probing studies show that subword token embeddings robustly encode character membership through systematic morphosyntactic correlations and tokenization variability. Larger models yield higher F1 scores in character probes, even across scripts (Sato et al., 5 Feb 2026, Kaushal et al., 2022).
Emergence via Percolation: Character-level abilities emerge late and suddenly in pretraining, requiring the model to reconstruct token-to-character mappings—a process theoretically captured by percolation models in which "properties" (characters) only become reliably linked to "concepts" (subword tokens) after a critical mass of learning (Cosma et al., 20 May 2025).

5. Architectures and Training Schemes

To circumvent or mitigate the limitations of subword tokenization for character-level tasks, several specialized architectural and training strategies have been developed:

Token Internal Position Awareness (TIPA): Augments standard LLMs with a reverse character-prediction head that, for every vocabulary token, learns to reconstruct its character sequence and positions. Fine-tuning with this auxiliary objective raises position prediction accuracy and downstream F1 in Chinese spelling correction (Xu et al., 2024).
One-to-one Character Tokenization with Embedding Retuning: For character-sensitive tasks, intentionally pruned BPE vocabularies (no multi-character tokens) paired with continued pretraining restore alignment, reduce sequence length errors, and cut mispredictions in error correction (Li et al., 2024).
Dynamic Pooling and Boundary Prediction: Toucan’s dynamic boundary mechanism pools characters into token-level representations according to learned, context-sensitive priors, striking a balance between sequence length reduction and pure character modeling (Fleshman et al., 2023).
Sub-character and Multiresolution Tokenization: Recent work on Chinese NLU augments character-level tokenization with sub-character information (glyph or phonetic encoding), further reducing sequence length and conferring homophone robustness (Si et al., 2021).

6. Applications, Benchmarks, and Limitations

Character-level tokenization is the optimal solution for:

Fine-grained error correction, spelling/phonetic normalization (Xu et al., 2024, Li et al., 2024, Bommarito et al., 21 Mar 2025).
Robust dialog act recognition, especially when context windows capture sub-word cues (affix, lemma boundaries) and punctuation/capitalization is informative (Ribeiro et al., 2018).
Model diagnostics, interpretability studies, and psycholinguistic experiments that require exact measurement of surprisal over arbitrary substrings ("focal areas") (Giulianelli et al., 2024).

However, it imposes substantial computational burdens (longer sequence lengths; $\Sigma$ 0 transformer scaling), and may discard higher-order context advantages (collapsing word or phrase boundaries). Hybrid and dynamic tokenization schemes, or architectures that explicitly propagate character information into token models, offer promising directions for combining the respective strengths of character and subword tokenizations (Fleshman et al., 2023, Cosma et al., 20 May 2025, Yang et al., 11 Mar 2026).

7. Open Challenges and Future Directions

The interplay between tokenization granularity and model inductive bias remains a central challenge:

Hybrid tokenizers: Development of domain-aware, hybrid, or dynamically switchable tokenization schemes (e.g., atomizing rare words, named entities, or code identifiers) (Uzan et al., 4 Aug 2025, Xiong et al., 12 Feb 2025, Bommarito et al., 21 Mar 2025).
Auxiliary Character-level Objectives: Use of explicit character-level prediction tasks to regularize or supplement standard language modeling (Xu et al., 2024, Uzan et al., 4 Aug 2025, Cosma et al., 20 May 2025).
Architecture Co-design: Integration of character-level pathways (block-causal attention, explicit embedding fusion) to eliminate inverse scaling and accelerate concept emergence (Cosma et al., 20 May 2025, Sun et al., 2023).
Evaluation methodologies: Expansion of character-level reasoning benchmarks, including multilingual and low-resource scripts, more complex string manipulation tasks, and alignment to human psycholinguistic metrics via marginalization-based surprisal (Giulianelli et al., 2024, Uzan et al., 4 Aug 2025).
Efficient decoding: Dynamic boundary predictors and pooling (as in Toucan) show near-linear decoding speedups without loss in perplexity, indicating fruitful directions for bridging practical and theoretical efficiency gaps (Fleshman et al., 2023, Libera et al., 30 Jan 2026).
Robustness and Security: Understanding the limits of model resilience to obfuscated, adversarial, or noisy character-level inputs, and leveraging (or mitigating) this robustness in real-world deployments (Zhuo et al., 16 Oct 2025, Yang et al., 11 Mar 2026).

Research thus continues to advance beyond the dichotomy of fixed subword vs. character tokenization, toward increasingly adaptive, task-aware, and interpretable models of linguistic structure.