Character-Level LLMs: Insights & Advances

Updated 9 April 2026

Character-Level LLMs are language models that operate on individual characters to perform tasks like word retrieval and reordering.
They employ hybrid techniques such as explicit character embeddings, block-causal attention, and decoupled character heads to mitigate subword tokenization bias.
Recent studies show that integrating architectural interventions and auxiliary tasks significantly boosts character-level performance, narrowing the gap with human accuracy.

Character-level LLMs are neural LLMs that operate at the granularity of individual characters or explicitly incorporate character-level representations for tasks involving text understanding, manipulation, or generation. They are investigated both in purely character-level settings and as hybrid models that combine subword/token and character-aware architectures. Despite the documented success of LLMs on tokenized or word-level tasks, their ability to precisely process, compose, and manipulate character sequences remains limited due to architectural and training biases originating from subword tokenization. Recent research details both the shortcomings and emerging solutions for robust character-level performance.

1. Character-Level Task Taxonomy and Evaluation

Character-level tasks are defined as operations or queries conducted exclusively on the sequence of raw characters within words, without recourse to syntactic or semantic cues beyond direct character composition. Comprehensive benchmarks have formalized six archetypal tasks:

Word Retrieval: Given a text and target character, identify all words containing that character, potentially filtered by character position.
Character Insertion, Deletion, Replacement: Modify words in-place according to prescriptions at the level of individual characters (e.g., insert ‘a’ at the second position of every word).
Character Reordering (Anagram): Output any permutation of a word’s characters.
Character Counting: Tally the occurrences of a specified character (or category, e.g., vowels) within an input.

For each character-level task, token-level counterparts exist (e.g., word or sentence retrieval) where the atomic unit is the word or phrase, not the character. The essential distinction is that token-level tasks do not require models to expose or manipulate the internal orthographic structure of words (Shin et al., 2024).

Evaluation employs task-specific metrics:

Precision, Recall, F1-score: For retrieval, insertion, deletion, and replacement.
Accuracy: For anagram (reordering) and count prediction. Human annotators consistently perform at or near 1.00 F1/acc on these tasks, while state-of-the-art LLMs show a sharp divergence between token-level and character-level performance, often dropping to F1 ≤ 0.6 for character-level manipulations, with the most severe degradations in insertion/deletion operations involving explicit positional reasoning (Shin et al., 2024).

2. Mechanistic Limitations of Subword LLMs on Character Tasks

Contemporary LLMs are universally trained on segmented token sequences (word-pieces or subwords), with tokenization obscuring intra-token character structure:

Token-Centric Representation: Transformer input embeddings and self-attention mechanisms process token or subword IDs, discarding explicit positional or identity information of constituent characters. There is no representation of, e.g., “the token ‘house’ is composed of h-o-u-s-e in sequence” (Shin et al., 2024).
Emergence Bottleneck: “The Strawberry Problem” formalizes the information-theoretic reason: during pretraining, mutual information between model context representations and the underlying character sequence is nearly zero, and any character-level mapping must be discovered incidentally through rare co-occurrence statistics (Cosma et al., 20 May 2025).
Concept-Percolation Dynamics: Character-level generalization in tokenized LMs emerges abruptly and late, following a percolation threshold in the concept-property bipartite mapping—manifesting as sudden, phase-transition-like jumps in task accuracy after protracted stagnation during training (Cosma et al., 20 May 2025).
Failure to Utilize Encoded Information: In controlled setups (e.g., character counting), linear probes show that early and mid-layer activations encode accurate character counts, but late-layer components—identified as “negative circuits”—suppress this information before output, resulting in correct internal representation but incorrect model output (Datta et al., 1 Apr 2026).

A table summarizes the gap between character- and token-level tasks:

Task Type	Human / SOTA LLM (F1/Acc)	Smaller LLMs (F1/Acc)
Token-level	≥ 0.9	≥ 0.9
Character-level	0.95–1.0	0.3–0.8

(Shin et al., 2024)

3. Mechanisms for Implicit and Explicit Character Knowledge

Studies reveal several mechanisms by which LLMs acquire or express character-level information:

Tokenization-Dependent

BPE Merge Rules and Boundary Effects: The statistical properties of tokenization merge rules encode regularities of character adjacency, leading to partial recovery of character structure even in BPE-tokenized models (Sato et al., 5 Feb 2026).
Orthographic Constraints: Real-word orthographic transitions (e.g., valid vs. invalid letter pairs, morpheme boundaries) imprint detectable signatures in token and embedding statistics, leveraged by probing classifiers.

Tokenization-Independent

Semantic Substring Clustering: Shared substrings among words induce similarity in embedding space, supporting character inference through co-clustering effects.
Syntactic and Morphological Cues: Inflectional or derivational morphemes (e.g., “-ed”, “-s”) supply indirect character-level information via correlations with token categories (Sato et al., 5 Feb 2026).

Empirical analysis partitions total probing accuracy into tokenization-dependent (~55 points, from merge and orthography) and tokenization-independent (~7 points, semantic and syntactic) components.

4. Architectural and Algorithmic Approaches for Character-Level Performance

A sequence of interventions to enhance character-level reasoning in LLMs have been proposed and empirically validated:

Hybrid and Multilevel Embedding Strategies

Explicit Character and Position Embeddings: Augment input representations with dedicated character-identity, intra-token position, and word-ID embeddings to recover fine-grained orthographic structure (Shin et al., 2024, Xu et al., 2024).
Block-Causal Cross-Attention: Parallel character encoders and token-to-character cross-attention inject character information at every transformer layer, yielding phase-transition-like improvements in robustness and early emergence of character manipulation skills (Cosma et al., 20 May 2025).
Reverse Character Prediction Auxiliary Tasks: Training with objectives such as Token Internal Position Awareness (TIPA), where the model is required to predict token-internal character positions in reverse order, boosts both position and content accuracy in tasks like spelling correction, even when used as a plug-in fine-tuning step without modifying the core architecture (Xu et al., 2024).

Multi-Head Character Decoding

Decoupled Output Heads: Approaches like SpeLLM replace the monolithic output projection with k independent “character heads,” each predicting one character position. This enables efficient representation of large output spaces and competitive performance with significant inference savings (Ben-Artzy et al., 22 Jul 2025).

Ensemble and Divide-and-Conquer Methods

CharED (Character-wise Ensemble Decoding): Produces character-level outputs by averaging character-wise marginals from multiple models, enabling ensemble decoding without shared tokenizers (Gu et al., 2024).
Divide-and-Conquer Prompting: Atomizing words into space-separated characters before manipulation prompts, followed by controlled reconstruction, yields large gains in deletion, insertion, and substitution tasks even in zero-shot settings (Xiong et al., 12 Feb 2025).

Causal Mechanisms in Tokenization Robustness

Word Recovery: Character-level tokenization robustness is attributed to internal “word recovery” dynamics, where early-layer self-attention aggregates character tokens into higher-level subword representations, maintained and applied through a subspace in the residual stream. Disabling in-group (within-word) attention or ablating the corresponding subspace sharply reduces downstream performance, causally verifying the mechanism (Yang et al., 11 Mar 2026).

Character Prediction Algorithms for Specialized Applications

Character Forecasting for AAC: In augmentative and alternative communication (AAC) interfaces, practical algorithms can convert subword LLM outputs to character-level distributions via beam search over next-token probabilities. This approach outperforms naive byte-level or simple classification-head modifications, especially after curriculum-based domain adaptation (Gaines et al., 17 Jan 2025).

5. Quantitative and Comparative Performance

Empirical findings consistently demonstrate a stark performance divide:

Token-level tasks: All major LLMs, including GPT-4, Claude, Gemini 1.5, and Mistral 7B, achieve F1/Acc ≥ 0.9 across token-level retrieval, insertion, deletion, and reordering (Shin et al., 2024).
Character-level tasks: Only GPT-4 approaches human performance on some tasks (F1 ≈ 1.0, likely via “cheating” through sub-token heuristics), while smaller models fall to F1 ≈ 0.3–0.6, with the largest deficit in insertion/deletion using explicit positions. For example, Mistral 7B achieves F1 = 0.268–0.641 across insertion, deletion, retrieval, and counting.
Ensemble, divide-and-conquer, and hybrid explicit-character methods close the gap: ToCAD achieves deletion/insertion/substitution exact match accuracy of 0.898–0.948 on GPT-3.5 compared to 0.159–0.903 for prompt-only baselines (Xiong et al., 12 Feb 2025). TIPA improves position prediction accuracy by up to 8 points and character-level F1 by 5–6 points over strong baselines (Xu et al., 2024).
Downstream tasks (BoolQ, GSM8K, HumanEval): Character-level decoding models maintain or exceed token-level baseline performance after architectural adaptation or distillation, while reducing inference costs (Ben-Artzy et al., 22 Jul 2025).

6. Interpretability and Theoretical Insights

Studies of internal model dynamics reveal failure modes and sources of emergent character-level competence:

Suppression via Negative Circuits: In symbolic tasks such as character counting, LLMs encode the correct information in mid-layer activations but systematically suppress the correct output in late MLP and attention heads—demonstrating a utilization, not representational, bottleneck (Datta et al., 1 Apr 2026).
Competitive Decoding Paradigm: Theories conceptualize layerwise computation as competitive decoding, where hypotheses compete via successive amplification and suppression, with model errors traced to maladaptive weighting of correct vs. heuristic strategies.
Phase-Transition Learning: Task accuracy for character manipulations does not improve gradually but exhibits abrupt emergence once a critical coverage of token-character edges is achieved in training (Cosma et al., 20 May 2025).

7. Open Challenges and Future Directions

Key directions for robust character-level LLM development include:

Architectural remedies: Multi-level embeddings, explicit position-awareness auxiliaries, and block-causal token-character cross-attention have shown promise but require further study for scaling to broader tasks and languages (Xu et al., 2024, Cosma et al., 20 May 2025).
Interpretability-driven training: Monitoring and regularizing negative circuit suppression may preserve symbolic signals for reliable output (Datta et al., 1 Apr 2026).
Tool-augmented and hybrid reasoning: Integrating deterministic routines (e.g., string manipulation APIs) or chain-of-thought decompositions remains a practical mitigation for character-level reasoning until models natively close the gap (Shin et al., 2024).
Low-resource and morphologically rich languages: Extending analyses and remedies beyond English, especially to languages where character order and identity crucially encode meaning (e.g., Chinese, Turkish, agglutinative scripts).
Efficiency and coverage: Decoupled output heads and character-level decoding architectures, as in SpeLLM, provide an efficient path to support underrepresented languages and domains without prohibitive output projection cost (Ben-Artzy et al., 22 Jul 2025).

A plausible implication is that, while baseline LLMs with subword tokenization recover some character structure due to intrinsic boundary cues and compositional clustering, significant advances in both performance and interpretability for character-level tasks will depend on multi-granular representations and targeted auxiliary objectives designed to bridge the gap between token and character compositionality. The field remains active, with a convergence of interpretability, architecture, and task-specific prompting approaches informing the next generation of character-aware LLMs.