CharBench: Evaluating Character-Level Reasoning

• CharBench is a benchmark designed to evaluate large language models on character-level reasoning by testing operations like character counting and indexing.
• It uses four distinct tasks on uniformly sampled English words to rigorously assess the impact of tokenization schemes on model accuracy.
• The findings reveal that while counting tasks perform relatively well, indexing tasks suffer from tokenization-induced errors, guiding future enhancements in LLM design.

CharBench is a large-scale, diagnostic benchmark designed to systematically evaluate the character-level reasoning abilities of LLMs and to quantify the impact of tokenization schemes on such tasks. Developed in response to persistent weaknesses observed in LLM performance on simple surface-level operations—counting characters or locating positions within words—CharBench introduces controlled, high-coverage datasets and rigorous analysis methodologies to uncover the interplay between word structure, token segmentation, and model accuracy (Uzan et al., 4 Aug 2025).

1. Task Definitions and Formal Benchmark Structure

CharBench consists of four distinct task categories targeting the retrieval or computation of specific character-level properties within words:

1. count_char: Computes the frequency of a specified character within a given word. Prompt: “How many times does the character <c> appear in the string <word>?” Implementation: count(<c>, <word>).
2. count_unique: Recognizes the number of unique characters present in a word. Prompt: “How many unique characters appear in the string <word>?” Implementation: len(set(<word>)).
3. find_first: Determines the index of the first occurrence of a character in a word, indexed from zero. Prompt: “What is the index of the first occurrence of the character <c> in the string <word>? Start counting from 0.” Implementation: <word>.index(<c>).
4. find_last: Identifies the index of the last occurrence of a character in a word, indexed from zero. Prompt: “What is the index of the last occurrence of the character <c> in the string <word>? Start counting from 0.” Implementation: <word>.rindex(<c>).

For each of 10,000 unique English words (lengths 4–10), CharBench generates one instance per task, producing a total of 40,000 examples. Words are uniformly sampled from the MiniPile corpus, with stratification across word lengths to eliminate skew. Each example isolates only the task-relevant prompt line and enforces case sensitivity (i.e., lowercase and uppercase letters are treated as different characters).

2. Dataset Construction and Prompt Methodology

The design of CharBench carefully controls for both linguistic and modeling confounds:

• Word Sampling: Words are drawn to ensure uniform representation across lengths 4–10, with no bias toward especially short or long words.
• Target Character Selection: For tasks involving a character <c>, the character is uniformly chosen from the set of characters present in <word>, ensuring that all prompts are well-posed and free of ambiguous or unsolvable cases.
• Meta-Prompting: A unified meta-prompt is used for all evaluated models: “Answer this question only with the final number, without any other text. Lowercase and uppercase letters are considered different characters. Question: {question}”
• Tokenization Tracking: All words undergo BPE (Byte-Pair Encoding) segmentation according to each model’s tokenizer. These tokenizations are stored for statistical analysis of compression effects and token boundaries in downstream evaluation.

3. Evaluation Protocol and Model Coverage

Seven leading LLMs, spanning both proprietary (OpenAI) and open-weight (Together.AI API) families, are assessed:

• Proprietary: GPT-4o, GPT-4o-mini, GPT-3.5-turbo (all using tiktoken BPE)
• Open-weight: DeepSeek-V3, Meta-Llama-3.1-405B, Llama-3.3-70B, Mistral-7B

Evaluation uses temperature zero (deterministic decoding) and scores only exact matches to enforce strict accuracy standards.

Aggregate Performance:

• Overall average accuracy across all models and tasks: 50.33%
• Per-task accuracy averages:
  • count_char: 77.34%
  • count_unique: 47.96%
  • find_first: 43.64%
  • find_last: 32.37%

Model-specific Highlights (Averages):

Model	count_char	count_unique	find_first	find_last	Overall
GPT-4o	89.45%	65.51%	64.65%	63.14%	70.73%
DeepSeek-V3	74.52%	57.31%	57.85%	38.72%	57.18%
Meta-Llama-3.1-405B	81.21%	44.83%	54.24%	41.01%	55.49%
GPT-3.5-turbo	79.11%	52.08%	28.96%	20.72%	45.22%

These figures reveal that even state-of-the-art LLMs struggle with ostensibly simple character-level queries, especially when required to determine positions (find_first, find_last).

4. Statistical Analysis of Tokenization Effects

CharBench employs Pearson correlation analysis between task correctness and a suite of intrinsic word and tokenization features, with the goal of disentangling the sources of model errors.

• Features Examined:
  • WL: Word length (number of characters)
  • GT: Gold truth (e.g., the actual count or index)
  • GT/WL: Proportion of the word exhibiting the target property
  • NT: Number of BPE tokens in the word
  • CR: Compression rate (WL / NT)
  • TTL: Target token length (number of characters in the token containing <c>, for indexing tasks)
• Key Correlation Findings:
  • Word length (WL) is consistently negatively correlated with accuracy across all tasks ($r\approx -0.21$ on average; up to $r\approx -0.38$ for count_unique).
  • Counting tasks (count_char, count_unique): Gold truth values show moderate negative correlation for count_char ($r\approx -0.31$), but NT and CR are only weakly correlated ($|r|<0.1$).
  • Indexing tasks (find_first, find_last): Target token length (TTL) is a dominant factor; longer tokens containing the queried character significantly lower accuracy ($r\approx -0.23$ for find_first, $-0.15$ for find_last via average, up to $r=-0.259$ on GPT-4o).
  • Number of BPE tokens and the compression ratio generally show negligible correlation with task performance for all tasks ($|r|<0.04$).

Task	WL	GT	GT/WL	NT	CR	TTL
find_first	–0.185	–0.306	–0.267	+0.000	–0.160	–0.230
find_last	–0.185	+0.071	+0.183	–0.039	–0.134	–0.148
count_char	–0.209	–0.307	–0.136	–0.048	–0.136	n/a
count_unique	–0.383	–0.233	+0.359	–0.143	–0.219	n/a

This suggests that surface-level factors—specifically word length and the length of tokens containing queried characters—are more critical to LLM errors than the total number of tokens or the degree of compression.

5. Implications for Tokenization and Modeling Practice

CharBench’s results challenge common assumptions regarding tokenization as the primary bottleneck in character-level reasoning. Empirical findings demonstrate:

• Compression-induced information loss is secondary to word length and intra-token structure for these tasks.
• For “counting” tasks, neither number of tokens nor compression rate reliably predicts model accuracy.
• For “indexing” tasks, loss of character position information within long tokens is a dominant factor.
• The detrimental effect of long tokens on positional accuracy highlights the challenge in tasks that require precise intra-token resolution.

A plausible implication is that surface-level inconsistencies in LLM outputs reflect architectural and training data limitations, not merely flaws in their tokenization schemes.

6. Recommendations and Future Directions

To address the observed deficiencies, the authors recommend multiple research and engineering avenues:

• Tokenization Schemes: Investigation of hybrid or adaptive tokenizers that reliably preserve character-level distinctions, and development of reversible schemes that allow backoff to finer granularity during inference.
• Architectural Enhancements: Addition of explicit character-level supervision, auxiliary loss functions during pretraining or fine-tuning, and incorporation of specialized “surface analysis” modules capable of attending directly to raw character embeddings.
• Benchmark Expansion: Extending CharBench to cover a broader range of languages and scripts, and inclusion of more complex character manipulation tasks (e.g., edit distance, substring operations).
• Evaluation Practice: Adoption of CharBench’s stratified sampling and correlation analysis in the evaluation of future LLMs, and application of causal interventions such as ablated tokenizers to further isolate sources of error (Uzan et al., 4 Aug 2025).

7. Significance and Broader Impact

By releasing the CharBench dataset and analytical framework, the authors establish a new standard for rigorous, high-coverage evaluation of LLMs on character-level reasoning. CharBench elucidates persistent blind spots in leading models, provides tools for quantifying tokenization impacts, and motivates the development of architectures equipped for fine-grained surface linguistic reasoning. Its methodology—rigorous task definition, careful construction, and correlation-based error analysis—serves as a template for future benchmarks targeting similar linguistic pathologies.