SpeLLM: Efficient Character-Level Decoding

• SpeLLM is a method that replaces a token-level softmax with multiple character-level heads to reduce output layer computational cost.
• It converts a standard BPE-based LLM via self-distillation by fine-tuning new character heads and select transformer layers.
• The approach enhances tokenization efficiency and rare language support while maintaining competitive downstream performance with fallback mechanisms.

SpeLLM is a method for LLM decoding that decouples input and output vocabularies by predicting character-level strings through multiple output heads rather than a single token-level softmax. Introduced in "SpeLLM: Character-Level Multi-Head Decoding" (Ben-Artzy et al., 22 Jul 2025), the method keeps a large BPE input vocabulary to preserve short input sequences while replacing the large output projection with $k$ smaller character heads. The stated objective is to reduce the linear dependence of output-layer cost on vocabulary size, while maintaining competitive downstream performance and increasing support for underrepresented languages and domains.

1. Motivation and problem setting

Modern decoder-only LLMs commonly use large BPE vocabularies, with $S \approx 100\text{K}+$, to shorten input sequences and reduce per-layer attention cost $O(n^2)$. However, the final output projection and softmax cost is $O(S \cdot d)$ per token, where $d$ is the hidden size, and this scales linearly with vocabulary size. SpeLLM is motivated by this asymmetry: expanding the vocabulary can improve tokenization, but the corresponding output projection becomes increasingly expensive (Ben-Artzy et al., 22 Jul 2025).

The method therefore separates the role of the vocabulary on the input side from its role on the output side. The goal is to keep a large input BPE vocabulary so that text is represented with few tokens, while replacing the large token-level output softmax with a much smaller character-level output. This suggests a design in which vocabulary scaling remains useful for encoding efficiency, but generation no longer pays the full $O(S \cdot d)$ cost of a large output layer.

A recurrent misconception is to treat SpeLLM as a fully character-based LLM. The reported design does not do that. It starts from an existing BPE-based LLM and modifies only its output head. Input encoding remains BPE-based, and some inference procedures still refer back to the original token head.

2. Architecture and probabilistic formulation

SpeLLM takes an existing BPE-based LLM $M$ and replaces its single token-level output head with character-level multi-head decoding. Let $h \in \mathbb{R}^d$ denote the final-layer hidden state at the current decoding position. A fixed character vocabulary $C$ of size $|C| = s$ is defined; one example given is $S \approx 100\text{K}+$0, including letters, digits, punctuation, and pad. The output layer then consists of $S \approx 100\text{K}+$1 independent linear heads $S \approx 100\text{K}+$2, each mapping $S \approx 100\text{K}+$3 (Ben-Artzy et al., 22 Jul 2025).

At each token-generation step $S \approx 100\text{K}+$4, SpeLLM predicts $S \approx 100\text{K}+$5 characters in parallel. For the $S \approx 100\text{K}+$6-th head,

$S \approx 100\text{K}+$7

The joint probability of the generated $S \approx 100\text{K}+$8-character string $S \approx 100\text{K}+$9 is factorized across heads and steps:

$O(n^2)$0

This factorization yields an effective output space of size $O(n^2)$1. The paper contrasts the original token-level output space size $O(n^2)$2 with the SpeLLM output space size $O(n^2)$3; an explicit example is $O(n^2)$4, $O(n^2)$5, giving a capacity of approximately $O(n^2)$6 possible strings. Because each head has only $O(n^2)$7 outputs, the output-layer cost drops from $O(n^2)$8 to $O(n^2)$9 when $O(S \cdot d)$0.

The architectural claim is therefore not merely that characters are predicted, but that multiple characters are predicted simultaneously. The multi-head design is central: each head emits one character, and the concatenated characters form a candidate token string.

3. Self-distillation and model conversion

SpeLLM is not trained from scratch as an independent architecture. It is obtained by converting a standard pre-trained BPE model into a SpeLLM variant by self-distillation. In the reported setup, almost all of the original model is frozen; training updates are applied only to the $O(S \cdot d)$1 new character heads, the original token output head for an auxiliary loss, and the last 5 transformer feed-forward layers (Ben-Artzy et al., 22 Jul 2025).

The conversion pipeline runs the teacher model over FineWeb-Edu and records, at each position, the teacher’s top-5 token predictions and their probabilities, excluding the gold token. For each position, the training target is selected from those top-5 candidates. Each candidate token is converted to a character sequence of length $O(S \cdot d)$2 by padding or truncation. The model then computes

$O(S \cdot d)$3

and chooses as the label the candidate whose $O(S \cdot d)$4-character sequence has the largest number of matching positions with $O(S \cdot d)$5, breaking ties by the highest teacher probability. The stated purpose is to prevent “blended” spellings when the teacher is uncertain.

The loss is the sum of a character-level term and a token-level auxiliary term:

$O(S \cdot d)$6

The character loss is

$O(S \cdot d)$7

while the token-level auxiliary loss is computed on the teacher’s top-5 tokens only:

$O(S \cdot d)$8

The reported interpretation is that training preserves teacher token-level information while forcing correct character spelling.

4. Inference procedure and computational profile

At inference time, SpeLLM computes the hidden state $O(S \cdot d)$9, applies all $d$0 character heads, selects one character per head, concatenates them into a $d$1-character string, strips any pad suffix, and treats the result as one token candidate. If the resulting string is a valid BPE token, it is emitted directly (Ben-Artzy et al., 22 Jul 2025).

Two optional recovery mechanisms are described. The first is AutoCorrect. If the produced string is not a valid token, the system enumerates all BPE tokens whose each character lies within the top-3 characters of each head, then re-scores that subset using the student’s token head and selects the highest-scoring candidate. The second is Entropy-Fallback. If the average entropy across the $d$2 character heads exceeds a threshold $d$3—an example threshold is $d$4—the decoder skips the character heads and falls back to the original token-level head.

The complexity analysis compares the standard token head cost, $d$5 per decoding step, with the SpeLLM head cost of $d$6. In end-to-end terms,

$d$7

becomes

$d$8

where $d$9 is the number of transformer layers. The paper emphasizes that most compute still resides in the transformer body, and therefore head-level acceleration does not translate into proportional end-to-end speedup. The reported end-to-end decode result, using prompt length $O(S \cdot d)$0 and generating $O(S \cdot d)$1 additional tokens, is an average runtime reduction of $O(S \cdot d)$2 across four models.

5. Experimental protocol and empirical results

The reported teacher models are Llama3-3B, Llama3-8B, Gemma2-2B, and Gemma2-9B. The character heads use $O(S \cdot d)$3 and $O(S \cdot d)$4 symbols. Distillation uses 500 K samples from FineWeb-Edu, truncated to 1,400 tokens, with training limited to the new heads and the last 5 feed-forward layers, using FP16 AdamW (Ben-Artzy et al., 22 Jul 2025).

Intrinsic evaluation is performed on 5,000 unseen FineWeb-Edu samples by comparing SpeLLM’s output against the teacher’s top-5 tokens. The metrics are full exact match, 10-character match, and prefix-only match. Averaged over the four models, the reported results are:

Setting	Total partial matches	Full exact
Without AutoCorrect	94.89%	91.75%
With AutoCorrect	97.57%	94.93%

Downstream evaluation covers BoolQ and ARC-Easy with 500 samples each in zero-shot mode, GSM8K with 500 samples using 6-shot for smaller models and 3-shot for the 9B model, and CNN/DailyMail summarization with 100 samples in zero-shot mode. The reported comparison between SpeLLM and the teacher is that BoolQ and ARC-Easy are roughly equal or improve by up to $O(S \cdot d)$5, whereas GSM8K and CNN/DailyMail show small degradation that is largely recovered by AutoCorrect or Entropy-Fallback.

Ablations vary $O(S \cdot d)$6 on Llama3-3B. Exact matches increase with $O(S \cdot d)$7, while total partial match remains stable at approximately $O(S \cdot d)$8–$O(S \cdot d)$9. Accuracy remains above $M$0 on tokens of length at most 4 characters and decays mildly for longer tokens. The paper also reports a high correlation between low character-head entropy and correctness, which is the basis for entropy-triggered fallback.

6. Interpretation, limitations, and terminological ambiguity

The discussion in the paper attributes two principal benefits to the design. First, decoupling the output head from a very large BPE vocabulary allows practitioners to expand input vocabularies for low-resource languages without inflating inference cost. Second, character-level decoding naturally handles unseen or rare strings, including proper names and code symbols (Ben-Artzy et al., 22 Jul 2025).

The limitations are explicit. SpeLLM still relies on BPE for input encoding and for AutoCorrect candidate generation, so it is not a fully end-to-end character model. The overall speedup is bounded because most cost remains in the transformer body rather than in the output head. Slight performance gaps remain on more difficult tasks such as GSM8K and CNN/DailyMail when fallback mechanisms are not used. These constraints place SpeLLM in a hybrid design space: the method alters the output interface and decoding procedure, but does not replace the underlying tokenized transformer architecture.

The name can also be confused with unrelated systems. A technical summary of "SPELL: Synthesis of Programmatic Edits using LLMs" discusses implications for a hypothetical system called “SpeLLM,” but that discussion concerns programmatic edits and API migration rather than character-level decoding (Ramos et al., 1 Feb 2026). Separately, "SPELL: Self-Play Reinforcement Learning for evolving Long-Context LLMs" uses the acronym SPELL for a multi-role self-play reinforcement learning framework for long-context reasoning (Yang et al., 28 Sep 2025). In the literature represented here, SpeLLM therefore denotes a character-level multi-head decoding method, not either of the two SPELL frameworks.