Accuracy per Token (APT): Efficiency Metrics
- Accuracy per Token (APT) is defined as the efficiency ratio between correctness and token expenditure across various model settings.
- Studies reveal that marginal accuracy gains often demand disproportionately higher token usage, highlighting cost‐efficiency trade-offs.
- APT metrics guide optimal system design by balancing token consumption against performance in document grounding, multilingual evaluation, vision-language compression, and reasoning models.
Accuracy per Token (APT) denotes an efficiency-oriented perspective in which model quality is evaluated jointly with token consumption rather than by raw accuracy alone. Recent work does not present a single universally standardized scalar called APT; instead, closely related formulations recur across several areas. In document-grounded generation, the relevant quantity is the trade-off between epistemic accuracy and token cost; in multilingual evaluation, it is accuracy as a function of tokenization fertility; in vision-LLMs, it is performance retained under aggressive token-budget reduction; and in reasoning models, it is benchmark accuracy relative to output length (Hamilton et al., 18 Jun 2026, Lundin et al., 5 Sep 2025, Ma et al., 23 Feb 2026, Liu et al., 16 Oct 2025). Taken together, these studies suggest that APT is best understood as a family of efficiency criteria for assessing how effectively systems convert token budgets into correct behavior.
1. Definition and formal scope
APT is most naturally interpreted as an efficiency ratio between some notion of correctness and some notion of token expenditure. One explicit approximation appears in work on document-grounded assistants:
or, more concretely,
A closely related multilingual formulation treats tokenization inefficiency itself as the explanatory variable, defining fertility as
where is token count and is word count. In reasoning-model training, the same idea is operationalized through benchmark accuracy together with average output length, rather than through a standalone scalar, with reward written as , where is correctness reward and is a length penalty (Hamilton et al., 18 Jun 2026, Lundin et al., 5 Sep 2025, Liu et al., 16 Oct 2025).
The absence of a canonical formula is itself a notable feature of the literature. The papers converge on a shared principle—accuracy normalized by token-processing burden—while differing on what counts as “accuracy” and which token costs matter most. In document-grounded systems, token cost is dominated by input evidence loaded into context. In multilingual modeling, token burden arises from tokenizer behavior across languages. In VLMs, the relevant budget is the number of visual tokens. In reasoning LMs, the salient quantity is generated output length (Hamilton et al., 18 Jun 2026, Lundin et al., 5 Sep 2025, Ma et al., 23 Feb 2026, Liu et al., 16 Oct 2025).
| Setting | APT-style quantity | Token term |
|---|---|---|
| Document-grounded QA | Correctness relative to token cost | Processed/generated tokens, especially input tokens |
| Multilingual evaluation | Accuracy as a function of fertility | Tokens per word |
| Vision-language compression | Performance retention under token reduction | Visual tokens |
| Reasoning LMs | Accuracy together with response length | Generated output tokens |
This suggests that APT is less a single benchmark metric than a normalization principle. The central analytical question is not merely whether one system is more accurate, but whether the incremental accuracy gain justifies the added token budget.
2. Epistemic accuracy and the token tax in document-grounded systems
A detailed APT-style treatment appears in the study of document-grounded assistants for manufacturing safety training, which compares retrieval-augmented generation (RAG) with long-context prompting (Hamilton et al., 18 Jun 2026). That work defines epistemic accuracy as “the extent to which a document-grounded AI system produces a correct answer because the necessary evidence is both available and appropriately utilized.” Correctness is operationalized as a binary LLM-judge outcome using GPT-5 Pro, with judgments made relative to the expert gold answer and helpfulness; correctness is the main metric because safety training requires technically correct guidance.
The paper frames RAG and long-context prompting as two regimes on an accuracy-cost frontier. RAG retrieves a few relevant passages and is therefore cheaper, but it may miss relevant evidence. Long-context prompting loads the whole document collection in context, broadening evidentiary access and reducing retrieval failure, but incurring what the paper calls the “token tax”: “the additional input-token cost paid for broader evidentiary access, and therefore for the possibility of higher epistemic accuracy.” This is the clearest direct precursor to an APT interpretation.
The empirical comparison is concrete. On 324 matched question/evaluation pairs, long context achieved 73.1% correctness, whereas semantic RAG achieved 65.4%, a gain of 7.7 percentage points. The difference was supported by McNemar’s test with uncorrected and with Yates correction. Yet the cost gap was much larger than the accuracy gap. Average input tokens processed were approximately 247,754 for long context, 8,569 for keyword RAG, and 8,399 for semantic RAG. Mean cost by approach was about \$\text{APT-style efficiency} = \frac{\text{Correctness}}{\text{Total token cost}}.$00.0046 for keyword RAG, and \$0.0045 for semantic RAG, yielding the paper’s headline statement that long-context prompting cost 26 times more per query relative to semantic RAG, while input-token volume was about 29.5 times larger.
This asymmetry is the paper’s central APT result. Higher raw correctness does not imply higher token efficiency. The study explicitly cautions that “the relationship between epistemic accuracy and cost might not be linear; modest gains in correctness may require disproportionately larger increases in token expenditure.” Under an APT-style reading, long-context prompting has higher absolute correctness, but RAG has higher correctness per unit token usage. The paper’s practical conclusion follows directly: long context may be preferable when correctness is critical and budget allows it, whereas RAG may be preferable when recurring cost matters, especially in resource-constrained organizations.
A further accounting nuance is important. Cost is estimated from total token usage and includes processed and generated tokens, but retrieval-specific costs are omitted from the main analysis. The paper notes embeddings via text-embedding-3-small at \$\text{APT-style efficiency} = \frac{\text{Correctness}}{\text{Total token cost}}.$10.10 per GB per day, but treats them as comparatively small and not decision-changing. This sharpens the APT interpretation: the dominant cost differential came from the evidence-access architecture itself.
3. Tokenization inefficiency as multilingual APT degradation
In multilingual evaluation, APT takes on a fairness dimension. “The Token Tax: Systematic Bias in Multilingual Tokenization” studies how tokenization fertility affects both cost and accuracy across 16 African languages on AfriMMLU, a benchmark of 9,000 MCQA items across 5 subjects and 10 LLMs (Lundin et al., 5 Sep 2025). The central claim is that tokenization inefficiency is not neutral: languages with higher fertility require more tokens to express the same content, which increases sequence length, cost, and latency while depressing accuracy.
Fertility is defined as $\text{APT-style efficiency} = \frac{\text{Correctness}}{\text{Total token cost}}.$2, tokens per word. The paper calculates token counts with each model’s tokenizer, computes fertility for each language, runs MCQA inference, and fits linear regressions of accuracy on fertility. Across all 10 models and all 5 subjects, higher fertility consistently predicts lower accuracy. Reported slopes range from $\text{APT-style efficiency} = \frac{\text{Correctness}}{\text{Total token cost}}.$3 to $\text{APT-style efficiency} = \frac{\text{Correctness}}{\text{Total token cost}}.$4, interpreted as each additional token per word reducing accuracy by 8–18 percentage points, depending on model and subject. Fertility explains 20–50% of the variance in accuracy, with highlighted examples including Llama-3.1-405B on Microeconomics with slope $\text{APT-style efficiency} = \frac{\text{Correctness}}{\text{Total token cost}}.$5, $\text{APT-style efficiency} = \frac{\text{Correctness}}{\text{Total token cost}}.$6, and Qwen-2.5-32B on Geography with slope $\text{APT-style efficiency} = \frac{\text{Correctness}}{\text{Total token cost}}.$7, $\text{APT-style efficiency} = \frac{\text{Correctness}}{\text{Total token cost}}.$8.
Under an APT interpretation, this result means that semantically comparable inputs can yield systematically different accuracy-per-token behavior purely because languages are tokenized differently. Languages with lower fertility obtain more usable predictive signal per tokenized representation; high-fertility languages obtain less. The paper therefore treats the token tax as a structural inequity rather than a small preprocessing artifact.
The economic consequences amplify the effect. Because transformer self-attention scales as $\text{APT-style efficiency} = \frac{\text{Correctness}}{\text{Total token cost}}.$9, a doubling in tokens implies a 4 times increase in training cost and time. The paper gives explicit examples: <a href="https://www.emergentmind.com/topics/large-language-model-meta-ai-llama" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">LLaMA</a> 2 (69B) rises from \$F = \frac{T}{W},$020M, <a href="https://www.emergentmind.com/topics/llama-3" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">LLaMA 3</a> (70B) from \$F = \frac{T}{W},$196M, and <a href="https://www.emergentmind.com/topics/llama-3-1" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">LLaMA 3.1</a> (405B) from \$F = \frac{T}{W},$2420M when moving from English to a 2 times fertility language. A model taking 90 days in English would take about 360 days in the high-fertility language. At inference time, cost scales approximately linearly with token count: for 1M English-equivalent tokens, <a href="https://www.emergentmind.com/topics/vocabulary-assistant-llm-gpt-4o" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">GPT-4o</a> is reported as \$F = \frac{T}{W},$320 English input/output versus \$F = \frac{T}{W},$440 for 2 times fertility, and a prompt-plus-completion taking 2 seconds in English may take about 4 seconds in the higher-fertility language.
The paper also reports that African languages trail English by about 25 accuracy points on average, with French usually between English and African languages. Reasoning-oriented models such as DeepSeek and o1 improve African-language performance by 8–12 points and reduce the English–African gap from about 25 points to 12–14 points in some settings, especially Global Facts, but do not remove the fertility effect. This indicates that better reasoning can partially offset tokenization disadvantage, while leaving the underlying APT inequality in place.
A supplementary notion in the paper is parity,
$F = \frac{T}{W},$5
where $F = \frac{T}{W},$6 and $F = \frac{T}{W},$7 are tokenized sequence lengths for translated counterparts, with English used as the reference. The emphasis, however, remains on fertility because it is the variable directly used in the accuracy regressions.
4. Visual-token compression and multimodal APT
In vision-LLMs, APT is expressed as performance retained after large reductions in visual token count. “ApET: Approximation-Error Guided Token Compression for Efficient VLMs” addresses the inefficiency created by redundant visual tokens and the quadratic attention burden they induce (Ma et al., 23 Feb 2026). The paper’s premise is that many token-pruning methods depend on attention weights, which introduces positional bias and is incompatible with efficient kernels such as FlashAttention. ApET replaces attention-based token importance with an information-theoretic, reconstruction-error criterion.
The method selects a small set of basis tokens, reconstructs other visual tokens as linear combinations of those basis tokens, and uses approximation error as the importance score. The core objective is framed as preserving information between the original token set $F = \frac{T}{W},$8 and a subset $F = \frac{T}{W},$9,
$T$0
A cited bound motivates reconstruction error as a surrogate for information preservation:
$T$1
For each token $T$2, the approximation step is
$T$3
with token importance scored by
$T$4
Low-error tokens are treated as redundant; high-error tokens are treated as informative. Removed tokens are not simply discarded but merged into retained tokens using similarity-based average merging.
The quantitative results are strongly APT-aligned. On image-understanding tasks, ApET retains 95.2% of the original performance while compressing token budgets by 88.9%. On video-understanding tasks, it attains 100.4% of the original performance while compressing token budgets by 87.5%. In the LLaVA-1.5-7B setting, retaining 64 tokens from an original 576 yields 95.2% of upper-bound average performance, compared with 92.7% for the second-best VisionZip. On Video-LLaVA-7B, using 256 tokens out of 2048 yields 100.7% average accuracy relative to the original model and 100.4% average score, outperforming VisionZip by 6.3% on average.
These results indicate that aggressive token reduction can improve the accuracy-token frontier rather than merely move along it. The likely interpretation offered in the paper is that pruning removes redundant or noisy visual tokens. Practical deployment evidence supports the same point. On a single A100 GPU, LLaVA-1.5-7B at 11.1% tokens shows total inference time 10:08, speedup 1.46 times, prefilling time 62.8 ms, prefilling speedup 1.38 times, and TFLOPs 2.09 compared with baseline 8.82. For Qwen2.5-VL-7B at 10% tokens, total inference time is 26:18, speedup 1.30 times, prefilling time 104.7 ms, prefill speedup 1.51 times, and TFLOPs 5.70 compared with baseline 14.52. Because ApET does not require attention weights, it remains fully compatible with FlashAttention.
The broader implication is that multimodal APT need not be restricted to text generation length. In VLMs, token efficiency can be improved by changing the representation before the LLM consumes it, with substantial gains in memory usage, prefilling latency, and FLOPs.
5. Intelligence per token in reasoning LLMs
Reasoning LMs make APT explicit at the output side: the issue is not only whether a model answers correctly, but how many generated tokens it needs to do so. “DLER: Doing Length pEnalty Right” frames the objective as maximizing “intelligence per token,” meaning accuracy relative to response length (Liu et al., 16 Oct 2025). The paper argues that earlier failures of length-penalized RL were primarily optimization failures rather than evidence that simple length penalties are intrinsically harmful.
The training setup combines correctness reward with a length penalty through 5, and uses GRPO with group-relative advantage
6
The paper diagnoses three optimization problems under naive truncation-based RL: large bias in advantage estimation, entropy collapse, and sparse reward signal. Many rollouts receive zero reward because they exceed the target length, producing extreme variance; PPO-style clipping suppresses low-probability, high-entropy transitional tokens such as “Wait,” “Alternatively,” “Thus,” and “Also”; and early batches often contain prompts where all rollouts truncate, yielding degenerate feedback.
DLER addresses these failures with four ingredients: batch-wise reward normalization, higher clipping, dynamic sampling, and a simple truncation length penalty. Batch-wise reward normalization replaces prompt-wise normalization with
7
Dynamic sampling discards prompts whose rollouts are all zero reward or all positive reward and resamples until the batch is filled. The key claim is that, once optimization is corrected, the simplest penalty—zero reward for responses longer than a fixed limit—is sufficient to produce state-of-the-art accuracy-efficiency trade-offs.
The empirical results are substantial. DLER-R1-1.5B reduces average output length from 10,499 tokens to 2,466, about 77% shorter, while improving benchmark results such as MATH from 84.31 to 86.95, AIME-24 from 29.79 to 34.38, AMC from 61.97 to 70.48, Minerva from 38.41 to 43.59, and Olympiad from 44.07 to 48.31. DLER-R1-7B reduces average length from 7,747 tokens to 2,405, about a 69% reduction, while increasing MATH from 93.60 to 94.21, AIME-24 from 55.40 to 55.62, AMC from 82.90 to 84.41, Minerva from 49.79 to 53.88, and Olympiad from 58.21 to 60.48.
APT also changes test-time scaling. On AIME-24, DLER-R1-7B achieves a 27–28% higher accuracy than DeepSeek-R1-7B within the same wall-clock “thinking time.” Single-response latency falls from 93.43s to 23.73s. The paper further states that even when generating 256 parallel responses, DLER remains faster than DeepSeek-R1-7B generating a single response in average request time. This shows that shorter reasoning traces increase the usefulness of parallel sampling under fixed compute budgets.
The adaptive extension, Difficulty-Aware DLER, tightens truncation on easier questions based on the model’s correctness ratio over a sampled response set. It yields an additional 15% reduction in average length for the 1.5B model and 11–12% for the 7B model while maintaining accuracy. A separate update-selective merging method keeps only the top 25% largest-update deltas from the DLER model, scales them by 0.7, and adds them to the original model. In the Nemotron-8B setting, this preserves concise reasoning while recovering accuracy, cutting average length by 46% after merging.
A central interpretive point in the paper is that APT here is not a single universal scalar. The evaluation relies on benchmark accuracy, average output length, pass@K under token cutoffs, and latency under parallel sampling. The common theme is nonetheless clear: a model that matches or exceeds a baseline’s accuracy with far fewer generated tokens occupies a strictly better deployment position.
6. Adjacent efficiency metrics and acronym ambiguity
APT should be distinguished from adjacent efficiency objectives. In sparse MoE LLMs, “LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts” argues that the most relevant serving metrics are often accuracy per FLOP and accuracy per parameter rather than a literal token-normalized score (Elango et al., 26 Jan 2026). The reason is systems-level: online inference is frequently memory-bandwidth bound, and offline high-throughput serving can be dominated by all-to-all communication. The paper therefore evaluates efficiency through tokens/s/GPU, throughput–latency Pareto frontiers, and an Effective Parameter Multiplier, while designing a latent-space routing architecture that preserves or improves model quality at similar or lower serving cost. This does not replace APT, but it constrains its interpretation: token counts alone may not capture the dominant bottleneck in every deployment regime.
There is also a recurring acronym ambiguity. In cybersecurity, “APT” commonly denotes Advanced Persistent Threat rather than Accuracy per Token. “High-Precision APT Malware Attribution with Out-of-Scope Resilience” uses ranked binary classifiers with abstention for malware attribution and reports, in its most challenging setting, that 87% of test samples came from 60 APT groups excluded from training, with 94% of out-of-scope samples abstained while maintaining 92% precision and 95% selective accuracy on classified samples (Williams et al., 2 Jun 2026). This usage is unrelated to token-efficiency evaluation.
A second unrelated meaning appears in multimodal causal reasoning. “APT: Atomic Physical Transitions for Causal Video-Language Understanding” defines APTs as minimal, temporally localized state changes that bind visible cues to active physical mechanisms and before/after regimes (Wu et al., 17 Jun 2026). That paper constructs a 14-type transition taxonomy with 27,303 timed instances over 1,246 trials and shows that zero-shot recall of current VLMs reaches at most 14%, while a parameter-efficient recipe called APT-Tune raises recall substantially without event-level forgetting. Here too, “APT” is an established acronym, but it refers to a causal supervision representation rather than to any token-efficiency measure.
These collisions matter because they create a persistent risk of conceptual conflation. In current arXiv usage, Accuracy per Token is best treated as a contextual term whose meaning must be inferred from the accompanying efficiency setup: evidence-access cost, tokenizer fertility, visual token budget, or output length. The shared intellectual core is an insistence that accuracy should be normalized by the token burden required to obtain it. That principle has become increasingly important as model deployment shifts from one-off demonstrations to repeated, budget-constrained, and latency-sensitive use.