Token Arena: A Continuous Benchmark Unifying Energy and Cognition in AI Inference

Abstract: Public inference benchmarks compare AI systems at the model and provider level, but the unit at which deployment decisions are actually made is the endpoint: the (provider, model, stock-keeping-unit) tuple at which a specific quantization, decoding strategy, region, and serving stack is exposed. We introduce TokenArena, a continuous benchmark that measures inference at endpoint granularity along five core axes (output speed, time to first token, workload-blended price, effective context, and quality on the live endpoint) and synthesizes them, together with a modeled energy estimate, into three headline composites: joules per correct answer, dollars per correct answer, and endpoint fidelity (output-distribution similarity to a first-party reference). The framework's novelty is empirical and methodological. Across 78 endpoints serving 12 model families, the same model on different endpoints differs in mean accuracy by up to 12.5 points on math and code, in fingerprint similarity to first party by up to 12 points, in tail latency by an order of magnitude, and in modeled joules per correct answer by a factor of 6.2. We further show that workload-aware blended pricing reorders the leaderboard substantially: 7 of 10 top-ranked endpoints under the chat preset (3:1 input:output) fall out of the top 10 under the retrieval-augmented preset (20:1), and the reasoning preset (1:5) elevates frontier closed models that the chat preset penalizes on price. We release the framework, schema, probe and eval harness, and a v1.0 leaderboard snapshot under CC BY 4.0. TokenArena is a methodology, not a single ranking; we publish full provenance and limitations and welcome external replication.

• The paper presents a novel endpoint-level benchmarking framework that quantifies energy per correct answer and consolidates diverse inference metrics.
• It introduces composite metrics such as Joules per Correct Answer and Dollars per Correct Answer, revealing significant endpoint heterogeneity even within identical models.
• The results highlight the need for workload-aware evaluation in AI inference, enabling procurement decisions that consider cost, speed, and energy efficiency.

Authoritative Summary of "Token Arena: A Continuous Benchmark Unifying Energy and Cognition in AI Inference"

Motivation and Problem Statement

Token Arena introduces a fundamentally revised methodology for benchmarking AI inference at endpoint granularity, targeting the growing constraints due to energy consumption and inference costs as LLMs transition from a training-centric bottleneck to inference-dominated production. It asserts three pervasive blind spots in the incumbent benchmarking paradigm: aggregation at the model or provider level (obscuring endpoint-specific heterogeneity), static and unrealistic workload assumptions (ignoring true input:output token ratios in production workloads), and the lack of longitudinal energy and reliability accounting. The motivation derives from observed disparities in AI endpoint behavior—such as significant variations in accuracy, latency, price, and energy consumption—despite sharing the same underlying model.

Framework Design and Core Methodology

Token Arena’s innovation is the construction of a continuous, endpoint-level benchmarking pipeline, formalizing a set of composite metrics that synthesize multiple facets of inference quality and efficiency. The core unit, the endpoint, is defined rigorously as a tuple $$(\mathrm{provider}, \mathrm{model}, \mathrm{SKU}, \mathrm{precision}, \mathrm{decoding}, \mathrm{region})$$, capturing all variables impacting production deployment decisions.

Composite metrics include:

• Joules per correct answer ($J_{\mathrm{CA}}$), computed as $J_{\mathrm{CA}}(e) = \frac{j_e \cdot T_e}{A_e}$, where $j_e$ is energy per output token, $T_e$ tokens-to-solution, and $A_e$ endpoint-specific accuracy.
• Dollars per correct answer ($C_{\mathrm{CA}}$), structurally analogous to $J_{\mathrm{CA}}$, integrating real-time pricing.
• Endpoint fidelity, quantified via symmetrized KL divergence ($F(e)$) between endpoint outputs and first-party references, distinguishing quantization, silent weight substitution, or serving-stack drift.

Composite scoring aggregates five key factors: output speed ($S$), time-to-first-token ($J_{\mathrm{CA}}$0), workload-blended price ($J_{\mathrm{CA}}$1), live endpoint quality ($J_{\mathrm{CA}}$2) via a broad suite of evals, and reliability ($J_{\mathrm{CA}}$3), each dynamically weighted by workload preset $J_{\mathrm{CA}}$4. All computations are normalized in-cohort to prevent domination by smaller/faster models.

The benchmarking pipeline executes three measurement loops in parallel: probe (continuous latency, jitter, throughput, etc.), eval (quality and fingerprint distribution), and energy/pricing, each writing to a common time-series store. Systematic region and concurrency rotation mitigates adversarial endpoint optimization.

Figure 1: Token Arena pipeline showing parallel probe, eval, and energy/pricing loops continuously measuring live endpoints and writing to a time-series store keyed on endpoint identity and probe conditions.

Empirical Results and Analyses

Strong empirical claims are substantiated across three analyses:

1. Endpoint-level divergence: Even for identical models (e.g., 19 endpoints serving gpt-oss-120B), accuracy on math/code benchmarks varies by up to 12.5 points, endpoint fidelity by up to 8.2 points, and modeled joules per correct answer by a factor of 6.2—a clear rejection of the sufficiency of model- or provider-level aggregation.
2. Undisclosed quantization detection: The fidelity metric directly detects undisclosed quantization. FP8 "Turbo" SKUs exhibit $J_{\mathrm{CA}}$5 (vs. $J_{\mathrm{CA}}$6 for BF16), with commensurate 4–7 point drops on math/code benchmarks—observable in output fingerprints even before accuracy measurement.
3. Workload-aware leaderboard re-ranking: Varying the input:output workload ratio (e.g., chat 3:1, RAG 20:1) radically alters endpoint ordering. Across six workload presets, top-10 overlap is <50% between pairs, demonstrating the inadequacy of static blended price conventions for real-world decision-making.

Notably, ablation and sensitivity analyses demonstrate the compositional robustness of the framework: headline rankings are stable to reasonable perturbations in factor weighting, and conclusions are not artifacts of tuning.

Theoretical and Practical Implications

Token Arena’s architecture elevates the endpoint to the proper unit of procurement-relevant comparison. For providers, this exposes "silent" degradations (quantization, hardware substitutions) not reflected in API or SKU branding. For users, it enables procurement grounded in actual delivered quality, latency, and cost-per-correct-answer—including energy, the new industrial constraint.

By making modeled energy per token—and thus per correct answer—a headline metric, Token Arena aligns AI system evaluation with emergent sustainability concerns, incentivizing providers to disclose, improve, or directly meter energy usage. Methodologically, the framework enables reproducible, falsifiable claims: released probe and eval harnesses, daily-rotating prompt sets, and fingerprint reference scores support persistent public auditing.

The provision for custom workload presets directly addresses the heterogeneous spectrum of inference workloads in production—voice agents, batch jobs, coding assistants, RAG, long-context analysis—empowering practitioners to compute use-case-aligned rankings.

Limitations and Open Problems

The energy metric is modeled, not directly metered, introducing potential systematic bias—though the authors bias parameters toward conservative (higher) energy estimates. The fingerprint reference depends on first-party API availability; for solely third-party-served models, the method is intrinsically weaker. While adversarial gaming is considered and mitigated, continuous arms-race dynamics may affect long-term signal integrity. Benchmark contamination (e.g., saturated eval splits) and coverage gaps in the endpoint registry are acknowledged challenges.

Future Directions

Token Arena enables several future advances:

• Direct integration of hardware telemetry from providers or future open-infrastructure initiatives could eliminate energy-modeling uncertainty.
• Expansion to sovereign and decentralized providers increases coverage and robustness, facilitating global policy interventions (e.g., regulatory carbon accounting).
• Preferential procurement by public or enterprise actors can be directly tied to Token Arena-derived headline metrics—linking dollars-per-correct-answer and joules-per-correct-answer to broader ESG frameworks.
• Hybrid or adaptive workload presets reflecting fine-grained production traces may drive per-organization benchmarking.

Methodologically, the token-level compositionality principle introduced here sets a precedent for multilevel, workload-aware benchmarking in other AI modalities (e.g., vision, speech, multimodal agents).

Conclusion

Token Arena establishes a public methodology for continuous, endpoint-level benchmarking of inference, unifying cognitive evaluation and modeled energy/pricing into composite procurement-relevant metrics. Empirically, it demonstrates large within-model endpoint divergence in all axes of concern, the feasibility of undisclosed quantization detection, and the necessity of workload-aware evaluation for meaningful endpoint selection. As AI inference cost and energy become dominant ecosystem constraints, Token Arena provides a reproducible, externally auditable, and extensible solution, facilitating both practical procurement and theoretical analysis of AI deployment efficiency.