Test-Time-Compute Aware Evaluation

• Test-Time-Compute Aware Evaluation is a framework that quantifies inference efficiency by measuring computational costs such as token counts, FLOPs, latency, and monetary expenses.
• The framework introduces adaptive metrics like ARISE and enforces standardized protocols to ensure reproducible, fair, and cost-aware comparisons of machine learning models.
• It drives resource-conscious optimization by enabling robust benchmarking and principled evaluations that account for fluctuating system and pricing conditions.

Test-time-compute (TTC) aware evaluation encompasses the formal methodologies, metrics, and protocols for rigorously quantifying the trade-off between solution quality and computational resources allocated at inference in modern machine learning systems, particularly in large language and reasoning models. Emerging as an essential discipline, TTC-aware evaluation incorporates standardized cost accounting, model-agnostic pipelines, and sample-level analysis to ensure that conclusions about algorithmic advances or architectural improvements remain robust across fluctuating system, API, or economic conditions. This framework supports reproducible, fair, and economically rational comparisons of both classical and adaptive inference paradigms, underlining the practical significance of resource-aware research for deployment at scale.

1. Formalization of Test-Time Compute and Core Metrics

Test-time compute (TTC) is the total computational expenditure during inference, quantifying all post-training computation devoted to processing an input instance. This is not limited to a single forward pass; rather, TTC aggregates all such passes (plus any auxiliary computation for, e.g., sampling, verification, adaptation, or search). Standard units include token counts, floating-point operations (FLOPs), wall-clock latency, and monetary costs (dollars per query).

Let $N_\mathrm{in}$ and $N_\mathrm{out}$ denote input and output token counts; $C_i$ and $C_o$ are input/output costs per million tokens (for API-backed LLMs). FEval-TTC introduces a unified cost function: $$\mathrm{DollarCost}(\mathrm{INP},\mathrm{OUT}) = 10^{-6}( C_i N_\mathrm{in} + C_o N_\mathrm{out} )$$ This abstraction allows direct, fair cost comparison across models, decoding strategies, and time periods, isolating evaluation from pricing or system drift (Rumiantsev et al., 3 Nov 2025).

For sample-level evaluation, metrics include not only overall accuracy, but also per-query compute consumption, time-weighted accuracy, and area under the accuracy–compute or accuracy–time curve (AATC, AUATC) (Alfarra et al., 2023, Yin et al., 7 Oct 2025).

2. Protocol Design: Standardization, Reproducibility, Fairness

To ensure fairness and reproducibility, TTC-aware frameworks enforce rigid standardization at multiple levels:

• Dataset and prompt standardization: Fixed prompt templates, identical sampling temperatures, frozen few-shot exemplars, and deterministic answer extraction (Rumiantsev et al., 3 Nov 2025).
• Response pre-recording: Canonicalizing all model outputs (inputs, outputs, costs) to enable future evaluation without new API or model invocations. Experiments are deterministic replays over this static corpus, eliminating confounding from backend updates or fluctuating API costs (Rumiantsev et al., 3 Nov 2025).
• Cost table freezing: Internal cost tables are locked at a historical snapshot date (e.g., “as of June 2, 2025”), preventing future changes in provider pricing from undermining cross-paper comparisons (Rumiantsev et al., 3 Nov 2025).

This pipeline supports answers to key evaluation questions—such as “which TTC method delivers the best accuracy per dollar on AQuA using Qwen 32B?”—that remain stable over time and for arbitrary researchers.

3. Sample-Level and Adaptive Metrics: ARISE and Beyond

Classical aggregate metrics (e.g., average accuracy at fixed budget) obscure sample-level dynamics—critical when extra computation can both correct and degrade performance depending on the instance. In response, the ARISE metric (Yin et al., 7 Oct 2025): $$\operatorname{ARISE}_i = \sum_{j=1}^m \Delta a_i^{(j)} \cdot W_i^{(j)},$$ where $\Delta a_i^{(j)}$ encodes accuracy change at step $j$ and $W_i^{(j)}$ is a compute-efficiency weight, captures:

• Positive scaling: Extra compute that turns wrong answers into right ones, rewarded by a fractional gain proportional to token efficiency;
• Negative scaling: Over-generation or compute allocation that degrades performance, penalized proportional to wasted tokens;
• Dynamic variance control: Budget is adaptively allocated per sample–resolution pair, ensuring statistical stability under stochastic inference.

Such adaptive, per-query metrics provide strict penalties for negative scaling and increase discriminative power over traditional metrics, exposing models that “waste” compute by worsening specific answers (Yin et al., 7 Oct 2025).

4. Cost-Aware Comparative Methodology and Pseudocode Templates

TTC-aware protocol design mandates that all comparative experiments operate under uniform, transparent, and deterministic pipelines. Example: comparing Self-Consistency (SC) and Best-of-N (BoN) under a fixed sample budget and cost table.

from feval_ttc import load, DatasetType, LLMType dataset, [qwen_model] = load(DatasetType.AQUA, [LLMType.QWEN32B]) for qid, entry in dataset: response = qwen_model(qid, N=20) cot_answers = [cot.answer for cot in response.cots if cot.answer is not None] total_cost = response.request.dollar_cost + sum(c.metadata.dollar_cost for c in response.cots) # SC: majority vote; BoN: highest internal confidence

For every method, the final report always aggregates accuracy and average dollar cost per query. Notably, equal sampling budgets yield identical costs for different algorithms, ensuring that the compute–performance trade-off is dictated only by the aggregation logic and not artifacts of sampling (Rumiantsev et al., 3 Nov 2025).

5. Impact and Empirical Utility Across Tasks

TTC-aware evaluation fundamentally alters the comparison between methods and models. For example, FEval-TTC demonstrates that, for 20-sample inference on AQuA with Mixtral 8×22B:

• Self-Consistency: 78.7% accuracy at $1.76
• Best-of-N: 60.6% accuracy at $1.76

Thus, the only variable distinguishing the two methods is the logical processing of fixed, pre-recorded chains-of-thought, not stochastic sampling, pricing, or prompt drift (Rumiantsev et al., 3 Nov 2025).

This determinism and precision facilitate:

• Cost efficiency: Conversion of hours-to-seconds runtime reduction via offline replay, and elimination of API costs for iterative research.
• Reproducibility: Pin-point replication of published results without dependence on cloud providers or access restrictions.
• Research focus: Shifts effort from method implementation or data plumbing to genuinely algorithmic advances in TTC reasoning.

6. Limitations, Extensions, and Open Directions

Key limitations of current TTC-aware frameworks include:

• Reliance on pre-recorded outputs restricts evaluation to the set of frozen model–prompt–dataset configurations, requiring periodic package updates for new models or data (Rumiantsev et al., 3 Nov 2025).
• Cost models encapsulate only per-token costs; deployment contexts may demand incorporation of wall-clock latency, memory, or energy models.
• ARISE assumes monotonic scaling budgets; non-monotonic or mixed-compute backend designs require custom metric adaptation (Yin et al., 7 Oct 2025).

Potential extensions include:

• Continuous-budget generalizations, e.g., integrating ARISE over a smooth compute axis.
• Extension to interactive, multi-agent, or cross-lingual reasoning domains.
• Architectural analysis tying metric outcomes to underlying model components (e.g., depth-of-thought, reranking, or verification modules) (Yin et al., 7 Oct 2025).

7. Representative Output Table: Comparative Results

Algorithm	Model/Dataset	Budget (Samples)	Accuracy (%)	Avg Cost ($)
SC-20	Mixtral 8×22B / AQuA	20	78.7	1.76
BoN-20	Mixtral 8×22B / AQuA	20	60.6	1.76

Both methods are evaluated on the same set of pre-recorded output samples, under fixed prompts, tokenization, and normalized cost structure (Rumiantsev et al., 3 Nov 2025).

---

Test-Time-Compute Aware Evaluation, as embodied by frameworks such as FEval-TTC and adaptive metrics like ARISE, defines a technical and reproducible standard for quantifying the efficiency of reasoning models under practical cost constraints. By eliminating sources of spurious variance and rigorously modeling overhead, it enables principled comparison, robust benchmarking, and resource-conscious optimization in both academic and deployed settings (Rumiantsev et al., 3 Nov 2025, Yin et al., 7 Oct 2025).