Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation

Abstract: Every public LLM cost calculator we surveyed treats GPU utilization as a fixed input -- entered by the user, baked in as a preset, or silently assumed at 100% -- never measured against the operator's actual load. We show that this assumption is the dominant source of error: on identical H100 hardware, effective cost spans $0.21 to $15.25 per million output tokens, an underutilization penalty of 2.5-24x across low-to-moderate enterprise loads (1-10 rps) and up to 36.3x near idle -- driven by one operator-controlled variable, offered request rate lambda, which sets in-flight concurrency via Little's Law and which no open-source calculator exposes. Because calculators take utilization as a user-supplied input, any utilization-naive estimate understates true cost by exactly 1/U, systematically mispricing self-hosting -- most severely over-selling it for low-traffic workloads. We propose a measurement methodology that parameterizes the relationship as C_eff = f(H, M, Q, lambda, L), validate it with 42 benchmarks across dense, ultra-sparse MoE, and sparse MoE models, and release vllm-cost-meter, an open-source cost meter that attaches to a live vLLM server and reports real $/M-tokens against the operator's own traffic. We further show that FP8 quantization benefits the MoE architectures we tested roughly 2.2-2.4x more than the dense model (+69 to +74% vs. +31% peak throughput; n=3, broader validation needed), and our data are consistent with active parameter count, not total model size, being a primary predictor of saturation economics. To rule out single-hardware confounding we repeat the core sweep on A100 80GB PCIe (56 runs): the load-driven spread reproduces at 7.0-11.4x, the active-parameters ordering survives at FP8, and the dense-FP8 advantage inverts on silicon without native FP8 tensor cores -- a hardware-conditional caveat the framework already accommodates.

• The paper shows that offered request rate, rather than token volume alone, determines GPU utilization and can create a 17.5–36.3× cost penalty at low traffic on identical H100 systems.
• The methodology models utilization as an outcome of workload, hardware, model, quantization, and latency requirements, while vllm-cost-meter measures real-time cost per million tokens from live Prometheus data.
• The results show that cost estimates must account for SLO-feasible load, with latency constraints adding up to a 1.91× premium over saturation costs and potentially making API inference cheaper for low-volume workloads.

The core claim

This paper argues that the dominant source of error in self-hosted LLM inference cost estimation is not hardware pricing or model choice, but offered request rate $\lambda$—the operator-controlled variable that determines in-flight concurrency via Little's Law and therefore realized GPU utilization. The author shows empirically that on identical H100 hardware, effective cost per million output tokens spans $0.21 to $15.25, an underutilization penalty of 17.5–36.3× within a single configuration between idle ($\lambda{=}1$ rps) and saturation. Because every public cost calculator surveyed (15+ tools, including Helicone, LiteLLM, and academic break-even frameworks) treats utilization as a user-supplied input or silently assumes full utilization, any utilization-naive estimate understates true cost by exactly $1/U$—systematically and in one direction: toward over-selling self-hosting for low-traffic workloads.

The paper formalizes this as $C_{\mathrm{eff}} = f(H, M, Q, \lambda\,;\, L)$, with hardware, model architecture, quantization, and offered load as swept variables and latency SLO held implicit, and releases vllm-cost-meter, an open-source tool that scrapes a live vLLM server's Prometheus endpoint and reports real $/M-tokens against the operator's own traffic rather than assumed utilization.

Framework

The central conceptual move is to treat GPU utilization as a dependent variable rather than an input:

$$U(\lambda, L \mid H, M, Q) = \frac{\Theta_{\text{achieved}}(\lambda, L)}{\Theta_{\max}(H, M, Q)}$$

so that effective cost becomes $$C_{\mathrm{eff}} = P_{\text{GPU}} \times 10^6 / (3600 \times \Theta_{\text{achieved}})$$, and the ratio to the naive token-volume estimate is exactly $1/U$. This reframing inverts the causal direction embedded in existing calculators: utilization is the output of the load-architecture-quantization interaction, not a number the operator can guess. The crossover between self-hosting and API pricing consequently ceases to be a single break-even point (e.g., "cheaper above 50M tokens/month") and becomes a surface in $(\lambda, L, H)$ space.

Five factors are deliberately held constant: engine (vLLM), on-demand Azure pricing, uniform 512:256 I/O shape, fixed tensor-parallel degree, and Poisson arrivals. The paper is explicit that these choices establish a conservative performance floor—cost figures are upper bounds—and that production workloads with prefix caching and optimized pipelines would only improve on them. Notably, the paper distinguishes its economic utilization $U = \Theta_{\text{achieved}}/\Theta_{\max}$ from nvidia-smi occupancy, which can read high even at $0.21 to $0 because a single resident decode kernel keeps the GPU nominally busy.

Headline results

The main sweep covers 6 server configurations (Llama 3.1 8B dense, Qwen3-30B-A3B ultra-sparse MoE, Mixtral 8x7B sparse MoE; FP16 and FP8 each) × 7 arrival rates on 2×H100 NVL, for 42 benchmark runs. Key findings:

Result	Magnitude
Underutilization penalty at $0.21 to $1 rps	17.5× (Mixtral FP16) to 36.3× (Qwen FP8)
Penalty at $0.21 to $2 rps	2.5× (dense Llama) to 3.8× (Qwen)
Penalty at $0.21 to $3 rps	≈1.0× across all configurations
FP8 peak-throughput gain, MoE vs. dense	+69–74% vs. +31%
Saturation cost, Qwen FP8 vs. Llama FP8	$0.21 to $40.238/MTok

The penalty declines non-linearly: by $0.21 to $5 rps it is 1.18× for the dense reference model, and beyond $0.21 to $6 throughput plateaus while TTFT P50 rises to 2.5–3.7 seconds, indicating queue saturation. The consistency of the cliff across architectures and precisions supports the claim that the underutilization trap is a structural property of discrete GPU rental cost rather than a model-specific artifact.

Three secondary results carry planning weight. First, the FP8 benefit is roughly 2.2–2.4× larger for MoE than for dense models, consistent with MoE's memory-bandwidth-bound profile under sparse activation. Second, at saturation Qwen3-30B-A3B FP8 is cheaper per token than the smaller dense Llama FP8—active parameter count, not total size, governs saturation economics—though the paper correctly flags this as resting on a single architecture pair with a ~12% margin. Third, the crossover analysis shows Mixtral FP16 at 1 rps costs $15.25/MTok, more expensive than Claude Sonnet 4.6's output tier, whereas naive math places every self-hosted configuration permanently below every API tier. The paper also notes that API prompt-caching, batch discounts, and asymmetric input/output pricing (output tokens priced 5–6× higher) would all shift crossover thresholds upward, making its self-hosting comparison conservative.

SLO-conditioned operating points

A particularly useful section prices the latency SLO itself. Under a fixed example SLA (TTFT p99 ≤ 300 ms, TPOT p99 ≤ 50 ms), the unconstrained saturation floor is unreachable for every configuration—$0.21 to $7 lives at $0.21 to $8–200 where TTFT p99 runs to multiple seconds. The SLA-feasible floor carries a premium of 1.13× to 1.91× over $0.21 to $9, largest for Mixtral FP16 (1.91×), whose memory budget forces the SLA-feasible load down to 10 rps. The implication is that a committed SLA is, through its effect on sustainable $\lambda{=}1$0, a directly priced quantity that no token-volume calculator can express. The raw-saturation floors reported elsewhere in the paper are therefore lower bounds on goodput-constrained cost.

Sensitivity and robustness

Five targeted probes bound the scope of the headline claim, and none overturns it. RAG-shaped (4096:1024) and agentic (1024:4096) I/O mixes change the cost level non-monotonically (RAG is cheaper than chat at $\lambda{=}1$1, 2.4–2.7× costlier at $\lambda{=}1$2, 3.2–3.7× at $\lambda{=}1$3) but preserve the concave cliff shape. Gamma arrivals at CV=2 shift cost by under 1% versus Poisson on the single configuration probed. Variable-length log-normal arrivals widen the spread rather than compressing it (39.3× and 47.6× vs. 31.9× and 36.3× fixed). Prefix caching is inert on random tokens (adding up to +11% bookkeeping overhead) but cuts saturation cost 20–22% on a shared-prefix workload—again without altering the curve's shape. A 12-run repeat campaign on one configuration shows $\lambda{=}1$4 reproducibility of ≤0.31% CV, roughly four orders of magnitude below the reported cross-load spread; the corresponding confidence intervals for the MoE headline penalties are extrapolated, not directly measured, and the paper says so.

Cross-hardware validation

A 56-run replication on A100 80GB PCIe addresses single-hardware confounding. The load-driven spread reproduces at 7.0–11.4× per configuration, compressed relative to H100 because cheaper, slower silicon brings the saturation plateau closer to idle-edge cost. Two results are hardware-conditional in instructive ways. The MoE FP8 advantage persists, but the dense FP8 advantage inverts on A100 (+17% cost for Llama FP8), since SM80 silicon lacks native FP8 tensor cores and the software-emulated path costs more than the bandwidth savings—a caveat the framework accommodates by treating $\lambda{=}1$5 as first-class. A TP=4 Mixtral run inverts the usual scaling intuition: doubling GPU count raises saturation cost 1.50× (from $\lambda{=}1$61.30/MTok) because sub-linear throughput scaling loses to the 2× cost multiplier. The active-parameters ordering survives on both silicon families, though it still rests on three architectures.

Limitations

The paper is candid about scope. The headline sweep uses synthetic fixed-length workloads with prefix caching and speculative decoding disabled, so absolute costs are upper bounds, not typical-case estimates. Only three architectures, two hardware families, and (for headline numbers) one engine are tested; the same configuration spreads 24.4× on vLLM versus 12.0× on SGLang, so multipliers are engine-conditional. The latency SLO $\lambda{=}1$7 is observed as an output rather than swept as an input, and the Gamma burstiness probe covers one configuration only. The 653× minute-level cost swing in the live validation is presented as a directional illustration of minute-window dispersion near idle, not a reproducible datum—per-minute Prometheus ticks are not in the released corpus. Multi-tenant GPU sharing is not modeled. The stability-derived confidence intervals on the MoE penalties assume the C2 noise profile transfers, which the author acknowledges is a simplification.

Conclusion

The paper's contribution is a measured correction to a widespread estimation practice: utilization is not an assumption an operator supplies but an outcome of offered load interacting with architecture, quantization, and SLO, and ignoring this makes cost estimates wrong by 2.5–36.3×—always in the direction of underpricing self-hosting at low traffic. The framework, the 140-run open corpus, and the live meter together give practitioners a defensible path to a load-conditioned cost number. The open questions are specific: whether the active-parameters ordering holds for larger dense models and additional sparsity ratios, whether the SLO dimension $\lambda{=}1$8 can be swept as a first-class input, and how the penalty structure behaves on newer silicon and under multi-tenant sharing.