Normalized Time Per Output Token (NTPOT)

• NTPOT is a latency metric that divides total end-to-end generation time by the number of output tokens, enabling standardized performance comparisons.
• It decomposes into front-loaded delay (TTFT) and steady per-token decode time (TPOT), reflecting impacts of queueing, prefill, and decoding cadence.
• NTPOT guides system optimization and benchmarking while revealing workload-dependent performance, measurement bias, and potential security implications.

Searching arXiv for papers on Normalized Time Per Output Token and related LLM latency metrics. Normalized Time Per Output Token (NTPOT) is a latency metric for LLM inference that amortizes response time by the number of generated output tokens. In the recent literature, the term is not fully standardized. Some work defines NTPOT explicitly as end-to-end latency divided by output length, some derives it from the more common pair of metrics TTFT (time to first token) and TPOT (time per output token), and some uses baseline-normalized variants to express relative speedup rather than absolute latency. Despite these differences, the common purpose is stable: to collapse queueing, prefill, decode cadence, and sometimes network delay into a per-token quantity that remains comparable across sequence lengths, workloads, and serving architectures (Chandrasekar et al., 22 May 2026, Tan et al., 26 May 2026, Zhang et al., 2024).

1. Definition and terminological status

The clearest formalization appears in production benchmarking work that defines NTPOT per request $i$ as

$\mathrm{NTPOT}_i = \frac{L_i}{N_i},$

where $L_i$ is total end-to-end latency from client dispatch to the last token received, and $N_i$ is the total number of output tokens. That formulation is explicitly designed to amortize front-loaded costs such as queueing and prefill across the response length (Chandrasekar et al., 22 May 2026).

Other papers use equivalent or closely related constructions without naming them identically. In "Pair-In, Pair-Out: Latent Multi-Token Prediction for Efficient LLMs" (Tan et al., 26 May 2026), the reported wall-clock metrics are TTFT and TPOT rather than NTPOT, but the paper’s latency accounting supports two derived forms: an absolute per-token time matching TPOT, and an end-to-end amortized variant

$\mathrm{NTPOT}_{\mathrm{total}} = \frac{\mathrm{TTFT} + T_{\mathrm{post\mbox{-}first}}}{N_{\mathrm{out}}}.$

The same source also defines a normalized form relative to regular decoding,

$$\mathrm{NTPOT}_{\mathrm{norm}}(\mathrm{method}) = \frac{\mathrm{NTPOT}_{\mathrm{method}}}{\mathrm{NTPOT}_{\mathrm{baseline}}},$$

so that speedup is its reciprocal (Tan et al., 26 May 2026).

In timing-side-channel analysis, the relevant quantity is likewise not always named NTPOT in the original system, but the formal timing model

$T = L + \tau N + \epsilon$

naturally induces

$\mathrm{NTPOT} = \frac{T-L}{N},$

with $L$ capturing startup latency and $\tau$ approximating steady per-token decode time (Zhang et al., 2024). Endpoint-level benchmarking work reaches the same object through throughput, writing

$\mathrm{NTPOT}_i = \frac{L_i}{N_i},$0

where TPS is streaming output speed in tokens per second (Gao et al., 1 May 2026).

Source	Status of the term	Representative formulation
(Chandrasekar et al., 22 May 2026)	Explicit formalization	$\mathrm{NTPOT}_i = \frac{L_i}{N_i},$1
(Tan et al., 26 May 2026)	Derived from TTFT and TPOT	$\mathrm{NTPOT}_i = \frac{L_i}{N_i},$2, or baseline-normalized
(Zhang et al., 2024)	Derived from timing model	$\mathrm{NTPOT}_i = \frac{L_i}{N_i},$3
(Gao et al., 1 May 2026)	Endpoint-throughput equivalent	$\mathrm{NTPOT}_i = \frac{L_i}{N_i},$4
(Lai et al., 3 Dec 2025)	Not explicit in the paper	TPOT-based normalizations consistent with Token Velocity

This distribution of definitions indicates that NTPOT is best understood as a metric family rather than a single universally fixed equation. The invariant element is per-output-token normalization; the variable element is which components of latency are included and what reference baseline, if any, is used.

2. Relation to TTFT, TPOT, throughput, and sequence length

NTPOT is most informative when decomposed into the latency terms from which it is constructed. In the production-benchmarking formulation, end-to-end latency can be written as TTFT plus decode contributions:

$\mathrm{NTPOT}_i = \frac{L_i}{N_i},$5

with

$\mathrm{NTPOT}_i = \frac{L_i}{N_i},$6

This yields the exact identity

$\mathrm{NTPOT}_i = \frac{L_i}{N_i},$7

which makes clear that NTPOT interpolates between first-token latency and steady-state decode cadence. When $\mathrm{NTPOT}_i = \frac{L_i}{N_i},$8 is small, the $\mathrm{NTPOT}_i = \frac{L_i}{N_i},$9 term dominates; when $L_i$0 is large, NTPOT converges toward TPOT (Chandrasekar et al., 22 May 2026).

Streaming benchmarks further separate initial response delay from inter-token rhythm. TokenArena defines output speed as “tokens per second received during streaming” and records “time to first chunk” together with inter-token timing distribution. From that instrumentation one obtains a streaming-aware NTPOT

$L_i$1

and a steady-state version

$L_i$2

which excludes first-token delay after generation begins (Gao et al., 1 May 2026).

The throughput equivalence $L_i$3 is exact only when the numerator and denominator refer to the same timing window. TokenArena therefore distinguishes the mean of per-request ratios from the ratio implied by average TPS. If per-request speeds are $L_i$4, then

$L_i$5

differs in general from

$L_i$6

where $L_i$7 is the arithmetic mean of TPS across probes. The distinction matters when output lengths or inter-token cadences vary substantially across requests (Gao et al., 1 May 2026).

This decomposition is also the basis for workload interpretation. In retrieval-augmented workloads with large inputs and short answers, TTFT amortization dominates. In long-reasoning workloads, TTFT is spread over many tokens and steady-state decode cadence dominates. TokenArena encodes this difference operationally through presets such as chat $L_i$8, retrieval-augmented generation $L_i$9, and reasoning $N_i$0, which reorder endpoints precisely because they alter the balance between front-loaded and per-token costs (Gao et al., 1 May 2026).

3. Measurement methodology and benchmark bias

Because NTPOT is defined from latency measurements, its reliability depends on how timestamps are collected. The production-benchmarking literature shows that widely used single-process, asyncio-driven clients introduce systematic bias at high concurrency by timestamping tokens when the client processes them rather than when they arrive. Modeling the client as an $N_i$1 queue yields an expected client-side waiting time

$N_i$2

and this contaminates measured TTFT and TPOT:

$N_i$3

Consequently, biased NTPOT inherits both inflated front-loaded delay and inflated per-token delay (Chandrasekar et al., 22 May 2026).

The empirical magnitude of that distortion is large. At a 1,000 QPS target, single-process tools saturated far below target rate, with vLLM Bench at 146 QPS and Inference X at 443 QPS, whereas the multi-process framework sustained the requested 1,000 QPS. In a zero-server-latency simulator, single-process utilities injected overhead ranging from 8 ms to 58 s, while the multi-process design maintained 0.63 ms end-to-end overhead (Chandrasekar et al., 22 May 2026). Within this framework, NTPOT is intended not merely as another latency metric but as a length-normalized statistic whose p50, p95, and p99 expose saturation once measurement bias is removed.

The timing-side-channel literature develops a complementary methodology for remote or closed-source systems. There the observable is total wall-clock response time $N_i$4, possibly without token boundaries. Calibration estimates $N_i$5 from RTT and TTFT, and estimates $N_i$6 from requests with known or inferable token counts. The resulting estimator

$N_i$7

relies on the same linear structure that underlies NTPOT. The paper uses median-based calibration, concurrent profiling, and 5-minute sliding windows for GPT-4o because server-side variability can dominate network variability. Locally, Pearson correlations between token count and generation time were at least 0.987 across Tower, M2M100, MBart50, and Gemma2; for remote GPT-4o the correlation was 0.370, which required stronger calibration to track time-varying $N_i$8 and TTFT (Zhang et al., 2024).

Taken together, these studies establish that NTPOT is meaningful only if the measurement path is explicit. Client-side queueing, batching interference, speculative decoding failures, and server load can all alter the observed per-token time. The metric is therefore inseparable from instrumentation design.

4. Architectural and algorithmic determinants of NTPOT

Architectural changes that reduce prefill cost, increase tokens emitted per decoding step, or remove redundant verification directly reduce NTPOT. PIPO is an archetypal example because it couples input-side and output-side acceleration. Its "Pair-In" compressor folds two input tokens into one latent representation, and its "Pair-Out" multi-token prediction head unfolds one hidden state into one additional output token. The paper reports that PIPO “halves the effective prefill length by compressing every two input tokens into one,” yielding a TTFT speedup from 1.65× at 2K input to 2.64× at 128K input, and that “since MTP and PIPO both emit two tokens per pass, their TPOT is roughly half of Regular at all lengths” (Tan et al., 26 May 2026).

The same paper removes verifier cost through a lightweight confidence head trained with the rejection-sampling acceptance probability used in speculative decoding:

$N_i$9

The confidence head adds approximately $\mathrm{NTPOT}_{\mathrm{total}} = \frac{\mathrm{TTFT} + T_{\mathrm{post\mbox{-}first}}}{N_{\mathrm{out}}}.$0M parameters on a 4B backbone, about $\mathrm{NTPOT}_{\mathrm{total}} = \frac{\mathrm{TTFT} + T_{\mathrm{post\mbox{-}first}}}{N_{\mathrm{out}}}.$1 of the model, and is described as orders of magnitude cheaper than a full backbone verifier pass. The paper therefore gives an approximate per-token time model

$\mathrm{NTPOT}_{\mathrm{total}} = \frac{\mathrm{TTFT} + T_{\mathrm{post\mbox{-}first}}}{N_{\mathrm{out}}}.$2

contrasted with regular decoding at approximately $\mathrm{NTPOT}_{\mathrm{total}} = \frac{\mathrm{TTFT} + T_{\mathrm{post\mbox{-}first}}}{N_{\mathrm{out}}}.$3, and reports up to $\mathrm{NTPOT}_{\mathrm{total}} = \frac{\mathrm{TTFT} + T_{\mathrm{post\mbox{-}first}}}{N_{\mathrm{out}}}.$4 TPOT speedup and up to $\mathrm{NTPOT}_{\mathrm{total}} = \frac{\mathrm{TTFT} + T_{\mathrm{post\mbox{-}first}}}{N_{\mathrm{out}}}.$5 TTFT speedup on Qwen3.5-4B and 9B backbones over AIME 2025, GPQA-Diamond, LiveCodeBench v6, and LongBench v2 (Tan et al., 26 May 2026).

A different route to low NTPOT is to reduce the asymptotic cost of attention itself. "Softmax Attention with Constant Cost per Token" replaces conventional scaled dot-product similarity with a log-domain construction based on exponentials and log-sum-exp reductions, maintaining a fixed-size latent state instead of a growing KV cache. The resulting sequential update has time complexity $\mathrm{NTPOT}_{\mathrm{total}} = \frac{\mathrm{TTFT} + T_{\mathrm{post\mbox{-}first}}}{N_{\mathrm{out}}}.$6 and space complexity $\mathrm{NTPOT}_{\mathrm{total}} = \frac{\mathrm{TTFT} + T_{\mathrm{post\mbox{-}first}}}{N_{\mathrm{out}}}.$7 per token, both independent of sequence length $\mathrm{NTPOT}_{\mathrm{total}} = \frac{\mathrm{TTFT} + T_{\mathrm{post\mbox{-}first}}}{N_{\mathrm{out}}}.$8, whereas conventional softmax attention scales as $\mathrm{NTPOT}_{\mathrm{total}} = \frac{\mathrm{TTFT} + T_{\mathrm{post\mbox{-}first}}}{N_{\mathrm{out}}}.$9 in both time and KV-cache space (Heinsen, 2024). The same source derives an attention-side normalized per-token cost relative to softmax,

$$\mathrm{NTPOT}_{\mathrm{norm}}(\mathrm{method}) = \frac{\mathrm{NTPOT}_{\mathrm{method}}}{\mathrm{NTPOT}_{\mathrm{baseline}}},$$0

which reduces to approximately $$\mathrm{NTPOT}_{\mathrm{norm}}(\mathrm{method}) = \frac{\mathrm{NTPOT}_{\mathrm{method}}}{\mathrm{NTPOT}_{\mathrm{baseline}}},$$1 when $$\mathrm{NTPOT}_{\mathrm{norm}}(\mathrm{method}) = \frac{\mathrm{NTPOT}_{\mathrm{method}}}{\mathrm{NTPOT}_{\mathrm{baseline}}},$$2. The paper does not report end-to-end tokens-per-second, but the theoretical implication is that attention’s contribution to NTPOT decreases inversely with context length as the baseline slows linearly while the modified method remains constant-cost per token (Heinsen, 2024).

At the system level, TokenScale links TPOT to decode capacity through Token Velocity rather than defining NTPOT explicitly. Its core relation is $$\mathrm{NTPOT}_{\mathrm{norm}}(\mathrm{method}) = \frac{\mathrm{NTPOT}_{\mathrm{method}}}{\mathrm{NTPOT}_{\mathrm{baseline}}},$$3, where $$\mathrm{NTPOT}_{\mathrm{norm}}(\mathrm{method}) = \frac{\mathrm{NTPOT}_{\mathrm{method}}}{\mathrm{NTPOT}_{\mathrm{baseline}}},$$4 is decode velocity in tokens per second. On top of that relation, the supporting formulation proposes baseline-normalized and bottleneck-normalized TPOT variants consistent with the paper’s PD-disaggregated autoscaling model, but these are explicitly presented as principled extensions rather than original paper definitions (Lai et al., 3 Dec 2025).

5. Benchmarking, autoscaling, and endpoint-level interpretation

In production benchmarking, NTPOT serves as a composite latency statistic for service-level objectives. The formal motivation is that TTFT and TPOT, considered separately, can obscure saturation: a workload may show nearly flat decode cadence while first-token latency explodes, or vice versa. By reporting

$$\mathrm{NTPOT}_{\mathrm{norm}}(\mathrm{method}) = \frac{\mathrm{NTPOT}_{\mathrm{method}}}{\mathrm{NTPOT}_{\mathrm{baseline}}},$$5

and then aggregating p50, p90, or p99 across requests, one directly captures the end-to-end penalty that a user experiences per generated token. The benchmark paper explicitly recommends throughput-versus-NTPOT quantile curves to identify the operating point just below saturation (Chandrasekar et al., 22 May 2026).

PD-disaggregated serving introduces another interpretation. TokenScale defines TTFT SLOs of 250 ms, 400 ms, and 2000 ms for short, medium, and long inputs, together with a TPOT SLO of 100 ms. It then scales prefillers according to

$$\mathrm{NTPOT}_{\mathrm{norm}}(\mathrm{method}) = \frac{\mathrm{NTPOT}_{\mathrm{method}}}{\mathrm{NTPOT}_{\mathrm{baseline}}},$$6

and decoders per bucket according to

$$\mathrm{NTPOT}_{\mathrm{norm}}(\mathrm{method}) = \frac{\mathrm{NTPOT}_{\mathrm{method}}}{\mathrm{NTPOT}_{\mathrm{baseline}}},$$7

Although the paper does not explicitly formalize NTPOT, these equations determine its operational behavior because TTFT affects the amortized front-loaded term and $$\mathrm{NTPOT}_{\mathrm{norm}}(\mathrm{method}) = \frac{\mathrm{NTPOT}_{\mathrm{method}}}{\mathrm{NTPOT}_{\mathrm{baseline}}},$$8 determines the reciprocal of TPOT. On production traces, TokenScale improved SLO attainment from 50–88% to 80–96% and reduced costs by 4–14% relative to DistServe, BlitzScale, and AIBrix; on H100 systems, SLO attainment improved from 43–77% to 85–98% while GPU usage fell by 38–47% (Lai et al., 3 Dec 2025).

At the endpoint level, TokenArena makes NTPOT a property of a concrete deployment tuple $$\mathrm{NTPOT}_{\mathrm{norm}}(\mathrm{method}) = \frac{\mathrm{NTPOT}_{\mathrm{method}}}{\mathrm{NTPOT}_{\mathrm{baseline}}},$$9. The benchmark measures “Output speed (S)” as tokens per second during streaming and “Time to first token (T)” as time to first chunk arrival, with a default condition of 10K input length, single stream, US-East. Under this setup, NTPOT is simply the inverse of measured streaming speed when aggregated consistently (Gao et al., 1 May 2026).

The endpoint perspective shows how much NTPOT can vary for ostensibly the same model. For the 19 endpoints serving gpt-oss-120B, TokenArena reports output speed from 248 to 2,988 tokens per second, a 12.0× ratio. Inverting those values yields an approximate endpoint-level NTPOT range from $T = L + \tau N + \epsilon$0 s/token to $T = L + \tau N + \epsilon$1 s/token. The same model family also showed TTFT P50 from 0.18 s to 0.36 s and TTFT P99 from 0.42 s to 1.20 s across endpoints (Gao et al., 1 May 2026). This confirms that NTPOT is not solely a model property; it is a deployment property shaped by precision, decoding settings, serving stack, and region.

The same benchmark also distinguishes TTFT from TTFV for reasoning models, noting that TTFT and TTFV can diverge by tens of seconds when the model has a hidden “thinking phase.” In such systems, a steady-state NTPOT or a TTFT-amortized NTPOT may understate user-perceived interactivity unless TTFV is reported alongside it (Gao et al., 1 May 2026).

6. Security implications, limitations, and interpretive caveats

NTPOT is not only a performance metric; it can also become an information leak. The timing-side-channel paper shows that because autoregressive generation time grows linearly with output token count, an adversary can recover sensitive attributes from wall-clock timing alone. Using timing-derived token-count estimates, the paper recovered target language in translation with more than 75% precision across Tower, M2M100, and MBart50, and recovered output class in classification tasks with more than 70% precision on models including Llama-3.1, Llama-3.2, Gemma2, and GPT-4o (Zhang et al., 2024). In that setting, NTPOT is operationally a sufficient statistic for reconstructing output length after startup latency is removed.

A second limitation is that NTPOT is sensitive to what is counted as output. Tokenization differences across languages and model families alter $T = L + \tau N + \epsilon$2, which changes any per-token normalization even when semantic content is unchanged. TokenArena therefore normalizes comparisons “within the cohort of endpoints serving the same model class,” and the side-channel paper explicitly exploits cross-language token-density differences as the mechanism of leakage (Gao et al., 1 May 2026, Zhang et al., 2024).

A third limitation is that low NTPOT does not guarantee quality, correctness, or fidelity. TokenArena reports that FP8 or Turbo variants can reduce per-token time but also correlate with fidelity drift and quality losses; for gpt-oss-120B endpoints, FP8/Turbo SKUs are associated with mean endpoint fidelity around 92.1 and drops versus BF16 references of $T = L + \tau N + \epsilon$3 on MATH-500 and $T = L + \tau N + \epsilon$4 on AIME-25 (Gao et al., 1 May 2026). PIPO likewise shows that naïvely accepting all draft tokens lowers pass@4 substantially; its confidence head and on-policy distillation are necessary to preserve reliability while lowering latency (Tan et al., 26 May 2026).

Finally, NTPOT can be unstable or misleading at very short outputs. When $T = L + \tau N + \epsilon$5 is small, $T = L + \tau N + \epsilon$6 becomes large and high-variance; the benchmark paper explicitly recommends quantile reporting and adequate sample sizes in this regime (Chandrasekar et al., 22 May 2026). PIPO makes the same point from another angle: when outputs are tiny, halving prefill helps less because TTFT is amortized over too few emitted tokens to expose the full benefit (Tan et al., 26 May 2026). A plausible implication is that NTPOT should almost always be reported together with its constituents—at minimum TTFT, TPOT or TPS, sequence-length conditions, and the counting convention for output tokens—rather than as a standalone scalar.

In current usage, then, NTPOT is best understood as a unifying latency abstraction for autoregressive inference: a metric that converts heterogeneous serving behavior into per-token time, but whose meaning depends on instrumentation, workload structure, tokenization, and whether the objective is benchmarking, systems optimization, or adversarial analysis.