Variance Amplification Factor (VAF)

• Variance Amplification Factor (VAF) is defined as the ratio between the variance of pass@1 scores on long-horizon tasks and on short-horizon tasks, capturing task-level outcome dispersion.
• It reveals that high VAF indicates selective success in long tasks, serving as a signature of a model's capability rather than a sign of instability.
• VAF integrates into a broader reliability-science framework alongside metrics like RDC, GDS, and MOP to comprehensively assess long-horizon performance.

Variance Amplification Factor (VAF) is a ratio used to quantify how strongly a change in regime increases variability. In the most explicit recent arXiv definition, within long-horizon LLM-agent evaluation, it is the ratio of the variance of pass@1 on long-horizon tasks to the variance of pass@1 on short-horizon tasks, with variance computed across tasks within a duration bucket (Khanal et al., 31 Mar 2026). Other literatures use related but non-identical constructs—such as variance quality diagnostics, variance inflation factors, or domain-specific amplification factors—rather than a single universal VAF. This suggests that the term is domain-specific and must be interpreted together with the underlying random object, the conditioning structure, and the evaluation regime.

1. Formal definition in long-horizon agent evaluation

In the reliability-science framework for long-horizon LLM agents, VAF is defined as

$$\text{VAF}(\mathcal{M}) = \frac{\sigma^2\!\left[\text{pass@1}(\mathcal{M}, \mathcal{T}) \mid d = \text{long}\right]}{\sigma^2\!\left[\text{pass@1}(\mathcal{M}, \mathcal{T}) \mid d = \text{short}\right]}.$$

The same study states that, in the full-study results table, the metric is operationalized as

$$\text{VAF} = \frac{\sigma^2(\text{pass@1} \mid \text{long+vlong})}{\sigma^2(\text{pass@1} \mid \text{short+medium})}.$$

The variance is computed across tasks within the duration bucket, not across repeated runs of the same task in the final table (Khanal et al., 31 Mar 2026).

Under this definition, VAF is a task-level dispersion ratio. It does not measure the mean success level directly; rather, it measures how much the task-level success-rate distribution spreads out when evaluation moves from shorter to longer horizons. The basic computation is correspondingly simple: collect pass@1 scores across the relevant short-duration bucket, compute their variance; collect pass@1 scores across the relevant long-duration bucket, compute their variance; then take the ratio. The paper introduces the metric because pass@1 alone can miss whether longer tasks create more heterogeneous outcomes across tasks and across model capability tiers (Khanal et al., 31 Mar 2026).

A key interpretive point is that VAF tracks variability induced by duration, not merely noise in repeated sampling. In the paper’s framing, long-horizon tasks amplify stochastic errors and compounding uncertainty across many steps. VAF is intended to summarize whether that amplification produces a more dispersed success landscape across tasks.

2. Interpretation: heterogeneity, capability, and the high-VAF inversion

A high VAF means that long-horizon tasks produce more variance than short-horizon tasks. The paper’s central empirical claim is that this is not a sign of instability in the ordinary sense. Instead, it indicates a regime in which some long tasks are still solvable and some are not, so outcomes are mixed rather than uniformly bad. Conversely, a low VAF often means that long-horizon tasks do not amplify variance much because the model is already failing uniformly. The paper emphasizes the resulting interpretation trap: low VAF does not mean reliability; it may simply mean the model is consistently bad (Khanal et al., 31 Mar 2026).

The empirical study evaluates 10 models across 23,392 episodes on a 396-task benchmark spanning four duration buckets and three domains. Its reported VAF values bifurcate sharply by capability tier. Frontier models have VAF $\ge 2.37$, whereas mid-tier and weaker models have VAF $\le 1.26$, often below $1$ (Khanal et al., 31 Mar 2026).

Model	VAF
MiniMax M2.5	2.60
DeepSeek V3	2.49
Kimi K2.5	2.48
GLM-4.5 Air	2.37
Qwen3 32B	1.26
Mistral 24B	1.02
Llama 3.3 70B	0.98
Qwen3 30B	0.71
Mistral Nemo	0.42
Llama 3.1 8B	0.26

The paper also reports that the confidence intervals for the frontier group lie above the rest, and that the scatter plot of VAF versus long+very-long pass@1 shows two clusters: a top-right group with both high pass@1 and high VAF, and a bottom-left group with low pass@1 and low VAF. Crucially, there is no model in a high-VAF/low-pass regime. On that basis, the authors argue that high VAF is a capability signature, not an instability signal (Khanal et al., 31 Mar 2026).

This use of variance reverses a naive reading. In many contexts, more variance is immediately read as worse calibration or worse robustness. Here, the reported pattern is the opposite: only capable models retain enough selective success on long tasks for duration to amplify variance meaningfully.

3. Position of VAF within the broader reliability-science framework

VAF is one of four metrics in the framework introduced for long-horizon agent evaluation: Reliability Decay Curve (RDC), Variance Amplification Factor (VAF), Graceful Degradation Score (GDS), and Meltdown Onset Point (MOP). The paper’s starting claim is that pass@1 on short tasks is structurally blind to the divergence between capability and reliability as task duration grows (Khanal et al., 31 Mar 2026).

Metric	Definition	Role relative to VAF
RDC	$$\text{RDC}(\mathcal{M}, k) : d \mapsto \text{pass}^k(\mathcal{M}, d)$$	Mean trajectory over duration
GDS	$$\text{GDS}(\tau, \mathcal{T}) = \sum_i w_i \cdot \mathbf{1}[\text{subtask } s_i \text{ completed correctly}]$$	Partial-credit complement to binary pass@1
MOP	$$H(t) = -\sum_i p_t(\text{tool}_i)\log p_t(\text{tool}_i)$$ with onset at the first entropy spike above thresholds	Trajectory-collapse diagnostic
VAF	Variance ratio across duration buckets	Spread of task-level outcomes

RDC and VAF separate mean degradation from heterogeneity. RDC measures how average reliability changes as duration increases. VAF measures whether the spread of outcomes expands with duration. GDS is useful when pass@1 is zero or too coarse because it gives partial credit for correctly completed subtasks; VAF, by contrast, remains a dispersion metric over binary task success. MOP operates at the within-episode trajectory level, identifying collapse behavior through sliding-window entropy of tool-call sequences, whereas VAF operates at the benchmark level across tasks (Khanal et al., 31 Mar 2026).

The broader study uses this framework to show that reliability decay is domain-stratified: software engineering GDS drops from $0.90$ to $0.44$, while document processing is nearly flat from $$\text{VAF} = \frac{\sigma^2(\text{pass@1} \mid \text{long+vlong})}{\sigma^2(\text{pass@1} \mid \text{short+medium})}.$$0 to $$\text{VAF} = \frac{\sigma^2(\text{pass@1} \mid \text{long+vlong})}{\sigma^2(\text{pass@1} \mid \text{short+medium})}.$$1. It also reports that capability and reliability rankings diverge substantially, with multi-rank inversions at long horizons; frontier models have the highest meltdown rates, up to $$\text{VAF} = \frac{\sigma^2(\text{pass@1} \mid \text{long+vlong})}{\sigma^2(\text{pass@1} \mid \text{short+medium})}.$$2, because they attempt ambitious multi-step strategies that sometimes spiral; and memory scaffolds universally hurt long-horizon performance across all 10 models (Khanal et al., 31 Mar 2026). Within that framework, VAF is not a stand-alone score but one component of a multi-axis characterization of long-horizon behavior.

4. VAF-adjacent variance diagnostics in experimentation and generative simulation

A conceptually related, but differently formulated, problem appears in experimentation platforms. The paper "Evaluating Variance Estimates with Relative Efficiency" does not define or use a metric called VAF explicitly. Instead, it studies variance quality in A/B and A/A experimentation through false positive rate (FPR), average $$\text{VAF} = \frac{\sigma^2(\text{pass@1} \mid \text{long+vlong})}{\sigma^2(\text{pass@1} \mid \text{short+medium})}.$$3, and kurtosis. Its motivation is confidence-interval validity: variance estimates are central to whether confidence intervals are reliable, and A/A testing is used to diagnose when variance estimates are biased or noisy. The paper’s main methodological conclusion is that empirical FPR is a lossy diagnostic because it turns each A/A test into a binary variable and discards magnitude information; average $$\text{VAF} = \frac{\sigma^2(\text{pass@1} \mid \text{long+vlong})}{\sigma^2(\text{pass@1} \mid \text{short+medium})}.$$4 is proposed as a more sample-efficient alternative, with relative efficiency around $$\text{VAF} = \frac{\sigma^2(\text{pass@1} \mid \text{long+vlong})}{\sigma^2(\text{pass@1} \mid \text{short+medium})}.$$5 to $$\text{VAF} = \frac{\sigma^2(\text{pass@1} \mid \text{long+vlong})}{\sigma^2(\text{pass@1} \mid \text{short+medium})}.$$6 compared with FPR, and kurtosis also often outperforms FPR to a lesser extent (Karhadkar et al., 20 Nov 2025).

A second adjacent construction appears in "Forecasting Generative Amplification," where amplification is defined through effective sample size rather than pass@1 variance. The paper defines $$\text{VAF} = \frac{\sigma^2(\text{pass@1} \mid \text{long+vlong})}{\sigma^2(\text{pass@1} \mid \text{short+medium})}.$$7, where $$\text{VAF} = \frac{\sigma^2(\text{pass@1} \mid \text{long+vlong})}{\sigma^2(\text{pass@1} \mid \text{short+medium})}.$$8 is obtained by matching a chosen metric $$\text{VAF} = \frac{\sigma^2(\text{pass@1} \mid \text{long+vlong})}{\sigma^2(\text{pass@1} \mid \text{short+medium})}.$$9 between generated data and a true-equivalent sample. It proposes two complementary estimators: averaging amplification, based on integrals over phase-space regions, and differential amplification, based on hypothesis testing with a KS statistic on a one-dimensional summary statistic. The reported empirical conclusion is that amplification is possible in specific regions of phase space, but not yet across the entire distribution (Bahl et al., 9 Sep 2025).

These cases are related to VAF at the level of intent rather than terminology. Each asks how a procedure changes effective variability or statistical power under a regime shift. The underlying random quantity, however, differs substantially: pass@1 across tasks in long-horizon evaluation, $\ge 2.37$0-statistics across A/A tests in experimentation, and effective sample size in generative simulation.

5. Relation to variance inflation factors in regression

A recurrent source of confusion is the proximity between VAF and the older regression diagnostic VIF, the variance inflation factor. In linear models with fixed regressors, the VIF result states that adding covariates cannot reduce the conditional variance of an OLS coefficient. For a treatment or focal regressor $\ge 2.37$1, the adjusted variance can be written as

$\ge 2.37$2

where $\ge 2.37$3 is the variance inflation factor. Since $\ge 2.37$4, the VIF is at least $\ge 2.37$5 (Ding, 2019).

The same literature shows that the interpretation depends on conditioning. In the VIF setting, one conditions on the realized regressors, so adding covariates can only inflate variance. In the ANCOVA setting for randomized experiments, one averages over treatment assignments, and covariate adjustment can reduce asymptotic variance by removing outcome variation explained by prognostic covariates. The apparent paradox disappears once one distinguishes conditional variance inflation from unconditional variance reduction under randomization (Ding, 2019).

A further refinement appears in the distinction between centered and non-centered auxiliary regressions. The centered VIF is computed from an auxiliary regression with intercept. The non-centered version is

$\ge 2.37$6

with the auxiliary regression run without intercept. The paper "Centered and non-centered variance inflation factor" states that $\ge 2.37$7 equals Stewart’s $\ge 2.37$8 and satisfies

$\ge 2.37$9

Accordingly, $\le 1.26$0, with equality only when $\le 1.26$1. The centered VIF mainly diagnoses essential multicollinearity, whereas $\le 1.26$2 can detect non-essential multicollinearity involving the intercept (Gómez et al., 2019).

For VAF, the relevance of this regression literature is mainly conceptual. VIF is a multiplicative variance ratio tied to collinearity and a specific conditioning structure. It is not the same quantity as the long-horizon agent VAF, but it is a canonical example of how variance-ratio measures depend on what is being held fixed and what source of randomness is being modeled.

6. Terminological cautions across metrology and the physical sciences

Several papers use “amplification factor” in ways that are not statistical VAFs. In weak-value metrology, the relevant object is often signal amplification together with a separate variance or noise analysis. "Weak-Value Amplification for Longitudinal Phase Measurements Approaching the Shot-Noise Limit Characterized by Allan Variance" does not define VAF explicitly, but it compares weak-value-amplified and conventional measurement through Allan variance. It reports a $\le 1.26$3–$\le 1.26$4 reduction in $\le 1.26$5 under WVA compared with no WVA, near-shot-noise-limited performance at $\le 1.26$6–$\le 1.26$7 s, and the scaling $\le 1.26$8 under fixed detected photon number and detector saturation (Huang et al., 19 Feb 2026). By contrast, "Weak value amplification in a shot-noise limited interferometer" states that the shot-noise contribution is always larger than the final variance of the pointer variable, with a difference in estimating noise level up to a factor of $\le 1.26$9 for a Gaussian initial state (Nishizawa et al., 2012). "Weak Value Amplification Can Outperform Conventional Measurement in the Presence of Detector Saturation" adds that saturation alone does not confer an advantage to WVA, whereas saturation combined with digitization and/or intrinsic pixel noise can make WVA superior in Fisher-information terms (Harris et al., 2016). In these papers, amplification and variance are related, but no formal VAF is introduced.

Other fields use amplification-factor language for entirely different observables. In gravitational-wave lensing, the relevant object is the complex, frequency-dependent wave-optics amplification factor computed from the time-delay function; the wolensing package is designed for this quantity and not for a variance ratio (Yeung et al., 2024). In ferroelectric field-effect transistors, the relevant amplification metric is the surface-potential gain or body factor

$1$0

with $1$1 corresponding to amplification, and the paper’s main conclusion is that multi-domain ferroelectricity bounds the achievable gain (Cano et al., 2011). In seismic site response, the relevant object is a 3D/1D displacement amplification factor,

$1$2

used to compare basin response with a free-field reference, again without any statistical variance meaning (Fajardo et al., 2016).

The resulting terminological lesson is straightforward. “Variance Amplification Factor” has a precise, explicit meaning in the long-horizon LLM-agent framework; closely related literatures often study analogous variance-ratio or variance-quality problems; and many papers use “amplification factor” for quantities that have no direct connection to statistical variance at all. A correct use of VAF therefore requires explicit identification of the stochastic object being measured, the baseline regime, and the source of variability under comparison.