Instance-level Randomization for LLM Evaluation

• Instance-level Randomization (ILR) is an evaluation method that randomizes factors like prompts and few-shot examples for each instance to minimize variance in LLM performance assessments.
• ILR reorganizes the evaluation process by independently sampling random factors per input, reducing both instance-wise and run-wise covariances compared to fixed-prompt methods.
• Empirical studies on benchmarks such as Winogrande and HellaSwag demonstrate that ILR can achieve similar stability with roughly 50% fewer LLM calls, enhancing cost efficiency and fairness.

Instance-level Randomization (ILR) is an evaluation method for LLMs in which the random factors that affect evaluation scores are sampled separately for every individual input instance rather than fixed across an entire benchmark. In the formulation introduced in "Instance-level Randomization: Toward More Stable LLM Evaluations" (Li et al., 16 Sep 2025), these random factors include few-shot examples, prompt template, option labels, and task description. The method is motivated by the observation that evaluations of LLMs suffer from instability, where small changes of random factors can lead to drastic fluctuations of scores and even model rankings, and that different LLMs can have different preferences for a certain setting of random factors. ILR addresses these effects by randomizing all factors per instance, repeating the resulting evaluation multiple times, and reporting the averaged score, with the stated goal of reducing variance and enhancing fairness in model comparisons (Li et al., 16 Sep 2025).

1. Definition and motivating problem

In a standard few-shot or prompt-based evaluation, one random setting is fixed across all instances in a dataset. ILR replaces that regime with per-instance sampling: given a dataset of $m$ examples $\{(x_k,y_k)\}$ and a set of random-factor samplers $\mathcal{F}$, ILR draws for each instance $x_k$ an independent $f_k \sim \mathcal{F}$ and feeds $M(f_k(x_k))$ to the model. By repeating this instance-level draw $n$ times and averaging, ILR greatly reduces correlations across instances and across experimental runs (Li et al., 16 Sep 2025).

The motivating failure mode is evaluation volatility under fixed prompts or fixed few-shot sets. Benchmarks using a fixed prompt suffer from high variance: scores can swing by 5–12 pts and model-rankings can even invert. Different LLMs may also prefer different prompts, which creates the possibility of unfair model comparisons when only one prompt is used. Naïve multi-run averaging with different fixed prompts reduces variance slowly and at high cost because the variance falls only as $1/n$, while strong instance-to-instance correlations remain. ILR is designed to target two specific variance sources—instance-wise correlation and run-wise correlation—by randomizing all factors per instance (Li et al., 16 Sep 2025).

A common misconception is that averaging over a small number of prompt settings is sufficient to stabilize evaluation. The variance analysis associated with ILR treats this as incomplete, because a fixed random setting within each run preserves correlations across dataset instances. In that sense, ILR is not merely a larger sample of prompts; it changes the dependence structure of the evaluation procedure.

2. Formal framework and variance decomposition

The notation used in the ILR framework is as follows. Let

• $D = \{(x_k,y_k)\}_{k=1}^m$ be the dataset.
• $\mathcal{F}$ be the joint distribution over all random factors.
• For one experiment $\{(x_k,y_k)\}$0, sample $\{(x_k,y_k)\}$1 fixed across $\{(x_k,y_k)\}$2, evaluate $\{(x_k,y_k)\}$3, and record accuracy

$\{(x_k,y_k)\}$4

• In vanilla multi-run evaluation, average

$\{(x_k,y_k)\}$5

The total-variance decomposition is stated as

$\{(x_k,y_k)\}$6

where

$\{(x_k,y_k)\}$7

Expanding $\{(x_k,y_k)\}$8, with $\{(x_k,y_k)\}$9, yields three terms:

$\mathcal{F}$0

These terms correspond respectively to the inherent Bernoulli variance of each instance-prompt pair, the instance-level covariance induced by sharing the same $\mathcal{F}$1 within a run, and the run-level covariance across runs (Li et al., 16 Sep 2025).

ILR replaces each $\mathcal{F}$2 with per-instance draws $\mathcal{F}$3. Under that modification, the instance-level covariance $\mathcal{F}$4 is approximately $\mathcal{F}$5 for $\mathcal{F}$6, and the run-level covariance $\mathcal{F}$7 is also reduced. This suggests that ILR is best understood not simply as repeated randomization, but as a reorganization of evaluation that suppresses the covariance terms that dominate instability under fixed-setting protocols.

3. Procedure and computational scaling

The ILR procedure uses two parameters: $\mathcal{F}$8, the number of dataset instances, and $\mathcal{F}$9, the number of randomizations per instance. The total number of LLM calls is $x_k$0 (Li et al., 16 Sep 2025).

The pseudocode sketch is: $n$6

The cost comparison is direct. One vanilla single-run evaluation uses $x_k$1 LLM calls. Naïve multi-run evaluation with $x_k$2 fixed-prompt runs costs $x_k$3 calls. ILR with $x_k$4 per-instance runs also costs $x_k$5, but it is reported to achieve the same variance reduction with roughly half the $x_k$6 (Li et al., 16 Sep 2025).

This computational comparison clarifies an important point. ILR is not presented as a zero-cost stabilization method; it still requires multiple calls per instance. Its claim is instead cost-efficiency relative to naïve multi-prompt averaging at the same robustness target.

4. Theoretical variance-reduction claims

The variance-in-$x_k$7 analysis states that, after ILR, the average covariance $x_k$8 between runs goes down. Since

$x_k$9

a smaller $f_k \sim \mathcal{F}$0 speeds up the $f_k \sim \mathcal{F}$1 decay (Li et al., 16 Sep 2025).

The variance-in-$f_k \sim \mathcal{F}$2 analysis considers the variance within one experiment:

$f_k \sim \mathcal{F}$3

Because ILR removes instance-level covariances, the second term nearly vanishes, and $f_k \sim \mathcal{F}$4 shrinks faster as $f_k \sim \mathcal{F}$5 grows (Li et al., 16 Sep 2025).

An empirical bound on cost saving is reported for Winogrande: to reach a standard deviation of $f_k \sim \mathcal{F}$6, vanilla multi-prompt needs $f_k \sim \mathcal{F}$7 runs, whereas ILR needs only $f_k \sim \mathcal{F}$8 runs, described as less than 50% cost (Li et al., 16 Sep 2025). The phrasing is significant: the result is empirical rather than a universal theorem, and it is attached to a specific benchmark and target standard deviation.

A broader implication is that ILR reframes evaluation stability as a covariance-management problem. That interpretation follows directly from the decomposition above: the first variance term is inherent, while the second and third terms are protocol-dependent.

5. Empirical results and evaluation criteria

The empirical study covers Winogrande (100 instances), HellaSwag (100), MMLU-Pro (500), and BIG-Bench Hard (100 per subtask). The models listed include Qwen2-72B, Llama3-70B, GLM4-9B, Llama3-8B, and Qwen2.5-7B (Li et al., 16 Sep 2025).

Three classes of metrics are emphasized: the variance, measured as standard deviation of averaged accuracy versus $f_k \sim \mathcal{F}$9; instance-level and run-level Pearson correlations as proxies for covariance terms; and Observed Reversal Probability (ORP, Sec 4.3) as a fairness/stability metric (Li et al., 16 Sep 2025).

The main reported findings are concise:

Finding	Reported result
Covariance reduction	Winogrande fixed→0.177, ILR→0.091; HellaSwag fixed→0.457, ILR→0.119
Variance reduction rate	Standard-deviation curve for $M(f_k(x_k))$0 falls $M(f_k(x_k))$1 faster under ILR
Fairness/stability	Ranking-reversal risk is consistently lower under ILR for all dataset/model pairs
Cost saving	Same stability level with $M(f_k(x_k))$2 fewer total LLM calls

The experiment-level correlation also drops, but less than the instance-level correlation. The ranking result is expressed through ORP, where the area under the ORP–$M(f_k(x_k))$3 curve is consistently lower under ILR for all dataset/model pairs (Li et al., 16 Sep 2025).

These results are framed as evidence for two linked claims: reduced variance and reduced unfair comparisons caused by random factors. The fairness claim is operationalized through ranking-reversal risk rather than through a normative definition of fairness.

6. Practical use, limitations, and terminological scope

For the benchmarks studied, the practical guideline is that in 100–500-instance benchmarks, $M(f_k(x_k))$4–$M(f_k(x_k))$5 suffices to drive $M(f_k(x_k))$6. A small pilot is recommended: measure $M(f_k(x_k))$7 versus $M(f_k(x_k))$8 on a held-out 10% and choose $M(f_k(x_k))$9 where returns diminish. The compute–robustness trade-off remains proportional to $n$0, but ILR is described as typically needing $n$1, where $n$2 is the fixed-prompt baseline. A mixed strategy is also mentioned for constrained settings, such as ILR on 50% of instances (Li et al., 16 Sep 2025).

The stated limitations are equally specific. The method treats all random factors as independent, though prompt format and few-shot content may interact. The reported evaluation covers 4 common factor types; other sources such as decoding hyperparameters or chain-of-thought style warrant study. ILR also still requires multiple calls per instance, so extremely large datasets might need sampling (Li et al., 16 Sep 2025).

The term "instance-level randomization" is not unique to LLM evaluation. In "Insta-RS: Instance-wise Randomized Smoothing for Improved Robustness and Accuracy" (Chen et al., 2021), an instance-wise variant appears in randomized smoothing, where each input $n$3 has its own Gaussian noise standard deviation $n$4 and the certified radius is computed with the local $n$5. In "InstanceDiffusion: Instance-level Control for Image Generation" (Wang et al., 2024), the phrase denotes training-time random dropout of each instance’s text-and-location tokens together with an inference-time Multi-instance Sampler. These usages share the idea of customizing randomness at the instance level, but they address different objects—certified robustness, image generation control, and LLM evaluation stability, respectively.

Within LLM evaluation, ILR is characterized as an easy-to-implement wrapper over any existing evaluation pipeline that requires no changes to the model or prompt-engineering while aiming to yield lower variance and more fair model comparisons than naïve multi-prompt averaging (Li et al., 16 Sep 2025).