---
title: Instance-level Randomization for LLM Evaluation
url: https://www.emergentmind.com/topics/instance-level-randomization-ilr
type: topic
---

# Instance-level Randomization for LLM Evaluation

Instance-level Randomization (ILR) is an evaluation method for large language models in which the random factors that affect evaluation scores are sampled separately for every individual input instance rather than fixed across an entire benchmark. In the formulation introduced in "Instance-level Randomization: Toward More Stable LLM Evaluations" [2509.12678], these random factors include few-shot examples, prompt template, option labels, and task description. The method is motivated by the observation that evaluations of LLMs suffer from instability, where small changes of random factors can lead to drastic fluctuations of scores and even model rankings, and that different LLMs can have different preferences for a certain setting of random factors. ILR addresses these effects by randomizing all factors per instance, repeating the resulting evaluation multiple times, and reporting the averaged score, with the stated goal of reducing variance and enhancing fairness in model comparisons [2509.12678].

## 1. Definition and motivating problem

In a standard few-shot or prompt-based evaluation, one random setting is fixed across all instances in a dataset. ILR replaces that regime with per-instance sampling: given a dataset of $m$ examples $\{(x_k,y_k)\}$ and a set of random-factor samplers $\mathcal{F}$, ILR draws for each instance $x_k$ an independent $f_k \sim \mathcal{F}$ and feeds $M(f_k(x_k))$ to the model. By repeating this instance-level draw $n$ times and averaging, ILR greatly reduces correlations across instances and across experimental runs [2509.12678].

The motivating failure mode is evaluation volatility under fixed prompts or fixed few-shot sets. Benchmarks using a fixed prompt suffer from high variance: scores can swing by 5–12 pts and model-rankings can even invert. Different LLMs may also prefer different prompts, which creates the possibility of unfair model comparisons when only one prompt is used. Naïve multi-run averaging with different fixed prompts reduces variance slowly and at high cost because the variance falls only as $1/n$, while strong instance-to-instance correlations remain. ILR is designed to target two specific variance sources—instance-wise correlation and run-wise correlation—by randomizing all factors per instance [2509.12678].

A common misconception is that averaging over a small number of prompt settings is sufficient to stabilize evaluation. The variance analysis associated with ILR treats this as incomplete, because a fixed random setting within each run preserves correlations across dataset instances. In that sense, ILR is not merely a larger sample of prompts; it changes the dependence structure of the evaluation procedure.

## 2. Formal framework and variance decomposition

The notation used in the ILR framework is as follows. Let
- $D = \{(x_k,y_k)\}_{k=1}^m$ be the dataset.
- $\mathcal{F}$ be the joint distribution over all random factors.
- For one experiment $i$, sample $f_i \sim \mathcal{F}$ fixed across $k$, evaluate $M(f_i(x_k))$, and record accuracy
$$
A_i = \frac{1}{m} \sum_{k=1}^m \mathbf{1}\{M(f_i(x_k))=y_k\}.
$$
- In vanilla multi-run evaluation, average
$$
\bar A = \frac{1}{n}\sum_{i=1}^n A_i.
$$

The total-variance decomposition is stated as
$$
\mathrm{Var}(\bar A)
= \frac{1}{n^2}\left[ \sum_{i=1}^n \mathrm{Var}(A_i) + 2\sum_{i<j} \mathrm{Cov}(A_i,A_j) \right]
= \frac1{n}\Bigl(\overline{Var} - \overline{Cov}\Bigr) + \overline{Cov},
$$
where
$$
\overline{Var} = \frac{1}{n}\sum \mathrm{Var}(A_i), \qquad
\overline{Cov} = \frac{2}{n(n-1)}\sum_{i<j}\mathrm{Cov}(A_i,A_j).
$$
Expanding $A_i = (1/m)\sum_{k=1}^m f_i(x_k)$, with $f_i(x_k)=\mathbf{1}(M(f_i(x_k))=y_k)$, yields three terms:
$$
\mathrm{Var}(\bar A)
= \frac{1}{n m^2}\sum_{i,k}\mathrm{Var}(f_i(x_k))
+ \frac{2}{n m^2}\sum_{i,k<l}\mathrm{Cov}(f_i(x_k),f_i(x_l))
+ \frac{2}{n^2 m^2} \sum_{i<j, k,l}\mathrm{Cov}(f_i(x_k),f_j(x_l)).
$$
These terms correspond respectively to the inherent Bernoulli variance of each instance-prompt pair, the instance-level covariance induced by sharing the same $f_i$ within a run, and the run-level covariance across runs [2509.12678].

ILR replaces each $f_i$ with per-instance draws $f_i^k$. Under that modification, the instance-level covariance $\mathrm{Cov}(f_i^k(x_k),f_i^k(x_l))$ is approximately $0$ for $k \neq l$, and the run-level covariance $\mathrm{Cov}(f_i^k(x_k),f_j^l(x_l))$ is also reduced. This suggests that ILR is best understood not simply as repeated randomization, but as a reorganization of evaluation that suppresses the covariance terms that dominate instability under fixed-setting protocols.

## 3. Procedure and computational scaling

The ILR procedure uses two parameters: $m$, the number of dataset instances, and $n$, the number of randomizations per instance. The total number of LLM calls is $m \times n$ [2509.12678].

The pseudocode sketch is:
```text
Input: model M, dataset D = {(x₁,y₁),…,(x_m,y_m)}, random‐factor sampler 𝓕, n
Initialize per‐instance scores s̄[1..m] ← 0
For k in 1..m:                    # loop over instances
  For j in 1..n:                  # loop over random draws
    f ← sample(𝓕)                # e.g. pick few‐shot set, prompt format, …
    ŷ ← M( f(x_k) )               # one LLM call
    s ← 𝟙(ŷ = y_k)                # 0/1
    s̄[k] ← s̄[k] + s
  s̄[k] ← s̄[k] / n               # average over j
Return overall score S_ILR = (1/m) ∑_{k=1}^m s̄[k]
```

The cost comparison is direct. One vanilla single-run evaluation uses $m$ LLM calls. Naïve multi-run evaluation with $n$ fixed-prompt runs costs $n \cdot m$ calls. ILR with $n$ per-instance runs also costs $n \cdot m$, but it is reported to achieve the same variance reduction with roughly half the $n$ [2509.12678].

This computational comparison clarifies an important point. ILR is not presented as a zero-cost stabilization method; it still requires multiple calls per instance. Its claim is instead cost-efficiency relative to naïve multi-prompt averaging at the same robustness target.

## 4. Theoretical variance-reduction claims

The variance-in-$n$ analysis states that, after ILR, the average covariance $\overline{Cov}$ between runs goes down. Since
$$
\mathrm{Var}_{ILR}(\bar A) = \frac1n(\overline{Var} - \overline{Cov}) + \overline{Cov},
$$
a smaller $\overline{Cov}$ speeds up the $1/n$ decay [2509.12678].

The variance-in-$m$ analysis considers the variance within one experiment:
$$
\mathrm{Var}(A_i) = \frac{1}{m^2} \sum \mathrm{Var}(f_i^k(x_k))
+ \frac{2}{m^2}\sum_{k<l} \mathrm{Cov}(f_i^k(x_k),f_i^k(x_l)).
$$
Because ILR removes instance-level covariances, the second term nearly vanishes, and $\mathrm{Var}(A_i)$ shrinks faster as $m$ grows [2509.12678].

An empirical bound on cost saving is reported for Winogrande: to reach a standard deviation of $0.02$, vanilla multi-prompt needs $n \approx 6.5$ runs, whereas ILR needs only $n \approx 3$ runs, described as less than 50% cost [2509.12678]. The phrasing is significant: the result is empirical rather than a universal theorem, and it is attached to a specific benchmark and target standard deviation.

A broader implication is that ILR reframes evaluation stability as a covariance-management problem. That interpretation follows directly from the decomposition above: the first variance term is inherent, while the second and third terms are protocol-dependent.

## 5. Empirical results and evaluation criteria

The empirical study covers Winogrande (100 instances), HellaSwag (100), MMLU-Pro (500), and BIG-Bench Hard (100 per subtask). The models listed include Qwen2-72B, Llama3-70B, GLM4-9B, Llama3-8B, and Qwen2.5-7B [2509.12678].

Three classes of metrics are emphasized: the variance, measured as standard deviation of averaged accuracy versus $n$; instance-level and run-level Pearson correlations as proxies for covariance terms; and Observed Reversal Probability (ORP, Sec 4.3) as a fairness/stability metric [2509.12678].

The main reported findings are concise:

| Finding | Reported result |
|---|---|
| Covariance reduction | Winogrande fixed→0.177, ILR→0.091; HellaSwag fixed→0.457, ILR→0.119 |
| Variance reduction rate | Standard-deviation curve for $\bar A$ falls $\sim 2\times$ faster under ILR |
| Fairness/stability | Ranking-reversal risk is consistently lower under ILR for all dataset/model pairs |
| Cost saving | Same stability level with $\approx 50\%$ fewer total LLM calls |

The experiment-level correlation also drops, but less than the instance-level correlation. The ranking result is expressed through ORP, where the area under the ORP–$\Delta$ curve is consistently lower under ILR for all dataset/model pairs [2509.12678].

These results are framed as evidence for two linked claims: reduced variance and reduced unfair comparisons caused by random factors. The fairness claim is operationalized through ranking-reversal risk rather than through a normative definition of fairness.

## 6. Practical use, limitations, and terminological scope

For the benchmarks studied, the practical guideline is that in 100–500-instance benchmarks, $n=3$–$8$ suffices to drive $\sigma(\bar A)<0.02$. A small pilot is recommended: measure $\sigma(\bar A)$ versus $n$ on a held-out 10% and choose $n$ where returns diminish. The compute–robustness trade-off remains proportional to $n \cdot m$, but ILR is described as typically needing $\lesssim n_0/2$, where $n_0$ is the fixed-prompt baseline. A mixed strategy is also mentioned for constrained settings, such as ILR on 50% of instances [2509.12678].

The stated limitations are equally specific. The method treats all random factors as independent, though prompt format and few-shot content may interact. The reported evaluation covers 4 common factor types; other sources such as decoding hyperparameters or chain-of-thought style warrant study. ILR also still requires multiple calls per instance, so extremely large datasets might need sampling [2509.12678].

The term "instance-level randomization" is not unique to LLM evaluation. In "Insta-RS: Instance-wise Randomized Smoothing for Improved Robustness and Accuracy" [2103.04436], an instance-wise variant appears in randomized smoothing, where each input $x$ has its own Gaussian noise standard deviation $\sigma(x)$ and the certified radius is computed with the local $\sigma(x)$. In "InstanceDiffusion: Instance-level Control for Image Generation" [2402.03290], the phrase denotes training-time random dropout of each instance’s text-and-location tokens together with an inference-time Multi-instance Sampler. These usages share the idea of customizing randomness at the instance level, but they address different objects—certified robustness, image generation control, and LLM evaluation stability, respectively.

Within LLM evaluation, ILR is characterized as an easy-to-implement wrapper over any existing evaluation pipeline that requires no changes to the model or prompt-engineering while aiming to yield lower variance and more fair model comparisons than naïve multi-prompt averaging [2509.12678].

Source: https://www.emergentmind.com/topics/instance-level-randomization-ilr