Paired Testing Protocol

• Paired testing protocol is an experimental design that compares matched units (e.g., items, users, prompts) using within-pair differences to isolate treatment effects.
• It employs various inferential tools, including permutation tests, paired t-tests, and bootstrap methods, to robustly quantify system contrasts.
• Rigorous experimental controls and artifact isolation ensure that observed differences are due to treatment variables rather than hidden variation.

Paired testing protocol denotes a class of experimental and inferential designs in which two systems, treatments, framings, or serving conditions are evaluated on the same experimental units, and the primary object of analysis is the within-pair contrast rather than an unpaired difference of averages. In the cited literature, the paired unit may be a test item, a matched experimental pair, a random seed, a user, a benchmark instance, a prompt, a question answered by two complete pipelines, or a before/after matrix observation. Across these settings, the null hypothesis is typically that the paired difference is centered at zero or that labels are exchangeable within each pair, while the alternative is directional or two-sided depending on the application (Zmigrod et al., 2022, Wu et al., 2019, Zhang et al., 2024, Zhang et al., 2024).

1. Core formal structure

A canonical formulation appears in the paired-permutation framework for comparing two systems $U$ and $V$ on $N$ test items. Each system produces vectors $u=(u_1,\dots,u_N)$ and $v=(v_1,\dots,v_N)$, the null hypothesis is that the system label is independent of the per-item outcomes, and the test statistic has the additively decomposable form

$t(u,v)=h\!\Bigl(\sum_{n=1}^N g(u_n,v_n)\Bigr),$

with integer-valued local effect $g$ and summary function $h$. Under random label-swapping, all $2^N$ swaps are equally likely, and the $p$-value is the tail probability of the swapped statistic $V$0, where $V$1 is the sum of the independently swapped local gains (Zmigrod et al., 2022).

Matched-pair experimentation uses the same logic with different notation. In paired experiments with $V$2 matched pairs, exactly one unit in each pair is randomly assigned to treatment, the observed pairwise contrast is $V$3, and the classical estimator is

$V$4

This estimator is unbiased under random assignment, and paired covariate adjustment can then be layered on top through leave-one-out potential-outcome imputation, as in the P-LOOP estimator (Wu et al., 2019).

Recent LLM evaluation papers express the same design at the level of prompts, seeds, or questions. A paired gap may be written as $V$5 for binary success indicators, or more generally as $V$6 for shared-prompt score differences. This notation makes explicit that the experimental unit is held fixed while only the system or condition varies (Kaliyev et al., 15 Jun 2026, Kotawala, 28 May 2026).

In high-dimensional settings, the protocol remains paired but the contrast is first projected into a univariate representation. The MWSR framework constructs a perpendicular bisecting hyperplane for each pair $V$7, aggregates those hyperplanes through a Hodges–Lehmann-style pseudomedian, and then applies a Wilcoxon signed-rank test to the resulting scalar paired differences $V$8 (Bargiotas et al., 2023).

2. Test statistics and inferential machinery

The statistical core of paired testing protocols is heterogeneous. Exact paired-permutation testing for structured statistics computes the probability mass function of the swapped sum $V$9 by convolving $N$0 independent two-point distributions. A dynamic-programming implementation runs in $N$1 time, while an FFT-based balanced binary-tree convolution runs in

$N$2

with space $N$3. On a POS-tagging dataset with $N$4, the exact FFT-based test runs in approximately $N$5 s, whereas Monte Carlo with $N$6 swaps takes approximately $N$7 s, yielding a reported $N$8 speedup (Zmigrod et al., 2022).

When only a few random seeds are affordable, a different inferential stack is used. A paired multi-seed protocol records per-seed deltas $N$9, constructs a BCa bootstrap confidence interval for the mean delta, and combines it with a sign-flip permutation test based on random $u=(u_1,\dots,u_N)$0 multipliers on the observed deltas. The reported decision rule is deliberately conservative: an improvement is called significant only if the BCa lower bound exceeds zero and the two-sided permutation $u=(u_1,\dots,u_N)$1-value is below $u=(u_1,\dots,u_N)$2 (Du, 24 Nov 2025).

Binary paired outcomes often invite discrete paired tests. In a cost-aware audit of VideoQA pipelines, McNemar’s test is used for final-answer accuracy, while paired bootstrap confidence intervals are used for $u=(u_1,\dots,u_N)$3, $u=(u_1,\dots,u_N)$4, $u=(u_1,\dots,u_N)$5, and $u=(u_1,\dots,u_N)$6. The same paper sorts each paired question-level outcome into six groups defined jointly by correctness and cost change: safe, neutral, overhead, ideal, costly-gain, and loss, with both-fail reported separately (Mohamed et al., 1 Jul 2026).

Other paired protocols tailor the test statistic to the data source. PaCoST defines confidence differences $u=(u_1,\dots,u_N)$7 between each benchmark instance and its paraphrased counterpart and applies a one-sided paired-sample $u=(u_1,\dots,u_N)$8-test under the null $u=(u_1,\dots,u_N)$9 and the alternative $v=(v_1,\dots,v_N)$0, where $v=(v_1,\dots,v_N)$1 is the mean confidence difference (Zhang et al., 2024). Evaluation-context divergence in open-weight LLMs is analyzed at pilot scale by paired-item Wilcoxon signed-rank tests and, in the primary specification, by marginal logistic regression with item-clustered standard errors (Burnat et al., 7 May 2026).

A recurrent implication is that paired testing is not tied to one test family. Exact convolution, paired $v=(v_1,\dots,v_N)$2-tests, Wilcoxon procedures, McNemar tests, bootstrap intervals, clustered GLMs, and FDR-controlled multiple testing all appear in the literature, but they are all anchored in the same matched-unit design (Ye et al., 2019).

3. Experimental controls that make the pairing credible

The validity of a paired testing protocol depends on whether the two sides of each pair are genuinely matched. In the paired noise-floor protocol for multi-agent LLM benchmarks, Kaliyev and Maryanskyy require configuration-equivalent API inputs at trial $v=(v_1,\dots,v_N)$3: code inspection is used to verify identical system-prompt templates, tool-list definitions, sampling parameters, and message-array construction; a SHA-256 byte audit records headers, system prompt, user messages, and tool list; and a wire-byte audit live-captures the actual TCP/IP payload for a subset of requests to confirm bitwise identity. The same protocol bounds server-side $v=(v_1,\dots,v_N)$4 stochasticity by a paired within-protocol replicate with $v=(v_1,\dots,v_N)$5, and treats trial $v=(v_1,\dots,v_N)$6 as the local noise floor because the coordination store is empty for all protocols at that stage (Kaliyev et al., 15 Jun 2026).

The paired-prompt protocol for evaluation-context divergence imposes a different set of controls. Each benchmark item is written in two semantically matched paraphrases, three fixed frame prefixes are inserted for evaluation, deployment, and neutral conditions, and each prompt is decoded on a grid of one deterministic sample plus multiple stochastic samples. The primary judge is blind to framing because the frame prefix is stripped from the judge input; a second judge is then used for cross-judge sensitivity analysis (Burnat et al., 7 May 2026).

Batch-conditioned refusal robustness introduces yet another layer of control. The same prompt is evaluated under two serving conditions that differ only in batch size, dispatch synchronization, or co-batch composition, and safety prompts are paired with capability controls so that generic output churn is observable on a non-safety axis. Because automated scoring produced many candidate flips that were not genuine behavioral flips, the protocol includes scorer correction and manual adjudication. In the reported Study A, $v=(v_1,\dots,v_N)$7 candidate changed rows reduced to $v=(v_1,\dots,v_N)$8 genuine boundary flips after adjudication (Kadadekar, 26 May 2026).

These controls are not interchangeable, but they serve a common purpose: to ensure that the observed paired difference can be attributed to the treatment variable under study rather than to hidden variation in prompts, judges, harnesses, schedulers, or serving kernels. This suggests that protocol design is not merely an inferential choice; it is also an exercise in artifact isolation.

4. Domain-specific instantiations

The paired testing template has been instantiated across a wide range of research programs.

Setting	Paired unit	Main reported objective
NLP system comparison	same test item	exact paired-permutation $v=(v_1,\dots,v_N)$9-value for structured statistics (Zmigrod et al., 2022)
Multi-agent LLM coordination	same task under matched protocols and seeds	estimate a trial-0 noise floor and report $t(u,v)=h\!\Bigl(\sum_{n=1}^N g(u_n,v_n)\Bigr),$0 and $t(u,v)=h\!\Bigl(\sum_{n=1}^N g(u_n,v_n)\Bigr),$1 (Kaliyev et al., 15 Jun 2026)
Benchmark contamination detection	original benchmark item and paraphrased counterpart	test whether confidence is higher on the original data (Zhang et al., 2024)
Evaluation-context divergence	same task under $t(u,v)=h\!\Bigl(\sum_{n=1}^N g(u_n,v_n)\Bigr),$2, $t(u,v)=h\!\Bigl(\sum_{n=1}^N g(u_n,v_n)\Bigr),$3, and $t(u,v)=h\!\Bigl(\sum_{n=1}^N g(u_n,v_n)\Bigr),$4 framing with two paraphrases	estimate within-item frame contrasts (Burnat et al., 7 May 2026)
Agentic VideoQA audit	same question answered by two complete systems	report joint accuracy–cost differences and six paired outcome groups (Mohamed et al., 1 Jul 2026)
Paired A/B experiments	same user participating in two experiments	collaboratively estimate test effects using a BLUE (Zhang et al., 2024)

Several additional instantiations deepen the picture. In matched-pair causal inference, P-LOOP adjusts the classical paired estimator by leaving out each pair and imputing its potential outcomes with a prediction algorithm such as lasso or random forests. The method is explicitly designed to resolve the trade-off between ignoring pair identities and including pair-specific structure in the imputation model (Wu et al., 2019).

In collaborative analysis for paired A/B tests, Zhang, Kang, and Deng consider two A/B experiments run on the same user pool. Their collaborative estimator combines the single-test estimator and the paired-difference estimator, and is asymptotically the best linear unbiased estimator under the stated mixed-model assumptions (Zhang et al., 2024).

In neuroscience, paired matrix-graph testing compares partial-correlation edges in correlated matrix observations before and after a stimulus. The protocol uses bias-corrected residual covariances, variance correction, standardized edgewise statistics $t(u,v)=h\!\Bigl(\sum_{n=1}^N g(u_n,v_n)\Bigr),$5, and a multiple-testing procedure that asymptotically controls the false discovery rate (Ye et al., 2019).

In high-dimensional paired-sample testing, the MWSR construction turns each paired observation into a local linear decision rule, aggregates those rules via coordinate-wise medians of Walsh averages, and applies a Wilcoxon signed-rank test to the induced scalar differences. The stated advantage is improved testing accuracy over traditional multivariate and multiple-testing baselines while also estimating each feature’s contribution (Bargiotas et al., 2023).

5. Sample size, power, and resolution

Paired testing protocols increasingly treat sample-size adequacy as part of the protocol rather than as an external planning exercise. In the multi-agent LLM noise-floor paper, sample-size recommendations for $t(u,v)=h\!\Bigl(\sum_{n=1}^N g(u_n,v_n)\Bigr),$6-bench retail at baseline $t(u,v)=h\!\Bigl(\sum_{n=1}^N g(u_n,v_n)\Bigr),$7 are given for both independent and paired designs. For a paired design, the reported $t(u,v)=h\!\Bigl(\sum_{n=1}^N g(u_n,v_n)\Bigr),$8 targets are $t(u,v)=h\!\Bigl(\sum_{n=1}^N g(u_n,v_n)\Bigr),$9 for a $g$0 pp effect, $g$1 for $g$2 pp, $g$3 for $g$4 pp, $g$5 for $g$6 pp, and $g$7 for $g$8 pp. The same paper states that detecting a $g$9 pp paired gain requires approximately $h$0 tasks in one seed, and that two seeds, or $h$1 tasks, bring the paired upper Wilson confidence bound under $h$2 pp (Kaliyev et al., 15 Jun 2026).

Resolution diagnostics generalize this concern. For paired LLM comparisons on shared prompts, the required sample size to detect an effect $h$3 with level $h$4 and power $h$5 is

$h$6

where $h$7. The primary diagnostic is the resolution ratio

$h$8

If $h$9, the benchmark has at least the target power for a gap of the observed size; if $2^N$0, the comparison is unresolved at the stated $2^N$1 target (Kotawala, 28 May 2026).

The same paper identifies a specific pitfall in practice. The widely used unpaired Cohen-$2^N$2-plus-$2^N$3 shortcut deviates from the correct paired $2^N$4 by approximately a factor of two in the close-comparison regime, and three of five off-the-shelf calculators are said to silently inherit this deficit when the user post-multiplies their per-arm output by $2^N$5 (Kotawala, 28 May 2026).

Small-budget paired protocols make a related point from the opposite direction. The paired bootstrap protocol for small gains recommends a minimum of $2^N$6 seeds and notes that $2^N$7 increases power while retaining the same guardrails. Its central argument is that tight-budget evaluation should default to conservative paired inference rather than to optimistic single-run claims (Du, 24 Nov 2025).

6. Distinction from pair-wise testing and combinatorial pair testing

Despite the lexical similarity, paired testing protocol is not synonymous with pair-wise testing in software engineering. Pair-wise testing, also called $2^N$8-wise or all-pairs testing, is a black-box combinatorial approach whose goal is to exercise every possible pair of input values at least once. Its formal object is the covering array $2^N$9, and the standard generation families include orthogonal-array methods, IPOG-style algorithms, and AETG-style greedy constructions (Sanchez, 2016).

This software-testing line includes metaheuristic generators such as the Pairwise Gravitational Search Algorithm Strategy. PGSAS treats each candidate test case as an object in a GSA population, uses a One-Test-At-a-Time procedure, and was benchmarked against existing $p$0-way strategies in terms of test-suite size. The reported parameter settings include $p$1, $p$2, $p$3, $p$4, $p$5, and $p$6 ratio $p$7 (Htay et al., 2021).

Industrial studies use the same terminology in yet another sense. Charbachi et al. encode PLC input models into SEAFOX, invoke the IPOG algorithm with $p$8, and compare automatically generated pairwise suites against manually handcrafted suites across $p$9 industrial programs. Their abstract reports that pairwise testing is almost as effective in fault detection as manual testing and is just as good as manual testing at fault detection for $V$00 of the programs (Charbachi et al., 2017).

Combinatorial Pair Testing is different again. In that framework, tests are performed on unordered participant pairs $V$01, a pair fails only when both members are slackers, and the objective is to identify the full slacker set using as few synchronous matching rounds as possible. Adaptive algorithms achieve $V$02 rounds, whereas non-adaptive randomized algorithms require $V$03 rounds (Eppstein et al., 2013).

A plausible implication is that the word pair identifies at least three distinct methodological traditions: matched-unit statistical comparison, combinatorial coverage of parameter pairs, and pair-based identification or screening problems. The protocols are related by structure, not by identity.

7. Limitations, controversies, and reporting norms

Several limitations recur across the paired-testing literature. Exactness is not always free. For structured integer-valued statistics, the exact FFT-based paired-permutation algorithm is attractive when $V$04 is up to tens of thousands and $V$05 is small, but the same paper notes that Monte Carlo may be the only practical route for very large $V$06 or multi-dimensional gains (Zmigrod et al., 2022).

Small observed gains remain difficult to interpret even under pairing. In the paired bootstrap protocol, single runs and unpaired $V$07-tests often suggest significance for $V$08–$V$09 point improvements, but with only three seeds the paired BCa-plus-permutation protocol never declares significance in the studied small-gain and medium-gain scenarios. The protocol is explicitly conservative and is proposed as a guardrail against over-claiming (Du, 24 Nov 2025).

Labeling and judging can also destabilize paired conclusions. In batch-conditioned refusal robustness, automated discovery overstated the number of meaningful flips until scorer correction and adjudication were applied, and in the targeted kernel ablation the same score-flip candidates dropped from $V$10 label flips under standard vLLM to $V$11 under a batch-invariant kernel. In evaluation-context divergence, the within-OLMo eval-cautious direction persisted under a second judge, but the cross-family heterogeneity flattened, indicating judge-dependent operationalization of the underlying construct (Kadadekar, 26 May 2026, Burnat et al., 7 May 2026).

The literature also warns against interpreting aggregate null results as proof of invariance. The continuous-batch composition study reported null omnibus tests at a $V$12 pp minimum detectable effect, yet the directional caveat was that $V$13 of $V$14 observed flips leaned unsafe. Similarly, the multi-agent noise-floor protocol treats gains that fall inside the empirical trial-$V$15 envelope as possibly noise rather than as confirmed coordination benefits (Kadadekar, 26 May 2026, Kaliyev et al., 15 Jun 2026).

A common reporting norm therefore emerges. The cited papers recommend matched conditions, transparent per-pair deltas, explicit confidence intervals and $V$16-values, sensitivity to multiplicity or judge dependence when relevant, and a clear distinction between local paired evidence and broader claims. This suggests that paired testing protocol is best understood not as a single test, but as a disciplined way of converting matched observations into credible statistical evidence under controlled variation (Kotawala, 28 May 2026).