Paired-Formula Protocol in Evaluation Design

• Paired-Formula Protocol is a method that matches controlled conditions to enable within-pair contrasts, reducing noise in experimental comparisons.
• It employs rigorous statistical tests such as BCa bootstrap and sign-flip permutation to validate small algorithmic improvements under constrained settings.
• The protocol’s versatile design is applied across machine learning evaluation, dynamic tool auditing, and coordination benchmark settings to ensure precise, actionable insights.

“Paired-Formula Protocol” denotes a class of formal evaluation or specification procedures in which two conditions are matched on the same experimental unit and analyzed through within-pair contrasts rather than independent aggregate summaries. In recent arXiv usage, the term appears in several distinct but structurally related settings: conservative validation of small machine-learning gains through paired multi-seed deltas, measurement of frame-induced behavioral shifts in open-weight LLMs through paired prompts, cost-aware auditing of tool-augmented video agents through paired per-question outcomes, and estimation of local trial-0 noise floors in multi-agent coordination benchmarks through paired discordance counts (Du, 24 Nov 2025, Burnat et al., 7 May 2026, Mohamed et al., 1 Jul 2026, Kaliyev et al., 15 Jun 2026). A broader reading is also supported by earlier work on head-to-head pairwise dialog evaluation, game semantics for formulas as network protocols, and pairing-based cryptographic constructions, although these literatures use the underlying idea in different technical senses (Lee et al., 2020, 0708.1480, Kumar et al., 2021).

1. Core structure and domain scope

Across the recent literature, the paired unit is the central design choice: a seed in controlled benchmarking, an item under alternative prompt framings, a question answered by two complete systems, or a task evaluated under configuration-equivalent protocols. The protocol then defines a per-pair contrast, a summary statistic over those contrasts, and an inferential rule that is explicitly paired rather than unpaired.

Setting	Paired unit	Primary inferential machinery
Small-gain ML evaluation	Matched seed	BCa bootstrap CI; sign-flip permutation test
Evaluation-context divergence	Matched item across frames	Paired-by-item Wilcoxon; cluster-robust logit
Dynamic tool synthesis audit	Same question under two systems	McNemar’s test; paired bootstrap CI
Coordination noise-floor measurement	Same task under two protocols and seed	Paired sign/McNemar; Wilson CI

The common motivation is that small reported differences can be overwhelmed by stochasticity, prompt framing, judge choice, action-space changes, or harness-level perturbations. The paired design is therefore used to reduce nuisance variance, isolate the manipulated factor, and make the estimand correspond to a within-unit difference rather than a between-run comparison. This suggests a family resemblance rather than a single canonical protocol: the “formula” is the formal rule set that maps paired observations into a decision procedure, but the concrete estimands, assumptions, and failure modes vary materially by domain.

2. Conservative validation of small algorithmic gains

In “When +1% Is Not Enough: A Paired Bootstrap Protocol for Evaluating Small Improvements,” the protocol is a compute-aware procedure for validating small gains under tight budgets (Du, 24 Nov 2025). For seeds $s = 1,\dots,n$, both baseline and new method are run under identical conditions, with per-seed delta

$d_s = M_{new}(s) - M_{baseline}(s),$

and target estimand

$\theta = \frac{1}{n}\sum_{s=1}^n d_s.$

The protocol uses a BCa bootstrap confidence interval over the paired deltas and a sign-flip permutation test on the same $d_s$. The null for the test is

$H_0:\, \mathbb{E}[d_s]=0,$

with symmetry of the delta distribution around zero, and the test statistic is

$t=\bar d = \frac{1}{n}\sum_{s=1}^n d_s.$

Under sign flips $\sigma_s\in\{-1,+1\}$, the null distribution is generated through

$t_{flip} = \frac{1}{n}\sum_{s=1}^n \sigma_s d_s.$

The decision rule is intentionally conservative: declare a significant improvement only if the BCa interval excludes $0$ and the sign-flip permutation test yields $p<\alpha$. Recommended settings are $d_s = M_{new}(s) - M_{baseline}(s),$0–$d_s = M_{new}(s) - M_{baseline}(s),$1 seeds under limited compute, $d_s = M_{new}(s) - M_{baseline}(s),$2 bootstrap resamples, and $d_s = M_{new}(s) - M_{baseline}(s),$3 Monte Carlo sign patterns when full enumeration is not used. The paper notes that with very small $d_s = M_{new}(s) - M_{baseline}(s),$4, BCa can be numerically fragile; if jackknife estimates degenerate, percentile bootstrap is the fallback.

The empirical illustrations on CIFAR-10, CIFAR-10N, and AG News are designed to show how single runs and unpaired tests can overstate evidence. With only three seeds, the paired protocol never declares significance for the synthetic small-gain settings discussed. The most striking examples are AG News S1 and S2, where BCa intervals are strictly positive—$d_s = M_{new}(s) - M_{baseline}(s),$5 and $d_s = M_{new}(s) - M_{baseline}(s),$6—yet permutation $d_s = M_{new}(s) - M_{baseline}(s),$7-values remain approximately $d_s = M_{new}(s) - M_{baseline}(s),$8, so the decision rule is not satisfied. The paper’s stated rationale is guardrail behavior: under $d_s = M_{new}(s) - M_{baseline}(s),$9, coarse sign-flip null distributions and high variance on text or noisy-label tasks make conservative non-endorsement a safer default for reported gains in the $\theta = \frac{1}{n}\sum_{s=1}^n d_s.$0 to $\theta = \frac{1}{n}\sum_{s=1}^n d_s.$1 percentage-point range.

3. Evaluation-context divergence in open-weight LLMs

In “Measuring Evaluation-Context Divergence in Open-Weight LLMs,” the protocol formalizes evaluation-context divergence (ECD) as a within-item behavioral change caused solely by reframing a fixed task as evaluation, deployment, or neutral (Burnat et al., 7 May 2026). Frames are

$\theta = \frac{1}{n}\sum_{s=1}^n d_s.$2

and for item $\theta = \frac{1}{n}\sum_{s=1}^n d_s.$3, paraphrase $\theta = \frac{1}{n}\sum_{s=1}^n d_s.$4, decode $\theta = \frac{1}{n}\sum_{s=1}^n d_s.$5, and outcome category $\theta = \frac{1}{n}\sum_{s=1}^n d_s.$6, the binary indicator is

$\theta = \frac{1}{n}\sum_{s=1}^n d_s.$7

The protocol focuses on refusal and harmful compliance.

Per-item outcome probabilities under frame $\theta = \frac{1}{n}\sum_{s=1}^n d_s.$8 are defined as

$\theta = \frac{1}{n}\sum_{s=1}^n d_s.$9

Within-item contrasts include

$d_s$0

with analogous $d_s$1 and $d_s$2 contrasts. Aggregate ECD over $d_s$3 items is

$d_s$4

with

$d_s$5

and a $d_s$6 interval $d_s$7. At pilot scale $d_s$8, the primary test is a paired-by-item Wilcoxon; the trial-level model is a cluster-robust marginal logit with item-clustered Liang–Zeger standard errors.

The design controls are unusually explicit. Each source item is rewritten into two semantically equivalent paraphrases, frame wrappers are held constant except for a short framing clause, and the classifier that labels responses never sees the frame wrapper. The pilot used five instruction-tuned checkpoints from four open-weight families, plus a matched OLMo-3 base/instruct ablation. With $d_s$9 paired items and a $H_0:\, \mathbb{E}[d_s]=0,$0 decoding grid, each checkpoint yields $H_0:\, \mathbb{E}[d_s]=0,$1 generations.

The pilot’s headline result is heterogeneity rather than a uniform frame effect. OLMo-3-7B-Instruct is evaluation-cautious under the primary judge: evaluation versus neutral raises refusal by $H_0:\, \mathbb{E}[d_s]=0,$2 pp with paired-by-item Wilcoxon $H_0:\, \mathbb{E}[d_s]=0,$3, and evaluation versus deployment reduces harmful compliance by $H_0:\, \mathbb{E}[d_s]=0,$4 pp with $H_0:\, \mathbb{E}[d_s]=0,$5; moreover, $H_0:\, \mathbb{E}[d_s]=0,$6 items invert that harmful-compliance direction. By contrast, Mistral-Small-3.2-24B, Phi-3.5-mini, and Llama-3.1-8B are deployment-cautious on refusal, with $H_0:\, \mathbb{E}[d_s]=0,$7 approximately $H_0:\, \mathbb{E}[d_s]=0,$8 pp, $H_0:\, \mathbb{E}[d_s]=0,$9 pp, and $t=\bar d = \frac{1}{n}\sum_{s=1}^n d_s.$0 pp, respectively. The matched OLMo-3 base also shows the deployment-cautious pattern, and the paper interprets this as localizing the inversion to post-training alignment rather than pretraining alone. A further caveat is judge dependence: re-judging all $t=\bar d = \frac{1}{n}\sum_{s=1}^n d_s.$1 generations with Llama-Guard-3-8B preserves the within-OLMo harmful direction but flattens much of the cross-family contrast.

4. Cost-aware paired auditing of dynamic tool synthesis

In “A Cost-Aware, Paired Protocol for Auditing Dynamic Tool Synthesis in Agentic Video Question Answering,” the paired unit is the question-video pair evaluated under two complete systems: $t=\bar d = \frac{1}{n}\sum_{s=1}^n d_s.$2, the baseline Static-SAGE, and $t=\bar d = \frac{1}{n}\sum_{s=1}^n d_s.$3, Dynamic-SAGE (Mohamed et al., 1 Jul 2026). For

$t=\bar d = \frac{1}{n}\sum_{s=1}^n d_s.$4

both systems use the same orchestrator $t=\bar d = \frac{1}{n}\sum_{s=1}^n d_s.$5, and differ only in action space: $t=\bar d = \frac{1}{n}\sum_{s=1}^n d_s.$6 versus $t=\bar d = \frac{1}{n}\sum_{s=1}^n d_s.$7. Final-answer correctness is judged semantically by an evaluator $t=\bar d = \frac{1}{n}\sum_{s=1}^n d_s.$8,

$t=\bar d = \frac{1}{n}\sum_{s=1}^n d_s.$9

and the audit records multiple cost axes per question: $\sigma_s\in\{-1,+1\}$0 for visible tool calls, reasoning turns, tokens, monetary cost, and primitive operations.

The paired differences are

$\sigma_s\in\{-1,+1\}$1

and

$\sigma_s\in\{-1,+1\}$2

A six-group taxonomy classifies each question by joint correctness and sign of $\sigma_s\in\{-1,+1\}$3: safe, neutral, overhead, ideal, costly-gain, and loss. Both wrong $\sigma_s\in\{-1,+1\}$4 is reported separately. Aggregate accuracy difference is

$\sigma_s\in\{-1,+1\}$5

tested with continuity-corrected McNemar, while confidence intervals for paired statistics are obtained by paired bootstrap resampling; the paper uses percentile CIs.

Applied to SAGE-Bench with $\sigma_s\in\{-1,+1\}$6, the protocol yields a deliberately multi-axis conclusion. Accuracy rises from $\sigma_s\in\{-1,+1\}$7 to $\sigma_s\in\{-1,+1\}$8, for $\sigma_s\in\{-1,+1\}$9 points with McNemar $t_{flip} = \frac{1}{n}\sum_{s=1}^n \sigma_s d_s.$0. Mean reasoning turns fall from $t_{flip} = \frac{1}{n}\sum_{s=1}^n \sigma_s d_s.$1 to $t_{flip} = \frac{1}{n}\sum_{s=1}^n \sigma_s d_s.$2 and visible calls from $t_{flip} = \frac{1}{n}\sum_{s=1}^n \sigma_s d_s.$3 to $t_{flip} = \frac{1}{n}\sum_{s=1}^n \sigma_s d_s.$4, both about $t_{flip} = \frac{1}{n}\sum_{s=1}^n \sigma_s d_s.$5 reductions. However, primitive ops increase from $t_{flip} = \frac{1}{n}\sum_{s=1}^n \sigma_s d_s.$6 to $t_{flip} = \frac{1}{n}\sum_{s=1}^n \sigma_s d_s.$7 $t_{flip} = \frac{1}{n}\sum_{s=1}^n \sigma_s d_s.$8, tokens from $t_{flip} = \frac{1}{n}\sum_{s=1}^n \sigma_s d_s.$9k to $0$0k $0$1, and cost from $0$2 $0$3. Latency decreases from $0$4s to $0$5s $0$6. The paper’s explicit interpretation is that Dynamic-SAGE compresses orchestration while shifting rather than uniformly reducing inference cost.

The subgroup analysis sharpens that conclusion. Gains are largest on open-ended, visual, hard, very hard, and long-video questions; they are neutral on verbal and multimodal subsets. The six-group counts are also diagnostic: safe $0$7, neutral $0$8, overhead $0$9, ideal $p<\alpha$0, costly-gain $p<\alpha$1, loss $p<\alpha$2, and both wrong $p<\alpha$3. Net accuracy improvement follows directly as $p<\alpha$4 points. Residual failures cluster on hard and open-ended questions, especially where Dynamic-SAGE uses more calls than the baseline.

5. Paired noise-floor estimation for coordination benchmarks

“How Much Coordination Gain Is Real? A Paired Noise-Floor Protocol for Multi-Agent LLM Benchmarks” uses pairing not to validate a proposed improvement directly, but to estimate a local floor below which reported coordination gains may be indistinguishable from protocol-induced disagreement (Kaliyev et al., 15 Jun 2026). The core contrast is at trial $p<\alpha$5, where coordination hooks are intended to be inert because the task-scoped store is empty. Two protocol inputs are treated as configuration-equivalent when code inspection plus a SHA-256 audit verify identical first-request behavior in system prompt prefix, tool list, sampling parameters, message-array construction, request headers, and pass-through hooks. The paper reports $p<\alpha$6 cells byte-identical on a separate $p<\alpha$7 sample and $p<\alpha$8 live wire-byte audit on user-simulator opens.

For seed $p<\alpha$9, task set $d_s = M_{new}(s) - M_{baseline}(s),$00, and two protocols $d_s = M_{new}(s) - M_{baseline}(s),$01, the trial-0 pass indicator is $d_s = M_{new}(s) - M_{baseline}(s),$02. Discordances are counted as

$d_s = M_{new}(s) - M_{baseline}(s),$03

with ties $d_s = M_{new}(s) - M_{baseline}(s),$04. The per-seed paired gap is

$d_s = M_{new}(s) - M_{baseline}(s),$05

and pooled across seeds,

$d_s = M_{new}(s) - M_{baseline}(s),$06

Inference uses Wilson intervals for the pooled gap through discordant-pair mapping, paired sign or McNemar tests, and Bonferroni correction over multiple contrasts. The paper also defines coordination-active $d_s = M_{new}(s) - M_{baseline}(s),$07 as the minimum reporting metric: $d_s = M_{new}(s) - M_{baseline}(s),$08 where $d_s = M_{new}(s) - M_{baseline}(s),$09 is the subset of tasks for which coordination is logically active.

On Claude Haiku 4.5 against $d_s = M_{new}(s) - M_{baseline}(s),$10-bench retail, the clean configuration-equivalent contrast no_coord versus intercept gives trial-0 gaps of $d_s = M_{new}(s) - M_{baseline}(s),$11 pp and $d_s = M_{new}(s) - M_{baseline}(s),$12 pp across two $d_s = M_{new}(s) - M_{baseline}(s),$13 seeds. Pooled, the gap is $d_s = M_{new}(s) - M_{baseline}(s),$14 pp with Wilson CI $d_s = M_{new}(s) - M_{baseline}(s),$15 and Bonferroni-corrected $d_s = M_{new}(s) - M_{baseline}(s),$16, hence not significant. The largest single-seed contrast is pull versus intercept at $d_s = M_{new}(s) - M_{baseline}(s),$17 pp with corrected $d_s = M_{new}(s) - M_{baseline}(s),$18, but this does not reproduce at the second seed, where the gap is $d_s = M_{new}(s) - M_{baseline}(s),$19 pp and corrected $d_s = M_{new}(s) - M_{baseline}(s),$20. No trial-0 contrast remains significant after Bonferroni at either seed or pooled. The envelope of observed paired gaps spans $d_s = M_{new}(s) - M_{baseline}(s),$21 pp across two seeds, with pooled upper Wilson CI approximately $d_s = M_{new}(s) - M_{baseline}(s),$22 pp.

The paper’s substantive claim is that this envelope constitutes a local noise floor for that model-benchmark regime. Seven of ten recent coordination architectures reportedly have headline effects below that floor, and one more lies inside the envelope. The protocol therefore treats same-model paired replication, multiplicity correction, and improvement in coordination-active $d_s = M_{new}(s) - M_{baseline}(s),$23 as necessary conditions before attributing a benchmark delta to genuine coordination.

6. Antecedents and alternative technical meanings

The paired logic is not unique to the recent works. “An Evaluation Protocol for Generative Conversational Systems” formalized head-to-head pairwise comparison for dialog models, using randomized A/B judgments on the same prompt, majority-vote win–loss–tie counts $d_s = M_{new}(s) - M_{baseline}(s),$24, WMT-style major and distinct scores, and ranking by Bradley–Terry and TrueSkill (Lee et al., 2020). That protocol does not use the label “Paired-Formula Protocol,” but it is an antecedent for paired outcome aggregation in modern evaluation practice. Its application to ten dialog systems found DialoGPT and Blender superior across several datasets, with Blender strongest on multi-turn ESL and DialoGPT very strong on single-turn sets.

A different lineage appears in “Valid formulas, games and network protocols,” where formulas in normal form define interactive protocols through game semantics (0708.1480). The initial state is

$d_s = M_{new}(s) - M_{baseline}(s),$25

which exposes an explicit pair $d_s = M_{new}(s) - M_{baseline}(s),$26 of dual roles. Abelard chooses from $d_s = M_{new}(s) - M_{baseline}(s),$27, Eloise chooses from $d_s = M_{new}(s) - M_{baseline}(s),$28, and Eloise wins exactly when $d_s = M_{new}(s) - M_{baseline}(s),$29 becomes empty. The paper uses this machinery to specify acknowledgement of one packet, acknowledgement with loss and retries, ordered acknowledgement of several packets, and sequential composition by replacing each final negative atomic occurrence $d_s = M_{new}(s) - M_{baseline}(s),$30 in $d_s = M_{new}(s) - M_{baseline}(s),$31 with $d_s = M_{new}(s) - M_{baseline}(s),$32. In this literature, the “paired formula” is literal: protocol execution is organized by a dual formula pair or by a compositional pair $d_s = M_{new}(s) - M_{baseline}(s),$33.

A third usage is cryptographic. “Pairing for Greenhorn: Survey and Future Perspective” treats bilinear pairings as maps

$d_s = M_{new}(s) - M_{baseline}(s),$34

with bilinearity

$d_s = M_{new}(s) - M_{baseline}(s),$35

non-degeneracy, and efficient computability (Kumar et al., 2021). The survey recommends Type-3 pairings on ordinary curves such as BLS12-381 for $d_s = M_{new}(s) - M_{baseline}(s),$36-bit security, describes Miller’s algorithm and final exponentiation, and reports lightweight-device timings including eta-t versus Tate pairing on MICA2, Tmote Sky, and Imote2. Here the pairing is algebraic rather than statistical, but the provided synthesis explicitly frames these constructions as useful for designing and analyzing a “Paired-Formula Protocol” that leverages bilinear or multilinear pairings to encode and verify algebraic relations.

7. Shared assumptions, controversies, and interpretive cautions

Despite their diversity, these protocols converge on a narrow methodological claim: pairing is valuable only when the matched conditions are genuinely controlled. In the small-gain protocol, pairing requires identical seeds, data split and order, epochs, optimizer, and fixed test set; in the ECD protocol, wrappers and roles are held fixed and frame text is stripped from judge inputs; in the Dynamic-SAGE audit, the orchestrator is held constant and only the action space changes; in the coordination noise-floor protocol, configuration-equivalence is substantiated by code inspection and SHA-256 byte audit of first-request payloads (Du, 24 Nov 2025, Burnat et al., 7 May 2026, Mohamed et al., 1 Jul 2026, Kaliyev et al., 15 Jun 2026).

The main controversies concern what pairing can and cannot guarantee. A paired design reduces nuisance variance, but it does not automatically yield stable inference at small $d_s = M_{new}(s) - M_{baseline}(s),$37. The small-gain protocol explicitly warns that BCa can be unstable when $d_s = M_{new}(s) - M_{baseline}(s),$38 is extremely small and deltas are discrete or nearly identical, and that sign-flip validity depends on approximate symmetry. The ECD study shows that cross-family conclusions can be judge-dependent even when within-item contrasts are cleanly defined. The Dynamic-SAGE audit shows that a paired accuracy gain can coexist with increased primitive operations, tokens, and monetary cost, so scalar performance summaries can be misleading. The coordination protocol emphasizes that a measured floor is local rather than universal: cross-model and cross-domain probes shift both magnitude and direction.

A common misconception is that a positive confidence interval, a significant $d_s = M_{new}(s) - M_{baseline}(s),$39-value, or a favorable win count is sufficient in isolation. The recent paired protocols reject that simplification. The conservative small-gain procedure requires both $d_s = M_{new}(s) - M_{baseline}(s),$40 and $d_s = M_{new}(s) - M_{baseline}(s),$41; the ECD study insists on paired within-item contrasts, judge controls, and matched base/instruct ablations; the Dynamic-SAGE audit requires joint reporting of correctness and cost axes; the coordination paper requires an effect to exceed the local noise-floor envelope, survive multiplicity correction, and reproduce across seeds. Taken together, these works define the paired protocol not merely as a statistical convenience, but as a design discipline for making small or fragile differences interpretable under constrained, high-variance experimental conditions.