TAIS: Invariance Screen for Speculative Decoding

• TAIS is a behavioural-equivalence screen that tests whether speculative decoding outputs remain statistically indistinguishable from target-only decoding on preset safety and capability tasks.
• The methodology combines three criteria—byte-identity, two one-sided tests (TOST) at ±3 percentage points, and Cohen’s h effect-size cutoff—to rigorously assess output equivalence.
• Empirical findings show that, under T=0 conditions and using vLLM stacks, TAIS achieves negligible safety divergence, with maximal observed effect sizes far below conventional thresholds.

Searching arXiv for the specified paper to ground the article in the cited source. Typical-Acceptance Invariance Screen (TAIS) is a behavioural-equivalence protocol for greedy speculative decoding at temperature zero. It is designed to test whether, on a fixed battery of safety and capability tasks, the refusal or safety-pass rates produced by a speculative-decoding stack are statistically indistinguishable—within a pre-specified tolerance—from those produced by target-only greedy decoding. In the formulation introduced in "Speculative Decoding at Temperature Zero: A Scoped Safety-Invariance Screen with a 48,072-Sample Expansion," TAIS operationalizes this objective by pairing target-only and speculative outputs on identical prompts and evaluating three invariance criteria per task: byte-identity, TOST equivalence at $\pm 3$ percentage points, and a small-effect cutoff on Cohen’s $h$ (Kadadekar, 23 Jun 2026).

1. Definition and acceptance criterion

TAIS is defined as a behavioural-equivalence screen rather than as a general safety guarantee. Its immediate objective is narrow and explicit: to certify, for greedy decoding at $T=0$, that speculative decoding does not induce detectable divergence in safety-scored outputs relative to target-only decoding on the tested task battery. The protocol therefore compares matched prompt-level outputs from two execution modes—target-only and speculative—under the same evaluation conditions.

A stack “passes TAIS” only if every well-conditioned per-task contrast meets all three criteria. This requirement is conjunctive rather than aggregate: byte-level agreement, statistical equivalence, and bounded effect size are all required for each eligible task-level comparison. The protocol is therefore scoped to invariance evidence on a defined benchmark set, not to model safety in a broader sense.

The name reflects the inclusion of probabilistic “typical acceptance” among the tested serving modes, but the screen itself is not limited to that policy. In the reported evaluation, the serving stack uses vLLM v0.19 with both rejection sampling and probabilistic “typical acceptance” modes under greedy temperature-zero execution.

2. Formal criteria

For each task $t$, TAIS compares the target-only safety-pass rate $p_{\text{target}}^{(t)}$ and the speculative-decoding safety-pass rate $p_{\text{spec}}^{(t)}$, with tolerance $\Delta = 0.03$. The protocol applies three criteria (Kadadekar, 23 Jun 2026).

First, Criterion C1: byte-identity. At $T=0$, TAIS compares the raw output bytes of target-only and speculative decoding on each prompt. The comparison is labeled “strong” if at least $99.5\%$ of byte positions match; otherwise it is labeled “moderate” if Criteria C2 and C3 still pass. This criterion addresses prompt-level output identity directly, rather than only downstream agreement in task-level labels.

Second, Criterion C2: Two One-Sided Tests (TOST) at $\pm \Delta$. TAIS rejects both null hypotheses

$h$0

and

$h$1

at significance level $h$2 after Holm–Bonferroni correction across tasks. Passing both one-sided tests establishes equivalence in the sense that $h$3.

Third, Criterion C3: Cohen’s $h$4 effect-size cutoff. TAIS defines

$h$5

and requires

$h$6

The reported cutoff $h$7 was post-hoc calibrated so that it sits approximately four times above the largest observed null-consistent $h$8 and remains below Cohen’s conventional “trivial effect” floor of $h$9. The paper states that future users should re-calibrate on held-out data or conservatively adopt $T=0$0.

Taken together, these criteria combine exact-output evidence, equivalence testing, and effect-size control. This suggests that TAIS is intended to screen for both visible output perturbations and smaller rate-level deviations that might survive byte mismatch.

3. Evaluation design and experimental scope

The reported evaluation has four main components: a confirmatory core, a matched expansion, a one-arm production-scale probe, and a fixed serving and scoring stack (Kadadekar, 23 Jun 2026).

The confirmatory core (E0) contains 16,783 paired samples across three model-pair families: Llama-3.2-3B+1B, Qwen-2.5-3B+1.5B, and Qwen-2.5-1.5B+0.5B. It covers four safety tasks—AdvBench refusal, jailbreak amplification, BBQ bias, and TruthfulQA—plus two capability controls, MMLU and ARC-Challenge.

The matched expansion adds 44,066 paired samples through four orthogonal stress tests:

• E2: DPO-adversarial 1B draft (4,006 samples)
• E3: GPTQ-4bit 1B draft (4,006 samples)
• E4: seed replication with seeds 123 and 456 (24,036 samples)
• E5: accumulator dtype swap to bf16 (12,018 samples)

The one-arm production-scale probe (E1) consists of 4,006 samples on Llama-3.1-70B target + 8B draft, but it lacks a matched target-only arm.

The serving stack is vLLM v0.19, run with greedy temperature zero, under both rejection sampling and probabilistic “typical acceptance” modes, with fp16 accumulator in the core and E2–E4, and bf16 in E5. Safety scoring uses deterministic regex classifiers for safety tasks; all expansion-cell conclusions use regex labels only. An LLM judge (Gemma 3 12B) provides a cross-check on the core.

This design is explicitly structured around matched comparisons. The one-arm 70B probe is therefore informative but not sufficient for a TAIS pass, because the protocol’s central logic depends on paired target-only versus speculative contrasts.

4. Empirical findings

Within the matched evaluations, the paper reports no detectable safety divergence under TAIS for the tested temperature-zero vLLM stacks (Kadadekar, 23 Jun 2026). The most prominent summary statistic is that the maximum observed $T=0$1 across all matched AdvBench-refusal contrasts (E0+E2–E5) is 0.024, described as roughly an order of magnitude below the conventional trivial-effect floor.

For TOST, the core per-task ledger comprises 27 contrasts across six tasks. Of these, 25 pass at $T=0$2 percentage points. The two non-passes occur only on capability tasks (MMLU, ARC) at rate = 1.0 boundary (Wald-CI degenerate) and are flagged but not counted as failures. In the expansion, all 17 matched AdvBench-refusal contrasts pass TOST at $T=0$3 percentage points with Holm–Bonferroni–adjusted $T=0$4. For E5 (bf16), all nine per-task contrasts pass TOST at $T=0$5 percentage points, with max $T=0$6.

For byte-identity, the reported outcomes differ by stress condition. E2 (DPO-adversarial draft vs canonical) yields 100.00% byte-identity on 4,006 samples. E3 (GPTQ-4bit) and E4 (seed replication) likewise yield 100.00% identity. By contrast, E5 (bf16) yields only 36–53% identity per pair, yet all C2 and C3 comparisons still pass.

The paper also reports that the DPO-adversarial draft produces byte-identical output to the canonical draft across 4,006 samples, and that bf16 changes 36%–53% of output bytes without moving any per-task safety rate outside equivalence. These two observations jointly separate exact string identity from task-level invariance: one can observe full byte agreement under some perturbations, and substantial byte change under others, while still retaining equivalence at the level of measured refusal or safety-pass rates.

For the 70B production probe (E1), which is not counted as a TAIS pass, the paper reports AdvBench refusal = 0.839 over 700 AdvBench completions with 95% Wilson CI [0.809, 0.864]. It also reports that the dose-response over speculation lengths $T=0$7 is flat, with slope $T=0$8.

5. Interpretation and methodological significance

TAIS is best understood as a scoped invariance screen for speculative decoding rather than as a universal safety benchmark. Its central methodological contribution is the combination of prompt-level raw-byte comparison with task-level equivalence testing and effect-size control. This produces a three-layer acceptance logic: exact output comparison, bounded rate difference, and bounded standardized effect.

The reported findings support a narrow conclusion. Within the explicitly stated regime—$T=0$9, vLLM v0.19, Llama 3.x and Qwen 2.5 families, six tasks, and rejection sampling or typical acceptance—the paper states that TAIS finds no detectable safety divergence between target-only and speculative stacks. The summary given in the paper is that max $t$0, all qualifying TOST contrasts pass at $t$1 percentage points, byte-identity is at least 99.5% except when modulated by dtype, and the 70B probe yields a plausible refusal rate of 0.839 [0.809, 0.864] (Kadadekar, 23 Jun 2026).

The reported non-pass cases are also methodologically important. They occur only in capability-domain contrasts at a boundary condition with identical ceiling rates and degenerate Wald confidence intervals. The paper characterizes these as edge cases, not as evidence of genuine non-equivalence. This matters because TAIS is framed as an equivalence procedure: failure to establish equivalence under such conditions is not automatically evidence of difference.

A plausible implication is that TAIS is intended to distinguish between three situations that are often conflated: exact deterministic reproducibility, statistical equivalence in downstream safety rates, and broader claims about safety preservation under deployment. The protocol addresses the first two directly and leaves the third outside scope.

6. Limitations, non-claims, and boundary conditions

The paper defines its limitations explicitly and repeatedly restricts the scope of its conclusions (Kadadekar, 23 Jun 2026). Temperature is the foremost boundary: TAIS applies only to $t$2 greedy decoding; behavior at $t$3 is untested. Framework coverage is limited to vLLM v0.19; TGI, TensorRT-LLM, and SGLang are untested. Model coverage is limited to the Llama 3.x and Qwen 2.5 families.

The tested speculation policies are limited to strict rejection sampling and probabilistic typical acceptance. The paper does not make claims about tree-speculation variants such as EAGLE, Medusa, and SpecInfer, nor about retrieval-based drafting or lookahead/parallel schemes. The task battery is also limited: single-turn refusal, bias, and truthfulness tasks—AdvBench, jailbreak, BBQ, and TruthfulQA—plus the capability controls MMLU and ARC-Challenge. Multi-turn, agentic, and red-teaming benchmarks are outside scope.

The statistical limits are also stated. The paper reports per-cell MDE $t$4–$t$5 percentage points in the core and pooled $t$6 percentage points. It further notes that high-ceiling cells ($t$7–$t$8) have weaker power. Accordingly, a failure to detect divergence outside these sensitivity limits would not be ruled out by the reported experiments.

Finally, the paper highlights a calibration caveat for the effect-size threshold: $t$9 was post-hoc tuned on the same expansion data, although a held-out check preserved the cutoff structure within 3% of 0.1. The recommendation is that independent recalibration be used for new regimes. This limitation is central to interpretation: TAIS is presented as a transferable screening framework, but not as a fixed thresholding rule that can be applied unchanged across all model, framework, or benchmark settings.