Stochastic Paraphrasing Protocol

• Stochastic Paraphrasing Protocol is an evaluation paradigm that generates randomized, meaning-preserving paraphrases to test NLP models’ stability.
• It leverages LLM-based paraphrase generation and statistical aggregation to expose vulnerabilities to token-level perturbations.
• The protocol shifts evaluation from static benchmarks to dynamic, compute-leveraged stress tests for robust model performance analysis.

The stochastic paraphrasing protocol is an evaluation paradigm for NLP models, particularly sentence embedding models, that leverages randomized, meaning-preserving paraphrase generation at evaluation time. By dynamically perturbing the input data using LLM-driven paraphrasing, the protocol aims to reveal model brittleness and quantify robustness to token-level perturbations that do not affect sentence semantics. In contrast to static benchmark approaches, the stochastic paraphrasing protocol introduces controlled semantic invariance stress tests and statistically grounded reporting, fostering a shift toward dynamic, compute-leveraged evaluation at test time (Frank et al., 8 Oct 2025).

1. Motivation and Rationale

Contemporary embedding evaluations such as the Massive Text Embedding Benchmark (MTEB) employ fixed, human-curated sentence collections. Models exposed repeatedly to these static datasets are susceptible to domain overfitting, data leakage during large-scale pretraining, or over-tuning, potentially inflating benchmark performance while masking deficiencies in real-world generalization. The stochastic paraphrasing protocol addresses these issues by generating fresh paraphrastic variants of test inputs during evaluation, thus decoupling reported robustness from static surface forms and exposing vulnerabilities in token-level invariance—even when underlying semantics are preserved (Frank et al., 8 Oct 2025).

This dynamic protocol enables:

• Shift from static to dynamic evaluation by test-time paraphrase generation.
• Stress-testing of models' ability to remain invariant to lexical/syntactic perturbations.
• Statistically robust measurement via aggregation across multiple paraphrase-based trials.

2. Formal Protocol Definition

Let $D = \{x_1, \ldots, x_n\}$ denote the original test sentence set. The protocol consists of the following formalized steps:

1. Paraphrase-judge selection: Define a semantic similarity scoring function $s(x, y)$, calibrated to semantic textual similarity (STS) gold ratings and yielding values in $[0,5]$.
2. Paraphrase generation: For each $x_i$, generate $R$ stochastic paraphrases $\{y_i^{(r)}\}_{r=1}^R$ using an LLM-based paraphraser.
3. Semantic filtering: Accept only candidate paraphrases $y_i^{(r)}$ satisfying $s(x_i, y_i^{(r)}) \geq \tau$, with default $\tau = 4.0$.
4. Evaluation: For each paraphrase run $r$, replace inputs with $s(x, y)$0, evaluate embedding models on the downstream task, and obtain metric $s(x, y)$1.
5. Statistical aggregation: Aggregate over runs to compute sample means $s(x, y)$2 and variances $s(x, y)$3:

$s(x, y)$4

1. Reporting: Present $s(x, y)$5 and, optionally, bootstrap confidence intervals.

3. Paraphrase Generation and Filtering Methodology

The paraphrasing component employs an LLM constrained to output grammatically correct, meaning-retaining paraphrases exhibiting high token-level diversity. The default prompting template is:

"Rephrase the following text while keeping its original meaning. Only reply with the paraphrased text and provide a single response—no alternatives or notes."

Sampling leverages nucleus sampling (top-$s(x, y)$6 with $s(x, y)$7) and temperature $s(x, y)$8, with reproducibility via fixed random seeds. Quantized models, such as gemma-3-27b Q4_K_M, are favored for efficiency. Each candidate paraphrase undergoes semantic similarity filtering (with threshold $s(x, y)$9) to ensure near-synonymy.

Pseudocode for a single paraphrase run for $[0,5]$0 inputs:

$x_i$7

4. Task Evaluation Pipeline

For each paraphrase set $[0,5]$1, embedding model $[0,5]$2, and evaluation trial:

• Encode originals and paraphrases: $[0,5]$3, $[0,5]$4
• Execute standard downstream evaluations (per MTEB):
  • STS tasks: Compute Spearman's ρ on embedding pairs
  • Classification: Evaluate model using train/test splits on paraphrased representations
  • Retrieval: Report MRR or nDCG@10 using paraphrased queries
• Metric $[0,5]$5 is collected for each run

To assess paraphrase diversity, the protocol reports the average normalized edit distance (ED):

$[0,5]$6

A high ED score signals substantive token-level variation while keeping semantic fidelity (Frank et al., 8 Oct 2025).

5. Statistical Analysis and Confidence Reporting

Using $[0,5]$7 paraphrase runs (default seeds 1337--1342), the stochastic paraphrasing protocol yields a distribution of $[0,5]$8 for robust inference. Uncertainty estimates are computed as:

• Standard Error (SE): $[0,5]$9
• 95% Bootstrap CI: resample $x_i$0 $x_i$1 times and extract 2.5th/97.5th percentiles
• Optionally, paired Wilcoxon signed-rank test yields Hodges-Lehmann location shift $x_i$2 and its CI

These statistical instruments provide a rigorous characterization of embedding model sensitivity to stochastic lexical perturbations.

6. Practical Extensions and Limitations

The protocol is intrinsically model-agnostic; any new embedding model $x_i$3 can be substituted with no modification to the pipeline. For multilingual evaluation, the prompt and paraphraser can be adapted to enforce language consistency, and the same paraphrasing seeds and run count $x_i$4 are deployed. Compute efficiency is a consideration, with paraphrasing via gemma-3-27b requiring approximately $x_i$5 seconds per text on an Nvidia 5090 GPU. For large test suites, subsampling or reducing $x_i$6 is possible.

Potential protocol extensions include:

• Human-in-the-loop semantic equivalence verification to detect rare drift cases
• Integration of adversarial paraphrase generators for stress-testing beyond naturally occurring lexical diversity
• Online continuous benchmarking via synthetic data augmentation

7. Broader Significance

By operationalizing stochastic test-time perturbations, the protocol exposes weaknesses and lexical shortcuts within sentence encoders that may otherwise appear robust on saturated static test beds. A plausible implication is that benchmark-driven research should incorporate such protocols routinely to ensure reported improvements generalize beyond curated inputs. The stochastic paraphrasing protocol, as introduced in the PTEB benchmark, supports a shift toward dynamic, statistically robust evaluation methodologies in NLP (Frank et al., 8 Oct 2025).