Random Selection Probing (RSP) Framework

• RSP is a hypothesis-testing framework that detects statistical correlations between models by transferring optimized prefixes.
• It rigorously computes a one-sided p-value under a binomial null hypothesis to assess whether a target model is correlated with a reference model.
• The framework incorporates baseline adjustments and diverse optimization methods to control false positives across varied access regimes.

Searching arXiv for the specified paper and closely related usage of “Random Selection Probing”. Random Selection Probing (RSP) is a hypothesis-testing framework for model correlation detection that determines whether a target model is statistically correlated with a reference model, for example because it was fine-tuned from that model or is even identical to it. The method operates by optimizing textual or visual prefixes on a reference model for a random selection task and then evaluating the transferability of those prefixes to a target model. Its defining output is a rigorous one-sided p-value rather than a heuristic similarity score, and the framework incorporates an unrelated baseline model to filter generic, transferable features that could otherwise inflate false positives (Chen et al., 29 Sep 2025).

1. Definition and problem formulation

RSP addresses the problem of deciding whether two models share lineage or other statistically detectable dependence under diverse access conditions. In the formulation given for LLMs and vision-LLMs (VLMs), the reference model is denoted $M_r$, the target model $M_t$, and an unrelated baseline model $M_u$ when baseline adjustment is used (Chen et al., 29 Sep 2025).

The framework defines a null hypothesis $H_0$ under which $M_r$ and $M_t$ are independent, so that probing prefixes optimized on $M_r$ produce no better-than-random performance on $M_t$. The alternative hypothesis $H_1$ states that the models are correlated and that prefixes tuned on $M_r$ transfer with success rate above random (Chen et al., 29 Sep 2025).

The random selection task is discrete with $M_t$0 equally likely choices. The exposition gives the example $M_t$1 letters. RSP optimizes $M_t$2 prefixes $M_t$3 on the reference model, each associated with a designated target token index $M_t$4. Evaluation on the target model counts how often the predicted index matches that designated target (Chen et al., 29 Sep 2025).

Formally, the observed number of successes is

$M_t$5

Under $M_t$6, each trial succeeds with probability $M_t$7 independently, so

$M_t$8

The one-sided p-value is then

$M_t$9

where $M_u$0 is the regularized incomplete beta-function (Chen et al., 29 Sep 2025).

This construction makes the inferential target explicit: RSP does not merely score similarity, but instead tests whether transfer performance is too strong to be explained by the binomial null.

2. Prefix optimization and random selection tasks

RSP parameterizes prefixes by a continuous or discrete vector $M_u$1. For textual prefixes, $M_u$2 may index the choice of $M_u$3 tokens; for visual prefixes, $M_u$4 is an image of size $M_u$5 (Chen et al., 29 Sep 2025).

The method defines a selection loss on the reference model:

$M_u$6

Equivalently, optimization may be stated as maximizing the likelihood of the designated target:

$M_u$7

When an unrelated baseline model $M_u$8 is incorporated, the optimization objective becomes

$M_u$9

This objective is intended to amplify model-specific rather than generic features (Chen et al., 29 Sep 2025).

The detailed exposition further states that, in practice, the method samples $H_0$0 pairings $H_0$1 and optimizes each prefix accordingly. This suggests a two-stage structure: first learn probe inputs on the reference side, then assess transfer on the target side. A plausible implication is that RSP uses probe construction itself as the vehicle for statistical identification, rather than treating probe inputs as externally fixed.

3. Statistical testing and access regimes

After prefix optimization, RSP evaluates the optimized prefixes on the target model and converts the resulting success count into a p-value under the binomial null (Chen et al., 29 Sep 2025).

In the gray-box setting with logit access, evaluation proceeds by computing, for each prefix $H_0$2,

$H_0$3

and then counting successes via $H_0$4.

In the black-box setting with text-only access, the method repeats each query $H_0$5 times, estimates empirical frequencies $H_0$6, sets $H_0$7, and forms the same success statistic $H_0$8 (Chen et al., 29 Sep 2025).

The algorithmic description emphasizes that the test statistic is unchanged across these regimes; only the manner of estimating the relevant argmax differs. This is significant because the abstract explicitly frames the framework as applicable under diverse access conditions for reference and test models, and the details enumerate gradient-accessible, logits-only, gray-box, and black-box variants for both LLMs and VLMs (Chen et al., 29 Sep 2025).

RSP therefore couples a fixed inferential core—the binomial test—with multiple operational instantiations. This suggests that its principal claim to generality lies less in any specific optimizer than in the invariance of the testing logic across access modalities.

4. Baseline model adjustment and false-positive control

A distinctive feature of RSP is the use of an unrelated baseline model $H_0$9 during prefix optimization. The baseline-adjusted objective is

$M_r$0

According to the exposition, this “differencing” ensures that the learned prefix relies on idiosyncrasies of $M_r$1 (Chen et al., 29 Sep 2025).

The statistical test itself is unchanged after this adjustment; the baseline acts only in the optimization stage. The reason for this design is stated explicitly: prefixes optimized without $M_r$2 occasionally transfer to truly unrelated models, inflating Type I error (Chen et al., 29 Sep 2025). The abstract summarizes the same point more broadly, stating that RSP incorporates an unrelated baseline model to filter out generic, transferable features and thereby mitigate false positives (Chen et al., 29 Sep 2025).

The reported ablation findings reinforce this role. Removing the unrelated baseline $M_r$3 increases false positives on some unrelated LLMs (Chen et al., 29 Sep 2025). The discussion section then characterizes this component as part of RSP’s false positive control, with the stated effect of keeping Type I error near the chosen significance level $M_r$4 (Chen et al., 29 Sep 2025).

This component is central to distinguishing correlation-specific transfer from broad prompt transferability. A plausible implication is that RSP is not merely a transfer test, but a transfer test conditioned against a nuisance source of generic elicitation effects.

5. Algorithmic structure and implementation regimes

The framework is described procedurally in two stages. Stage 1 performs prefix optimization. Stage 2 performs correlation detection on the target model and computes the p-value (Chen et al., 29 Sep 2025).

The pseudocode specifies the following inputs: a reference model $M_r$5, an unrelated model $M_r$6, a target model $M_r$7, a random-selection prompt $M_r$8, vocabulary size $M_r$9, number of prefixes $M_t$0, and significance $M_t$1, with the example $M_t$2 (Chen et al., 29 Sep 2025). In Stage 1, each prefix is initialized and iteratively updated using either gradients or alternative search procedures; the code sketch explicitly mentions a gradient step using $M_t$3 and $M_t$4, with the update $M_t$5, “or genetic/ZO step” (Chen et al., 29 Sep 2025).

Stage 2 branches on target-model access. In gray-box mode it takes the argmax over $M_t$6. In black-box mode it samples outputs repeatedly and selects the most-frequent sampled token. After computing $M_t$7, it evaluates

$M_t$8

and rejects $M_t$9 when $M_r$0 (Chen et al., 29 Sep 2025).

The implementation regimes described for experiments are diverse. For LLMs, the access conditions include gradient-accessible versus logits-only for $M_r$1, and gray-box versus black-box for $M_r$2. For VLMs, the access conditions include PGD-accessible gradients versus logits-only zeroth-order (Chen et al., 29 Sep 2025). This indicates that RSP is formulated independently of a single optimization backend; the optimizer is adapted to what the interface permits.

6. Experimental setups, results, and ablations

The experiments reported for RSP span both LLMs and VLMs. The LLM reference models are Llama-3-8B-Instruct, Qwen2.5-3B, and Phi-4-mini. The fine-tuning datasets are GSM8K, Dolly-15k, and Alpaca instructions. For VLMs, the reference models are Qwen2.5-VL-7B and Llama-3.2-11B-Vision, and the fine-tuning datasets are Visual7W and MathV360k (Chen et al., 29 Sep 2025).

The common hyperparameters reported are $M_r$3 prefixes, token prefix length $M_r$4, image prefix resolution $M_r$5, black-box sampling $M_r$6, and significance threshold $M_r$7 (Chen et al., 29 Sep 2025).

The following summary collects the experimental configurations and findings explicitly stated in the source.

Category	Reported items	Reported findings
LLM setups	Llama-3-8B-Instruct, Qwen2.5-3B, Phi-4-mini; GSM8K, Dolly-15k, Alpaca instructions	Correlated models yield extremely small p-values; unrelated models yield p-values near 1
VLM setups	Qwen2.5-VL-7B, Llama-3.2-11B-Vision; Visual7W, MathV360k	RSP is evaluated under PGD-accessible gradients and logits-only zeroth-order
Common hyperparameters	$M_r$8, $M_r$9, $M_t$0, $M_t$1, significance threshold $M_t$2	$M_t$3 is chosen; $M_t$4 is a good trade-off

The principal result is that correlated models, including true fine-tunes or identical backbones, yield extremely small p-values, quantified as $M_t$5 and often $M_t$6, while unrelated models produce p-values near 1 (Chen et al., 29 Sep 2025). The abstract states the same qualitative pattern: RSP consistently yields small p-values for related models while maintaining high p-values for unrelated ones (Chen et al., 29 Sep 2025).

The ablation studies further characterize robustness. Increasing $M_t$7 monotonically decreases p-values, corresponding to greater statistical power. Shorter prefix length $M_t$8 still works but can violate independence, so $M_t$9 is chosen. Mutation probability in logits-only search can vary widely from $H_1$0 to $H_1$1 with limited impact. For images, $H_1$2 is reported as a good trade-off: too low loses signal and too high slows optimization (Chen et al., 29 Sep 2025).

These findings support the framework’s claim to robustness across modalities and access conditions, although the details also make clear that this robustness depends on specific design choices such as baseline adjustment and sufficiently long prefixes.

7. Interpretation, scope, and limitations

RSP is described as the first principled and general statistical framework for model correlation detection (Chen et al., 29 Sep 2025). In the discussion, its advantages are stated in four terms: transparent p-values, minimal assumptions, false positive control, and broad applicability to settings where only query access to the target model is available (Chen et al., 29 Sep 2025).

The p-value is central to the interpretability claim. Unlike heuristic similarity scores, the output has a defined interpretation under the binomial null. This distinguishes RSP from methods that require access to parameters or provide thresholdless scores, a limitation highlighted in the abstract (Chen et al., 29 Sep 2025).

At the same time, the stated limitations are nontrivial. The test requires independence of prefixes, and short prefixes can collide. Very expensive black-box optimization for VLMs in logits-only mode is also identified as a limitation (Chen et al., 29 Sep 2025). These caveats bear directly on the validity and cost of the procedure rather than on incidental implementation details.

The discussion also mentions possible extensions to more complex tasks such as multiple choice and regression outputs, and to other modalities such as speech and code (Chen et al., 29 Sep 2025). Because these are presented as extensions rather than demonstrated results, they are best understood as prospective generalizations rather than established capabilities.

A potential source of confusion is terminological rather than methodological: “RSP” may be mistaken for unrelated uses of similar initials in other security literatures. In the present context, the term refers specifically to “Random Selection Probing,” the framework for detecting model correlation by transferring optimized prefixes and evaluating the transfer with a statistical test (Chen et al., 29 Sep 2025).