---
title: Random Selection Probing (RSP) Framework
url: https://www.emergentmind.com/topics/random-selection-probing-rsp
type: topic
---

# Random Selection Probing (RSP) Framework

Searching arXiv for the specified paper and closely related usage of “Random Selection Probing”.
Random Selection Probing (RSP) is a hypothesis-testing framework for model correlation detection that determines whether a target model is statistically correlated with a reference model, for example because it was fine-tuned from that model or is even identical to it. The method operates by optimizing textual or visual prefixes on a reference model for a random selection task and then evaluating the transferability of those prefixes to a target model. Its defining output is a rigorous one-sided p-value rather than a heuristic similarity score, and the framework incorporates an unrelated baseline model to filter generic, transferable features that could otherwise inflate false positives [2509.24171].

## 1. Definition and problem formulation

RSP addresses the problem of deciding whether two models share lineage or other statistically detectable dependence under diverse access conditions. In the formulation given for large language models (LLMs) and vision-language models (VLMs), the reference model is denoted \(M_r\), the target model \(M_t\), and an unrelated baseline model \(M_u\) when baseline adjustment is used [2509.24171].

The framework defines a null hypothesis \(H_0\) under which \(M_r\) and \(M_t\) are independent, so that probing prefixes optimized on \(M_r\) produce no better-than-random performance on \(M_t\). The alternative hypothesis \(H_1\) states that the models are correlated and that prefixes tuned on \(M_r\) transfer with success rate above random [2509.24171].

The random selection task is discrete with \(N\) equally likely choices. The exposition gives the example \(N=26\) letters. RSP optimizes \(K\) prefixes \(\{\bm x_k\}_{k=1}^K\) on the reference model, each associated with a designated target token index \(t_k\). Evaluation on the target model counts how often the predicted index matches that designated target [2509.24171].

Formally, the observed number of successes is
$$
T_{\rm obs} \;=\; \sum_{k=1}^K \mathbf{1}\Bigl(\arg\max_{j\in\{1,\dots,N\}} P_{M_t}(j\mid \bm x_k, p_r) \;=\;t_k\Bigr).
$$

Under \(H_0\), each trial succeeds with probability \(1/N\) independently, so
$$
T \;\sim\; \mathrm{Binomial}\bigl(K,\;1/N\bigr).
$$

The one-sided p-value is then
$$
p \;=\; \Pr_{Z\sim H_0}\bigl(T(Z)\ge T_{\rm obs}\bigr)
\;=\;
\sum_{x=T_{\rm obs}}^{K}\binom{K}{x}\Bigl(\tfrac1N\Bigr)^{x}\Bigl(1-\tfrac1N\Bigr)^{\,K-x}
\;=\;
I_{1/N}\bigl(T_{\rm obs},\,K-T_{\rm obs}+1\bigr),
$$
where \(I\) is the regularized incomplete beta-function [2509.24171].

This construction makes the inferential target explicit: RSP does not merely score similarity, but instead tests whether transfer performance is too strong to be explained by the binomial null.

## 2. Prefix optimization and random selection tasks

RSP parameterizes prefixes by a continuous or discrete vector \(\theta \in \Theta\). For textual prefixes, \(\theta\) may index the choice of \(L\) tokens; for visual prefixes, \(\theta\) is an image of size \(H \times W \times 3\) [2509.24171].

The method defines a selection loss on the reference model:
$$
\mathcal{L}_{\mathrm{select}}(M_r,\theta)
\;=\;
\frac1N \sum_{k=1}^N -\log\,P_{M_r}(o_k\mid \bm x(\theta),\,p_r)
\quad \longrightarrow \quad \min_\theta.
$$

Equivalently, optimization may be stated as maximizing the likelihood of the designated target:
$$
\theta_k^* \,=\,\arg\max_{\theta}\; P_{M_r}\bigl(t_k\mid \bm x(\theta),p_r\bigr).
$$

When an unrelated baseline model \(M_u\) is incorporated, the optimization objective becomes
$$
\theta_k^* \,=\,\arg\max_{\theta}\;\Bigl[ P_{M_r}(t_k\mid \bm x(\theta),p_r)\;-\; P_{M_u}(t_k\mid \bm x(\theta),p_r) \Bigr].
$$
This objective is intended to amplify model-specific rather than generic features [2509.24171].

The detailed exposition further states that, in practice, the method samples \(K\) pairings \((\bm x_k^0,t_k)\) and optimizes each prefix accordingly. This suggests a two-stage structure: first learn probe inputs on the reference side, then assess transfer on the target side. A plausible implication is that RSP uses probe construction itself as the vehicle for statistical identification, rather than treating probe inputs as externally fixed.

## 3. Statistical testing and access regimes

After prefix optimization, RSP evaluates the optimized prefixes on the target model and converts the resulting success count into a p-value under the binomial null [2509.24171].

In the gray-box setting with logit access, evaluation proceeds by computing, for each prefix \(\bm x_k\),
$$
j_k^*=\arg\max_j P_{M_t}(j\mid\bm x_k,p_r),
$$
and then counting successes via \(T_{\rm obs}=\sum_k \mathbf{1}(j_k^*=t_k)\).

In the black-box setting with text-only access, the method repeats each query \(T\) times, estimates empirical frequencies \(\hat P_{M_t}(j\mid\bm x_k,p_r)\), sets \(j_k^*=\arg\max_j \hat P\), and forms the same success statistic \(T_{\rm obs}\) [2509.24171].

The algorithmic description emphasizes that the test statistic is unchanged across these regimes; only the manner of estimating the relevant argmax differs. This is significant because the abstract explicitly frames the framework as applicable under diverse access conditions for reference and test models, and the details enumerate gradient-accessible, logits-only, gray-box, and black-box variants for both LLMs and VLMs [2509.24171].

RSP therefore couples a fixed inferential core—the binomial test—with multiple operational instantiations. This suggests that its principal claim to generality lies less in any specific optimizer than in the invariance of the testing logic across access modalities.

## 4. Baseline model adjustment and false-positive control

A distinctive feature of RSP is the use of an unrelated baseline model \(M_u\) during prefix optimization. The baseline-adjusted objective is
$$
\max_\theta\;\bigl[P_{M_r}(t\mid\theta)-P_{M_u}(t\mid\theta)\bigr].
$$
According to the exposition, this “differencing” ensures that the learned prefix relies on idiosyncrasies of \(M_r\) [2509.24171].

The statistical test itself is unchanged after this adjustment; the baseline acts only in the optimization stage. The reason for this design is stated explicitly: prefixes optimized without \(M_u\) occasionally transfer to truly unrelated models, inflating Type I error [2509.24171]. The abstract summarizes the same point more broadly, stating that RSP incorporates an unrelated baseline model to filter out generic, transferable features and thereby mitigate false positives [2509.24171].

The reported ablation findings reinforce this role. Removing the unrelated baseline \(M_u\) increases false positives on some unrelated LLMs [2509.24171]. The discussion section then characterizes this component as part of RSP’s false positive control, with the stated effect of keeping Type I error near the chosen significance level \(\alpha\) [2509.24171].

This component is central to distinguishing correlation-specific transfer from broad prompt transferability. A plausible implication is that RSP is not merely a transfer test, but a transfer test conditioned against a nuisance source of generic elicitation effects.

## 5. Algorithmic structure and implementation regimes

The framework is described procedurally in two stages. Stage 1 performs prefix optimization. Stage 2 performs correlation detection on the target model and computes the p-value [2509.24171].

The pseudocode specifies the following inputs: a reference model \(M_r\), an unrelated model \(M_u\), a target model \(M_t\), a random-selection prompt \(p_r\), vocabulary size \(N\), number of prefixes \(K\), and significance \(\alpha\), with the example \(\alpha = 0.05\) [2509.24171]. In Stage 1, each prefix is initialized and iteratively updated using either gradients or alternative search procedures; the code sketch explicitly mentions a gradient step using
\(\nabla_\theta \log P_{M_r}(t_k \mid \theta,p_r)\) and \(\nabla_\theta \log P_{M_u}(t_k \mid \theta,p_r)\), with the update
\(\theta_k \leftarrow \theta_k + \eta (\mathrm{grad}_r - \mathrm{grad}_u)\), “or genetic/ZO step” [2509.24171].

Stage 2 branches on target-model access. In gray-box mode it takes the argmax over \(P_{M_t}(j \mid \theta_k,p_r)\). In black-box mode it samples outputs repeatedly and selects the most-frequent sampled token. After computing \(T_{\rm obs}\), it evaluates
$$
p \leftarrow \sum_{x=T_{\rm obs}}^K \binom{K}{x}(1/N)^x(1-1/N)^{K-x},
$$
and rejects \(H_0\) when \(p < \alpha\) [2509.24171].

The implementation regimes described for experiments are diverse. For LLMs, the access conditions include gradient-accessible versus logits-only for \(M_r\), and gray-box versus black-box for \(M_t\). For VLMs, the access conditions include PGD-accessible gradients versus logits-only zeroth-order [2509.24171]. This indicates that RSP is formulated independently of a single optimization backend; the optimizer is adapted to what the interface permits.

## 6. Experimental setups, results, and ablations

The experiments reported for RSP span both LLMs and VLMs. The LLM reference models are Llama-3-8B-Instruct, Qwen2.5-3B, and Phi-4-mini. The fine-tuning datasets are GSM8K, Dolly-15k, and Alpaca instructions. For VLMs, the reference models are Qwen2.5-VL-7B and Llama-3.2-11B-Vision, and the fine-tuning datasets are Visual7W and MathV360k [2509.24171].

The common hyperparameters reported are \(K=500\) prefixes, token prefix length \(L=50\), image prefix resolution \(280\times 280\), black-box sampling \(T=100\), and significance threshold \(0.05\) [2509.24171].

The following summary collects the experimental configurations and findings explicitly stated in the source.

| Category | Reported items | Reported findings |
|---|---|---|
| **LLM setups** | Llama-3-8B-Instruct, Qwen2.5-3B, Phi-4-mini; GSM8K, Dolly-15k, Alpaca instructions | Correlated models yield extremely small p-values; unrelated models yield p-values near 1 |
| **VLM setups** | Qwen2.5-VL-7B, Llama-3.2-11B-Vision; Visual7W, MathV360k | RSP is evaluated under PGD-accessible gradients and logits-only zeroth-order |
| **Common hyperparameters** | \(K=500\), \(L=50\), \(280\times280\), \(T=100\), significance threshold \(0.05\) | \(L=50\) is chosen; \(280\times280\) is a good trade-off |

The principal result is that correlated models, including true fine-tunes or identical backbones, yield extremely small p-values, quantified as \(<10^{-6}\) and often \(<10^{-300}\), while unrelated models produce p-values near 1 [2509.24171]. The abstract states the same qualitative pattern: RSP consistently yields small p-values for related models while maintaining high p-values for unrelated ones [2509.24171].

The ablation studies further characterize robustness. Increasing \(K\) monotonically decreases p-values, corresponding to greater statistical power. Shorter prefix length \(L=10\) still works but can violate independence, so \(L=50\) is chosen. Mutation probability in logits-only search can vary widely from \(0.1\) to \(1.0\) with limited impact. For images, \(280\times280\) is reported as a good trade-off: too low loses signal and too high slows optimization [2509.24171].

These findings support the framework’s claim to robustness across modalities and access conditions, although the details also make clear that this robustness depends on specific design choices such as baseline adjustment and sufficiently long prefixes.

## 7. Interpretation, scope, and limitations

RSP is described as the first principled and general statistical framework for model correlation detection [2509.24171]. In the discussion, its advantages are stated in four terms: transparent p-values, minimal assumptions, false positive control, and broad applicability to settings where only query access to the target model is available [2509.24171].

The p-value is central to the interpretability claim. Unlike heuristic similarity scores, the output has a defined interpretation under the binomial null. This distinguishes RSP from methods that require access to parameters or provide thresholdless scores, a limitation highlighted in the abstract [2509.24171].

At the same time, the stated limitations are nontrivial. The test requires independence of prefixes, and short prefixes can collide. Very expensive black-box optimization for VLMs in logits-only mode is also identified as a limitation [2509.24171]. These caveats bear directly on the validity and cost of the procedure rather than on incidental implementation details.

The discussion also mentions possible extensions to more complex tasks such as multiple choice and regression outputs, and to other modalities such as speech and code [2509.24171]. Because these are presented as extensions rather than demonstrated results, they are best understood as prospective generalizations rather than established capabilities.

A potential source of confusion is terminological rather than methodological: “RSP” may be mistaken for unrelated uses of similar initials in other security literatures. In the present context, the term refers specifically to “Random Selection Probing,” the framework for detecting model correlation by transferring optimized prefixes and evaluating the transfer with a statistical test [2509.24171].

Source: https://www.emergentmind.com/topics/random-selection-probing-rsp