--- title: Black-Box Token Verification url: https://www.emergentmind.com/topics/black-box-token-verification type: topic --- # Black-Box Token Verification Black-box token verification denotes a family of verification procedures in which token-level or token-derived properties are inferred without access to model internals, gradients, logits, or private keys. In the surveyed literature, the observable interface is limited to API-level predictions, top-\(k\) log-probabilities, sampled continuations, or text outputs, and the verification target may be a replaced token, an anomalous vocabulary entry, a key token that modulates output behavior, a membership-bearing perturbation signal, a watermark-bearing token sequence, or a hidden confirmation witness embedded in a black-box harness [2603.12582] [2406.19840] [2512.11573] [2506.19399] [2603.14968] [2410.02099] [2606.04443]. This suggests that the topic is best understood not as a single algorithmic paradigm but as a constraint class: verification under black-box access, with token granularity or token-induced evidence as the decisive signal. ## 1. Problem formulations and black-box observables A central formulation treats the input as a token sequence \(x=[x_1,\dots,x_n]\) and the protected model as a black-box function \(M\) or \(S\). In adversarial-text detection, the observable is the predicted class and its confidence before and after a token-level intervention [2603.12582]. In anomalous-token discovery for LLM vocabularies, the observable is the top-\(k\) next-token log-probabilities under a controlled single-token prompt at temperature \(T=0\), followed by confirmation runs at \(T=1.0\) [2406.19840]. In sensitivity analysis, the observable is a distribution over outputs \(S:X\to P(Y)\), accessible only through repeated sampling after discrete token substitutions [2512.11573]. In pre-training membership detection, the observable is the generated suffix \(S=\Theta(X)\) for a candidate prefix \(X\), possibly repeated to reduce randomness [2506.19399]. In watermark verification, the observable is a suspicious token sequence \(q\) together with reference samples from a service provider under watermark on/off modes, or, in a separate line of work, black-box samples from a language model used for distortion-free watermark insertion and key-based detection [2603.14968] [2410.02099]. In FO-based KEM testing, the observable is the execution transcript seen by an honest-reference harness, while the reencryption computation is hidden except through values that reach final key derivation [2606.04443]. These formulations share two structural features. First, verification is indirect: the property of interest is rarely observed directly, and must instead be inferred from shifts in confidence, suffix mappings, output distributions, or oracle-query patterns. Second, token-level evidence is operationalized through controlled perturbation, reference comparison, or keyed statistics rather than through white-box saliency or parameter inspection. A plausible implication is that black-box token verification is fundamentally a problem of designing sufficiently informative interventions and test statistics under severe observability constraints. ## 2. Replacement-token and anomaly detection RTD-Guard instantiates black-box token verification for word-level adversarial examples by leveraging a pre-trained Replaced Token Detection discriminator from ELECTRA exactly as released, without any fine-tuning on adversarial data [2603.12582]. During pre-training, ELECTRA uses a generator \(G\) trained with an MLM loss and a discriminator \(D_{\rm RTD}\) trained to predict whether each token in a corrupted input equals the original token. The discriminator loss is \[ \mathcal{L}_{\rm RTD} = -\,\mathbb{E}_{x,\tilde x_D}\;\sum_{i=1}^n \log p_{\theta_D}\bigl(\mathds{1}(\tilde x_{i,D}=x_i)\,\bigm|\;\tilde x_D,\;i\bigr), \] with a high weight \(\lambda\) placed on this term so that \(D_{\rm RTD}\) becomes very sensitive to “out-of-place” tokens. At detection time, each token receives a replacement probability \[ p_i = p_{\rm RTD}(x_i\text{ is replaced}\mid x)=\sigma\bigl(f(x)_i\bigr). \] RTD-Guard then selects the top-\(k\) suspicious indices, masks them, queries the victim model twice, and computes either \[ \Delta(x)=C_{\rm orig}-C_{\rm mask} \quad\text{or}\quad S(x)=(C_{\rm orig}-C_{\rm mask})^2. \] The input is declared adversarial if \(\Delta(x)>\tau\). The framework is strict black-box, zero-shot, and constant-query: it requires only predicted class and confidence, uses no gradients or internal features, and makes exactly two queries per example regardless of sequence length. On AG-News, IMDB, and Yelp with TextFooler, PWWS, BAE, and TF-adj, it reports across 12 dataset–attack combinations the highest or near-highest ROC-AUC at 95–99%, TPR10 above 95% in most cases, and strong F1, while outperforming PPL, FGWS, MLE, RDE, GradMask, WDR, and VoteTRANS; on AG-News/TextFooler with 2.4K examples, total detection time is approximately \(35\) s [2603.12582]. AnomaLLMy addresses a different verification target: anomalous tokens in black-box LLM vocabularies [2406.19840]. For each token \(v\in V\), a “REPEAT” prompt is issued at temperature \(T=0\) with top \(k=5\) log-probabilities returned, from which the method computes \(p_i=\exp(\log p_i)\), top-\(k\) entropy \[ H_5(v)=-\sum_{i=1}^5 p_i\log p_i, \] tail mass \[ p_{\rm tail}(v)=1-\sum_{i=1}^5 p_i, \] and best-vs-second-best gap \[ \Delta(v)=p_1-p_2. \] A token is flagged as a candidate iff \(H_5(v)>1.0\) or \(p_{\rm tail}(v)>0.1\) or \(\Delta(v)<0.5\). Confirmation then runs \(N_{\rm confirm}=10\) temperature-\(1.0\) queries; if the normalized top-1 output is wrong in \(1\) to \(5\) runs, the token is a minor anomaly, and if wrong in at least \(6\) runs, a major anomaly. On the cl100k\_base vocabulary of GPT-4, the initial scan cost is approximately \(\$19.00\), confirmation cost approximately \(\$5.39\), and total cost \(\$24.39\). The run took approximately \(20\) h and yielded \(478\) true anomalies, comprising \(413\) major and \(65\) minor anomalies, with mean values over non-anomalous tokens of \(p_1=0.9785\), \(\Delta=0.9615\), \(H_5=0.0668\), and \(p_{\rm tail}=0.00065\) [2406.19840]. Taken together, these methods exemplify two distinct black-box verification logics. RTD-Guard verifies whether a token appears contextually replaced by measuring downstream confidence collapse after masking. AnomaLLMy verifies whether a vocabulary entry is intrinsically unstable by measuring single-token low-confidence behavior under tightly controlled prompting. The first is intervention-centric; the second is confidence-profile-centric. ## 3. Perturbation, latent mapping, and output-distribution sensitivity VeilProbe frames black-box verification around pre-training membership rather than token correctness, but key-token perturbation is central to the signal extraction process [2506.19399]. Given a black-box LLM \(\Theta\), a candidate text \(X\), and generated suffix \(S=\Theta(X)\), the method trains an explicit seq2seq mapping model \(f_\phi\) on collected \((X,S)\) pairs using \[ \max_\phi \log P_\phi(S\mid X) =\sum_{t=1}^T \log P_\phi\bigl(w_t^S\mid w_1^S,\dots,w_{t-1}^S,X\bigr). \] From the trained encoder-decoder Transformer, it extracts a latent mapping feature \[ Z^s=\bigl[h_L(X,S)[\text{last}]\bigr]_{L=1..L_{\max}}\in\mathbb{R}^D. \] Key tokens are identified by a proxy LLM through removal-based importance scoring, averaging importance scores \(\mu_i\) across several proxy models and selecting the top \(\gamma\%\) tokens. These tokens are replaced by random synonyms to form \(R\), the model is re-queried to obtain \(S^R=\Theta(R)\), and a perturbation calibration feature is computed as \[ Z^c=Z^s-Z^r. \] After significance-test denoising, the final feature is \(Z=Z^s\Vert\tilde Z^c\). Membership classification is performed by a prototypical network with class prototypes \[ c_y=\frac{1}{|U_y|}\sum_{(Z_i,y_i)\in U_y}F_\theta(Z_i), \] squared-Euclidean distance \(d(u,v)=\|u-v\|^2\), and decision score \[ \delta(Z)=d(e,c_0)-d(e,c_1). \] The method is evaluated on WikiMIA, BookTection, and arXivTection against open-source and closed-source target LLMs, reporting average AUC gains of \(0.25\) over the prior best black-box method and roughly doubled TPR@5% FPR in many settings [2506.19399]. DBSA addresses token importance directly by quantifying how perturbing one input token alters the entire output distribution of a stochastic black-box LLM [2512.11573]. For each position \(i\), token \(t_i\) is replaced by one of its \(k\) nearest neighbors in embedding space. Let \(Y=\{y^{(1)},\dots,y^{(n)}\}\) be samples from \(S(x)\) and \(Y'=\{y'^{(1)},\dots,y'^{(m)}\}\) samples from the perturbed prompt. After embedding outputs with \(\phi(y)\), the method computes \[ A=\frac{1}{nm}\sum_{a=1}^n\sum_{b=1}^m s(\phi(y^{(a)}),\phi(y'^{(b)})), \] \[ B=\frac{1}{n^2}\sum_{a=1}^n\sum_{b=1}^n s(\phi(y^{(a)}),\phi(y^{(b)})), \qquad C=\frac{1}{m^2}\sum_{a=1}^m\sum_{b=1}^m s(\phi(y'^{(a)}),\phi(y'^{(b)})), \] and the empirical energy distance \[ E=2A-B-C. \] A permutation test yields a \(p\)-value for \(H_0:E\approx 0\). Effects are averaged across neighbors and repeated positions to produce a token score \(\omega_w\) and token-level \(p\)-value \(p_w\). The paper gives complexity \(O(n_t\cdot k\cdot n)\) in API calls and \(O(P\cdot n_t\cdot k\cdot n^2)\) for the permutation test; with \(n_t\approx 50\), \(k=3\), and \(n=40\), this corresponds to approximately \(12{,}000\) API calls. Illustrative examples show highest \(\omega_w\) for “agreement”, “California”, and “AI software” in a legal prompt, and top-\(k=5\) tokens “congestive, examination, Lower, mid, hypertensive” in a clinical prompt, with all \(p>0.05\) in the latter case [2512.11573]. These two systems operationalize black-box token verification through different latent objects. VeilProbe learns a latent fingerprint of the input-to-suffix mapping and amplifies it with key-token perturbation. DBSA measures output-distribution sensitivity directly, using energy distance and permutation testing rather than a learned detector. Both replace white-box token attribution with black-box perturbation design. ## 4. Provenance and watermark verification TTP-Detect addresses watermark verification as a third-party black-box auditing problem [2603.14968]. Given a suspicious token sequence \(q\), the objective is to decide between \[ H_0:q\sim P_0 \qquad\text{and}\qquad H_1:q\sim P_{\rm wm}, \] where \(P_0\) and \(P_{\rm wm}\) are unwatermarked and watermarked output distributions of a service provider \(S\). The framework samples \(N\) reference completions under both watermark settings, maps texts to \(\ell_2\)-normalized proxy features \(z(t)\in\mathbb{R}^d\), and computes four complementary scores: local consistency \(A_{\rm loc}\), Mahalanobis contrast \(A_{\rm Mah}\), energy contrast \(A_{\rm Ene}\), and adaptive rank test \(A_{\rm Ada}\). These are combined through \[ A_{\rm ens}(q)=\sigma(w^\top[A_1,\dots,A_M]^\top+b), \] followed by thresholding at \(\tau\). The adaptive rank test uses tokenwise NLLs \(\ell_i=-\log P_{\rm score}(x_i\mid x_{t\) giving \(\mathrm{FPR}=1-t\). The paper proves a distortion-free theorem, an exact FPR theorem under \(H_0\), and an ROC-AUC lower bound in the special case \(k=1\), \(F=\mathrm{Uniform}(0,1)\). On Mistral-7B-instruct and Gemma-7B-instruct with databricks-dolly-15k and eli5-category, it reports for the black-box watermark with \(k=1\), \(m=1024\), \(F=\mathrm{Uniform}\) an AUC of approximately \(97.7\%\), pAUC of approximately \(90.0\%\) on mixed lengths, and perplexity approximately \(2.61\) versus \(2.81\) for Aaronson’s scheme, while noting that paraphrasing attacks largely destroy watermark signals to below \(55\%\) AUC [2410.02099]. Within black-box token verification, watermark work therefore splits into two regimes. One regime verifies whether a given token sequence aligns more closely with watermarked than unwatermarked reference distributions. The other constructs a keyed token-sequence statistic whose null behavior is analytically controlled. Both treat the token sequence itself as the auditable artifact. ## 5. Query complexity, statistical tests, and decision rules The surveyed methods suggest a pronounced trade-off between query economy and statistical richness. RTD-Guard is at the low-query end: exactly two black-box queries per example, irrespective of sequence length, with a single forward pass through the frozen RTD discriminator and thresholding on \(\Delta(x)\) [2603.12582]. AnomaLLMy scales linearly in vocabulary size \(|V|\) but keeps cost low through a two-phase filter: an initial scan over approximately \(50{,}200\) tokens followed by confirmation of only a few hundred candidates, for a total API spend of \(\$24.39\) [2406.19840]. TTP-Detect requires \(2N=32\) generation calls per audit when \(N=16\), plus proxy-side representation extraction, geometric tests, and adaptive rank scoring [2603.14968]. DBSA is substantially more expensive, with API complexity \(O(n_t\cdot k\cdot n)\) and permutation testing \(O(P\cdot n_t\cdot k\cdot n^2)\), because it estimates distributional shifts rather than confidence deltas [2512.11573]. Decision rules are likewise heterogeneous. RTD-Guard uses a scalar confidence-drop threshold \(\Delta(x)>\tau\) tuned on clean validation data [2603.12582]. AnomaLLMy uses hard thresholds on entropy, tail mass, and best-vs-second-best gap, followed by a count-based confirmation rule over \(10\) high-variance runs [2406.19840]. DBSA and the black-box watermarking scheme rely on explicit hypothesis testing: a permutation test for energy distance in the former, and exact null calibration through \(p=1-F_T(\sum_i R_i)\) in the latter [2512.11573] [2410.02099]. TTP-Detect combines relative measurements through a sigmoid ensemble with threshold \(\tau\) chosen on held-out benign text to guarantee \(\mathrm{FPR}\le \alpha\) [2603.14968]. VeilProbe uses a prototypical classifier trained with cross-entropy over negative squared-Euclidean distances, reflecting a few-shot classification perspective rather than a direct threshold test [2506.19399]. A plausible implication is that black-box token verification methods can be differentiated not only by their verification target but also by the statistical object they trust most: confidence collapse, low-confidence outliers, latent feature displacement, distributional two-sample deviation, or keyed null-calibrated scores. ## 6. Guarantees, obstructions, and limits of certifiability The strongest explicit characterization of black-box limits in the surveyed material appears in the FO-based KEM setting [2606.04443]. In an honest-reference harness, the hidden final-key point is \(x^*=\langle good,B,W\rangle\), and a \(q\)-localized system under test passes if it outputs the honest shared key \(K=G_K(x^*)\). The pass probability is bounded by \[ \mathrm{Pass}^{\rm href}_{\Pi^C,\mathcal{H}}(S) \le \delta_{\Pi^C} +\epsilon_{\rm alias}(S,\mathcal{H}) +\epsilon_{\rm fresh}(S,\mathcal{H}) +\Pr[W\in Q_G(B)] +2^{-\kappa}. \] The one-query matching construction shows that this is tight up to the fresh-key coincidence term: \[ \mathrm{Pass}^{\rm href}_{\Pi^C,\mathcal{H}}(S_A) = \Pr[A(V)=W] + (1-\Pr[A(V)=W])\,2^{-\kappa}. \] The “list-hit” event \(\Pr[W\in Q_G(B)]\) is therefore the fundamental black-box obstruction measured by the harness. It can be bounded either via a cUP-faithful harness certificate, yielding \[ \mathrm{Pass}\le \delta_{\Pi^C}+\epsilon_{\rm alias}+\epsilon_{\rm fresh}+q\,\epsilon_{\rm cUP}+2^{-\kappa}, \] or via conditional min-entropy, yielding \[ \mathrm{Pass}\le \delta+\epsilon_{\rm alias}+\epsilon_{\rm fresh}+q\,2^{-\lambda}+2^{-\kappa}. \] The same framework proves a dependency-cone lower bound for non-certification: if observation factors only through the confirmation-observable final-key target, then for operation sets outside the support-active cone, soundness and completeness errors satisfy \(\alpha+\beta\ge 1\). This is not merely a performance limitation but a certifiability limit [2606.04443]. Other black-box token verification papers state more task-specific limits. RTD-Guard is designed around the observation that adversarial word substitutions resemble RTD “replaced tokens,” so its mechanism is explicitly aligned with word-substitution perturbations and adversarial synonym swaps [2603.12582]. TTP-Detect reports weaker signals for extremely distribution-preserving watermarks, degraded local consistency under aggressive paraphrasing such as Dipper-2, reduced reliability for very short queries below \(50\) tokens, and possible compromise if an adversary fine-tunes a generator to mimic the reference distributions [2603.14968]. The distortion-free watermarking scheme likewise finds that paraphrasing attacks largely destroy watermark signals, motivating semantic-level or robust schemes [2410.02099]. DBSA makes no distributional assumptions about the LLM, but its statistical reliability depends on repeated sampling, nearest-neighbor substitutions, and permutation testing, which raises compute and query costs [2512.11573]. VeilProbe explicitly addresses scarcity of labeled member/non-member examples through a prototype-based classifier to alleviate overfitting [2506.19399]. Taken together, the surveyed work indicates that black-box token verification is bounded less by lack of ingenuity than by the observables exposed through the interface. When the observable carries a strong token-level signature—such as RTD replacement likelihood, low-confidence single-token failure, or a keyed watermark statistic—verification can be both efficient and accurate. When observability is mediated through stochastic generations or hidden confirmation points, verification becomes a problem of relative testing, entropy transfer, or, in the strongest negative results, non-certification. Source: https://www.emergentmind.com/topics/black-box-token-verification