Black-Box Token Verification

• Black-box token verification is a method that infers token-level properties using API outputs, controlled perturbations, and statistical tests.
• It employs strategies like replaced-token detection, entropy measures, and key-token perturbation to identify adversarial or watermark-embedded signals.
• These techniques balance query efficiency with rigorous statistical assessment, ensuring reliable token verification under limited observability.

Black-box token verification denotes a family of verification procedures in which token-level or token-derived properties are inferred without access to model internals, gradients, logits, or private keys. In the surveyed literature, the observable interface is limited to API-level predictions, top-$k$ log-probabilities, sampled continuations, or text outputs, and the verification target may be a replaced token, an anomalous vocabulary entry, a key token that modulates output behavior, a membership-bearing perturbation signal, a watermark-bearing token sequence, or a hidden confirmation witness embedded in a black-box harness (Zhu et al., 13 Mar 2026, Witold, 2024, Rauba et al., 12 Dec 2025, Hu et al., 24 Jun 2025, Wang et al., 16 Mar 2026, Bahri et al., 2024, Jiménez, 3 Jun 2026). This suggests that the topic is best understood not as a single algorithmic paradigm but as a constraint class: verification under black-box access, with token granularity or token-induced evidence as the decisive signal.

1. Problem formulations and black-box observables

A central formulation treats the input as a token sequence $x=[x_1,\dots,x_n]$ and the protected model as a black-box function $M$ or $S$. In adversarial-text detection, the observable is the predicted class and its confidence before and after a token-level intervention (Zhu et al., 13 Mar 2026). In anomalous-token discovery for LLM vocabularies, the observable is the top-$k$ next-token log-probabilities under a controlled single-token prompt at temperature $T=0$, followed by confirmation runs at $T=1.0$ (Witold, 2024). In sensitivity analysis, the observable is a distribution over outputs $S:X\to P(Y)$, accessible only through repeated sampling after discrete token substitutions (Rauba et al., 12 Dec 2025). In pre-training membership detection, the observable is the generated suffix $S=\Theta(X)$ for a candidate prefix $X$, possibly repeated to reduce randomness (Hu et al., 24 Jun 2025). In watermark verification, the observable is a suspicious token sequence $x=[x_1,\dots,x_n]$0 together with reference samples from a service provider under watermark on/off modes, or, in a separate line of work, black-box samples from a LLM used for distortion-free watermark insertion and key-based detection (Wang et al., 16 Mar 2026, Bahri et al., 2024). In FO-based KEM testing, the observable is the execution transcript seen by an honest-reference harness, while the reencryption computation is hidden except through values that reach final key derivation (Jiménez, 3 Jun 2026).

These formulations share two structural features. First, verification is indirect: the property of interest is rarely observed directly, and must instead be inferred from shifts in confidence, suffix mappings, output distributions, or oracle-query patterns. Second, token-level evidence is operationalized through controlled perturbation, reference comparison, or keyed statistics rather than through white-box saliency or parameter inspection. A plausible implication is that black-box token verification is fundamentally a problem of designing sufficiently informative interventions and test statistics under severe observability constraints.

2. Replacement-token and anomaly detection

RTD-Guard instantiates black-box token verification for word-level adversarial examples by leveraging a pre-trained Replaced Token Detection discriminator from ELECTRA exactly as released, without any fine-tuning on adversarial data (Zhu et al., 13 Mar 2026). During pre-training, ELECTRA uses a generator $x=[x_1,\dots,x_n]$1 trained with an MLM loss and a discriminator $x=[x_1,\dots,x_n]$2 trained to predict whether each token in a corrupted input equals the original token. The discriminator loss is

$x=[x_1,\dots,x_n]$3

with a high weight $x=[x_1,\dots,x_n]$4 placed on this term so that $x=[x_1,\dots,x_n]$5 becomes very sensitive to “out-of-place” tokens. At detection time, each token receives a replacement probability

$x=[x_1,\dots,x_n]$6

RTD-Guard then selects the top-$x=[x_1,\dots,x_n]$7 suspicious indices, masks them, queries the victim model twice, and computes either

$x=[x_1,\dots,x_n]$8

The input is declared adversarial if $x=[x_1,\dots,x_n]$9. The framework is strict black-box, zero-shot, and constant-query: it requires only predicted class and confidence, uses no gradients or internal features, and makes exactly two queries per example regardless of sequence length. On AG-News, IMDB, and Yelp with TextFooler, PWWS, BAE, and TF-adj, it reports across 12 dataset–attack combinations the highest or near-highest ROC-AUC at 95–99%, TPR10 above 95% in most cases, and strong F1, while outperforming PPL, FGWS, MLE, RDE, GradMask, WDR, and VoteTRANS; on AG-News/TextFooler with 2.4K examples, total detection time is approximately $M$0 s (Zhu et al., 13 Mar 2026).

AnomaLLMy addresses a different verification target: anomalous tokens in black-box LLM vocabularies (Witold, 2024). For each token $M$1, a “REPEAT” prompt is issued at temperature $M$2 with top $M$3 log-probabilities returned, from which the method computes $M$4, top-$M$5 entropy

$M$6

tail mass

$M$7

and best-vs-second-best gap

$M$8

A token is flagged as a candidate iff $M$9 or $S$0 or $S$1. Confirmation then runs $S$2 temperature-$S$3 queries; if the normalized top-1 output is wrong in $S$4 to $S$5 runs, the token is a minor anomaly, and if wrong in at least $S$6 runs, a major anomaly. On the cl100k_base vocabulary of GPT-4, the initial scan cost is approximately $S$7, and total cost $S$8. The run took approximately $S$9 h and yielded $k$0 true anomalies, comprising $k$1 major and $k$2 minor anomalies, with mean values over non-anomalous tokens of $k$3, $k$4, $k$5, and $k$6 (Witold, 2024).

Taken together, these methods exemplify two distinct black-box verification logics. RTD-Guard verifies whether a token appears contextually replaced by measuring downstream confidence collapse after masking. AnomaLLMy verifies whether a vocabulary entry is intrinsically unstable by measuring single-token low-confidence behavior under tightly controlled prompting. The first is intervention-centric; the second is confidence-profile-centric.

3. Perturbation, latent mapping, and output-distribution sensitivity

VeilProbe frames black-box verification around pre-training membership rather than token correctness, but key-token perturbation is central to the signal extraction process (Hu et al., 24 Jun 2025). Given a black-box LLM $k$7, a candidate text $k$8, and generated suffix $k$9, the method trains an explicit seq2seq mapping model $T=0$0 on collected $T=0$1 pairs using

$T=0$2

From the trained encoder-decoder Transformer, it extracts a latent mapping feature

$T=0$3

Key tokens are identified by a proxy LLM through removal-based importance scoring, averaging importance scores $T=0$4 across several proxy models and selecting the top $T=0$5 tokens. These tokens are replaced by random synonyms to form $T=0$6, the model is re-queried to obtain $T=0$7, and a perturbation calibration feature is computed as

$T=0$8

After significance-test denoising, the final feature is $T=0$9. Membership classification is performed by a prototypical network with class prototypes

$T=1.0$0

squared-Euclidean distance $T=1.0$1, and decision score

$T=1.0$2

The method is evaluated on WikiMIA, BookTection, and arXivTection against open-source and closed-source target LLMs, reporting average AUC gains of $T=1.0$3 over the prior best black-box method and roughly doubled TPR@5% FPR in many settings (Hu et al., 24 Jun 2025).

DBSA addresses token importance directly by quantifying how perturbing one input token alters the entire output distribution of a stochastic black-box LLM (Rauba et al., 12 Dec 2025). For each position $T=1.0$4, token $T=1.0$5 is replaced by one of its $T=1.0$6 nearest neighbors in embedding space. Let $T=1.0$7 be samples from $T=1.0$8 and $T=1.0$9 samples from the perturbed prompt. After embedding outputs with $S:X\to P(Y)$0, the method computes

$S:X\to P(Y)$1

$S:X\to P(Y)$2

and the empirical energy distance

$S:X\to P(Y)$3

A permutation test yields a $S:X\to P(Y)$4-value for $S:X\to P(Y)$5. Effects are averaged across neighbors and repeated positions to produce a token score $S:X\to P(Y)$6 and token-level $S:X\to P(Y)$7-value $S:X\to P(Y)$8. The paper gives complexity $S:X\to P(Y)$9 in API calls and $S=\Theta(X)$0 for the permutation test; with $S=\Theta(X)$1, $S=\Theta(X)$2, and $S=\Theta(X)$3, this corresponds to approximately $S=\Theta(X)$4 API calls. Illustrative examples show highest $S=\Theta(X)$5 for “agreement”, “California”, and “AI software” in a legal prompt, and top-$S=\Theta(X)$6 tokens “congestive, examination, Lower, mid, hypertensive” in a clinical prompt, with all $S=\Theta(X)$7 in the latter case (Rauba et al., 12 Dec 2025).

These two systems operationalize black-box token verification through different latent objects. VeilProbe learns a latent fingerprint of the input-to-suffix mapping and amplifies it with key-token perturbation. DBSA measures output-distribution sensitivity directly, using energy distance and permutation testing rather than a learned detector. Both replace white-box token attribution with black-box perturbation design.

4. Provenance and watermark verification

TTP-Detect addresses watermark verification as a third-party black-box auditing problem (Wang et al., 16 Mar 2026). Given a suspicious token sequence $S=\Theta(X)$8, the objective is to decide between

$S=\Theta(X)$9

where $X$0 and $X$1 are unwatermarked and watermarked output distributions of a service provider $X$2. The framework samples $X$3 reference completions under both watermark settings, maps texts to $X$4-normalized proxy features $X$5, and computes four complementary scores: local consistency $X$6, Mahalanobis contrast $X$7, energy contrast $X$8, and adaptive rank test $X$9. These are combined through

$x=[x_1,\dots,x_n]$00

followed by thresholding at $x=[x_1,\dots,x_n]$01. The adaptive rank test uses tokenwise NLLs $x=[x_1,\dots,x_n]$02 to derive global cross-entropy $x=[x_1,\dots,x_n]$03 and local volatility $x=[x_1,\dots,x_n]$04, while the global geometry tests compare the query with watermark/no-watermark reference sets via PCA-projected Mahalanobis distance and set-based energy distance. On C4 and OpenGen with Llama-3.1-8B and OPT-6.7B, the framework reports AUC at least $x=[x_1,\dots,x_n]$05 and F1 approximately $x=[x_1,\dots,x_n]$06–$x=[x_1,\dots,x_n]$07 for KGW and Unigram, AUC $x=[x_1,\dots,x_n]$08–$x=[x_1,\dots,x_n]$09 for SWEET and MorphMark, AUC approximately $x=[x_1,\dots,x_n]$10–$x=[x_1,\dots,x_n]$11 for Unbiased and SynthID, and perfect AUC $x=[x_1,\dots,x_n]$12 for SymMark. Editing attacks reduce AUC by less than $x=[x_1,\dots,x_n]$13 on average, reference size $x=[x_1,\dots,x_n]$14 is reported as a good balance, performance saturates at $x=[x_1,\dots,x_n]$15 for local consistency when $x=[x_1,\dots,x_n]$16, and total third-party overhead is below $x=[x_1,\dots,x_n]$17 s per query plus upstream API calls (Wang et al., 16 Mar 2026).

A different watermark line, formulated as a watermark insertion and detection scheme for black-box LLMs, defines a distortion-free keyed procedure over $x=[x_1,\dots,x_n]$18-grams (Bahri et al., 2024). A pseudorandom function $x=[x_1,\dots,x_n]$19 maps $x=[x_1,\dots,x_n]$20-grams to random real scores under a continuous CDF $x=[x_1,\dots,x_n]$21. In encoding, the algorithm samples $x=[x_1,\dots,x_n]$22 candidate continuations $x=[x_1,\dots,x_n]$23, deduplicates them into unique sequences-with-counts $x=[x_1,\dots,x_n]$24, computes per-sequence scores

$x=[x_1,\dots,x_n]$25

and selects

$x=[x_1,\dots,x_n]$26

Detection extracts unique $x=[x_1,\dots,x_n]$27-gram seeds $x=[x_1,\dots,x_n]$28, draws corresponding $x=[x_1,\dots,x_n]$29, and computes

$x=[x_1,\dots,x_n]$30

with detection based on $x=[x_1,\dots,x_n]$31 or equivalently a score threshold $x=[x_1,\dots,x_n]$32 giving $x=[x_1,\dots,x_n]$33. The paper proves a distortion-free theorem, an exact FPR theorem under $x=[x_1,\dots,x_n]$34, and an ROC-AUC lower bound in the special case $x=[x_1,\dots,x_n]$35, $x=[x_1,\dots,x_n]$36. On Mistral-7B-instruct and Gemma-7B-instruct with databricks-dolly-15k and eli5-category, it reports for the black-box watermark with $x=[x_1,\dots,x_n]$37, $x=[x_1,\dots,x_n]$38, $x=[x_1,\dots,x_n]$39 an AUC of approximately $x=[x_1,\dots,x_n]$40, pAUC of approximately $x=[x_1,\dots,x_n]$41 on mixed lengths, and perplexity approximately $x=[x_1,\dots,x_n]$42 versus $x=[x_1,\dots,x_n]$43 for Aaronson’s scheme, while noting that paraphrasing attacks largely destroy watermark signals to below $x=[x_1,\dots,x_n]$44 AUC (Bahri et al., 2024).

Within black-box token verification, watermark work therefore splits into two regimes. One regime verifies whether a given token sequence aligns more closely with watermarked than unwatermarked reference distributions. The other constructs a keyed token-sequence statistic whose null behavior is analytically controlled. Both treat the token sequence itself as the auditable artifact.

5. Query complexity, statistical tests, and decision rules

The surveyed methods suggest a pronounced trade-off between query economy and statistical richness. RTD-Guard is at the low-query end: exactly two black-box queries per example, irrespective of sequence length, with a single forward pass through the frozen RTD discriminator and thresholding on $x=[x_1,\dots,x_n]$45 (Zhu et al., 13 Mar 2026). AnomaLLMy scales linearly in vocabulary size $x=[x_1,\dots,x_n]$46 but keeps cost low through a two-phase filter: an initial scan over approximately $x=[x_1,\dots,x_n]$47 tokens followed by confirmation of only a few hundred candidates, for a total API spend of $x=[x_1,\dots,x_n]$48 (Witold, 2024). TTP-Detect requires $x=[x_1,\dots,x_n]$49 generation calls per audit when $x=[x_1,\dots,x_n]$50, plus proxy-side representation extraction, geometric tests, and adaptive rank scoring (Wang et al., 16 Mar 2026). DBSA is substantially more expensive, with API complexity $x=[x_1,\dots,x_n]$51 and permutation testing $x=[x_1,\dots,x_n]$52, because it estimates distributional shifts rather than confidence deltas (Rauba et al., 12 Dec 2025).

Decision rules are likewise heterogeneous. RTD-Guard uses a scalar confidence-drop threshold $x=[x_1,\dots,x_n]$53 tuned on clean validation data (Zhu et al., 13 Mar 2026). AnomaLLMy uses hard thresholds on entropy, tail mass, and best-vs-second-best gap, followed by a count-based confirmation rule over $x=[x_1,\dots,x_n]$54 high-variance runs (Witold, 2024). DBSA and the black-box watermarking scheme rely on explicit hypothesis testing: a permutation test for energy distance in the former, and exact null calibration through $x=[x_1,\dots,x_n]$55 in the latter (Rauba et al., 12 Dec 2025, Bahri et al., 2024). TTP-Detect combines relative measurements through a sigmoid ensemble with threshold $x=[x_1,\dots,x_n]$56 chosen on held-out benign text to guarantee $x=[x_1,\dots,x_n]$57 (Wang et al., 16 Mar 2026). VeilProbe uses a prototypical classifier trained with cross-entropy over negative squared-Euclidean distances, reflecting a few-shot classification perspective rather than a direct threshold test (Hu et al., 24 Jun 2025).

A plausible implication is that black-box token verification methods can be differentiated not only by their verification target but also by the statistical object they trust most: confidence collapse, low-confidence outliers, latent feature displacement, distributional two-sample deviation, or keyed null-calibrated scores.

6. Guarantees, obstructions, and limits of certifiability

The strongest explicit characterization of black-box limits in the surveyed material appears in the FO-based KEM setting (Jiménez, 3 Jun 2026). In an honest-reference harness, the hidden final-key point is $x=[x_1,\dots,x_n]$58, and a $x=[x_1,\dots,x_n]$59-localized system under test passes if it outputs the honest shared key $x=[x_1,\dots,x_n]$60. The pass probability is bounded by

$x=[x_1,\dots,x_n]$61

The one-query matching construction shows that this is tight up to the fresh-key coincidence term: $x=[x_1,\dots,x_n]$62 The “list-hit” event $x=[x_1,\dots,x_n]$63 is therefore the fundamental black-box obstruction measured by the harness. It can be bounded either via a cUP-faithful harness certificate, yielding

$x=[x_1,\dots,x_n]$64

or via conditional min-entropy, yielding

$x=[x_1,\dots,x_n]$65

The same framework proves a dependency-cone lower bound for non-certification: if observation factors only through the confirmation-observable final-key target, then for operation sets outside the support-active cone, soundness and completeness errors satisfy $x=[x_1,\dots,x_n]$66. This is not merely a performance limitation but a certifiability limit (Jiménez, 3 Jun 2026).

Other black-box token verification papers state more task-specific limits. RTD-Guard is designed around the observation that adversarial word substitutions resemble RTD “replaced tokens,” so its mechanism is explicitly aligned with word-substitution perturbations and adversarial synonym swaps (Zhu et al., 13 Mar 2026). TTP-Detect reports weaker signals for extremely distribution-preserving watermarks, degraded local consistency under aggressive paraphrasing such as Dipper-2, reduced reliability for very short queries below $x=[x_1,\dots,x_n]$67 tokens, and possible compromise if an adversary fine-tunes a generator to mimic the reference distributions (Wang et al., 16 Mar 2026). The distortion-free watermarking scheme likewise finds that paraphrasing attacks largely destroy watermark signals, motivating semantic-level or robust schemes (Bahri et al., 2024). DBSA makes no distributional assumptions about the LLM, but its statistical reliability depends on repeated sampling, nearest-neighbor substitutions, and permutation testing, which raises compute and query costs (Rauba et al., 12 Dec 2025). VeilProbe explicitly addresses scarcity of labeled member/non-member examples through a prototype-based classifier to alleviate overfitting (Hu et al., 24 Jun 2025).

Taken together, the surveyed work indicates that black-box token verification is bounded less by lack of ingenuity than by the observables exposed through the interface. When the observable carries a strong token-level signature—such as RTD replacement likelihood, low-confidence single-token failure, or a keyed watermark statistic—verification can be both efficient and accurate. When observability is mediated through stochastic generations or hidden confirmation points, verification becomes a problem of relative testing, entropy transfer, or, in the strongest negative results, non-certification.