B3IT: Black-Box Border Input Tracking

• The paper introduces B3IT, a token-efficient method using border inputs to detect subtle changes in LLM API behavior without access to internal model data.
• B3IT exploits low-temperature prompts that yield multiple top tokens, making it highly sensitive to changes from fine-tuning, quantization, and other modifications.
• The method achieves robust change detection with ROC AUC ≈ 0.90 while reducing costs up to 30× compared to other white-box and grey-box techniques.

Black-Box Border Input Tracking (B3IT) is a token-efficient, strictly black-box method for remote change detection in LLM APIs. Its central object is the border input: a prompt for which, at low temperature, the first-token logits have at least two maximizers, so repeated queries can yield more than one distinct top token. B3IT exploits the fact that such prompts are highly sensitive to small changes in model parameters, system prompts, safety policies, routing, quantization, pruning, or further fine-tuning, while requiring access only to sampled output tokens rather than weights, logits, or log probabilities. The method is formalized in "Token-Efficient Change Detection in LLM APIs" (Chauvin et al., 11 Feb 2026).

1. Problem setting and access model

B3IT is posed as a hypothesis test over two stages of an API endpoint. At initialization, the endpoint corresponds to parameters $\theta_0$; at detection time, it corresponds to parameters $\theta_1$. The objective is to test

$$H_0: \theta_0 = \theta_1 \quad\text{vs.}\quad H_1: \theta_0 \neq \theta_1.$$

The observable is restricted to API outputs: one can send prompts $x$ with temperature $T$ and observe returned tokens $f_\theta(x,T)$, but not weights, logits, or log probabilities (Chauvin et al., 11 Feb 2026).

The motivating changes are broad. The formulation explicitly includes model-level changes such as fine-tuning, LoRA, distillation, quantization, pruning, and further RL fine-tuning; prompt- or system-level changes such as system prompt changes and safety policy adjustments; and infrastructure-level changes such as different GPUs, CUDA or compiler versions, or routing to a different or misconfigured model instance. The paper’s emphasis is query-efficient detection even for small changes (Chauvin et al., 11 Feb 2026).

This access model is stricter than both white-box and grey-box settings. White-box strategies such as ESF and TRAP require access to model weights or internal optimization signals. Grey-box approaches such as LT require token log probabilities. B3IT instead targets the most restrictive regime: only output tokens are available. A plausible implication is that B3IT is intended for commercial or proprietary endpoints where internal model state is inaccessible by design (Chauvin et al., 11 Feb 2026).

2. Border inputs and low-temperature behavior

For a fixed test prompt $x_{\text{test}}$, let the first-token logits be

$$z(\theta, x_{\text{test}}) := W r(\theta_{\text{pre}}, x_{\text{test}}) + b \in \mathbb{R}^d,$$

and define the set of maximum-logit indices

$$\mathcal{M} := \bigl\{ i \in \{1,\dots,d\} : z_i = \max_{1\le j\le d} z_j \bigr\}, \qquad k := |\mathcal{M}|.$$

An input is a Border Input if, at low temperature, the logits for its first-token prediction have at least two maximizers, i.e. $k \ge 2$. Operationally, border inputs are discovered empirically as prompts whose first output token, sampled repeatedly at very low temperature, takes more than one distinct value (Chauvin et al., 11 Feb 2026).

The underlying intuition is geometric. At sampling temperature $\theta_1$0, token probabilities are

$\theta_1$1

If $\theta_1$2, the distribution collapses to a Dirac mass on the unique top token as $\theta_1$3, so small parameter changes often leave the sampled token unchanged. If $\theta_1$4, the probability mass concentrates uniformly on $\theta_1$5, and small perturbations can break the tie and collapse the distribution onto a single token in $\theta_1$6. The paper therefore describes border inputs as prompts near a decision boundary in logit space, where multiple tokens are equally preferred (Chauvin et al., 11 Feb 2026).

This low-temperature sensitivity is the conceptual basis of B3IT. Rather than monitoring generic prompts, it monitors prompts that are already poised at a tie or near-tie in the first-token decision. This suggests that the method is deliberately designed to maximize sensitivity per token rather than to characterize the entire output distribution.

3. Statistical and information-theoretic foundations

The statistical analysis is formulated for the first token only. For a fixed prompt, let the categorical output distribution be $\theta_1$7, and let $\theta_1$8 denote empirical frequencies from $\theta_1$9 i.i.d. token samples. Asymptotically,

$$H_0: \theta_0 = \theta_1 \quad\text{vs.}\quad H_1: \theta_0 \neq \theta_1.$$0

where

$$H_0: \theta_0 = \theta_1 \quad\text{vs.}\quad H_1: \theta_0 \neq \theta_1.$$1

The paper then places the problem in a Local Asymptotic Normality regime by considering $$H_0: \theta_0 = \theta_1 \quad\text{vs.}\quad H_1: \theta_0 \neq \theta_1.$$2 with $$H_0: \theta_0 = \theta_1 \quad\text{vs.}\quad H_1: \theta_0 \neq \theta_1.$$3 and Jacobian

$$H_0: \theta_0 = \theta_1 \quad\text{vs.}\quad H_1: \theta_0 \neq \theta_1.$$4

where $$H_0: \theta_0 = \theta_1 \quad\text{vs.}\quad H_1: \theta_0 \neq \theta_1.$$5 is the reduced output map (Chauvin et al., 11 Feb 2026).

The key asymptotic quantity is

$$H_0: \theta_0 = \theta_1 \quad\text{vs.}\quad H_1: \theta_0 \neq \theta_1.$$6

Theorem 3.1 states that the Type-II error of optimal level-$$H_0: \theta_0 = \theta_1 \quad\text{vs.}\quad H_1: \theta_0 \neq \theta_1.$$7 tests is governed by this signal-to-noise ratio. Higher $$H_0: \theta_0 = \theta_1 \quad\text{vs.}\quad H_1: \theta_0 \neq \theta_1.$$8 implies more powerful detection (Chauvin et al., 11 Feb 2026).

For LLMs with a linear last-layer head, the paper derives the specialization

$$H_0: \theta_0 = \theta_1 \quad\text{vs.}\quad H_1: \theta_0 \neq \theta_1.$$9

where

$x$0

This expression yields the paper’s central phase transition. If $x$1, then

$x$2

If $x$3 and

$x$4

then

$x$5

The condition is stated to hold for almost every direction $x$6; if $x$7 is drawn from a continuous distribution, it holds with probability $x$8 (Chauvin et al., 11 Feb 2026).

The practical interpretation is explicit: non-border inputs are asymptotically useless for change detection in the zero-temperature limit, whereas border inputs are asymptotically maximally informative. This is the theoretical justification for monitoring support changes only on a carefully selected set of prompts.

4. Initialization and detection procedure

B3IT consists of an initialization stage and a detection stage, both operating at the lowest available temperature $x$9, typically $T$0. In initialization, a candidate pool $T$1 of short prompts is constructed. Each candidate is queried $T$2 times, with the paper recommending $T$3. If the first output token takes more than one distinct value across those $T$4 queries, the candidate is retained as a border input and added to the BI set $T$5. This continues until the desired number $T$6 of border inputs is found (Chauvin et al., 11 Feb 2026).

For each retained BI $T$7, the endpoint is then queried $T$8 times to build a reference support

$T$9

Appendix C analyzes the query cost of BI discovery and shows that $f_\theta(x,T)$0 is a near-optimal trade-off in typical settings: $f_\theta(x,T)$1 is dominated by $f_\theta(x,T)$2, and $f_\theta(x,T)$3 is better only if the BI frequency $f_\theta(x,T)$4, which the paper characterizes as unrealistic in practice (Chauvin et al., 11 Feb 2026).

At detection time, the method assumes that, for a fixed BI at very low temperature, the output distribution is well approximated by a uniform distribution over an unknown support. Under the reference model, $f_\theta(x,T)$5; under the candidate model, $f_\theta(x,T)$6. After collecting detection samples

$f_\theta(x,T)$7

B3IT rejects “no change” if the symmetric difference is non-empty: $f_\theta(x,T)$8 Operationally, the algorithm declares “Change detected” as soon as any monitored BI exhibits a support mismatch; otherwise it declares “No change detected” (Chauvin et al., 11 Feb 2026).

For ROC construction and threshold tuning, the paper also computes total variation (TV) distance between empirical first-token distributions for each BI and averages TV across a small number of BIs, using the averaged TV as a continuous statistic. The core B3IT decision rule, however, is the support-mismatch test (Chauvin et al., 11 Feb 2026).

5. Finite-sample guarantees and efficiency

The support-mismatch test admits explicit finite-sample bounds. Under $f_\theta(x,T)$9, if $x_{\text{test}}$0 with $x_{\text{test}}$1 and both distributions are uniform on $x_{\text{test}}$2, then

$x_{\text{test}}$3

The interpretation given in the paper is that false positives occur only when one empirical support fails to observe all true support tokens (Chauvin et al., 11 Feb 2026).

Under $x_{\text{test}}$4, with $x_{\text{test}}$5, $x_{\text{test}}$6, $x_{\text{test}}$7, $x_{\text{test}}$8, and $x_{\text{test}}$9, the bound is

$$z(\theta, x_{\text{test}}) := W r(\theta_{\text{pre}}, x_{\text{test}}) + b \in \mathbb{R}^d,$$0

A common case is support collapse from two tokens to one, such as $$z(\theta, x_{\text{test}}) := W r(\theta_{\text{pre}}, x_{\text{test}}) + b \in \mathbb{R}^d,$$1 and $$z(\theta, x_{\text{test}}) := W r(\theta_{\text{pre}}, x_{\text{test}}) + b \in \mathbb{R}^d,$$2. Then $$z(\theta, x_{\text{test}}) := W r(\theta_{\text{pre}}, x_{\text{test}}) + b \in \mathbb{R}^d,$$3, $$z(\theta, x_{\text{test}}) := W r(\theta_{\text{pre}}, x_{\text{test}}) + b \in \mathbb{R}^d,$$4, giving

$$z(\theta, x_{\text{test}}) := W r(\theta_{\text{pre}}, x_{\text{test}}) + b \in \mathbb{R}^d,$$5

The paper also proves a lower bound for the same $$z(\theta, x_{\text{test}}) := W r(\theta_{\text{pre}}, x_{\text{test}}) + b \in \mathbb{R}^d,$$6 regime: $$z(\theta, x_{\text{test}}) := W r(\theta_{\text{pre}}, x_{\text{test}}) + b \in \mathbb{R}^d,$$7 It therefore states that the simple support-mismatch rule is optimal up to a constant factor in the main practical regime (Chauvin et al., 11 Feb 2026).

Efficiency follows from the structure of the test. B3IT uses very short prompts, only the first output token of each query, a small number of BIs, and very few samples per BI. In the TinyChange benchmark, the operating point selected in the paper uses 5 BIs, 50 reference samples per BI, and 3 detection samples per BI. At about \$z(\theta, x_{\text{test}}) := W r(\theta_{\text{pre}}, x_{\text{test}}) + b \in \mathbb{R}^d,$8T=0$z(\theta, x_{\text{test}}) := W r(\theta_{\text{pre}}, x_{\text{test}}) + b \in \mathbb{R}^d,$967 / year. The paper summarizes this as roughly 30× cheaper for similar performance (Chauvin et al., 11 Feb 2026).

6. Empirical performance, limitations, and broader context

The in-vitro evaluation uses the TinyChange benchmark with 9 instruction-tuned open-weight models ranging from 0.5B to 9B parameters and perturbations including fine-tuning, LoRA, pruning, and additive Gaussian parameter noise. B3IT discovers a pool of border inputs per model by sampling 20k candidate prompts and querying each 3 times at $\mathcal{M} := \bigl\{ i \in \{1,\dots,d\} : z_i = \max_{1\le j\le d} z_j \bigr\}, \qquad k := |\mathcal{M}|.$0, yielding 105–1182 BIs/model. On subtle fine-tuning changes, the paper reports ROC AUC ≈ 0.87 even for single-step fine-tuning, while B3IT and grey-box LT remain substantially stronger than black-box baselines such as MMLU-ALG and MET on low-magnitude perturbations (Chauvin et al., 11 Feb 2026).

The in-vivo study filters OpenRouter endpoints to a set of 93 endpoints, spanning 64 models and 20 providers. Border-input prevalence is high but not universal. At $\mathcal{M} := \bigl\{ i \in \{1,\dots,d\} : z_i = \max_{1\le j\le d} z_j \bigr\}, \qquad k := |\mathcal{M}|.$1, about 62% of endpoints yield at least one BI within the stated search budget; at small non-zero temperature, about 80% do. The paper identifies 18 endpoints where no BIs are found under the budget, of which 16 are reasoning endpoints that do not return any first token under short output limits and 2 appear to require at least 2 output tokens before producing a non-empty response. This delimits the main practical coverage limitation of first-token B3IT (Chauvin et al., 11 Feb 2026).

For 54 endpoints with at least 5 BIs found at $\mathcal{M} := \bigl\{ i \in \{1,\dots,d\} : z_i = \max_{1\le j\le d} z_j \bigr\}, \qquad k := |\mathcal{M}|.$2, the continuous monitoring experiment uses <strong>5 BIs</strong>, <strong>50 reference samples per BI</strong>, and <strong>3 detection samples per BI per day</strong> over <strong>23 days</strong>. The reported hourly-equivalent monitoring cost is <strong>\$\mathcal{M} := \bigl\{ i \in \{1,\dots,d\} : z_i = \max_{1\le j\le d} z_j \bigr\}, \qquad k := |\mathcal{M}|.$30.0045 per endpoint. Using an average TV threshold of 0.5, together with a persistence criterion of remaining below threshold for at least 4 days and then above threshold for at least 4 days, the study detects 8 such changes across endpoints. One cited example is Together AI’s routing of Mistral-7B-Instruct-v0.3 to Ministral-3-14B-Instruct-2512, corroborated by a public changelog (Chauvin et al., 11 Feb 2026).

The paper also states several limitations. Coverage is incomplete because some endpoints do not expose usable first-token border inputs under short-output constraints. The theory is single-token and does not yet extend to multi-token dependencies. The phase transition relies on very low temperature, while some APIs may implement $\mathcal{M} := \bigl\{ i \in \{1,\dots,d\} : z_i = \max_{1\le j\le d} z_j \bigr\}, \qquad k := |\mathcal{M}|.$4 with special-case behavior that differs from the mathematical $\mathcal{M} := \bigl\{ i \in \{1,\dots,d\} : z_i = \max_{1\le j\le d} z_j \bigr\}, \qquad k := |\mathcal{M}|.$5 limit. The paper further notes that hardware and batching nondeterminism can matter when thresholds are set very low (Chauvin et al., 11 Feb 2026).

Related black-box input-tracking formulations appear in other domains. "RISE: Randomized Input Sampling for Explanation of Black-box Models" treats an image model as an oracle and estimates per-pixel importance by probing randomly masked inputs, thereby localizing regions and approximate borders relevant to a prediction (Petsiuk et al., 2018). "SCALE-UP: An Efficient Black-box Input-level Backdoor Detection via Analyzing Scaled Prediction Consistency" tracks label stability along a one-dimensional path of pixel-wise amplification and uses scaled prediction consistency to flag backdoored inputs in MLaaS settings (Guo et al., 2023). "Hacking in the Blind: (Almost) Invisible Runtime UI Attacks on Safety-Critical Terminals" uses the phrase in a different but closely related sense: a hardware device at the I/O border passively tracks hidden UI state and input location from raw events alone (Malisa et al., 2016). This suggests a broader family of techniques organized around black-box probing of boundary-sensitive or border-located inputs, but in the literature the acronym B3IT is formalized specifically for low-temperature LLM API change detection (Chauvin et al., 11 Feb 2026).