---
title: 'B3IT: Black-Box Border Input Tracking'
url: https://www.emergentmind.com/topics/black-box-border-input-tracking-b3it
type: topic
---

# B3IT: Black-Box Border Input Tracking

Black-Box Border Input Tracking (B3IT) is a token-efficient, strictly black-box method for remote change detection in LLM APIs. Its central object is the **border input**: a prompt for which, at low temperature, the first-token logits have at least two maximizers, so repeated queries can yield more than one distinct top token. B3IT exploits the fact that such prompts are highly sensitive to small changes in model parameters, system prompts, safety policies, routing, quantization, pruning, or further fine-tuning, while requiring access only to sampled output tokens rather than weights, logits, or log probabilities. The method is formalized in "Token-Efficient Change Detection in LLM APIs" [2602.11083].

## 1. Problem setting and access model

B3IT is posed as a hypothesis test over two stages of an API endpoint. At initialization, the endpoint corresponds to parameters $\theta_0$; at detection time, it corresponds to parameters $\theta_1$. The objective is to test
\[
H_0: \theta_0 = \theta_1
\quad\text{vs.}\quad
H_1: \theta_0 \neq \theta_1.
\]
The observable is restricted to API outputs: one can send prompts $x$ with temperature $T$ and observe returned tokens $f_\theta(x,T)$, but not weights, logits, or log probabilities [2602.11083].

The motivating changes are broad. The formulation explicitly includes model-level changes such as fine-tuning, LoRA, distillation, quantization, pruning, and further RL fine-tuning; prompt- or system-level changes such as system prompt changes and safety policy adjustments; and infrastructure-level changes such as different GPUs, CUDA or compiler versions, or routing to a different or misconfigured model instance. The paper’s emphasis is query-efficient detection even for small changes [2602.11083].

This access model is stricter than both white-box and grey-box settings. White-box strategies such as ESF and TRAP require access to model weights or internal optimization signals. Grey-box approaches such as LT require token log probabilities. B3IT instead targets the most restrictive regime: only output tokens are available. A plausible implication is that B3IT is intended for commercial or proprietary endpoints where internal model state is inaccessible by design [2602.11083].

## 2. Border inputs and low-temperature behavior

For a fixed test prompt $x_{\text{test}}$, let the first-token logits be
\[
z(\theta, x_{\text{test}}) := W r(\theta_{\text{pre}}, x_{\text{test}}) + b \in \mathbb{R}^d,
\]
and define the set of maximum-logit indices
\[
\mathcal{M} := \bigl\{ i \in \{1,\dots,d\} : z_i = \max_{1\le j\le d} z_j \bigr\},
\qquad
k := |\mathcal{M}|.
\]
An input is a **Border Input** if, at low temperature, the logits for its first-token prediction have at least two maximizers, i.e. $k \ge 2$. Operationally, border inputs are discovered empirically as prompts whose first output token, sampled repeatedly at very low temperature, takes more than one distinct value [2602.11083].

The underlying intuition is geometric. At sampling temperature $\tau$, token probabilities are
\[
p_i^{(\tau)}(\theta) =
\frac{\exp(z_i(\theta)/\tau)}
{\sum_{j=1}^d \exp(z_j(\theta)/\tau)}.
\]
If $k=1$, the distribution collapses to a Dirac mass on the unique top token as $\tau \to 0$, so small parameter changes often leave the sampled token unchanged. If $k\ge 2$, the probability mass concentrates uniformly on $\mathcal{M}$, and small perturbations can break the tie and collapse the distribution onto a single token in $\mathcal{M}$. The paper therefore describes border inputs as prompts near a decision boundary in logit space, where multiple tokens are equally preferred [2602.11083].

This low-temperature sensitivity is the conceptual basis of B3IT. Rather than monitoring generic prompts, it monitors prompts that are already poised at a tie or near-tie in the first-token decision. This suggests that the method is deliberately designed to maximize sensitivity per token rather than to characterize the entire output distribution.

## 3. Statistical and information-theoretic foundations

The statistical analysis is formulated for the first token only. For a fixed prompt, let the categorical output distribution be $\mathbf{p}=(p_1,\dots,p_d)$, and let $T_n=\hat{\mathbf{p}}$ denote empirical frequencies from $n$ i.i.d. token samples. Asymptotically,
\[
T_n \sim \mathcal{N}\!\Bigl(\mathbf{p}, \frac{1}{n}F(\mathbf{p})\Bigr),
\]
where
\[
F(\mathbf{p}) := \operatorname{diag}(\mathbf{p}) - \mathbf{p}\mathbf{p}^\top.
\]
The paper then places the problem in a Local Asymptotic Normality regime by considering $\theta_1=\theta_0+\epsilon_n h$ with $\epsilon_n=s/\sqrt{n}$ and Jacobian
\[
J := \nabla_\theta g(\theta_0, x_{\text{test}}),
\]
where $g(\theta,x_{\text{test}})$ is the reduced output map [2602.11083].

The key asymptotic quantity is
\[
\mathrm{SNR}^2(h)
:= h^\top \bigl(J^\top F(\mathbf{p}_0)^{-1}J\bigr) h.
\]
Theorem 3.1 states that the Type-II error of optimal level-$\alpha$ tests is governed by this signal-to-noise ratio. Higher $\mathrm{SNR}^2(h)$ implies more powerful detection [2602.11083].

For LLMs with a linear last-layer head, the paper derives the specialization
\[
\mathrm{SNR}^2(h)
=
\frac{1}{\tau^2}\,
h^\top \bigl(J_z(\theta)^\top \Sigma(\mathbf{p}^{(\tau)}(\theta)) J_z(\theta)\bigr) h,
\]
where
\[
\Sigma(\mathbf{p}) := \operatorname{diag}(\mathbf{p}) - \mathbf{p}\mathbf{p}^\top.
\]
This expression yields the paper’s central phase transition. If $k=1$, then
\[
\mathrm{SNR}^2(h)\to 0
\quad \text{as } \tau\to 0.
\]
If $k\ge 2$ and
\[
h^\top \left(J_z^\top \Sigma_{\mathcal{M}} J_z\right) h \neq 0,
\]
then
\[
\mathrm{SNR}^2(h)\to +\infty
\quad \text{as } \tau\to 0.
\]
The condition is stated to hold for almost every direction $h\in\mathbb{S}^{q-1}$; if $h$ is drawn from a continuous distribution, it holds with probability $1$ [2602.11083].

The practical interpretation is explicit: non-border inputs are asymptotically useless for change detection in the zero-temperature limit, whereas border inputs are asymptotically maximally informative. This is the theoretical justification for monitoring support changes only on a carefully selected set of prompts.

## 4. Initialization and detection procedure

B3IT consists of an initialization stage and a detection stage, both operating at the lowest available temperature $T$, typically $T\approx 0$. In initialization, a candidate pool $\mathcal{X}=\{x_1,\dots,x_n\}$ of short prompts is constructed. Each candidate is queried $m$ times, with the paper recommending $m=3$. If the first output token takes more than one distinct value across those $m$ queries, the candidate is retained as a border input and added to the BI set $\mathcal{B}$. This continues until the desired number $\bar n$ of border inputs is found [2602.11083].

For each retained BI $x\in\mathcal{B}$, the endpoint is then queried $n_1$ times to build a reference support
\[
\hat S_1(x):=\{y^{(1)}_1,\dots,y^{(1)}_{n_1}\}.
\]
Appendix C analyzes the query cost of BI discovery and shows that $m=3$ is a near-optimal trade-off in typical settings: $m=2$ is dominated by $m=3$, and $m=4$ is better only if the BI frequency $f_B>0.75$, which the paper characterizes as unrealistic in practice [2602.11083].

At detection time, the method assumes that, for a fixed BI at very low temperature, the output distribution is well approximated by a uniform distribution over an unknown support. Under the reference model, $P=\mathrm{Unif}(S_1)$; under the candidate model, $Q=\mathrm{Unif}(S_2)$. After collecting detection samples
\[
\hat S_2(x):=\{y^{(2)}_1,\dots,y^{(2)}_{n_2}\},
\]
B3IT rejects “no change” if the symmetric difference is non-empty:
\[
\hat S_1(x)\triangle \hat S_2(x)\neq \emptyset.
\]
Operationally, the algorithm declares **“Change detected”** as soon as any monitored BI exhibits a support mismatch; otherwise it declares **“No change detected”** [2602.11083].

For ROC construction and threshold tuning, the paper also computes total variation (TV) distance between empirical first-token distributions for each BI and averages TV across a small number of BIs, using the averaged TV as a continuous statistic. The core B3IT decision rule, however, is the support-mismatch test [2602.11083].

## 5. Finite-sample guarantees and efficiency

The support-mismatch test admits explicit finite-sample bounds. Under $H_0$, if $S_1=S_2=S$ with $|S|=k$ and both distributions are uniform on $S$, then
\[
\mathbb{P}_{H_0}(\mathcal{R})
\le
k\Bigl(1-\frac{1}{k}\Bigr)^{n_1}
+
k\Bigl(1-\frac{1}{k}\Bigr)^{n_2}
\le
k e^{-n_1/k} + k e^{-n_2/k}.
\]
The interpretation given in the paper is that false positives occur only when one empirical support fails to observe all true support tokens [2602.11083].

Under $H_1$, with $I=S_1\cap S_2$, $k_1=|S_1|$, $k_2=|S_2|$, $p_1=|I|/k_1$, and $p_2=|I|/k_2$, the bound is
\[
\mathbb{P}_{H_1}(\mathcal{R}^c)\le p_1^{n_1}p_2^{n_2}.
\]
A common case is support collapse from two tokens to one, such as $S_1=\{a,b\}$ and $S_2=\{a\}$. Then $p_1=1/2$, $p_2=1$, giving
\[
\mathbb{P}_{H_1}(\mathcal{R}^c)\le 2^{-n_1}.
\]
The paper also proves a lower bound for the same $k=2$ regime:
\[
\mathbb{P}_{H_0}(\mathcal{R}) + \mathbb{P}_{H_1}(\mathcal{R}^c)\ge 2^{-(n+1)}.
\]
It therefore states that the simple support-mismatch rule is optimal up to a constant factor in the main practical regime [2602.11083].

Efficiency follows from the structure of the test. B3IT uses very short prompts, only the first output token of each query, a small number of BIs, and very few samples per BI. In the TinyChange benchmark, the operating point selected in the paper uses **5 BIs**, **50 reference samples per BI**, and **3 detection samples per BI**. At about **\$2.2 / year per endpoint** for hourly monitoring, B3IT achieves **ROC AUC ≈ 0.90**. The best black-box baseline at comparable performance is MET at $T=0$, which achieves **ROC AUC ≈ 0.88** but costs **\$67 / year**. The paper summarizes this as roughly **30× cheaper** for similar performance [2602.11083].

## 6. Empirical performance, limitations, and broader context

The in-vitro evaluation uses the TinyChange benchmark with 9 instruction-tuned open-weight models ranging from 0.5B to 9B parameters and perturbations including fine-tuning, LoRA, pruning, and additive Gaussian parameter noise. B3IT discovers a pool of border inputs per model by sampling 20k candidate prompts and querying each 3 times at $T=0$, yielding **105–1182 BIs/model**. On subtle fine-tuning changes, the paper reports **ROC AUC ≈ 0.87** even for **single-step fine-tuning**, while B3IT and grey-box LT remain substantially stronger than black-box baselines such as MMLU-ALG and MET on low-magnitude perturbations [2602.11083].

The in-vivo study filters OpenRouter endpoints to a set of **93 endpoints**, spanning **64 models and 20 providers**. Border-input prevalence is high but not universal. At $T=0$, about **62%** of endpoints yield at least one BI within the stated search budget; at small non-zero temperature, about **80%** do. The paper identifies **18 endpoints** where no BIs are found under the budget, of which **16 are reasoning endpoints** that do not return any first token under short output limits and **2** appear to require at least 2 output tokens before producing a non-empty response. This delimits the main practical coverage limitation of first-token B3IT [2602.11083].

For **54 endpoints** with at least 5 BIs found at $T=0$, the continuous monitoring experiment uses **5 BIs**, **50 reference samples per BI**, and **3 detection samples per BI per day** over **23 days**. The reported hourly-equivalent monitoring cost is **\$0.52 per endpoint per year**, with initialization costing about **\$0.0045 per endpoint**. Using an average TV threshold of **0.5**, together with a persistence criterion of remaining below threshold for at least 4 days and then above threshold for at least 4 days, the study detects **8 such changes** across endpoints. One cited example is Together AI’s routing of `Mistral-7B-Instruct-v0.3` to `Ministral-3-14B-Instruct-2512`, corroborated by a public changelog [2602.11083].

The paper also states several limitations. Coverage is incomplete because some endpoints do not expose usable first-token border inputs under short-output constraints. The theory is single-token and does not yet extend to multi-token dependencies. The phase transition relies on very low temperature, while some APIs may implement $T=0$ with special-case behavior that differs from the mathematical $\tau\to 0$ limit. The paper further notes that hardware and batching nondeterminism can matter when thresholds are set very low [2602.11083].

Related black-box input-tracking formulations appear in other domains. "RISE: Randomized Input Sampling for Explanation of Black-box Models" treats an image model as an oracle and estimates per-pixel importance by probing randomly masked inputs, thereby localizing regions and approximate borders relevant to a prediction [1806.07421]. "SCALE-UP: An Efficient Black-box Input-level Backdoor Detection via Analyzing Scaled Prediction Consistency" tracks label stability along a one-dimensional path of pixel-wise amplification and uses scaled prediction consistency to flag backdoored inputs in MLaaS settings [2302.03251]. "Hacking in the Blind: (Almost) Invisible Runtime UI Attacks on Safety-Critical Terminals" uses the phrase in a different but closely related sense: a hardware device at the I/O border passively tracks hidden UI state and input location from raw events alone [1604.04723]. This suggests a broader family of techniques organized around black-box probing of boundary-sensitive or border-located inputs, but in the literature the acronym **B3IT** is formalized specifically for low-temperature LLM API change detection [2602.11083].

Source: https://www.emergentmind.com/topics/black-box-border-input-tracking-b3it