Black-Box Distillation: Transfer under Constraints

• Black-Box Distillation (B2D) is a knowledge transfer paradigm that trains a student model solely from teacher outputs via an external interface, without accessing internal parameters.
• B2D methods vary by using different transferable objects such as probability distributions, ranked lists, one-hot outputs, free-form responses, or reasoning traces tailored to specific applications.
• B2D tackles challenges like distribution shift, limited query budgets, and privacy concerns by employing techniques like dynamic weighting, surrogate extraction, and iterative refinement.

Searching arXiv for papers on black-box distillation and closely related formulations. Black-Box Distillation (B2D) denotes a family of teacher–student transfer procedures in which the teacher is accessed only through an external interface and the student is trained from observable outputs rather than from teacher parameters, gradients, hidden states, or full white-box supervision. In the literature represented here, the transferable object varies with the interface: class-probability vectors in black-box hypothesis transfer and source-free adaptation, ranked lists in retrieval and alignment, one-hot decisions in decision-only extraction, free-form responses in LLM distillation, and reasoning traces in anti-distillation work. What unifies these settings is that knowledge transfer proceeds through queryable behavior under restricted access, often in the presence of distribution shift, limited query budgets, privacy constraints, or safety concerns (Yu et al., 2020, Ma et al., 2022, Li et al., 2024, Ye et al., 13 Nov 2025).

1. Conceptual scope and problem formulation

B2D is not a single algorithmic template but an access model. In some papers, the teacher is a cloud model exposed by an API; in others it is a proprietary source hypothesis, a medical segmentation service, an open-vocabulary segmentation API, a simulator, or a closed LLM. The common constraint is that the teacher’s internals are inaccessible, while the student is locally trainable. This makes B2D distinct from standard knowledge distillation, which typically assumes access to soft teacher distributions or white-box internals, and distinct from ordinary domain adaptation, which often assumes access to source data or source parameters (Ma et al., 2022, Yu et al., 2020, Wang et al., 2023, Benigmim et al., 30 Aug 2025).

The supervision channel is therefore the key design variable. Some methods assume full probability vectors, as in dynamic knowledge distillation for black-box hypothesis transfer learning and black-box source-free domain adaptation. Others assume only rankings, such as CycleAlign’s preference rankings over candidate responses or Intermediate Distillation’s listwise document orders. More restrictive settings use only top-1 decisions or one-hot masks, as in decision-based black-box KD and open-vocabulary segmentation B2D. The LLM literature broadens the notion further: teacher outputs may be complete responses, preference judgments, or reasoning traces rather than token-level probabilities (Yu et al., 2020, Hong et al., 2023, Li et al., 2024, Wang, 2021, Benigmim et al., 30 Aug 2025, Jahan et al., 10 Dec 2025).

A recurring formal motif is that the student is optimized against an output-level surrogate for teacher behavior. In dkdHTL, for example, the per-instance objective is

$$\mathcal{L}(\mathbf{x}, \mathbf{y}; \mathbf{w}) = \alpha \mathcal{L}_1(\mathbf{p}^t, \mathbf{y}) + \beta \mathcal{L}_2(\mathbf{p}^t, \mathbf{p}^s; T),$$

where the supervised term and distillation term are reweighted instance-wise according to the consistency between teacher probabilities and target labels. In GAD, black-box LLM distillation is framed as a minimax game,

$$\max_G \min_D \ \mathcal{V}(G, D) = \mathbb{E}_{(x,y_t) \sim \mathcal{T}} \left[-\log \sigma\left( D(y_t) - D(G(x)) \right)\right],$$

so that the student is trained on-policy using a discriminator-derived reward rather than teacher logits (Yu et al., 2020, Ye et al., 13 Nov 2025).

2. Supervision interfaces and transfer objects

The broadest technical distinction inside B2D is the form of the teacher output. Probability-based B2D uses full class distributions and remains closest to classical KD. dkdHTL assumes that the only visible knowledge from the source hypothesis is its predicted probability on target data, reconstructs softened targets from probabilities rather than logits, and uses instance-wise dynamic weighting to reduce negative transfer under domain shift. Two-stage black-box source-free segmentation likewise consumes full soft pseudo-labels from the source model and trains a target model by KL minimization, first from the source API and then from a target-domain teacher produced in Stage I (Yu et al., 2020, Wang et al., 2023).

Ranking-based B2D replaces unavailable teacher probabilities with ordinal structure. CycleAlign distills alignment capabilities from a parameter-invisible LLM by querying it to rank multiple student-generated responses, then trains a white-box model with a ranking-plus-SFT objective

$$\mathcal{L} = \mathcal{L}_\mathrm{rank} + \lambda \mathcal{L}_\mathrm{sft}, \qquad \mathcal{L}_\mathrm{sft} = -\frac{1}{\lvert y^1 \rvert}\sum_{t}\log P_{\pi}(y^{1}_{t} \mid x, y^{1}_{<t}),$$

while feeding agreement rankings back into the prompt as dynamic in-context demonstrations. Intermediate Distillation in retrieval uses an LLM only for listwise reranking, trains an intermediate ranker with ListMLE on the teacher permutation, and then trains the retriever to match the ranker’s candidate-set distribution by

$D_{KL}(P_{RANK}\,\|\,P_{RETR}).$

In both cases, what is distilled is not next-token likelihood but a preference or relevance ordering (Hong et al., 2023, Li et al., 2024).

Decision-only B2D is stricter. DB3KD assumes the teacher returns only a top-1 class label, estimates distances to decision boundaries from hard-label queries, and converts those distances into pseudo-soft labels. Its zero-shot extension synthesizes pseudo-samples by pushing noise away from decision boundaries. In semantic segmentation, ATGC defines a one-hot-only B2D setting for open-vocabulary APIs and compensates for the lack of logits by selecting query scale through DINOv2 attention entropy,

$$\mathbf{S}(A_j) = - \sum_{u,v} A_j(u,v)\log A_j(u,v), \qquad s^* = \arg\min_{s_j \in \mathcal{S}} \mathbf{S}(A_j),$$

before querying the API for one-hot masks (Wang, 2021, Benigmim et al., 30 Aug 2025).

Generative and behavioral B2D use free-form text rather than structured probabilities. SeqKD-style imitation, CycleAlign’s teacher-written responses, GAD’s adversarially rewarded student outputs, and medical or safety-oriented behavioral cloning all fall in this category. Here the transfer object is the observable response distribution or response policy, not a token-level white-box teacher signal (Ye et al., 13 Nov 2025, Jahan et al., 10 Dec 2025, Hasan, 28 May 2026).

3. Algorithmic patterns

Several recurring algorithmic patterns organize the field. One is output imitation with correction for domain shift. dkdHTL explicitly balances teacher imitation against supervised target learning through a consistency score

$$S(\mathbf{y}, \mathbf{p}^s) = \exp(-H(\mathbf{y}, \mathbf{p}^s)),$$

and then sets

$$\alpha = \lambda + \delta (1 - S(\mathbf{y}, \mathbf{p}^s)), \qquad \beta = 1 - \alpha.$$

This converts black-box KD into a transfer-learning method that trusts the teacher more when teacher predictions align with ground truth. DDSR adopts a different correction mechanism: it fuses a black-box source model with CLIP, regularizes a full model and a subnetwork through Jensen–Shannon divergence and weighted gradient discrepancy, updates pseudo-labels by EMA, and finishes with prototype-based self-training. Both methods interpret black-box outputs as useful but noisy supervision under shift (Yu et al., 2020, Zhang et al., 24 Mar 2026).

A second pattern is teacher-interface translation. Intermediate Distillation inserts a ranker between LLM and retriever because a retriever cannot directly absorb a teacher’s listwise textual ranking. MEKD similarly inserts a generator that emulates an inverse mapping from logits to image space. Its two-stage workflow—deprivatization followed by distillation—uses a frozen generator to align student and teacher outputs through

$$\left\| G\!\left(S(x')/\tau\right) - G\!\left(T(x')/\tau\right) \right\|_{F} + \beta\, T(x') \log\frac{T(x')}{S(x')},$$

thereby replacing direct black-box logit matching with generative alignment (Li et al., 2024, Ma et al., 2022).

A third pattern is surrogate extraction for interpretation or analysis rather than compression. DiConStruct distills a black-box binary classifier into a concept-based structural causal model, jointly predicting semantic concepts and the teacher score. Its loss combines exogenous independence, concept prediction, and teacher-score distillation,

$$\mathcal L = \gamma \mathcal L_E + \beta \mathcal L_C + \mathcal L_D.$$

KDDT, by contrast, distills a black-box predictor into a decision tree by sampling pseudo-inputs over feature support and labeling them through teacher queries. In these works, B2D is used to build interpretable surrogates rather than smaller task models (Moreira et al., 2024, Lu et al., 2022).

A fourth pattern is iterative or on-policy refinement. CycleAlign updates prompts using agreement rankings; GAD updates a discriminator against current student responses; DDSR updates CLIP prompts and pseudo-labels online; BAHSD probes a recommender teacher across short, mid, and full sequence views to cope with head–tail heterogeneity. These methods reject the assumption that black-box supervision is fixed. Instead, the quality of the teacher signal is itself made prompt-conditioned, scale-conditioned, or policy-conditioned (Hong et al., 2023, Ye et al., 13 Nov 2025, Zhang et al., 24 Mar 2026, Zhou et al., 2 Jun 2026).

4. Application domains

B2D has been instantiated across a notably broad task spectrum. In language-model alignment, CycleAlign distills harmlessness and helpfulness from ChatGPT into LLaMA-7B or Alpaca-7B using only ranking feedback and teacher-written responses. On HH-RLHF, it reports that LLaMA-7B with RRHF rises from total reward 63.12 to 68.43 under CycleAlign$_\text{RRHF}$, while Alpaca-7B with PRO rises from 67.64 to 68.97 under CycleAlign$$\max_G \min_D \ \mathcal{V}(G, D) = \mathbb{E}_{(x,y_t) \sim \mathcal{T}} \left[-\log \sigma\left( D(y_t) - D(G(x)) \right)\right],$$0 (Hong et al., 2023).

In retrieval and RAG, Intermediate Distillation trains a dense retriever from black-box LLM rerankings using only 1,000 training instances. On NQ, a non-distilled retriever with HR@5 $$\max_G \min_D \ \mathcal{V}(G, D) = \mathbb{E}_{(x,y_t) \sim \mathcal{T}} \left[-\log \sigma\left( D(y_t) - D(G(x)) \right)\right],$$1 and HR@10 $$\max_G \min_D \ \mathcal{V}(G, D) = \mathbb{E}_{(x,y_t) \sim \mathcal{T}} \left[-\log \sigma\left( D(y_t) - D(G(x)) \right)\right],$$2 improves to as high as HR@5 $$\max_G \min_D \ \mathcal{V}(G, D) = \mathbb{E}_{(x,y_t) \sim \mathcal{T}} \left[-\log \sigma\left( D(y_t) - D(G(x)) \right)\right],$$3 and HR@10 $$\max_G \min_D \ \mathcal{V}(G, D) = \mathbb{E}_{(x,y_t) \sim \mathcal{T}} \left[-\log \sigma\left( D(y_t) - D(G(x)) \right)\right],$$4 with Claude3 Opus; on TriviaQA, HR@5 rises from $$\max_G \min_D \ \mathcal{V}(G, D) = \mathbb{E}_{(x,y_t) \sim \mathcal{T}} \left[-\log \sigma\left( D(y_t) - D(G(x)) \right)\right],$$5 to $$\max_G \min_D \ \mathcal{V}(G, D) = \mathbb{E}_{(x,y_t) \sim \mathcal{T}} \left[-\log \sigma\left( D(y_t) - D(G(x)) \right)\right],$$6 (Li et al., 2024).

In vision, B2D appears in classification, domain adaptation, and segmentation. dkdHTL improves over static KD on m-MNIST, Office-31, and MIMIC-III; the two-stage source-free segmentation method improves average DSC from 82.00 to 86.28 on fundus segmentation relative to the black-box baseline EMD and from 52.47 to 56.12 on prostate MRI (Yu et al., 2020, Wang et al., 2023). ATGC formalizes a one-hot-only segmentation B2D regime and shows ATGC reaching 50.1 mIoU on Cityscapes with SAN, compared with 48.4 for CoRTE and 48.8 for Naive Transfer, while under CLIP-DINOiser it reaches 37.9 versus 34.5 and 34.3 (Benigmim et al., 30 Aug 2025).

The paradigm also extends beyond conventional perception and NLP tasks. MEKD distills cloud classifiers for edge deployment under privacy constraints and limited communication (Ma et al., 2022). KDDT distills black-box models into stable decision trees for explanation (Lu et al., 2022). DiConStruct distills scores into causal concept models (Moreira et al., 2024). A simulation-heavy epidemiological system is distilled into an MLP student for COVID-19 forecasting, where projection MAPE on the US is 0.0433 for the student versus 0.0352 for the approximate teacher and 0.0727 for coarse search, while runtime drops from $$\max_G \min_D \ \mathcal{V}(G, D) = \mathbb{E}_{(x,y_t) \sim \mathcal{T}} \left[-\log \sigma\left( D(y_t) - D(G(x)) \right)\right],$$7 s for the approximate teacher to $$\max_G \min_D \ \mathcal{V}(G, D) = \mathbb{E}_{(x,y_t) \sim \mathcal{T}} \left[-\log \sigma\left( D(y_t) - D(G(x)) \right)\right],$$8 s for the student (Wang et al., 2021).

5. Evaluation, fidelity, and empirical criteria

B2D evaluation is similarly heterogeneous. Many papers report task utility: retrieval hit rates, segmentation mIoU or DSC, clinical auROC/auPRC, or recommendation Recall@10 and NDCG@10. Others evaluate teacher imitation more directly. DiConStruct reports fidelity as $$\max_G \min_D \ \mathcal{V}(G, D) = \mathbb{E}_{(x,y_t) \sim \mathcal{T}} \left[-\log \sigma\left( D(y_t) - D(G(x)) \right)\right],$$9 between student and teacher scores, with local variants reaching 98.79%–98.83% on CUB and 99.31%–99.47% on Merchant Fraud with a LightGBM teacher. GAD uses averaged GPT-4o score and human comparison; on LMSYS, a Qwen2.5-14B-Instruct student trained with GAD reaches 52.1 against GPT-5-Chat at 51.7, while SeqKD reaches 50.6 (Moreira et al., 2024, Ye et al., 13 Nov 2025).

A more recent development is the claim that output similarity is not sufficient. “Bounded Behavioral Indistinguishability” formalizes evaluation as a teacher–student distinguishing game. For adversary class $$\mathcal{L} = \mathcal{L}_\mathrm{rank} + \lambda \mathcal{L}_\mathrm{sft}, \qquad \mathcal{L}_\mathrm{sft} = -\frac{1}{\lvert y^1 \rvert}\sum_{t}\log P_{\pi}(y^{1}_{t} \mid x, y^{1}_{<t}),$$0,

$$\mathcal{L} = \mathcal{L}_\mathrm{rank} + \lambda \mathcal{L}_\mathrm{sft}, \qquad \mathcal{L}_\mathrm{sft} = -\frac{1}{\lvert y^1 \rvert}\sum_{t}\log P_{\pi}(y^{1}_{t} \mid x, y^{1}_{<t}),$$1

and $$\mathcal{L} = \mathcal{L}_\mathrm{rank} + \lambda \mathcal{L}_\mathrm{sft}, \qquad \mathcal{L}_\mathrm{sft} = -\frac{1}{\lvert y^1 \rvert}\sum_{t}\log P_{\pi}(y^{1}_{t} \mid x, y^{1}_{<t}),$$2 is $$\mathcal{L} = \mathcal{L}_\mathrm{rank} + \lambda \mathcal{L}_\mathrm{sft}, \qquad \mathcal{L}_\mathrm{sft} = -\frac{1}{\lvert y^1 \rvert}\sum_{t}\log P_{\pi}(y^{1}_{t} \mid x, y^{1}_{<t}),$$3-behaviorally indistinguishable from $$\mathcal{L} = \mathcal{L}_\mathrm{rank} + \lambda \mathcal{L}_\mathrm{sft}, \qquad \mathcal{L}_\mathrm{sft} = -\frac{1}{\lvert y^1 \rvert}\sum_{t}\log P_{\pi}(y^{1}_{t} \mid x, y^{1}_{<t}),$$4 if this advantage is at most $$\mathcal{L} = \mathcal{L}_\mathrm{rank} + \lambda \mathcal{L}_\mathrm{sft}, \qquad \mathcal{L}_\mathrm{sft} = -\frac{1}{\lvert y^1 \rvert}\sum_{t}\log P_{\pi}(y^{1}_{t} \mid x, y^{1}_{<t}),$$5. In experiments on Qwen and Llama families, LoRA distillation raises embedding similarity from 0.788 to 0.862 for Qwen and from 0.814 to 0.874 for Llama, yet learned discriminators and pairwise teacher-identification still retain nonzero advantage. For Qwen, pairwise distinguishing advantage falls from 0.158 to 0.081 after LoRA distillation, rather than disappearing (Hasan, 28 May 2026).

These results motivate a distinction between functional fidelity and behavioral indistinguishability. A student may match task outputs or reward scores while remaining detectably different in style/format, robustness, refusal policy, or domain-technical behavior. The same tension appears in medical LLM cloning: a LoRA-tuned LLaMA-3 8B surrogate trained on 25,000 benign Meditron-7B completions shows moderate benign fidelity, but on 50 handcrafted adversarial prompts its refusal failure reaches 94% and its unsafe violation rate reaches 86%, compared with 14% and 66% for Meditron-7B and 78% and 46% for the base model (Jahan et al., 10 Dec 2025).

6. Risks, defenses, and unresolved questions

B2D is technically productive but normatively ambivalent. Several papers treat it as a constructive tool for low-resource alignment, local deployment, or interpretability. Others show that the same machinery enables privacy attacks, model extraction, or safety degradation. GLiRA demonstrates that distilling a black-box classifier into shadow models improves black-box membership inference; for a ResNet-34 target on CIFAR100, [email protected]% FPR rises from 9.13% under LiRA to 17.62% under GLiRA(MSE) (Galichin et al., 2024). The medical LLM study argues that benign-only behavioral distillation exposes a “functional-ethical gap,” where task utility transfers while alignment collapses (Jahan et al., 10 Dec 2025).

This has stimulated a defensive subliterature. “Protecting the Trace” formulates antidistillation as a Stackelberg game:

$$\mathcal{L} = \mathcal{L}_\mathrm{rank} + \lambda \mathcal{L}_\mathrm{sft}, \qquad \mathcal{L}_\mathrm{sft} = -\frac{1}{\lvert y^1 \rvert}\sum_{t}\log P_{\pi}(y^{1}_{t} \mid x, y^{1}_{<t}),$$6

and proposes TraceGuard, a post-generation black-box defense that deletes high-importance branching sentences from reasoning traces. It reports that stronger students suffer larger accuracy drops as more thought-anchor sentences are poisoned, while random sentence removal is substantially less effective (Hartman et al., 25 Apr 2026). Defensive proposals in the medical LLM setting are more preliminary, centering on behavioral watermarking, prompt monitoring, and DistillGuard++ as a prototype detector for alignment drift (Jahan et al., 10 Dec 2025).

Several unresolved issues recur across the literature. One is the mismatch between what the API returns and what the student needs: rankings, one-hot outputs, and textual completions require different intermediate abstractions than logits. A second is teacher unreliability under shift, which motivates dynamic weighting, multi-scale probing, auxiliary teachers, or self-distillation. A third is evaluation mismatch: reward models, semantic similarity, or mIoU do not capture all dimensions of transferred behavior. A fourth is access realism: many methods described as black-box still assume rich outputs such as full logits or repeated large-scale querying. Finally, there is a terminological misconception worth excluding explicitly: (Gil et al., 2019), despite its title string, is not a substantive adversarial-distillation paper in the provided material and should not be treated as evidence for a B2D method (Gil et al., 2019).

In aggregate, B2D has evolved from output-level imitation into a broader framework for transfer under interface constraints. The surveyed works show that the transferable object may be probability mass, rankings, decisions, scores, concepts, free-form responses, or reasoning traces; that the student may optimize via KL, ranking losses, contrastive objectives, policy gradients, or surrogate-tree induction; and that the central scientific questions increasingly concern not only whether a student can match a teacher, but which parts of behavior transfer, which parts do not, and how that asymmetry should be measured or controlled (Yu et al., 2020, Hong et al., 2023, Li et al., 2024, Hasan, 28 May 2026, Hartman et al., 25 Apr 2026).