Black-White Box Prompt Learning (IOTA)

Updated 4 February 2026

The paper introduces IOTA’s framework that integrates black-box models with explicit white-box corrective knowledge to enhance prompt-based adaptation.
It employs a joint optimization objective that balances task cross-entropy with knowledge-guided loss, leading to measurable improvements in accuracy and correction rates.
This framework demonstrates versatility across vision and language tasks, outperforming traditional PET methods by offering increased interpretability and reduced API overhead.

A Black-White Box Prompt Learning Framework (IOTA) refers to a methodological paradigm that strategically unifies black-box pre-trained models and white-box knowledge systems to enable efficient and interpretable adaptation to downstream tasks through prompt-based learning. Unlike conventional parameter-efficient tuning (PET) approaches, which operate exclusively within the black-box regime and overlook the integration of explicit knowledge, Black-White Box frameworks—exemplified by IOTA—rely on corrective knowledge synthesis, knowledge-verbalized prompt selection, and joint optimization signals to bridge data-driven and knowledge-driven adaptation (Wang et al., 28 Jan 2026).

1. Conceptual Foundations and Motivation

Black-White Box Prompt Learning is grounded in the recognition that data-driven black-box models—such as frozen pre-trained vision transformers or LLMs—possess extensive representational power but are limited by their opaqueness, whereas white-box (knowledge-driven) modules can encode interpretable priors but lack capacity for high-dimensional modeling. The key insight motivating IOTA and related frameworks is that explicitly contrasting the wrong predictions (or behaviors) of a black-box with correct knowledge, extracting that contrast as interpretable prompts, and using these prompts to guide the adaptation of the black-box offers superior adaptation and interpretability over pure black-box or pure white-box approaches (Wang et al., 28 Jan 2026, Li et al., 2024, Ren et al., 14 Jun 2025, Chu et al., 29 Oct 2025, Cho et al., 2024).

2. Unified Architectural Principles

A prototypical IOTA-style architecture consists of:

Black Box Module: A fixed, pre-trained model (e.g., ViT encoder φ or LLM generator) whose internal parameters are unreachable. Adaptation occurs via trainable prompts or input modifications without backpropagation through internal weights.
White Box Module: An interpretable subsystem (e.g., CLIP text encoder ψ, open-source LLM, or rule-based engine) tasked with generating, verbalizing, and selecting corrective or guidance knowledge that can be mapped to actionable prompts or instruction candidates.
Prompt Injection and Selection: Corrective knowledge, distilled into prompt candidates, is injected at multiple points (e.g., input tokens, transformer layers, or intermediate guidance strings). Selection is typically informed by a matching or distributional softmax mechanism leveraging knowledge embeddings.
Joint Optimization: A combined objective balances task-relevant cross-entropy (or reward) with knowledge-guided constraints—typically enforcing alignment between the model’s intermediate representations and the knowledge-driven prompt or guidance distribution.

A representative example is the image classification method described in (Wang et al., 28 Jan 2026), where the black-box ViT is supplemented by a white-box language module generating prompts like “This is a photo of a [CLASS_A],” enabling the construction of a knowledge prompt set $V = \{(k_j, v_j)\}_{j=1}^S$ with interpretable knowledge embeddings $k_j$ and trainable soft-prompt vectors $v_j$ .

3. Core Methodologies: Corrective Knowledge and Prompt Learning

The distinctive methodological contribution of Black-White Box frameworks lies in the extraction and operationalization of corrective knowledge:

Contrastive Prompt Generation: For each misclassification (or model error), the framework explicitly constructs a correct prompt and an “incorrect cognition” prompt, producing a knowledge triplet $I = \{X, A, B\}$ where $A \ne B$ .
White Box Embedding and Selection: Both true and wrong prompts are embedded via a language encoder to produce $k_A, k_B$ . A soft probability distribution $P(k_j|\hat m)$ over knowledge prompts is computed as:

$P(k_j | \hat m) = \frac{ \exp( \text{cosine}(\hat m, k_j) / \tau ) }{ \sum_{i=1}^S \exp( \text{cosine}(\hat m, k_i) / \tau ) }$

where $\hat m$ is a match token derived from the black-box representation.

Prompt Injection and Soft Combination: The selected prompts are injected by forming a weighted sum $\hat v = \sum_j w_j v_j$ using $w_j = P(k_j|\hat m)$ ; $\hat v$ is concatenated to the input or intermediate tokens of the black-box model.
Knowledge-Guided Loss: The white-box’s corrective matching enforces a constraint:

$L_{ckg} = -\log P(k_A|\hat m) - 1_{[A\ne B]} \log(1 - P(k_B|\hat m) + \epsilon)$

Joint Loss Objective: Adaptation is driven by

$L = L_{cls} + \lambda L_{ckg}$

where $L_{cls}$ is conventional cross-entropy (or task reward) and $\lambda$ regulates knowledge guidance intensity (Wang et al., 28 Jan 2026).

This overall procedure can be instantiated in diverse domains—including vision (Wang et al., 28 Jan 2026, Cho et al., 2024), language modeling (Li et al., 2024, Ren et al., 14 Jun 2025, Chu et al., 29 Oct 2025), and instruction optimization—with adjustments for data flow, prompt form, and target tasks.

4. Algorithmic Workflow and Implementation

A typical Black-White Box Prompt Learning workflow includes:

Knowledge Prompt Set Construction: For each training example, the system generates true/wrong prompts as needed, embeds them via the white-box module ( $k_j = \psi(Z_j)$ ), and pairs with learnable prompt vectors.
Model Initialization: The black-box model is frozen; the only trainable components are the injected prompt vectors and any adapter parameters.
Prompt Selection and Injection: For each input, the black-box produces its internal representation ( $c_L$ ). The white-box module computes the match token and corresponding prompt distribution $P(k_j|\hat m)$ ; the resulting prompt vector $\hat v$ is injected.
Forward Pass and Loss Computation: The system computes prediction logits and the knowledge-guided loss. The optimizer updates only prompt vectors and associated parameters.
Iterative Training: Learning proceeds over epochs with jointly optimized losses; hard sample curricula can be incorporated, as in the two-stage easy-to-hard adaptation (Wang et al., 28 Jan 2026).
Evaluation and Correction Analysis: Performance is assessed both on standard accuracy and on the correction rate of originally misclassified samples, providing diagnostic insight into the corrective knowledge’s impact.

A condensed pseudocode is provided in (Wang et al., 28 Jan 2026), highlighting the roles of prompt matching, selection, concatenation, and update.

5. Empirical Results and Performance Characteristics

IOTA and related Black-White Box frameworks have been empirically validated across a broad spectrum of benchmark tasks, commonly outperforming state-of-the-art parameter-efficient or prompt optimization baselines. In visual domains (Wang et al., 28 Jan 2026), IOTA achieves:

16-shot protocol: 81.71% average accuracy (ViT-B/16), +4.70% over best PET baseline.
Correction rate: 65.8% of wrong-predicted samples are corrected, versus <60% for all PET baselines.
Easy-to-hard adaptation: Hard samples yield large further improvements, with full stage protocols outperforming all single-stage and black-box-only baselines.

Ablation studies evidence that removal of white-box corrective knowledge substantially reduces downstream accuracy. Sensitivity analysis indicates prompt length and full-layer injection as essential factors (Wang et al., 28 Jan 2026). In instruction optimization, black-white paradigms dramatically reduce API cost and search iterations compared to black-box bandit/evolution schemes by leveraging interpretable white-box surrogates for semantic similarity and representation fusion (Ren et al., 14 Jun 2025, Chu et al., 29 Oct 2025).

Several frameworks generalize or extend the Black-White Box paradigm:

Matryoshka: Uses a white-box controller LLM as a policy to decompose and steer black-box LLMs via intermediate sub-prompts, formalizing the setting as an MDP with RL-based preference optimization (Li et al., 2024).
Instruction IOTA: Fuses black-box-generated instruction initializations with white-box semantic refinement, employing cosine similarity constraints over hidden states for iterative optimization (Ren et al., 14 Jun 2025).
PRESTO: Exploits white-box-to-black-box mappings and many-to-one “preimage” structures to maximize data efficiency via score sharing and regularization (Chu et al., 29 Oct 2025).
Black-Box Vision Prompting: Integrates spatial-frequency domain prompt engineering and output clustering in vision black-box transfer, combining gradient-free input adaptation with probabilistic white-box output refinement (Cho et al., 2024).

A common thread is the explicit modularization of black-box environments/generators and white-box policies/controllers, with iterative policy improvement looped over black-box feedback.

7. Limitations and Future Perspectives

Key limitations identified include:

Data Regimes: In ultra-low-data scenarios (e.g., 4-shot), corrective knowledge is inherently sparse, reducing the efficacy margin over data-only tuning (Wang et al., 28 Jan 2026).
White-Box Expressivity: Current white-box modules are generally limited to single-step prompt selection or rule application; extensions to structured multi-hop reasoning or knowledge graph integration are underexplored.
Computational Overheads: While API call counts are dramatically reduced, overhead may arise from preimage construction, embedding computation, or white-box surrogate training in settings with massive prompt spaces (Chu et al., 29 Oct 2025).
Generalization and Dynamic Tasks: Extending frameworks to dynamic, multi-turn dialog, or multi-modal fusion presents new challenges for prompt synthesis and knowledge incorporation.

Promising future research avenues include leveraging ontological sources (e.g., WordNet, ConceptNet) for richer white-box priors and distilling complex, multi-step reasoning into structured prompt-action policies. These directions are expected to amplify generalization in transfer, reasoning, and complex real-world interaction settings (Wang et al., 28 Jan 2026, Chu et al., 29 Oct 2025).

References:

(Wang et al., 28 Jan 2026, Li et al., 2024, Ren et al., 14 Jun 2025, Chu et al., 29 Oct 2025, Cho et al., 2024)