Customized Black-box Prompt Tuning

• The paper presents a two-stage architecture where server-side prompt generation is combined with user-side gradient-free optimization to achieve task-specific tuning.
• CBP-Tuning is a prompt-based adaptation framework that customizes large language models in black-box settings while preserving both provider and user privacy.
• It integrates methodologies from low-dimensional soft-prompt search, modular prompt selection, and privacy-aware local customization into a coherent strategy.

Searching arXiv for the cited CBP-Tuning paper and closely related black-box prompt-tuning work to ground the article in current literature. Customized Black-box Prompt Tuning (CBP-Tuning) denotes a prompt-based adaptation regime for LLMs in which customization is performed under black-box access constraints, typically through API-mediated inference rather than parameter access, gradient access, or full-model fine-tuning. In its named formulation, CBP-Tuning is presented as a framework for efficient local customization with bidirectional privacy, using a two-stage design in which a server-side prompt generator captures domain-specific and task-agnostic capability, while user-side gradient-free optimization learns only a single customized vector per task (Zhao et al., 15 Sep 2025). More broadly, the surrounding literature places CBP-Tuning within a larger family of black-box prompt adaptation methods that includes low-dimensional soft-prompt search, modular prompt selection, discrete prompt optimization, and federated prompt aggregation (Sun et al., 2022).

1. Definition, scope, and historical emergence

The named framework appears in "CBP-Tuning: Efficient Local Customization for Black-box LLMs" (Zhao et al., 15 Sep 2025). Its motivating problem is the combination of three constraints: personalization, provider scalability, and privacy. The framework is designed for the regime in which LLMs are offered as cloud services, providers do not want to reveal model weights, and users do not want to upload private task data. The paper characterizes the goal as local task adaptation with low compute and storage cost, while preserving provider-side privacy and user-side privacy.

Earlier black-box prompt-tuning work established most of the technical ingredients from which CBP-Tuning is assembled. "Black-Box Tuning for Language-Model-as-a-Service" (Sun et al., 2022) formulated prompt learning under LMaaS as derivative-free optimization of a continuous prompt in a low-dimensional random subspace. "Reliable Gradient-free and Likelihood-free Prompt Tuning" (Shen et al., 2023) extended black-box prompt tuning to prompt distributions and uncertainty-aware inference, including a stricter label-only setting via ABC-SMC. "Black-box Prompt Tuning with Subspace Learning" (Zheng et al., 2023) reframed black-box tuning as learning a task-family- and model-specific prompt subspace. "Hyperband-based Bayesian Optimization for Black-box Prompt Selection" (Schneider et al., 2024) treated black-box prompt customization as finite optimization over modular prompt components, specifically instructions and few-shot exemplar tuples. "FedDTPT: Federated Discrete and Transferable Prompt Tuning for Black-Box LLMs" (Wu et al., 2024) added a federated, client-specific, and transferable discrete prompt setting.

This suggests that CBP-Tuning is best understood not as a single isolated algorithm, but as an overview of several black-box prompt-tuning lines: low-dimensional soft-prompt optimization, transferable subspace design, modular prompt composition, and privacy-aware local adaptation.

Paper	Core mechanism	Relation to CBP-Tuning
(Sun et al., 2022)	Random-subspace black-box soft prompt tuning	Foundational LMaaS formulation
(Zheng et al., 2023)	Meta-learned reusable prompt subspaces	Task/model-specific customization precursor
(Schneider et al., 2024)	BO + Hyperband over modular prompts	Static natural-language customization under API access
(Wu et al., 2024)	Federated discrete prompt optimization	Multi-client transferable extension

A persistent ambiguity across this literature is the meaning of “customized.” In some papers it means task-specific adaptation; in others it means task-family-specific, client-specific, or prompt-component-specific adaptation. The named CBP-Tuning framework uses the term in the sense of individual task specialization on the user side, while retaining a reusable server-side domain prior (Zhao et al., 15 Sep 2025).

2. Two-stage architecture of CBP-Tuning

CBP-Tuning is explicitly divided into Server-side Domain Training (SDT) and User-side Local Customization (ULC) (Zhao et al., 15 Sep 2025). In SDT, the provider trains a prompt generator $\mathcal{G}$ on a domain dataset $\{\mathcal{X}_D; \mathcal{Y}_D\}$ while the LLM is frozen. In ULC, the user optimizes only a low-dimensional vector locally through gradient-free search.

The prompt generator fuses instance information and task information. Its input is defined as

$$I(\mathbf{X}, \mathbf{z}) = Pooler(Emb(\mathbf{X})) + \mathbf{Az}.$$

Here, $Emb(\mathbf{X})$ is the embedding sequence of the input text, $Pooler(\cdot)$ is mean pooling over token hidden states, $\mathbf{z} \in \mathbb{R}^r$ is the low-dimensional task vector with $r \ll d$, and $\mathbf{A} \in \mathbb{R}^{d \times r}$ projects the task vector into the model hidden space.

The prompt generator itself is a bottleneck MLP with down-projection $\mathbf{L_D} \in \mathbb{R}^{m \times d}$, GeLU activation, and up-projection $\mathbf{L_U} \in \mathbb{R}^{(t \cdot d) \times m}$. The generated soft prompt is

$\{\mathcal{X}_D; \mathcal{Y}_D\}$0

Since the output dimension is $\{\mathcal{X}_D; \mathcal{Y}_D\}$1, $\{\mathcal{X}_D; \mathcal{Y}_D\}$2 can be reshaped into a prompt of length $\{\mathcal{X}_D; \mathcal{Y}_D\}$3, each token embedding lying in $\{\mathcal{X}_D; \mathcal{Y}_D\}$4. The final model input is then

$\{\mathcal{X}_D; \mathcal{Y}_D\}$5

Two structural properties follow directly from this formulation. First, customization is not performed by modifying the LLM itself, but by learning a task vector $\{\mathcal{X}_D; \mathcal{Y}_D\}$6 that modulates a generator already specialized to a domain. Second, the prompt is instance-conditioned as well as task-conditioned, because the generator depends jointly on $\{\mathcal{X}_D; \mathcal{Y}_D\}$7 and $\{\mathcal{X}_D; \mathcal{Y}_D\}$8 (Zhao et al., 15 Sep 2025). This distinguishes CBP-Tuning from static prompt tuning in which the same soft prompt is prepended to every instance in a task.

3. Local customization, efficient storage, and bidirectional privacy

The paper’s central systems claim is that CBP-Tuning enables local customization without exposing either model weights or user data (Zhao et al., 15 Sep 2025). The user does not receive the full LLM and does not upload private downstream examples to the server. Instead, the user receives the trained prompt generator and performs user-side optimization of a low-dimensional vector using CMA-ES. The resulting task artifact is only a single customized vector per task.

This design is positioned against several prior adaptation strategies. Full fine-tuning is too expensive and complicates weight sharing. LoRA and adapters remain parameter-efficient, but still require maintaining task modules. Prefix-style methods inject trainable vectors into all layers or into key/value states. By contrast, CBP-Tuning operates at the input stage and needs only the shared generator plus one low-dimensional vector per downstream task (Zhao et al., 15 Sep 2025).

The privacy claim is formulated as bidirectional privacy. Provider-side privacy is preserved because the user never obtains the full model weights. User-side privacy is preserved because the provider never receives the user’s private customization data. This systems property is directly aligned with the motivation of earlier LMaaS work, where adaptation must proceed without gradients and without exposing internal parameters (Sun et al., 2022).

A notable implication is that the user-side optimization problem is substantially smaller than conventional black-box prompt search from scratch. Earlier black-box soft-prompt methods optimized directly in a projected prompt space, for example $\{\mathcal{X}_D; \mathcal{Y}_D\}$9 in BBT (Sun et al., 2022). CBP-Tuning retains the low-dimensional user variable, but inserts a learned prompt generator between the task vector and the final prompt. This suggests that the effective search space is no longer an undifferentiated random subspace, but a domain-structured manifold learned on the server.

4. Relation to precursor black-box prompt-tuning methods

The immediate technical ancestor of CBP-Tuning is low-dimensional black-box prompt tuning. BBT parameterizes a soft prompt as $$I(\mathbf{X}, \mathbf{z}) = Pooler(Emb(\mathbf{X})) + \mathbf{Az}.$$0 and solves

$$I(\mathbf{X}, \mathbf{z}) = Pooler(Emb(\mathbf{X})) + \mathbf{Az}.$$1

using derivative-free optimization under LMaaS constraints (Sun et al., 2022). This established that prompt adaptation can be reduced to compact black-box search. CBP-Tuning inherits the compact task vector idea, but replaces the fixed projection with a learned prompt generator (Zhao et al., 15 Sep 2025).

A second precursor is the subspace-learning view. BSL assumes that nearly optimal prompts for similar tasks reside in a common subspace and meta-learns a projection matrix $$I(\mathbf{X}, \mathbf{z}) = Pooler(Emb(\mathbf{X})) + \mathbf{Az}.$$2 and offset $$I(\mathbf{X}, \mathbf{z}) = Pooler(Emb(\mathbf{X})) + \mathbf{Az}.$$3, after which only low-dimensional coordinates $$I(\mathbf{X}, \mathbf{z}) = Pooler(Emb(\mathbf{X})) + \mathbf{Az}.$$4 are optimized on the target task (Zheng et al., 2023). In conceptual terms, CBP-Tuning and BSL share the same thesis: direct black-box prompt search is too unconstrained, and a learned prior over the adaptation space is needed. The difference is that BSL learns reusable affine subspaces from source tasks, whereas CBP-Tuning learns a domain-specific prompt generator and then tunes a task vector locally (Zhao et al., 15 Sep 2025).

A third precursor is uncertainty-aware black-box prompt tuning. "Reliable Gradient-free and Likelihood-free Prompt Tuning" (Shen et al., 2023) learns a distribution over prompts rather than a single prompt, using CMA-ES in both likelihood-available and likelihood-free settings. That paper is highly relevant because it treats prompt tuning as black-box inference under API-style restrictions, but it still assumes soft prompt injection through the embedding interface. CBP-Tuning remains in the soft-prompt family, although its main contribution is the decomposition of domain learning and task specialization rather than posterior prompt inference.

A fourth precursor is modular static prompt customization. HbBoPs formulates prompt customization for API-only LLMs as finite optimization over

$$I(\mathbf{X}, \mathbf{z}) = Pooler(Emb(\mathbf{X})) + \mathbf{Az}.$$5

where $$I(\mathbf{X}, \mathbf{z}) = Pooler(Emb(\mathbf{X})) + \mathbf{Az}.$$6 is a set of instructions and $$I(\mathbf{X}, \mathbf{z}) = Pooler(Emb(\mathbf{X})) + \mathbf{Az}.$$7 is a set of few-shot exemplar tuples (Schneider et al., 2024). It uses a structural-aware deep-kernel Gaussian process together with Hyperband to obtain both sample efficiency and query efficiency. Relative to CBP-Tuning, HbBoPs customizes discrete natural-language prompt structure rather than a generator-controlled soft prompt. The contrast is important: CBP-Tuning addresses local customization through soft-prompt generation and low-dimensional latent search, whereas HbBoPs addresses black-box customization through structured selection among modular prompt components.

5. Adjacent paradigms, empirical themes, and conceptual boundaries

The broader black-box adaptation literature contains several methods that are adjacent to CBP-Tuning without being identical to it. "PromptBoosting: Black-Box Text Classification with Ten Forward Passes" (Hou et al., 2022) does not optimize prompts directly; instead it combines a small prompt pool, verbalizer learning, and AdaBoost-based ensembling. "Enhancing Black-Box Few-Shot Text Classification with Prompt-Based Data Augmentation" (Luo et al., 2023) treats the black-box model as a feature extractor and trains a downstream classifier, using a smaller prompt-tuned teacher for pseudo-labeled augmentation. These methods are black-box adaptation methods, but not prompt-customization methods in the same sense as CBP-Tuning.

Other neighboring methods operate directly on discrete prompts. "A Bayesian approach for prompt optimization in pre-trained LLMs" (Sabbatella et al., 2023) formulates hard prompt tuning as combinatorial Bayesian optimization over token sequences. "Rethinking Prompt Optimization: Reinforcement, Diversification, and Migration in Blackbox LLMs" (Davari et al., 14 Jul 2025) studies natural-language prompt rewriting through positive and negative reinforcement, feedback diversification, and prompt migration across models. These works differ from CBP-Tuning in representation: their optimized object is a human-readable prompt string, whereas CBP-Tuning optimizes a low-dimensional vector whose image through the generator is a soft prompt (Zhao et al., 15 Sep 2025).

Empirically, the field exhibits two recurring themes. The first is that low-budget black-box optimization benefits strongly from structure. HbBoPs reports an average normalized test error of 0.150 at full budget, outperforming TRIPLE-GSE (0.158), TRIPLE-SH (0.159), HDBO (0.185), and Random Search (0.214) across ten benchmarks and three LLMs (Schneider et al., 2024). The second is that carefully designed black-box prompt methods can rival or surpass white-box baselines in few-shot regimes. BBT reports an average score of 83.90, above Prompt Tuning (63.46), P-Tuning v2 (70.47), and Model Tuning (78.88) in its evaluated setting (Sun et al., 2022). These results do not evaluate CBP-Tuning directly, but they establish that black-box prompt adaptation is not intrinsically marginal or merely approximate.

A common misconception is that all black-box prompt-tuning methods are equally deployable on commercial text-only APIs. The literature shows otherwise. Several prominent methods, including BBT (Sun et al., 2022), the uncertainty-aware soft-prompt work (Shen et al., 2023), BBT-RGB (Sun et al., 2023), and CBBT for vision-LLMs (Guo et al., 2023), assume some form of embedding-level prompt injection, probability access, or feature access. CBP-Tuning remains in this soft-prompt lineage, even though its local-customization design is motivated by realistic privacy and deployment concerns (Zhao et al., 15 Sep 2025).

6. Limitations, privacy risks, and open directions

CBP-Tuning is explicitly motivated by privacy, but privacy is not automatic in prompt-based customization. "Does Prompt-Tuning LLM Ensure Privacy?" (Xie et al., 2023) shows that a frozen pretrained LM paired with a user-specific prompt generator can leak private information through black-box querying when prompts are driven by user-specific signals. In that work, the prompt is produced from an N-gram frequency vector derived from a user’s email corpus, and the resulting system is vulnerable to memorization-style leakage under first-person and third-person attacks. This is directly relevant to CBP-Tuning because its architecture also separates a frozen backbone from a prompt-generating customization layer. A plausible implication is that bidirectional privacy as a systems property does not by itself eliminate prompt-level leakage risk.

A second limitation concerns access assumptions. Much of black-box prompt tuning, including CBP-Tuning, still operates in a regime that is black-box with respect to gradients and weights but not necessarily with respect to interface richness. Earlier works often require logits, masked-token distributions, hidden-state access, or embedding-layer prompt insertion (Sun et al., 2022, Shen et al., 2023, Sun et al., 2023, Guo et al., 2023). CBP-Tuning reduces user-side burden and weight exposure, but it remains a soft-prompt method rather than a purely discrete text-only customization method (Zhao et al., 15 Sep 2025).

A third limitation is representational scope. The named CBP-Tuning framework is task-specific and domain-conditioned, but the available description does not position it as a method for open-ended prompt generation, per-user conversational preference alignment, or federated multi-client prompt sharing (Zhao et al., 15 Sep 2025). Those directions are instead explored by neighboring lines such as FedDTPT for federated discrete prompts (Wu et al., 2024) and BReAD/CPO for migration across model versions and providers (Davari et al., 14 Jul 2025).

The main open direction suggested by this literature is the unification of three properties that currently appear separately: structured local customization, strong query efficiency, and privacy robustness. CBP-Tuning contributes the local-customization and bidirectional-privacy architecture (Zhao et al., 15 Sep 2025). HbBoPs contributes sample-efficient and query-efficient modular prompt selection (Schneider et al., 2024). Privacy work on prompt-conditioned systems contributes a warning that customization itself can become an attack surface (Xie et al., 2023). This suggests that the next stage of CBP-Tuning research is likely to combine domain-structured generators, stronger text-only black-box compatibility, and explicit privacy defenses rather than assuming that prompt-based customization is secure by construction.