À-la-carte Prompt Tuning (APT)

• À-la-carte Prompt Tuning is a modular method that tunes isolated soft prompts for individual data sources, achieving model accuracy within 2–5% of jointly-trained baselines.
• It employs structured attention masks and per-prompt memory tokens to ensure prompt compartmentalization while supporting ensemble-like aggregation at inference.
• APT facilitates scalable and privacy-preserving adaptation across vision, audio, and vision-language models, enabling efficient federated and continual learning.

À-la-carte Prompt Tuning (APT) refers to a class of techniques for efficient, modular, and composable adaptation of large pre-trained models to diverse, domain-specific, or user-specified data sources. APT encompasses several instantiations across vision, audio, and vision-LLMs, but shares key properties: prompts or prompt-like parameters are tuned in isolation on specific data sources or tasks; these prompts are subsequently composed at inference time to construct a model that leverages exactly the designated subset of training data—without revisiting the raw data, altering the backbone weights, or retraining from scratch. This enables a-la-carte learning, privacy-preserving model customization, federated adaptation, and efficient unlearning, while maintaining accuracy close to a monolithic jointly-trained model (Bowman et al., 2023, Liu et al., 2023, Wu et al., 2023).

1. Foundational Motivation and Problem Setting

APT is motivated by the need for scalable methods to construct customized models in settings with diverse data sources and stringent compartmentalization requirements. In the general form, there exists a data pool $\mathcal{D} = \{D_1, \ldots, D_n\}$, each $D_i \subseteq \mathcal{X} \times \mathcal{Y}$ sharing the same input and label domains. At inference, a user specifies a subset $S \subseteq \{1,\ldots,n\}$, and seeks predictions $f(x; S)$ that depend only on $\bigcup_{i \in S} D_i$, without retraining for every possible $S$. Traditional solutions—either one model per subset ($O(2^n)$ complexity) or ensembling individual models per data source—are infeasible at scale due to prohibitive storage and computational cost, as well as accuracy and data-leakage issues.

APT addresses this by training, for each $i$, a parameter-efficient soft prompt $P_i$ encoding only $D_i$. These prompts can be composed at inference to create a tailored model for any $D_i \subseteq \mathcal{X} \times \mathcal{Y}$0, achieving paragon-level accuracy (within $D_i \subseteq \mathcal{X} \times \mathcal{Y}$1 of a model trained on $D_i \subseteq \mathcal{X} \times \mathcal{Y}$2), with storage and inference cost proportional to $D_i \subseteq \mathcal{X} \times \mathcal{Y}$3 (Bowman et al., 2023). This scheme generalizes to continual, federated, and unlearning settings.

2. Model Architecture, Prompt Design, and Structured Composition

In vision settings, as exemplified by APT with Vision Transformers (ViTs), the backbone model $D_i \subseteq \mathcal{X} \times \mathcal{Y}$4 is frozen. Each prompt $D_i \subseteq \mathcal{X} \times \mathcal{Y}$5 (with $D_i \subseteq \mathcal{X} \times \mathcal{Y}$6 prompt tokens of hidden dimension $D_i \subseteq \mathcal{X} \times \mathcal{Y}$7) is appended to the input stream, followed by a forward pass through $D_i \subseteq \mathcal{X} \times \mathcal{Y}$8 (Bowman et al., 2023). At inference, user-selected prompts $D_i \subseteq \mathcal{X} \times \mathcal{Y}$9 are concatenated to form $S \subseteq \{1,\ldots,n\}$0, which is processed in a single forward pass.

To prevent destructive self-attention interference between prompts, APT employs structured attention masks and per-prompt memory tokens $S \subseteq \{1,\ldots,n\}$1 (with $S \subseteq \{1,\ldots,n\}$2). The resulting attention rules enforce that (i) image tokens attend only to their standard previous layer; (ii) each prompt attends only to its associated image tokens and its own memory tokens; (iii) prompts are strictly isolated from each other; (iv) memory tokens do not attend to any other tokens. This compartmentalization ensures independence and scalability: the total token budget per prompt is minimal, and the overhead per composed model is negligible relative to the frozen backbone's parameters.

Training leverages only the prompt and a prompt-specific classification head, with the cross-entropy loss minimized over $S \subseteq \{1,\ldots,n\}$3. The backbone remains unchanged, and prompt isolation allows for distributed, asynchronous, or federated prompt updates (Bowman et al., 2023).

3. Compositional Inference and Output Aggregation

Given a test instance $S \subseteq \{1,\ldots,n\}$4 and user-selected subset $S \subseteq \{1,\ldots,n\}$5, APT constructs $S \subseteq \{1,\ldots,n\}$6 via concatenation. Inference proceeds by:

1. Processing $S \subseteq \{1,\ldots,n\}$7 with $S \subseteq \{1,\ldots,n\}$8 using the structured-attention backbone.
2. Generating separate logits $S \subseteq \{1,\ldots,n\}$9 for each constituent prompt.
3. Averaging these logits: $f(x; S)$0.

Empirically, this mechanism yields accuracy degradation $f(x; S)$1 bounded by $f(x; S)$2–$f(x; S)$3 for up to twenty data shards (Bowman et al., 2023). Output averaging across prompts is justified by the strictly-enforced independence within attention, ensuring that ensemble-like behavior closely mimics that of a model trained on the data union.

4. Empirical Evaluation

APT has been benchmarked on seven fine-grained vision datasets and continual learning tasks such as Split CIFAR-100 and CORe50 (Bowman et al., 2023). Key findings include:

• On splits into two equal shards, naive prompt concatenation or weight-averaging suffer severe accuracy loss on out-of-domain tasks (e.g., Aircrafts: $f(x; S)$4 for naive vs $f(x; S)$5 for APT, with paragon at $f(x; S)$6).
• For increasing shard counts (2 to 20), APT's mean in-domain accuracy falls marginally ($f(x; S)$7), always staying within $f(x; S)$8–$f(x; S)$9 of the paragon.
• Classifier-head-only adaptation is inferior (e.g., $\bigcup_{i \in S} D_i$0 for head-only vs $\bigcup_{i \in S} D_i$1 for APT out-of-domain), confirming that soft prompts encode richer domain structure.
• Continual learning benchmarks: On Split CIFAR-100, APT achieves $\bigcup_{i \in S} D_i$2 (vs next-best $\bigcup_{i \in S} D_i$3), improving to $\bigcup_{i \in S} D_i$4 with $\bigcup_{i \in S} D_i$5-means-based prompt weighting; on CORe50, APT / APT-W achieves $\bigcup_{i \in S} D_i$6 vs $\bigcup_{i \in S} D_i$7 for baseline.

Ablation studies indicate that removing structured attention results in high variance and catastrophic failures; using non-ImageNet pre-training accelerates accuracy decay with sharding. This highlights the critical role of backbone pretraining and prompt compartmentalization.

5. Extensions to Other Domains and Modalities

APT's general framework applies beyond vision:

• Audio Prompt Tuning for Universal Sound Separation (Liu et al., 2023): APT-USS adapts a frozen universal sound separator by tuning class-specific prompt vectors, initialized from averaged sound-event embeddings and injected as conditioning at the waveform or feature level. With only $\bigcup_{i \in S} D_i$8K–$\bigcup_{i \in S} D_i$9K tuned parameters ($S$0 of model size), APT improves signal-to-distortion ratio (SDR) by up to $S$1 dB on ESC-50 (50 classes), outperforming full-data baselines even in 5-shot regimes.
• Approximated Prompt Tuning for Vision-LLMs (Wu et al., 2023): For vision-language pre-trained models (e.g., ViLT, METER), APT replaces global prompt-input attention with independent, low-rank, additive diffusion steps, eliminating the quadratic cost of prompt-attention. On ViLT, APT reduces prompt-related computation by $S$2 with accuracy within $S$3 of full fine-tuning. On METER, APT closes $S$4 of the accuracy gap between deep prompt and full tuning, with $S$5 less extra computation.

6. Applications, Limitations, and Future Work

APT supports modular model construction based on user access rights and preferences: models can be tailored by adding or removing corresponding prompts, enabling privacy-preserving machine unlearning, federated or decentralized training, continual learning, and on-demand model customization (Bowman et al., 2023). Each prompt contains only information from its original data source; no training data or backbone weights need to be revisited for future compositions.

The principal limitation is a bounded loss of expressivity: prompt synergy across sources flows only through narrow cross-prompt memory tokens, resulting in increased $S$6 on highly out-of-domain or heterogeneous tasks. Major open questions include dynamic or learned prompt selection/composition, adaptive or per-domain prompt sizing, design of prompt-injection procedures for other modalities, and combining prompts with other parameter-efficient transfer learning approaches. Extensions may involve prompt selection policies beyond $S$7-means weighting and exploiting multi-modal or reinforcement-learning-based controller architectures for dynamic assembly (Bowman et al., 2023, Liu et al., 2023, Wu et al., 2023).

7. Relationship to Related Approaches

APT distinguishes itself from standard ensembling, fine-tuning, and prompt-tuning by its strong compartmentalization guarantees and composability at inference. In both vision and vision-language domains, APT outperforms LoRA and various adapter variants in parameter-efficiency vs. performance trade-off and matches or exceeds baseline accuracy in continual or multi-task learning without incurring corresponding privacy or compute penalties (Bowman et al., 2023, Wu et al., 2023). Prompt-only adaptation yields superior cross-domain performance compared to classifier-head adaptation, confirming the importance of deep feature-level alignment.

In summary, À-la-carte Prompt Tuning is an efficient, scalable, and privacy-preserving paradigm for constructing modular prediction systems from arbitrary combinations of training data sources, achieving accuracy near that of jointly-trained models with orders of magnitude lower computational and storage cost.