Descriptor-Based Prompting

• Descriptor-based prompting is a paradigm that decomposes prompts into composable, labeled descriptors to enable precise model customization.
• The APT framework trains isolated prompts from distinct data sources and uses structured attention to ensure modular, interference-free inference.
• Empirical results demonstrate that APT achieves near joint-training accuracy while offering scalable, privacy-aware, and cost-efficient continual learning.

Descriptor-based prompting is a paradigm in which prompts provided to LLMs or other foundation models are structured, modular, and explicitly composed from labeled “descriptors.” These descriptors—such as data source tags, user access rights, knowledge domains, or task-specific features—are implemented not as free-form text but as composable, isolated prompt parameters. The À-la-carte Prompt Tuning (APT) framework exemplifies this concept, offering a principled approach to model customization, modular continual learning, and privacy-preserving machine learning via prompt composition.

1. Modular Prompt Learning: Foundations of Descriptor-Based Prompting

APT introduces a transformer-based scheme where individual prompts are trained against distinct, potentially heterogeneous data sources, each acting as a standalone “descriptor” of the subset of information it encodes. This enables the following core properties:

• Prompt isolation: Each prompt is trained exclusively on data from a single source or class and only encodes information about that subset.
• Composable inference: At inference time, users can select any combination of trained prompts (“a-la-carte learning”) to assemble a model whose output reflects only the union of the chosen descriptors. This supports dynamic, on-demand creation of bespoke models tailored to user roles, access rights, or application scenarios.
• Data compartmentalization: Since prompts are not mixed or co-trained, adding or removing a descriptor (i.e., a prompt) equates to including or forgetting that data source, all without re-training or affecting the remaining prompts.

In the broader context, descriptor-based prompting as formalized in APT offers a scalable alternative to classic monolithic prompt engineering, supporting fine-grained model control and modular deployment.

2. Training and Inference Workflow in APT

Training of Individual Prompts

Given a pre-trained transformer backbone $f_\theta$ and a set of data sources $D = \{ D_1, D_2, ..., D_n \}$, APT trains for each source $D_i$:

• A prompt $p^{(i)}$ and a source-specific classifier head $\mathrm{head}_i$.
• The backbone weights $\theta$ are kept frozen.
• The loss for each prompt is:

$$L_{D_i}(p^{(i)}, \mathrm{head}_i) = \sum_{(x, y) \in D_i} \ell(f(x; p^{(i)}), y)$$

where $f(x; p^{(i)})$ is the model’s output on input $x$ with prompt $p^{(i)}$.

Prompts can be trained independently (even on different devices and schedules), which supports parallel, asynchronous, and decentralized learning.

Inference by Prompt Composition

At inference, a user specifies a subset $S \subset D$ representing the desired data descriptors. The model then:

• Concatenates the corresponding prompts $\{ p^{(i)} : D_i \in S \}$ into $p^{(S)}$.
• Inputs this to the backbone, using a specially designed structured attention mask (see Section 3 below) to prevent cross-prompt interference.
• Outputs are computed via each corresponding head, then aggregated (typically by averaging):

$$\hat{y}_S = \frac{1}{|S|} \sum_{i: D_i \in S} \hat{y}^{(i)}$$

This architecture allows bespoke models to be assembled on demand, with negligible inference-time cost compared to traditional ensembling.

3. Structured Attention and Modularity

A distinctive feature of APT is its structured attention mechanism, designed to maintain independence among composable prompts. The attention mask enforces:

• Data tokens do not attend to the prompts.
• Prompts do not attend to each other.
• Each per-prompt memory token interacts only with its own prompt and associated tokens.

This guarantees that each prompt’s impact is confined to its respective data source, preserving the modularity vital for security, privacy, and compositionality. The inference cost scales as $O(N^2 + (N + d_{mem})|S|)$, which is considerably less than naïve ensembling ($O(|S| N^2)$), where $N$ is the sequence length.

Table: Structured Attention Mask Example

Query\Key	Backbone	Prompt 1	Prompt 2	Memory 1	Memory 2
Backbone	✓	✗	✗	✗	✗
Prompt 1	✓	✓	✗	✓	✗
Prompt 2	✓	✗	✓	✗	✓

This design realizes the “descriptor compartment” principle, making it practical to add or delete prompt-knowledge slices without retraining or destructive interference.

4. Performance Analysis: Accuracy, Efficiency, and Continual Learning

Empirical results reveal that a-la-carte prompt composition achieves accuracy within 2–5% of models trained on the union of all sources (“paragon” models), even as the number of prompts (descriptors) grows to 20. This holds for both standard benchmarks and continual learning scenarios.

Table: APT Accuracy After Sharding and Recomposition

Dataset	No Sharding	10 Shards	20 Shards
MIT-67	86.2%	86.8%	86.3%
Cub-200	86.6%	86.5%	83.9%
Caltech-256	91.7%	89.7%	88.7%
Pets	93.3%	93.4%	93.3%
Aircrafts	71.0%	49.9%	45.4%
Flowers	99.1%	98.5%	97.6%
Stanford Cars	81.2%	52.1%	45.8%

Except for challenging domain-shift cases (e.g., Aircrafts, Stanford Cars), performance degrades slowly even as datasets are highly fragmented and prompts are composed, confirming APT’s robustness and modularity.

On continual learning tasks:

• In Split CIFAR-100 (class-incremental), APT and APT-W (prototype-weighted composition) achieve 83.6% and 85.2% accuracy, surpassing or matching state-of-the-art methods.
• For CORe50 (domain-incremental), they reach 90.9% and 91.1%, outperforming all baselines presented.

Efficiency:

• Training is linear in the number of data sources.
• Storage overhead is minimal (<0.06% of backbone parameters per prompt).
• Inference cost is nearly constant with respect to the number of active prompts.

This suggests that descriptor-based prompting via a-la-carte composition has strong theoretical and practical guarantees for both scalability and efficacy.

5. Privacy, Data Rights, and Practical Compartmentalization

A direct implication of APT’s design is support for privacy and granular information control in deployment:

• Bespoke models can be created to match the precise access rights and preferences of each user; e.g., regulatory constraints, business logic, or consent boundaries can all be represented at the prompt level.
• Forgetting or restricting certain data becomes as simple as removing a prompt, with no retraining or data exposure.
• Prompts can be distributed, stored, or regulated independently, and model retraining or sharing need not cross compartment boundaries—critical for federated or distributed AI use cases.

A plausible implication is that future model deployments for enterprise, healthcare, or regulated settings may strongly prefer descriptor-based approaches to traditional monolithic models for reasons of both legal compliance and operational efficiency.

6. Comparison to Classical Approaches and Broader Applicability

Unlike classical model ensembling (which incurs high inference or storage costs) or retraining-based continual learning (with catastrophic forgetting and slow adaptation), descriptor-based prompting with APT offers:

• Immediate composition of new capabilities or knowledge bases with no need for full model retraining.
• Parameter-efficient continual learning, especially in domains where data is fragmented or subject to frequent change.
• The ability to achieve high accuracy within 5% of joint-training “paragon” models on most real-world tasks.

Table: Continual Learning Benchmarks

Method	CIFAR-100	CORe50
APT	83.6	90.9
APT-W	85.2	91.1
L2P	83.8	78.3
S-liPrompts	--	89.1
LwF	60.7	75.4
EWC	47.0	74.8

For most benchmark datasets, APT outperforms naive ensembling, head-only methods, and many prior continual learning strategies.

7. Limitations and Open Directions

While APT’s a-la-carte learning supports efficient modularity and privacy, performance can degrade for datasets with severe domain or distribution shifts (e.g., fine-grained aircraft or car classes). In such cases, integrating more advanced similarity-weighted ensembling (APT-W) or regularization strategies can help, but some accuracy loss compared to full joint training may remain.

A plausible implication is that research into more expressive inter-prompt composition, prompt-knowledge transfer, or descriptor meta-learning may further enhance the flexibility and accuracy of descriptor-based prompting in challenging conditions.

---

In summary, descriptor-based prompting as realized by À-la-carte Prompt Tuning offers a compelling model engineering paradigm, enabling compositional, privacy-preserving, and highly efficient deployment of LLMs. By decoupling model knowledge and behavior into modular descriptors, APT achieves accuracy within 5% of full-data baselines, state-of-the-art continual learning results, and practical support for real-world customization of model capabilities. This framework is particularly pertinent for organizations and applications that require flexible, scalable, and compartmentalized AI solutions.