Conditional & Compositional Prompting

• Conditional and Compositional Prompting are modular approaches that transform task conditions into prompt embeddings, enabling systematic re-use in neural models.
• They employ techniques like prompt encoders and independent module training to steer pre-trained transformers towards task-specific outputs.
• These methods improve efficiency, privacy, and continual learning in diverse applications including vision-language, text, and multimodal tasks.

Conditional and compositional prompting are two foundational approaches that enable the modular adaptation and systematic re-use of trainable prompts in large neural models, particularly in the context of vision-language, language, and multimodal transformers. These approaches formalize the interaction between prompt modules, external conditions (such as task metadata), and sub-task composition, supporting generalization to novel combinations and facilitating privacy, continual learning, and efficient adaptation across domains.

1. Formal Definitions and Modularization

Conditional prompting refers to the transformation of task instructions or input metadata (the condition) into continuous prompt embeddings that steer a frozen backbone model—typically a pre-trained transformer—toward condition-specific outputs. Concretely, given a discrete condition $c$ (such as a task label, domain indicator, or user-specified option), a prompt encoder yields $E(c)\in\mathbb{R}^d$. This embedding conditions further model components, such as production rules or prompt tokens.

Compositional prompting extends this paradigm by enabling the composition of multiple independently learned prompt modules, so that the model can process novel combinations of conditions at inference that were not observed jointly during training. Compositional prompting strategies include the additive or parallel assembly of learned prompt embeddings, the staged execution of skills (sub-procedures), and fusion via inter/intra-modal adapters. This modularity allows for systematic generalization in, for example, attribute-object compositional recognition, adaptive bias detection, and compositional semantic parsing (Pilault et al., 2023, Nayak et al., 2022, Bowman et al., 2023).

2. Key Architectural Paradigms

Several models instantiate conditional and compositional prompting with varying mechanisms and scope:

• Neural Production Systems (PRopS): Uses a bank of $N$ learnable rule embeddings ($R\in\mathbb{R}^{N\times d}$) and a condition encoder $E(c)$. A score matrix $M=E(c)R^T$ is computed, and $k$ relevant rules are sparsely selected via a Gumbel-Top-$k$ operator. Each selected rule $j$ has an associated neural module $f_j$, and the final prompt embedding is $I(c)=\sum_{j\in S}f_j(E(c))$ (Pilault et al., 2023). This enables few-shot adaptation, compositional transfer, and parameter sharing.
• À-la-carte Prompt Tuning (APT): Defines prompts for distinct data sources, training each in isolation. At inference, any subset $I$ is selected, with prompt tokens $P^{(I)}$ and per-layer memories concatenated, and a structured attention mask enforcing conditional independence between prompt blocks. This supports a-la-carte learning, where adding or removing model capabilities equates to adding or deleting prompt modules—enabling privacy, selective unlearning, and continual compositional growth (Bowman et al., 2023).
• Compositional Soft Prompting (CSP): For compositional zero-shot learning in vision-language settings, CSP learns token embeddings for attributes and objects, which are concatenated with a fixed template. At test time, unseen combinations (e.g., new attribute-object pairs) are composed from the learned primitives with no further tuning, providing robust systematic compositionality (Nayak et al., 2022).

3. Training and Inference Workflows

Conditional and compositional prompting architectures share a modular workflow:

Stage	Process	Example Realization
Prompt Module Learning	Train prompt(s) for each discrete data shard, condition, or sub-task, with frozen backbone	PRopS rules, APT soft prompts
Isolation and Independence	Each prompt incorporates only its source's information and is trained/evaluated independently	Per-source or per-condition isolation
Conditional Assembly	At inference, a subset of prompt modules is selected/assembled according to input condition(s)	Rule selection; user/shard selection
Composition and Fusion	Prompt modules are composed (sum, concat, staged) and injected as a unified adaptation signal	PRopS sum; CSP concat; APT prompt stack
Structured Attention/Mask	Transformer attention blocks are masked to enforce no cross-talk between unrelated prompts	APT structured attention
Prediction/Aggregation	Output(s) from each module are combined (uniformly or weighted) for final prediction	APT ensemble; PRopS prompt injection

This design yields parameter-efficient adaptation ($<$1–3% of backbone parameters in PRopS and APT), supports continual/lifelong learning, and provides dynamic control at inference (Pilault et al., 2023, Bowman et al., 2023).

4. Theoretical and Empirical Properties

• Compositional Generalization: PRopS and CSP demonstrate that zero-shot generalization to new compositions (e.g., unseen attribute-object pairs or novel control tag combinations) is attainable via prompt module reuse. Theoretical claims (see (Pilault et al., 2023) Appendix) state that under sufficient embedding separation and prompt diversity, the generalization error on any $k$-way novel composition is bounded.
• Privacy and Forgetting: Since each prompt is trained on a subset of data and isolated, APT and related modular frameworks can forget sources simply by deleting corresponding prompts, with no impact on others—a direct security advantage (Bowman et al., 2023).
• Efficiency: Prompt-only adaptation avoids full model fine-tuning. APT’s inference cost grows only linearly in the number of selected prompts (dominated by the backbone’s $O(N^2)$ cost), and per-prompt parameter overhead is $<0.06\%$ of the backbone (Bowman et al., 2023).
• Empirical Performance: APT achieves within 2–5% of the accuracy of models tuned on the full union of sources, even with 10–20 shards each seeing only a small data fraction. PRopS matches or exceeds prior models on summarization and compositional benchmarks, tuning $<$1% of parameters and outperforming non-compositional baselines by 10–15pts (Bowman et al., 2023, Pilault et al., 2023).

5. Application Scenarios

• Vision-Language Systems: CSP and related architectures support robust recognition of new attribute-object combinations in zero-shot and continual settings. Modular prompt banks facilitate dynamic scene understanding, high-order composition (e.g., attribute-attribute-object), and efficient dataset extension (Nayak et al., 2022).
• Text and Multimodal Tasks: Compositional prompting applies to adaptive social bias detection (via per-instance prompt composition), compositional semantic parsing (via least-to-most in-context prompting), and systematic reasoning (via skills-in-context or staged chain-of-thought prompts) (Spliethöver et al., 10 Feb 2025, Chen et al., 2023, Drozdov et al., 2022).
• Continual and Privacy-Preserving Learning: The modularity, conditional composition, and explicit provenance tracking of prompt modules in APT enable dynamic model construction, legally compliant data deletion, and efficient continual learning. Statistical results show that APT achieves state-of-the-art accuracy for continual learning on Split CIFAR-100 and CORe50, with no buffer memory required (Bowman et al., 2023).

6. Limitations and Open Challenges

• Functional independence between prompt modules, as enforced via structured attention and training isolation, may limit cross-task synergy, especially for OOD or highly interdependent domains. Smaller shards may see degraded accuracy on out-of-domain classes (e.g., Aircrafts, Cars) (Bowman et al., 2023).
• Compositional prompting generally presumes a discrete, pre-labeled set of conditions or metadata. Automatic discovery and dynamic expansion of the condition vocabulary remain open research directions (Pilault et al., 2023).
• The modular assembly approach (as in APT) precludes information-sharing between prompt memories, which may constrain feature fusion or multi-source integration in some high-complexity settings (Bowman et al., 2023).

7. Future Directions

Ongoing work aims to advance conditional and compositional prompting through:

• Learnable gating or dynamic attention-over-prompts, replacing static uniform averaging, to enable more adaptive ensemble prediction;
• Prompt-to-prompt cross-attention with controlled sparsity, supporting richer information exchange between modules while limiting interference;
• Extension to hierarchical, nested, or graph-structured prompt grammars, to systematically encode complex domain relations and higher-order dependencies;
• Unified frameworks that combine semi-parametric retrieval and modular prompt libraries, including parametric selection of prompt sets per instance or user (Pilault et al., 2023, Bowman et al., 2023).

These directions seek to further improve systematic generalization, adaptability, privacy, and computational efficiency in large language and multimodal models.