---
title: 'CoPS: Conditional Prompt Synthesis'
url: https://www.emergentmind.com/topics/conditional-prompt-synthesis-cops
type: topic
---

# CoPS: Conditional Prompt Synthesis

Conditional Prompt Synthesis (CoPS) refers to a family of parameter-efficient approaches for synthesizing adaptive, input- or context-conditioned prompts that steer large, typically frozen, deep models (such as language models, vision models, or multimodal transformers) toward instance- or task-specific behavior. CoPS exploits conditionality—via learnable functions, routing mechanisms, or attention over expert pools—to move beyond static prompt templates and achieve strong performance and generalization with minimal trainable parameters, particularly in transfer (zero- and few-shot), multi-modal, and compositional settings.

## 1. Definitions and Conceptual Foundations

Conditional Prompt Synthesis is formally characterized by the goal of learning a prompt-synthesizing function $p(x)$ (with trainable parameters) that maps instance-specific information or task metadata $x$ into a continuous prompt, which is then injected into a frozen, pre-trained backbone model $\theta$ to adapt its predictions. The core feature distinguishing CoPS from vanilla prompting is that the prompts are not fixed or purely learned per class/task, but instead dynamically derived from input or side information—potentially including auxiliary modalities, semantic priors, learned expert pools, or structured rule systems.

Variants of CoPS exist in multiple domains:

- In language modeling, CoPS modules can transform task instructions or input tags into differentiable prompt vectors that enable task-specific outputs from a fixed LLM [2307.01446].
- In vision and multimodal problems, CoPS mechanisms synthesize text and/or visual prompts by conditioning on visual features, semantic prototypes, or even other modalities, as in prompt fusion or anomaly detection [2312.03734, 2507.08410, 2508.03447].

Common architectural design principles of CoPS include:
- Pooling a set of learnable "prompt experts," which are compositely mixed via softgating or routing;
- Using lightweight selector/routing networks (MLP, attention, production system modules) that output instance-dependent mixture weights;
- Integration of the conditional prompt into the model input, attention layers, or feature stages to bias computation toward relevant semantics.

## 2. Architectural Methodologies

### 2.1 Expert Pools and Soft Routing

A prevalent CoPS formulation employs a pool of $l$ learnable prompt experts $\mathcal{P} = \{P_1, ..., P_l\}$, each $P_i \in \mathbb{R}^{K \times D}$, where $K$ is the prompt token count and $D$ the backbone hidden dimension. For a given input $X$, a selector network synthesizes a weighted mixture:

$$
\alpha = \text{softmax}(W_2\,\sigma(W_1\,g(X)) + b_2), \qquad
P_{\text{cond}} = \sum_{i=1}^l \alpha_i\,P_i
$$

Here, $g(X)$ is an input embedding extractor (e.g., pooled features), and $W_1,W_2$ are learned [2305.12437]. This prompt is then concatenated or injected into the main model at the input or early transformer/CNN stages.

### 2.2 Mixture of Prompt Experts (MoPE)

Conditional Prompt Tuning for multimodal fusion [2312.03734] proposes an extension where, for each instance and each transformer layer $i$, a dynamic prompt is constructed as a soft mixture of prompt experts, with soft routing scores $r^{(i)}(\psi_y)$ (dependent on a prior modality):

$$
\mathbf{P}_d^{(i)}(\psi_y) = \sum_{j=1}^k r^{(i)}_j(\psi_y)\;\mathbf{E}_{i,j}
$$

To balance expert utilization, an "importance loss" regularizes the distribution of routing weights across a mini-batch, incentivizing balanced expert usage and preventing degeneracy.

### 2.3 Production-System Modules

In language modeling, CoPS can be realized via differentiable production systems, as in PRopS [2307.01446], where a set of rule modules (attention or MLP blocks) are selected and composed according to the input/task condition. A Gumbel-softmax-based selector induces sparsity in module choice:

$$
p(x) = \sum_{i \in S_k(x)} \alpha_i f_i(E(x))
$$

where $E(x)$ embeds the input/task, $f_i$ are the rule modules, and $S_k(x)$ is the selected set of k modules.

### 2.4 Cross-Modal Conditional Prompting

Recent VLM research [2507.08410] synthesizes both text and visual conditional prompts via mutual guidance. Semantic prompts are extracted with a multi-modal large language model (MLLM) using attention over the MLLM’s decoder cache, followed by adaptation into VLM space. Visual prompts are then constructed by mutually guiding visual and semantic features through self- and cross-attention (AMG module).

### 2.5 Prototype and Semantic Token Enhancement

For zero-shot anomaly detection, CoPS [2508.03447] leverages explicit state prototypes (extracted via cross-attention over patch features) and implicit semantic class tokens (sampled via VAE from global image features) to assemble context-rich, state-aware prompts. A spatially-aware alignment module further refines prompt effectiveness for both image-level and pixel-level detection.

## 3. Training Objectives and Regularization

These methods generally follow “frozen backbone, trainable prompt” paradigms, optimizing the conditional prompt parameters along with the selector/routing modules via a standard downstream task loss, often cross-entropy or binary cross-entropy, with added regularization for balanced expert usage, prompt diversity, or compositional sparsity.

For instance, [2312.03734] minimizes:
$$
\mathcal{L} = \mathcal{L}_{\mathrm{task}} + \lambda\sum_{i}\mathcal{L}_{\mathrm{imp}}^{(i)}
$$

while [2508.03447] optimizes joint modular losses for state prototype alignment, variational class sampling, and spatial text-image alignment:
$$
\min_{\theta,\psi,\omega,\varphi} \mathcal{L}_{\rm ESTS}(\theta) + \mathcal{L}_{\rm ICTS}(\psi) + \mathcal{L}_{\rm SAGA}(\psi,\omega,\varphi)
$$

Contrastive learning objectives are common for multimodal settings, with additional regularizers enforcing consistent prompt usage or feature alignment with augmentations.

## 4. Applications and Empirical Impact

Conditional Prompt Synthesis methods have been validated across a diverse range of applications:

| Domain                   | CoPS Implementation (Source) | Representative Gains                              |
|--------------------------|------------------------------|---------------------------------------------------|
| Video Action Recognition | Soft Conditional Prompt Learning (SCP) [2305.12437] | +3.17–10.2% accuracy on Okutama, NECDrone, SSV2   |
| Multimodal Fusion        | MoPE-based conditional tuning [2312.03734]        | SOTA with 0.7% params, matches or exceeds fine-tuning |
| Zero-shot Anomaly Detection | CoPS [2508.03447]                        | +2.5 pp AUROC vs. prior SOTA (92.5% vs. 90.0%)    |
| Vision-Language Models   | MuGCP [2507.08410]                            | +2.01% (few-shot HM metric) vs. previous best      |
| Language Model Adaptation | PRopS [2307.01446]                            | +15.5% compositional EM accuracy over baseline     |

In all settings, CoPS techniques consistently outperform non-conditional prompt methods or static prompt baselines—particularly in settings with limited data, task composition, or where parameter efficiency is critical.

## 5. Comparative Analysis and Ablations

Empirical analyses reveal several robust properties of CoPS designs:

- **Expressivity**: Instance-conditional prompt synthesis, especially via expert pooling or mixture modules, scales more effectively than simply enlarging prompt length [2312.03734].
- **Generalization**: Compositional/gated prompt systems (e.g., PRopS) enable zero- and few-shot transfer by reusing learned “subprompts” for novel input combinations, yielding sample-efficient generalization [2307.01446].
- **Parameter Efficiency**: Across benchmarks, CoPS approaches achieve high accuracy with 1–10% (often <1%) of the trainable parameters required for full fine-tuning or adapter-based transfer.
- **Ablations**: Removing dynamic routing, regularization, or mutual-attention modules causes significant drops in performance and generalization (e.g., importance loss prevents expert collapse, full AMG and multi-prompt fusion boost generalization in MuGCP [2507.08410]).
- **Prompt Diversity**: Balanced utilization of prompt experts (encouraged via importance loss or similar terms) prevents routings from collapsing onto a few experts and improves robustness to data scaling [2312.03734].

## 6. Limitations and Open Directions

Known constraints of CoPS methodologies include:

- **Selector Design Sensitivity**: Effectiveness depends on the capacity, architecture, and regularization of the selector/router. Overly simplistic routings cannot capture complex input variability; poorly regularized selectors collapse onto a small subset of experts.
- **Prompt Interpretation**: Learned prompt experts or composed modules do not always correspond to semantically interpretable factors or tasks; understanding prompt semantics remains an open problem [2307.01446].
- **Resource Overhead**: Some advanced schemes (e.g., MuGCP [2507.08410]) require substantial compute and memory due to reliance on MLLM decoders, offline caching, or multiple attention modules.
- **Domain/Task Portability**: Optimal pool size, token counts, and fusion topology are task-dependent and may require extensive ablation.
- **Noise and Overfitting**: External priors (such as MLLM-generated semantic embeddings) can encode irrelevant or spurious context, necessitating future research into content filtering and adaptive knowledge distillation [2507.08410].

Directions for future exploration include: lightweight distillation of prompt knowledge, dynamic gating, efficient memory management for prompt caches, and extension of CoPS principles to detection, segmentation, or video-language alignment [2507.08410].

## 7. Theoretical Properties and Interpretability

Initial theoretical evidence (e.g., [2307.01446]) suggests that CoPS-style modular prompt systems retain favorable sample complexity compared to monolithic prompt learning, provided module selection is sparse and compositionally structured. Proposition 1 in [2307.01446] formalizes that, under compositional reuse and sufficient expressivity, risk can be made arbitrarily close to the Bayes risk with polynomially fewer samples relative to the number of modules and composed subtasks. This suggests that prompt libraries can span large task spaces while preserving compactness and adaptation speed. Moreover, gating scores or module activations in these systems provide an interpretable basis for analyzing input-to-prompt mappings, though the alignment to human-interpretable subtasks is not always guaranteed.

## References

- "SCP: Soft Conditional Prompt Learning for Aerial Video Action Recognition" [2305.12437]
- "Conditional Prompt Tuning for Multimodal Fusion" [2312.03734]
- "CoPS: Conditional Prompt Synthesis for Zero-Shot Anomaly Detection" [2508.03447]
- "On Conditional and Compositional Language Model Differentiable Prompting" [2307.01446]
- "Multi-modal Mutual-Guidance Conditional Prompt Learning for Vision-Language Models" [2507.08410]

Source: https://www.emergentmind.com/topics/conditional-prompt-synthesis-cops