TaAM-CPT: Text as Any-Modality Prompt Tuning

• TaAM-CPT is a prompt-learning framework that treats text as a universal surrogate to generate class-specific prompts for video, image, and audio classification.
• It utilizes modality-specific prompt pools and aligned text encoders from pretrained models like CLIP and CLAP to enable scalable zero-shot classification.
• The method replaces modality-specific labels with LLM-assisted text construction, allowing efficient and extensible training without retraining for new classes or modalities.

Searching arXiv for the TaAM-CPT paper and closely related prompt-tuning context. TaAM-CPT, short for Text as Any-Modality for Zero-Shot Classification by Consistent Prompt Tuning, is a prompt-learning framework for zero-shot classification that treats text as a universal surrogate for arbitrary modalities and learns class-specific prompts directly in the shared embedding spaces of frozen multimodal foundation models (Wu et al., 8 Aug 2025). It is presented as a scalable approach for constructing a general representation model toward unlimited modalities using solely text data, with a design centered on modality prompt pools, text construction, and modality-aligned text encoders from pre-trained models. The method is evaluated on video, image, and audio classification, and is explicitly positioned as an alternative to prompt-tuning pipelines that depend on modality-specific labeled data or are customized for a single modality (Wu et al., 8 Aug 2025).

1. Problem setting and conceptual basis

TaAM-CPT targets a practical bottleneck in prompt tuning and multimodal recognition: many prior methods for zero-shot or weakly supervised classification still depend on modality-specific labeled data or are custom-built for one modality (Wu et al., 8 Aug 2025). The framework addresses the harder goal of learning a universal representation model toward unlimited modalities using solely text data, with the stated objective of remaining extensible as new modalities or categories appear.

The phrase “Text as Any-Modality” is used literally. Each class of a target modality is converted into a learnable prompt vector, and supervision is supplied by LLM-generated text descriptions rather than labeled modality-specific examples. Instead of introducing a modality-specific prompt encoder, TaAM-CPT directly optimizes prompts inside the embedding space of a pretrained multimodal model. The method assumes that pretrained models such as ViCLIP, CLIP, and CLAP already provide aligned embedding spaces where text can act as a proxy for video, image, and audio (Wu et al., 8 Aug 2025).

This design suggests a deliberate shift in the role of prompt learning. Rather than adapting a text encoder to a single downstream modality, TaAM-CPT uses text-only supervision to construct modality-specific classifier prompts in multiple aligned latent spaces. A plausible implication is that the classifier itself becomes a modular object: prompt pools can be expanded without requiring a full retraining of an end-to-end multimodal recognizer.

2. Architectural components

The framework has three main components: LLM-assisted data construction, prompt initializing and modality text encoding, and intra-modal and inter-modal learning (Wu et al., 8 Aug 2025).

For each modality $m$, TaAM-CPT maintains a modality-specific prompt pool

$${\mathbf{P}_m} = [\;{\mathbf p}_1^m, {\mathbf p}_2^m, {\mathbf p}_3^m, ..., {\mathbf p}_{N}^m\;],$$

where $m \in \{ \mathcal{V}, \mathcal{A}, \mathcal{I}\}$, $\mathbf{p}_{i}^m \in \mathbb{R}^{d}$ is the $i$-th class-specific prompt, and $N$ is the total number of labels (Wu et al., 8 Aug 2025). The prompt pool has shape $\mathbf{P}_m \in \mathbb{R}^{N \times d}$ and includes labels across modalities. A key stated property is that the prompt pool is identically initialized across modalities, while each modality retains its own pool; adding a new modality therefore does not disturb existing ones, and adding a new label means adding a new class-specific prompt to each pool.

Text features are extracted by modality-aligned text encoders from pretrained models:

• $\mathrm{Text}_v$ from ViCLIP
• $\mathrm{Text}_a$ from CLAP
• $\mathrm{Text}_i$ from CLIP (Wu et al., 8 Aug 2025)

An important implementation choice is that prompts are not passed through additional encoders or MLPs. They are optimized directly in the shared embedding space. The paper identifies this as part of the reason the architecture remains simple and scalable.

The modularity claim is central to TaAM-CPT. Each modality receives its own prompt pool and aligned text encoder; adding a new modality requires adding a new prompt pool and a new aligned text encoder; adding a new class requires appending a new class prompt to every modality pool (Wu et al., 8 Aug 2025). The paper states that, because prompts are directly optimized vectors, there is no retraining of already learned prompts required when new classes or modalities arrive. This is the basis for the framework’s “unlimited modalities” characterization.

3. Text construction and text-only supervision

TaAM-CPT replaces manual noun filtering or hand-crafted caption data with LLM-assisted data construction. The method uses an instruction template to query an LLM and synthesize text sentences that explicitly include target labels:

TEMPLATE: Making several English sentences to describe a { Modality }. Requirements: Generate 5 English sentences! Each sentence should be less than 25 words and includes: { Labels }. (Wu et al., 8 Aug 2025)

Here, $${\mathbf{P}_m} = [\;{\mathbf p}_1^m, {\mathbf p}_2^m, {\mathbf p}_3^m, ..., {\mathbf p}_{N}^m\;],$$0 is replaced with “video”, “audio”, or “image”, and $${\mathbf{P}_m} = [\;{\mathbf p}_1^m, {\mathbf p}_2^m, {\mathbf p}_3^m, ..., {\mathbf p}_{N}^m\;],$$1 is the set of class labels. For video, up to 2 labels are sampled per sentence; for image and audio, up to 3 labels are used (Wu et al., 8 Aug 2025). The stated purpose is to avoid issues from noun filtering and to make the generated text directly usable for supervision, because the ground-truth labels are the labels included in the prompt.

The appendix states that the authors generate on the order of 100k text sentences per modality with LLaMA-2-7B, and that generating about 300k total sentences for the main datasets takes roughly **2 hours on a Tesla V100