---
title: 'TaAM-CPT: Text as Any-Modality Prompt Tuning'
url: https://www.emergentmind.com/topics/taam-cpt
type: topic
---

# TaAM-CPT: Text as Any-Modality Prompt Tuning

Searching arXiv for the TaAM-CPT paper and closely related prompt-tuning context.
TaAM-CPT, short for **Text as Any-Modality for Zero-Shot Classification by Consistent Prompt Tuning**, is a prompt-learning framework for zero-shot classification that treats text as a universal surrogate for arbitrary modalities and learns class-specific prompts directly in the shared embedding spaces of frozen multimodal foundation models [2508.06382]. It is presented as a scalable approach for constructing a general representation model toward unlimited modalities using solely text data, with a design centered on modality prompt pools, text construction, and modality-aligned text encoders from pre-trained models. The method is evaluated on video, image, and audio classification, and is explicitly positioned as an alternative to prompt-tuning pipelines that depend on modality-specific labeled data or are customized for a single modality [2508.06382].

## 1. Problem setting and conceptual basis

TaAM-CPT targets a practical bottleneck in prompt tuning and multimodal recognition: many prior methods for zero-shot or weakly supervised classification still depend on **modality-specific labeled data** or are **custom-built for one modality** [2508.06382]. The framework addresses the harder goal of learning a **universal representation model toward unlimited modalities using solely text data**, with the stated objective of remaining extensible as new modalities or categories appear.

The phrase “Text as Any-Modality” is used literally. Each class of a target modality is converted into a **learnable prompt vector**, and supervision is supplied by **LLM-generated text descriptions** rather than labeled modality-specific examples. Instead of introducing a modality-specific prompt encoder, TaAM-CPT directly optimizes prompts inside the embedding space of a pretrained multimodal model. The method assumes that pretrained models such as **ViCLIP**, **CLIP**, and **CLAP** already provide aligned embedding spaces where text can act as a proxy for video, image, and audio [2508.06382].

This design suggests a deliberate shift in the role of prompt learning. Rather than adapting a text encoder to a single downstream modality, TaAM-CPT uses text-only supervision to construct modality-specific classifier prompts in multiple aligned latent spaces. A plausible implication is that the classifier itself becomes a modular object: prompt pools can be expanded without requiring a full retraining of an end-to-end multimodal recognizer.

## 2. Architectural components

The framework has three main components: **LLM-assisted data construction**, **prompt initializing and modality text encoding**, and **intra-modal and inter-modal learning** [2508.06382].

For each modality \(m\), TaAM-CPT maintains a **modality-specific prompt pool**
\[
{\mathbf{P}_m} = [\;{\mathbf p}_1^m, {\mathbf p}_2^m, {\mathbf p}_3^m, ..., {\mathbf p}_{N}^m\;],
\]
where \(m \in \{ \mathcal{V}, \mathcal{A}, \mathcal{I}\}\), \(\mathbf{p}_{i}^m \in \mathbb{R}^{d}\) is the \(i\)-th class-specific prompt, and \(N\) is the total number of labels [2508.06382]. The prompt pool has shape \(\mathbf{P}_m \in \mathbb{R}^{N \times d}\) and includes labels across modalities. A key stated property is that the prompt pool is **identically initialized across modalities**, while each modality retains its own pool; adding a new modality therefore does not disturb existing ones, and adding a new label means adding a new class-specific prompt to each pool.

Text features are extracted by **modality-aligned text encoders** from pretrained models:
- \(\mathrm{Text}_v\) from **ViCLIP**
- \(\mathrm{Text}_a\) from **CLAP**
- \(\mathrm{Text}_i\) from **CLIP** [2508.06382]

An important implementation choice is that prompts are **not passed through additional encoders or MLPs**. They are optimized directly in the shared embedding space. The paper identifies this as part of the reason the architecture remains simple and scalable.

The modularity claim is central to TaAM-CPT. Each modality receives its own prompt pool and aligned text encoder; adding a new modality requires adding a new prompt pool and a new aligned text encoder; adding a new class requires appending a new class prompt to every modality pool [2508.06382]. The paper states that, because prompts are directly optimized vectors, there is no retraining of already learned prompts required when new classes or modalities arrive. This is the basis for the framework’s “unlimited modalities” characterization.

## 3. Text construction and text-only supervision

TaAM-CPT replaces manual noun filtering or hand-crafted caption data with **LLM-assisted data construction**. The method uses an instruction template to query an LLM and synthesize text sentences that explicitly include target labels:
> **TEMPLATE**: Making several English sentences to describe a \{ Modality \}. Requirements: Generate 5 English sentences! Each sentence should be less than 25 words and includes: \{ Labels \}. [2508.06382]

Here, \(\{ \text{Modality} \}\) is replaced with “video”, “audio”, or “image”, and \(\{ \text{Labels} \}\) is the set of class labels. For video, up to 2 labels are sampled per sentence; for image and audio, up to 3 labels are used [2508.06382]. The stated purpose is to avoid issues from noun filtering and to make the generated text directly usable for supervision, because the ground-truth labels are the labels included in the prompt.

The appendix states that the authors generate on the order of **100k text sentences per modality** with **LLaMA-2-7B**, and that generating about **300k total sentences** for the main datasets takes roughly **2 hours on a Tesla V100

Source: https://www.emergentmind.com/topics/taam-cpt