TransPrompt v2: Meta-Learning for Text Classification

• TransPrompt v2 is a meta-learning framework that transfers prompt-based knowledge across both similar and heterogeneous text classification tasks.
• It integrates multi-task meta-knowledge acquisition with prototype-based and entropy-based debiasing techniques for robust adaptation.
• Extensive evaluations show that TransPrompt v2 outperforms state-of-the-art baselines, achieving significant gains in few-shot and full-supervision scenarios.

TransPrompt v2 is a meta-learning framework for cross-task text classification that enables effective transfer of prompt-based knowledge across both similar and distant text classification tasks, targeting few-shot and full-data regimes. The framework combines multi-task meta-knowledge acquisition with advanced debiasing techniques, leveraging task-type prompt injection to support heterogeneous label spaces and providing robust generalization to unseen tasks. Extensive empirical validation demonstrates that TransPrompt v2 outperforms state-of-the-art baselines in both few-shot and full-supervision scenarios (Wang et al., 2023).

1. Motivation and Conceptual Framework

Prompt-based fine-tuning with PLMs (pre-trained LLMs) has achieved notable success in few-shot text classification; however, traditional approaches such as PVP, LM-BFF, and P-tuning train templates and verbalizers in a single-task fashion. Given a limited data regime (e.g., $K\approx16$ per class), these approaches are prone to over-fitting, and the learned prompts remain narrowly scoped with limited cross-task transferability. Real-world scenarios often involve multiple related tasks—sentiment analysis, natural language inference (NLI), and paraphrase detection—none of which possesses abundant labeled instances, but which could benefit from prompt knowledge transfer. Moreover, previous work lacks a unified framework capable of simultaneously handling “similar” tasks (identical label spaces) and “distant” tasks (heterogeneous label spaces), limiting the potential for mutual reinforcement and knowledge leveraging (Wang et al., 2023).

TransPrompt v2 addresses this gap with a transfer-oriented architecture: a meta-learner jointly acquires and encodes prompt knowledge from multiple tasks during the training phase, then allows rapid adaptation to both seen and entirely novel tasks, significantly improving few-shot and generalization performance.

2. Multi-Task Meta-Knowledge Acquisition (MMA)

The MMA stage forms the core of cross-task prompt learning. Given $M$ source tasks $\{T_1, ..., T_M\}$, each with label set $\mathcal{Y}_m$ and $K$-shot training set $D_m$, the objective is to learn meta-learner parameters $\Theta$—comprising the underlying PLM and prompt encoders—that internalize transferable representations.

Prompt Encoding for Similar Tasks

Each task $m$ has both a task-specific pseudo prompt $P^{(m)} = [P_1^{(m)},...,P_I^{(m)}]$ and a universal pseudo prompt $P^{(*)}=[P_1^{(*)},...,P_I^{(*)}]$, with templates

$M$0

Prompt encoders are instantiated as:

$M$1

Outputs are fused via self-attention:

$M$2

$M$3, together with the tokenized input $M$4 and [MASK], constitutes the input to the PLM.

Prompt Encoding for Distant Tasks

Tasks grouped into $M$5 types $M$6. For type $M$7, a natural-language description $M$8 (e.g., “A sentiment analysis task”) and pseudo tokens $M$9 guide prompt construction:

• Task-specific template: $\{T_1, ..., T_M\}$0
• Type description: $\{T_1, ..., T_M\}$1
• Universal template: $\{T_1, ..., T_M\}$2

Embedding is computed as:

$\{T_1, ..., T_M\}$3

$\{T_1, ..., T_M\}$4

A type-specific combination:

$\{T_1, ..., T_M\}$5

Final $\{T_1, ..., T_M\}$6 is obtained by pooling over all $\{T_1, ..., T_M\}$7.

3. Debiasing Strategies and Meta-Learning Objective

To prevent the meta-learner from overfitting to specific tasks or dominant task domains—and to maintain task-agnostic transfer—the framework employs two debiasing strategies:

• Prototype-based Debiasing: For each task $\{T_1, ..., T_M\}$8 and label $\{T_1, ..., T_M\}$9, a prototype centroid $\mathcal{Y}_m$0 is computed as an average of [MASK] token embeddings across all instances of $\mathcal{Y}_m$1. Each instance receives a dynamic score $\mathcal{Y}_m$2 reflecting similarity to centroids in both the target and other tasks, scaled by parameter $\mathcal{Y}_m$3.
• Entropy-based Debiasing: To discourage over-confident, task-specific predictions, a negative entropy term is incorporated. For dataset $\mathcal{Y}_m$4, the entropy $\mathcal{Y}_m$5 penalizes peaked output distributions.

The total loss function is:

$\mathcal{Y}_m$6

Optimization iterates over batches, updating $\mathcal{Y}_m$7 and recalculating $\mathcal{Y}_m$8 until convergence.

4. Task-Aware Model Specification (TMS)

Following meta-learning, adaptation to a specific target task $\mathcal{Y}_m$9 proceeds via TMS:

• Seen Tasks ($K$0): Fine-tune only the task-specific encoder $K$1 and the PLM in a few steps, optimizing:

$K$2

• Unseen Tasks: Construct a new pseudo-template incorporating a type description, initialize the encoder with universal parameters $K$3, and fine-tune on $K$4 using the same objective as above. Thanks to MMA, this results in fast convergence and competitive performance.

5. Task-Type Prompt Injection

Prompt injection of task-type descriptions is critical for handling tasks with heterogeneous label spaces. Each distant task type $K$5 injects a human-interpretable natural language description $K$6 into the prompt:

$K$7

This mechanism serves as a “soft verbalizer,” providing the meta-learner with an explicit cue about task semantics and label meaning. The use of intra-type and inter-type prompt embeddings enables both the modeling of type-specific commonalities and the extraction of cross-type universality, facilitating robust transfer—even across distant NLP tasks.

6. Empirical Evaluation

Experiments cover seven datasets across three major groups: sentiment analysis (SST-2, MR, CR), NLI (MNLI, SNLI), and paraphrasing (MRPC, QQP). Few-shot evaluation uses $K$8 per class, with five random seeds and a RoBERTa-large backbone.

Baselines

Method	Description
Fine-tuning	Standard PLM fine-tuning
LM-BFF (auto/mtl)	Prompt-based, multi-task
P-tuning (mtl)	Soft prompt multi-task
TransPrompt v1	Previous-generation transfer
Meta Fine-tuning	Cross-task meta learning
Meta-DTL	Distillation transfer

Main Findings

Method	Avg(similar)	Avg(distant)
Fine-tuning	67.66	67.66
LM-BFF (auto)	77.28	77.28
P-tuning	75.70	75.70
LM-BFF (mtl)	79.24	77.43
P-tuning (mtl)	79.44	77.86
TransPrompt v1	82.15	77.62
TransPrompt v2	82.14	79.53

Ablation studies indicate that removing either prototype-based or entropy-based debiasing results in a 0.5–1.0 percentage point drop in accuracy, with the absence of both yielding a ∼2 point reduction. Omission of type descriptions or intra/inter-type prompts in the distant task setting decreases accuracy from 90.21% to 88.94%. When adapting the meta-learner to an unseen task within a task group, TransPrompt v2 shows gains of +1.0–1.3 points over LM-BFF or P-tuning initialization. Distant transfer (e.g., sentiment to NLI) also exhibits consistent improvements with multiple source types. On full training sets, TransPrompt v2 outperforms both single-task and cross-task baselines by 1–5% absolute on average.

Visualization (e.g., t-SNE plots of [MASK] token embeddings) reveals clustering according to class and task type, supporting the interpretability and discriminative power of the learned representations.

7. Practical Considerations and Limitations

TransPrompt v2 is particularly effective in environments with multiple small labeled datasets that may be either task-homogeneous or heterogeneous. For new tasks, constructing concise task-type descriptions and a pseudo-template enables leveraging the learned universal encoder for rapid and robust adaptation.

Current limitations include a focus exclusively on text classification; extension to generative or sequence-tagged tasks is not addressed. Prospective research avenues include the use of richer task descriptions, learned verbalizers, or application to alternate PLM architectures (e.g., T5) (Wang et al., 2023).