User-Aware Prefix-Tuning

• User-aware prefix-tuning is a personalization paradigm that conditions small, learnable prefix tokens on explicit user attributes.
• It integrates frozen foundation models with optimized prefix modules to achieve efficient adaptation for tasks like image captioning and dialogue generation.
• Experimental evaluations demonstrate that updating less than 1% of total parameters yields competitive performance compared to full fine-tuning.

User-aware prefix-tuning is a parameter-efficient personalization paradigm for adapting large language and vision-LLMs to individual user preferences or profiles through the insertion and optimization of small learnable prefix tokens. By leveraging frozen foundation models and solely updating prefix parameters or small adapters, this approach achieves strong user-specific adaptation on tasks such as image captioning and dialogue generation while requiring a fraction of the traditional parameter count.

1. Fundamental Principles and Definitions

User-aware prefix-tuning extends standard prefix-tuning by incorporating explicit user characteristics (user embeddings, profile attributes, or posting history) into the construction and training of the prefix vectors prepended to the inputs or internal states of transformer-based models. In both vision-language and language-only settings, the base model parameters—such as those of GPT-2, T5, or CLIP—are frozen, while a small set of continuous prefix parameters is optimized per user or user group.

In notation, for a Transformer model with parameters θ (frozen), and L layers, a set of learnable prefix vectors $P_i$ (or $P_{(u)}$ for user $u$) with $l_i$ tokens per layer are concatenated to the input at each layer:

$$h_i' = \mathrm{concat}(P_i, h_i), \qquad h_{i+1} = \mathrm{TransformerLayer}_i(h_i'; \theta)$$

For user-aware adaptation, $P_i$ or $P_{(u)}$ is conditioned on user embeddings or profiles, differencing this approach from vanilla prefix-tuning.

2. Model Architectures in Diverse Personalization Tasks

Two primary instantiations are demonstrated: personalized image captioning (Wang et al., 2023) and user-specific dialogue generation (Otsuka et al., 2024).

Personalized Image Captioning.

The User-Aware Prefix-Tuning (UAPT) framework for image captioning combines image content and user history as follows:

• A frozen CLIP image encoder extracts visual features $V_0$ from input image $I$.
• User posting history $u$ is processed to TF-IDF keywords and embedded via CLIP's text encoder into $P_{(u)}$0.
• A query-guided mapping network, composed of multi-head cross-attention and feed-forward layers, projects $P_{(u)}$1 onto language-aligned representations $P_{(u)}$2 using learnable queries $P_{(u)}$3.
• A fusion transformer merges $P_{(u)}$4 into a sequence of prefix tokens $P_{(u)}$5.
• $P_{(u)}$6 is sliced into per-layer prefixes $P_{(u)}$7 and prepended to the hidden states of a frozen GPT-2 model at all layers, after which caption generation proceeds autoregressively.

User-Specific Dialogue Generation.

A frozen encoder-decoder transformer (e.g., T5) serves as the base. Two user-aware conditioning mechanisms are used:

• Simple text prompts incorporating inferred user profile attributes (Persona Speaker/Pair Prompts), included as part of the input sequence.
• A learnable, continuous prefix $P_{(u)}$8 is prepended to each input and flows through all layers; only this prefix is updated during user-specific fine-tuning.

3. Prefix Construction and Conditioning on User Context

Image Captioning

Prefix tokens are generated through transformer-based fusion of the pooled image embedding from CLIP, $P_{(u)}$9, and the user context embedding $u$0. The formula for prefix generation is:

$u$1

The sequence $u$2 is then mapped to the individual layers or shared across layers for insertion as the prefix.

Dialogue Generation

For dialogue, each user $u$3 is assigned a separate prefix matrix $u$4, where $u$5 is the prefix length and $u$6 is the model hidden dimension. This prefix is prepended to the embedded token sequence at every layer, producing a user-specific initial hidden state. The user profile is also encoded as a textual prompt in pre-training, teaching the frozen parameters θ to utilize such information as a context signal.

4. Training Strategies and Parameter Efficiency

Frozen Model Paradigm.

All base model parameters are kept frozen throughout both pre-training and fine-tuning. Only:

• The mapping network, fusion network, and prefixes in UAPT (image captioning),
• The continuous per-user prefixes $u$7 (dialogue),

are updated during training.

User Profile Pre-Training.

In dialogue systems, pre-training the base model θ with explicit user profile prompts (Persona Speaker or Persona Pair Prompts) enables effective conditioning on real-world profile attributes. Millions of dialogue-response pairs, each tagged with user metadata extracted from historical posts and classified into demographic attributes (e.g., gender, age, location, occupation), are used for pre-training.

Loss Functions and Regularization.

Training minimizes the standard autoregressive cross-entropy loss, with optional ℓ₂ regularization on prefix or adapter weights. For dialogue,

$u$8

No explicit domain-alignment or additional user-specific objectives are employed beyond basic weight decay (Wang et al., 2023, Otsuka et al., 2024).

Model Size and Efficiency.

Trained parameters are on the order of 1–2 million (image captioning) or less than 0.01% of the frozen base model (dialogue), such as $u$9 parameters per user when $l_i$0, $l_i$1 (Otsuka et al., 2024).

5. Experimental Evaluations and Empirical Findings

Personalized Image Captioning.

UAPT demonstrates competitive performance on Instagram and YFCC100M datasets with significant gains over prior baselines such as CSMN-P5. Notably:

Dataset	BLEU-4 Baseline	BLEU-4 UAPT	CIDEr Baseline	CIDEr UAPT
Instagram	0.013	0.032	0.214	0.343
YFCC100M	0.004	0.008	0.064	0.170

Ablation studies show that removing user context, mapping, fusion, or learnable queries degrades BLEU-4 and CIDEr, confirming the necessity of all components (Wang et al., 2023).

Dialogue Generation.

Prefix-tuning (PFX) matches or closely approaches the performance of full fine-tuning across next-utterance reproducibility metrics—including embedding similarity, Acc@sim ≥ 0.9, and ROUGE-L—on Japanese Speed Dating, User-Profile Chat, and PersonaChatJP corpora. For example, on the Speed Dating corpus:

Method	sim	[email protected]	ROUGE-L
GPT-3.5	.27	1.3%	10.9
FULL	.52	6.6%	18.4
PFX	.52	6.2%	18.1

Diversity across users, measured by Distinct-1/2/Sent, is substantially higher for full and prefix-tuning compared to models that only switch on user IDs (Otsuka et al., 2024).

Prefix-tuning achieves these results with ≪1% added parameters and is robust to overfitting and “model destruction” even with modest amounts of per-user data. Pre-training with profile prompts is essential; neither full fine-tuning nor prefix-tuning can capture user style from small data without it.

6. Comparative Analysis With Prior Approaches

User-aware prefix-tuning offers considerable efficiency advantages over full model fine-tuning or user-specific adapters. In both major applications, all models parameterize personalization exclusively through small, learnable prefixes or shallow fuser modules, while the heavy foundation models are shared and remain static.

Prior approaches in personalized captioning often require updating the full model for each new user or domain, which is computationally intensive and does not scale (Wang et al., 2023). User-aware prefix-tuning addresses this limitation by drastically reducing the update set and storage requirements per user. In dialogue, direct prompting of LLMs with user information is insufficient to reproduce individual dialogue behaviors, as confirmed empirically (Otsuka et al., 2024). Prefix-based adaptation achieves high personalization with efficient, robust training and minimal extra storage.

7. Implications, Limitations, and Directions

The user-aware prefix-tuning paradigm demonstrates robust, scalable personalization across distinct domains. A plausible implication is that such parameter-efficient adaptation is well suited for large, personalized services where storing a full separate model per user is impractical. The consistent requirement for explicit user conditioning—either via profile prompts or embedding fusion—highlights a central methodological principle underpinning effectiveness of the approach.

Limitations include the reliance on high-quality user attribute extraction and sufficient historical data for effective profile estimation (Otsuka et al., 2024), as well as the abstraction level at which user preferences are represented. In multimodal cases, the mapping between domains (e.g., vision and language) requires careful semantic alignment, as shown by the use of a query-guided mapping network and fusion modules in UAPT (Wang et al., 2023).

Overall, user-aware prefix-tuning constitutes a rigorously validated, parameter-efficient mechanism for user modeling in deep generative systems. It enables high-fidelity personalization with minimal incremental costs and offers a general recipe applicable to both vision-language and language-only personalization tasks.