PersonaVLM: Personalized Vision-Language Models

• PersonaVLM is a paradigm where vision-language models condition multimodal inference on user identities, histories, and preferences for personalized tasks.
• It employs modular personalization by integrating frozen base models with external modules that extract user-specific signals through concept embeddings or retrieval.
• Empirical studies show high precision in personalized recognition and control while addressing challenges in ethical alignment, privacy, and generalization.

PersonaVLM denotes an emerging family of vision-language approaches in which multimodal inference is conditioned on user-specific identities, preferences, histories, or simulated personas rather than only on generic visual-semantic priors. In the literature, the term appears both as the proper name of a specific on-board autonomous-driving system and as a broader label for personalized or persona-conditioned VLM research, including user-specific concept grounding, referential dialogue, socially aware human-robot interaction, contextualized visual memory, and VLM personas used as proxy participants in embodied HCI studies (Cui et al., 2024, Alaluf et al., 2024, Pham et al., 2024, Rahimi et al., 15 Feb 2025, Oh et al., 3 Feb 2026, Gui et al., 18 Feb 2026).

1. Terminological scope and research setting

The unifying problem is that standard VLMs can usually recognize category-level content but do not naturally represent private referents such as a particular person, pet, toy, mug, or other user-specific object, nor do they reliably adapt responses to user state, demographic context, accumulated multimodal memory, or downstream control requirements. Across the recent literature, personalization is therefore framed not as generic multimodal competence, but as the ability to recognize a user-specific entity or context and then communicate, reason, or act on that recognition without damaging the base model’s unrelated capabilities (Alaluf et al., 2024, Rahimi et al., 15 Feb 2025, Oh et al., 3 Feb 2026).

Usage of PersonaVLM	Representative papers	Core formulation
User-specific concept grounding	MyVLM (Alaluf et al., 2024), PLVM (Pham et al., 2024), PeKit (Seifi et al., 4 Feb 2025)	Learn or retrieve referential concepts from reference images
Contextualized visual personalization	CoViP (Oh et al., 3 Feb 2026)	Interpret a query image using prior multimodal user context
User-aware interaction	USER-VLM 360 (Rahimi et al., 15 Feb 2025), PerPilot (Wang et al., 25 Aug 2025)	Adapt responses or actions using user modeling, memory, and tuning
Embodied or proxy personas	VLM personas in HCI (Gui et al., 18 Feb 2026), urban perception analysis (Silva et al., 27 May 2026)	Condition multimodal outputs on persona profiles
On-board control	PersonaVLM (Cui et al., 2024)	Map instruction, vision, and memory to controller parameters

A central distinction in this literature is between explicit personalization and contextualized personalization. Explicit personalization covers identity- or instance-level grounding, such as recognizing “your dog” or “my friend” from a few examples. Contextualized personalization is stronger: the model must connect the current image with a user’s accumulated visual-textual history and retrieve only the relevant facts for the present scene (Alaluf et al., 2024, Oh et al., 3 Feb 2026).

2. Core architectures for personalized vision-language modeling

A common architectural pattern is modular personalization: the pretrained VLM remains largely frozen, while external modules detect, align, retrieve, or inject user-specific information. MyVLM makes this decomposition explicit by separating recognition from language integration. It augments a frozen VLM with external concept heads that function as toggles for user-specific concepts and then learns a latent concept embedding in the model’s intermediate feature space. For BLIP-2, the embedding is appended before the Q-Former; for LLaVA, it is appended after the visual projection into the language-model embedding space. The embedding is optimized by standard caption-generation cross-entropy,

$$e_* = \arg \min_e \sum_{i=1}^{N} \mathcal{L}_{CE} \left( t_i, o(I_i, e) \right),$$

and MyVLM further adds an attention regularizer,

$$\mathcal{L}_{reg} = \left\lVert \text{softmax}(Q \cdot \hat{k}_*) \right\rVert_2^2,$$

to prevent the concept token from monopolizing Q-Former attention (Alaluf et al., 2024).

PLVM pursues a related goal through an online reference-image encoder called the Aligner. Given a reference image $I$, the reference encoder produces

$$z_{\text{ref}} = E_{\text{ref}}(I) \in \mathbb{R}^{L \times d},$$

and the global token is mapped to a personalized word embedding and head weight,

$$e^{\text{word}} = f_{\phi_1}(z_{\text{ref}[0]}), \qquad e^{\text{weight}} = f_{\phi_2}(z_{\text{ref}[0]}).$$

The remaining reference tokens are compressed into a small set of context tokens by cross-attention, allowing the LVLM to treat a user-provided identity as a compact prompt-time concept rather than a fine-tuned parameter block (Pham et al., 2024).

PeKit replaces optimization with retrieval. For each personalized object $p$, open-vocabulary localization yields

$S_p = F_\text{ext}(I_p, k_p),$

patch-level embeddings are pooled as

$e_p=\AvgPool(F_\text{emb}(I_p),S_p)\in\mathbb{R}^{D_h},$

and multiple reference views are stored in a memory bank,

$\mathcal{M}=\{ (E_p, (n_p, c_p)) \}_{p\in P}.$

At inference, proposal embeddings are matched against memory with cosine similarity and fixed threshold $\tau = 0.75$, after which the downstream LVLM is visually prompted with bounding boxes, object names, and optional context (Seifi et al., 4 Feb 2025).

CoViP generalizes personalization from concept grounding to multimodal episodic context. It formulates the personalized response as

$$\mathcal{L}_{reg} = \left\lVert \text{softmax}(Q \cdot \hat{k}_*) \right\rVert_2^2,$$0

where $$\mathcal{L}_{reg} = \left\lVert \text{softmax}(Q \cdot \hat{k}_*) \right\rVert_2^2,$$1 is the query image, $$\mathcal{L}_{reg} = \left\lVert \text{softmax}(Q \cdot \hat{k}_*) \right\rVert_2^2,$$2 the user prompt, and $$\mathcal{L}_{reg} = \left\lVert \text{softmax}(Q \cdot \hat{k}_*) \right\rVert_2^2,$$3 prior multimodal user experience. The paper argues that the key bottleneck is learning the internal process

$$\mathcal{L}_{reg} = \left\lVert \text{softmax}(Q \cdot \hat{k}_*) \right\rVert_2^2,$$4

and uses personalized image captioning as an explicit intermediate representation of that latent personalized state (Oh et al., 3 Feb 2026).

3. Adaptation regimes, memory, and inference control

The field spans several adaptation regimes. MyVLM and PLVM retain a frozen backbone and learn only small personalization modules, which is intended to preserve the pretrained visual semantics while avoiding catastrophic forgetting. MyVLM trains concept embeddings for 75 steps for objects and 100 for people on BLIP-2, and 100 steps for both on LLaVA, using AdamW with constant learning rate $$\mathcal{L}_{reg} = \left\lVert \text{softmax}(Q \cdot \hat{k}_*) \right\rVert_2^2,$$5, gradient clipping at max norm $$\mathcal{L}_{reg} = \left\lVert \text{softmax}(Q \cdot \hat{k}_*) \right\rVert_2^2,$$6, and regularization weights $$\mathcal{L}_{reg} = \left\lVert \text{softmax}(Q \cdot \hat{k}_*) \right\rVert_2^2,$$7 for BLIP and $$\mathcal{L}_{reg} = \left\lVert \text{softmax}(Q \cdot \hat{k}_*) \right\rVert_2^2,$$8 for LLaVA (Alaluf et al., 2024). PLVM likewise keeps the backbone frozen and trains only the Aligner, using AdamW with learning rate $$\mathcal{L}_{reg} = \left\lVert \text{softmax}(Q \cdot \hat{k}_*) \right\rVert_2^2,$$9, batch size $I$0, and a single A6000 GPU (Pham et al., 2024).

A second regime is training-free personalization. PeKit updates no LVLM weights at all: personalization is accomplished by frozen extraction models, frozen embedding models, proposal generation, nearest-neighbor retrieval, and prompt-time visual conditioning (Seifi et al., 4 Feb 2025). PerPilot adopts a similar philosophy for VLM-based mobile agents, but specializes it to personalized instructions. It formalizes personalization detection as

$I$1

then combines memory-based retrieval and reasoning-based exploration to resolve ambiguous user-specific elements such as “my home” or “my favorite song” before handing a clarified instruction to an existing mobile agent (Wang et al., 25 Aug 2025).

A third regime is staged parameter-efficient tuning with explicit user modeling and ethical alignment. USER-VLM 360 is trained in three stages: Vision Alignment, Instruction Tuning, and Bias Mitigation. The generation model is written as

$I$2

with a learned visual projection $I$3, followed by LoRA or MoLE adaptation and a final DPO-based bias-mitigation stage (Rahimi et al., 15 Feb 2025).

A fourth regime is RL-based post-training over verifiable personalization rewards. CoViP optimizes personalized captioning by maximizing

$I$4

with reward

$I$5

where $I$6 scores set-level recognition of contextual concepts and $I$7 scores whether the generated caption exposes the correct personalized facts while avoiding irrelevant ones. Optimization uses GSPO with sequence-level clipping and normalized group advantages (Oh et al., 3 Feb 2026).

Memory is the other dominant design axis. In the autonomous-driving PersonaVLM system, the operative mapping is

$I$8

implemented in practice as $I$9, where $$z_{\text{ref}} = E_{\text{ref}}(I) \in \mathbb{R}^{L \times d},$$0 is retrieved historical memory and $$z_{\text{ref}} = E_{\text{ref}}(I) \in \mathbb{R}^{L \times d},$$1 is a matrix of controller parameters for longitudinal PID and lateral MPC control. Human feedback is written back into memory as $$z_{\text{ref}} = E_{\text{ref}}(I) \in \mathbb{R}^{L \times d},$$2, producing an explicit retrieval-augmented personalization loop rather than repeated retraining (Cui et al., 2024).

4. Benchmarks and empirical performance

Evaluation has moved beyond generic caption quality toward task-specific measures of personalized grounding. MyVLM uses identifier recall, CLIPScore, and sentence similarity on a dataset of 45 user-specific concepts, consisting of 29 objects and 16 people, and reports that the concept heads achieve high recall and precision (Alaluf et al., 2024). PLVM evaluates recognition, text-only QA, and visual QA on 46 identities and 354 images, plus comparison on the YoLLaVA dataset (Pham et al., 2024). PeKit evaluates recognition and VQA on the MyVLM dataset, Yo’LLaVA dataset, and the harder This-Is-My-Img benchmark (Seifi et al., 4 Feb 2025). CoViP adds diagnostic evaluations, including CapEval-QAs, Last-Seen Detection, Last-Action Recall, and Instruction-Triggered Recall, explicitly designed to rule out textual shortcut solutions (Oh et al., 3 Feb 2026).

System	Evaluation setting	Reported results
MyVLM	Personalized captioning	Recall 87.11% on BLIP-2 and 95.97% on LLaVA
PLVM	Recognition / QA	Recognition 89.5 positive, 82.1 negative, 85.8 mean; text-only QA 85.9; visual QA 77.5
PeKit	Recognition / VQA	Weighted accuracy 98.3 on MyVLM with G-SAM; This-Is-My-Img precision 90.1, avg accuracy 74.8; VQA 95.9 with InternVL
USER-VLM 360	Personalized VQA / facial understanding / fairness	+35.3% F1 in personalized VQA, +47.5% F1 in facial features understanding, 15% bias reduction, 30X speedup
PersonaVLM	Real-world autonomous driving	Takeover-rate reduction up to 76.9%
CoViP	Contextual grounding	About a 40% average improvement in contextual grounding capability

Several recurrent empirical themes emerge. First, few-shot or even one-shot personalization is feasible when the method explicitly isolates user-specific signals. MyVLM reports strong few-shot behavior and generalization to unseen images while preserving unrelated behavior (Alaluf et al., 2024). Second, online or training-free formulations can match or exceed training-based approaches on several benchmarks: PLVM avoids the roughly 40 minutes per concept reported for YoLLaVA, while PeKit surpasses prior training-based methods on both recognition and VQA (Pham et al., 2024, Seifi et al., 4 Feb 2025). Third, contextualized personalization remains substantially harder than explicit concept naming; CoViP’s contribution is precisely to show that current VLMs, including proprietary ones, are still weak when current images must be linked to accumulated multimodal user context (Oh et al., 3 Feb 2026).

5. Embodied, social, and interpretive applications

PersonaVLM research is not confined to captioning and QA. In social HRI, USER-VLM 360 uses a LLaVA-style stack trained on PaliGemma 2 backbones in 3B and 10B sizes, integrates FairFace, GenUser, UserEmotion, DOCCI, FaceTask-VQA, AlpaGasus-VQA, Alexa-VQA, NLE-VQA, BiasVision-DPO, and VLM-DPO, and deploys on Pepper with ROS 2, Whisper-Large-V3, and Tacotron 2. Reported end-to-end latency is 1.8 s for the 3B model and 4.2 s for the 10B model, with about 320 ms due to ROS 2 serialization and deserialization (Rahimi et al., 15 Feb 2025).

The autonomous-driving PersonaVLM system instantiates personalization as control rather than language generation. A compact 9B-parameter Qwen-VL model receives system message, human instruction, camera observation, and retrieved memory, then outputs controller parameters

$$z_{\text{ref}} = E_{\text{ref}}(I) \in \mathbb{R}^{L \times d},$$3

for longitudinal PID and lateral MPC. It is deployed on a 2019 Lexus RX450h with Autoware.AI, radar, LiDAR, cameras, and an onboard RTX-A4000 system, and reports LLM latency of 1.98 s in acceleration, 1.83 s in lane change, and 1.64 s in turning, versus roughly 4.8–5.8 s for GPT-4o (Cui et al., 2024).

Personalization-aware mobile agents extend the same logic to GUI task execution. PerPilot introduces the 75-instruction PerInstruct benchmark over 27 mobile apps and reports that UI-TARS improves from 12.0% baseline success to 68.0% with o4-mini-driven PerPilot and human intervention, while the system progressively shifts from reasoning-based exploration to memory-based retrieval as repeated use accumulates personalized information (Wang et al., 25 Aug 2025).

Another branch uses personas as experimental instruments rather than end-user assistants. In embodied HCI, VLM personas are constructed from participant demographics, Big Five traits, and AV-related attitudes, then evaluated in a discretized interactive street-crossing video simulator. The VLM personas reproduce overall average crossing time closely—5.25 s versus 5.07 s for humans—but exhibit lower variability, greater conservatism, and narrower qualitative reasoning than real participants (Gui et al., 18 Feb 2026). In urban perception, persona-conditioned multimodal agents produce strong convergence in captions but systematic divergence in justifications: caption similarity lies in the 0.85 to 0.90 range across persona pairs, whereas justification similarity spans 0.44 to 0.70, with statistically significant effects concentrated in economic status and political orientation (Silva et al., 27 May 2026).

These embodied and interpretive uses clarify that PersonaVLM is not only a personalization mechanism for object naming. It is also a framework for user-aware control, social adaptation, and simulation of perspective-conditioned multimodal judgment.

6. Limitations, ethics, and open problems

A persistent technical concern is the trade-off between personalization utility and preservation of general behavior. MyVLM explicitly freezes the backbone to avoid catastrophic forgetting and reports that unrelated captioning ability remains close to the frozen VLM (Alaluf et al., 2024). CoViP similarly reports that contextual grounding gains do not significantly degrade general caption quality or substantially increase hallucination (Oh et al., 3 Feb 2026). Even so, these results do not eliminate the broader problem that personalized modules may overfit narrow signals, scene context, or proxy cues.

Fairness and utility are also not perfectly aligned. USER-VLM 360 reports that DPO-based bias mitigation improves fairness metrics and reduces racist, sexist, offensive, or otherwise unethical outputs, but often hurts task performance, including ROUGE and F1 on some general-purpose and personalization benchmarks. The paper therefore frames ethical optimization as a distinct alignment layer that must be balanced against utility loss rather than as a universally beneficial post-processing step (Rahimi et al., 15 Feb 2025).

Synthetic persona realism is another major limitation. The HCI street-crossing study concludes that VLM personas mimic human response patterns at the level of group central tendency but lack behavioral variability and depth, making them more suitable for formative studies, pilot preparation, and limited data augmentation than for replacing human participants (Gui et al., 18 Feb 2026). The urban-perception analysis sharpens this point by showing that persona prompting has little effect on descriptive grounding but clearer effects on interpretive framing. A common misconception is therefore that persona conditioning uniformly changes “what the model sees”; the reported evidence suggests stronger effects on explanation, evaluation, and thematic emphasis than on basic scene description (Silva et al., 27 May 2026).

Privacy risk is an additional structural issue. Everyday personal videos allow VLMs to infer gender, age, education, marital status, monthly income, location, and occupation, and the benchmark study on inferential privacy reports that top VLMs achieve superhuman performance on every attribute relative to human evaluators. The same work further shows that full video outperforms static frames for several sensitive attributes, that prompting strategies can amplify leakage, and that model-generated explanations are not reliably faithful to the evidence actually driving the inference (Zhang et al., 4 Nov 2025). For PersonaVLM-style systems, this implies that personalization and inferential profiling are closely adjacent capabilities.

The broader research trajectory suggests two simultaneous directions. One is stronger contextual grounding, memory use, and compositional personalization, as exemplified by CoViP, USER-VLM 360, and PerPilot (Oh et al., 3 Feb 2026, Rahimi et al., 15 Feb 2025, Wang et al., 25 Aug 2025). The other is stricter auditing of bias, safety, privacy, and causal fidelity, especially in systems that move from captioning into embodied action or socially consequential user modeling (Cui et al., 2024, Zhang et al., 4 Nov 2025). A plausible implication is that future PersonaVLM systems will be judged less by whether they can merely name a personalized concept and more by whether they can do so with calibrated retrieval, bounded control authority, ethical verification, and demonstrable resistance to shortcut and profiling failures.