PersonaBooth: Motion Personalization

• PersonaBooth is a multi-modal framework that synthesizes text-driven motion while preserving the unique stylistic cues of an actor using diffusion-based techniques.
• It leverages contrastive learning, adaptive self-attention, and context-aware fusion to integrate visual and textual features, ensuring high persona fidelity.
• The system sets new benchmarks in motion personalization by blending semantic directives with actor-specific identity, demonstrated on the PerMo dataset.

PersonaBooth is a multi-modal personalization framework designed for text-driven synthesis of motions that exhibit both semantic and identity-specific characteristics. It targets the novel task of Motion Personalization: generating entirely new motion sequences aligned with free-form text descriptions, but rendered in the idiosyncratic style (the "persona") of a specific actor, as inferred from a small number of atomic motion samples. The PersonaBooth system advances diffusion-based motion generative models through innovations in dataset design, multi-modal adaptation, contrastive learning, and input fusion, establishing new benchmarks for personalized motion generation and challenging prior style transfer approaches (Kim et al., 10 Mar 2025).

1. Task Definition and Motivation

PersonaBooth introduces Motion Personalization, in which a set of basic motion clips (e.g., "walk," "wave") for a given actor (the persona) is used alongside a free-form text prompt to drive a pretrained diffusion model that generates novel behaviors reflecting both the semantic intent of the text and the unique stylistic patterns of the individual actor. For example, given several clips of “walking” and “waving” by actor $A$ and a prompt such as “A person jumps forward and waves,” PersonaBooth aims to synthesize a “jump and wave” sequence matching actor $A$'s style.

The task is motivated by the need for personalized, avatar-driven experiences in virtual environments and the desire to facilitate text-driven motion authoring without requiring extensive actor-specific motion capture for each new action.

2. The PerMo Dataset

To enable research in motion personalization, PersonaBooth introduces the PersonaMotion (PerMo) dataset:

• Actors and Styles: Five professional motion-capture actors (2F, 3M) each perform 34 distinct styles (e.g., Ballerina, Drunken, Old, Happy), creating 170 persona-style combinations.
• Content Diversity: For each persona, 10 atomic contents (e.g., kick, punch, walk, jump) are captured, with 4–5 repetitions per action.
• Scale and Representation: After cropping and augmentation, PerMo provides over 13,200 motion clips in SMPL-H format (263D per frame, encoding joint rotations, positions, velocities, and foot contacts), with 20–30 textual descriptions per clip.
• Preprocessing: Marker-based motion capture streams (41 OptiTrack markers) are converted to SMPL-H meshes using the MoSh++ pipeline, with per-persona body shape optimization.

PerMo’s design ensures high intra-persona variability and robust coverage for data-hungry generative models, supporting the learning of both generic and unique stylistic cues.

3. Model Architecture and Finetuning Strategy

PersonaBooth builds upon a pretrained 50-step Motion Diffusion Model (MDM), $\mathcal D$, and introduces multi-modal adaptation to integrate persona-specific information:

• Motion Encoding: Motions $M \in \mathbb{R}^{f \times 263}$, with $f$ frames.
• Pretraining: MDM learns a denoising score function $\mathcal D$ conditioned on CLIP-derived text embeddings, with a loss

$$L_{D} = \mathbb{E}_{M^0,t}\left[ \|M^0 – \mathcal D(M^t,t,C)\|^2_2 \right] + L_{geo}.$$

PersonaBooth augments the model as follows:

• Visual Adaptation: Adaptive self-attention layers are injected into each UNet block. For input $z$ at block $b$, time step $t$:

$A$0

where $A$1 is the extracted visual persona feature and Adapt is an attention layer.

• Textual Adaptation: A learnable persona token $A$2 is prepended to the text prompt, and its embedding is replaced with $A$3. The personalized prompt representation is

$A$4

where $A$5 is the frozen CLIP text encoder.

• Contrastive Persona-Cohesion Loss: To enforce consistency among samples from the same persona, a supervised contrastive loss $A$6 is applied to persona projections $A$7:

$A$8

with indices $A$9 for samples of the same persona and $\mathcal D$0 as temperature.

• Classifer-Free Guidance: During training, persona and text features are randomly dropped with 10% probability. At inference, predictions are mixed using a balance coefficient $\mathcal D$1 and guidance weights $\mathcal D$2 for text and vision conditions.
• Context-Aware Fusion (CAF): When multiple motion samples $\mathcal D$3 are given, PersonaBooth uses similarity scoring to weight extracted persona features, forming fused representations $\mathcal D$4, $\mathcal D$5 by a softmax over top-$\mathcal D$6 samples selected according to their relevance to the prompt.

All modifications preserve the frozen backbone weights; adaptation uses lightweight, learnable components.

4. Training and Optimization

The PersonaBooth training regime employs:

• Optimizer: AdamW, learning rate $\mathcal D$7, batch size 64.
• Training Schedule: 500 epochs.
• Loss Function: Total loss

$\mathcal D$8

with $\mathcal D$9.

• Control and Inference: s_t, s_v set to 1 (training), 0.3 (inference); guidance strengths $M \in \mathbb{R}^{f \times 263}$0, $M \in \mathbb{R}^{f \times 263}$1; balance $M \in \mathbb{R}^{f \times 263}$2 for single input, 0.5 for multiple inputs.

This regime is designed to enable both text-motion alignment, persona coherence, and the integration of multiple motion inputs without catastrophic forgetting.

5. Evaluation and Empirical Performance

PersonaBooth is evaluated on both the PerMo benchmark and the established 100Style benchmark, using the following metrics:

• Frechet Inception Distance (FID): Assesses motion realism, lower values indicating higher fidelity.
• R-Precision (Top-$M \in \mathbb{R}^{f \times 263}$3 Retrieval Accuracy): Measures text–motion alignment.
• Diversity: Variance among generated samples.
• Persona Recognition Accuracy (PRA): Assesses how well a pretrained classifier can recover the correct persona from generated motion.

Empirical results:

Method & Setting	FID (↓)	R-Prec@3 (↑)	Diversity (↑)	PRA / SRA (%) (↑)
PersonaBooth / PerMo	3.18	0.33	7.74	18
PersonaBooth+CAF/PerMo	2.95	0.39	8.12	—
PersonaBooth/100Style	3.27	0.40	7.9	64.5
MCM-LDM (prior art)	9.16	—	—	—

Qualitatively, PersonaBooth outputs exhibit both prompt-appropriate semantic structure (e.g., “hop forward and turn in the air”) and distinctive individual stylistic features.

6. Connections, Limitations, and Extensions

PersonaBooth establishes the first robust framework for motion personalization by combining:

• A large-scale, persona-oriented dataset (PerMo).
• Integrated multi-modal adaptation (visual, textual).
• Contrastive learning for persona consistency.
• Classifier-free guidance for robust generation.
• Context-aware fusion for multi-sample personalization.

A notable limitation is the short duration of PerMo clips, sometimes leading to stationary artifacts at sequence ends; future directions include dynamic-length modeling and action-level fusion for compound prompts (e.g., “run then jump”).

PersonaBooth’s approach contrasts with tuning-free identity preservation methods from the text-to-image domain—such as FaceStudio, which uses a feed-forward hybrid guidance and skip fine-tuning entirely (Yan et al., 2023), and PhotoMaker, which leverages stacked ID embedding within cross-attention (Li et al., 2023). While these methods excel in data efficiency and speed for image synthesis, PersonaBooth's contribution is the first to address motion with high persona fidelity, offering a foundation for cross-modal personalization research.

Potential architectural enhancements indicated by recent work include decoupling tokens for understanding, generation, and editing, as shown in OmniPersona (Zhong et al., 11 Jan 2026), or introducing lightweight knowledge-replay at inference to improve attribute control. However, as of its publication, PersonaBooth remains state-of-the-art (particularly in FID and alignment metrics) for motion personalization tasks.