ID-LoRA: Identity-Driven Audio-Video Personalization with In-Context LoRA

Abstract: Existing video personalization methods preserve visual likeness but treat video and audio separately. Without access to the visual scene, audio models cannot synchronize sounds with on-screen actions; and because classical voice-cloning models condition only on a reference recording, a text prompt cannot redirect speaking style or acoustic environment. We propose ID-LoRA (Identity-Driven In-Context LoRA), which jointly generates a subject's appearance and voice in a single model, letting a text prompt, a reference image, and a short audio clip govern both modalities together. ID-LoRA adapts the LTX-2 joint audio-video diffusion backbone via parameter-efficient In-Context LoRA and, to our knowledge, is the first method to personalize visual appearance and voice in a single generative pass. Two challenges arise. Reference and generation tokens share the same positional-encoding space, making them hard to distinguish; we address this with negative temporal positions, placing reference tokens in a disjoint RoPE region while preserving their internal temporal structure. Speaker characteristics also tend to be diluted during denoising; we introduce identity guidance, a classifier-free guidance variant that amplifies speaker-specific features by contrasting predictions with and without the reference signal. In human preference studies, ID-LoRA is preferred over Kling 2.6 Pro by 73% of annotators for voice similarity and 65% for speaking style. On cross-environment settings, speaker similarity improves by 24% over Kling, with the gap widening as conditions diverge. A preliminary user study further suggests that joint generation provides a useful inductive bias for physically grounded sound synthesis. ID-LoRA achieves these results with only ~3K training pairs on a single GPU. Code, models, and data will be released.

• The paper introduces a unified audio-video diffusion model that jointly personalizes visual appearance and vocal identity using in-context LoRA.
• It employs a dual-stream transformer with negative RoPE encoding to effectively separate reference and target tokens, enhancing synchronization and identity retention.
• Empirical results demonstrate significant improvements in speaker similarity, lip sync, and environmental sound realism over cascaded approaches.

ID-LoRA: Identity-Driven Audio-Video Personalization with In-Context LoRA

Motivation and Overview

ID-LoRA introduces a unified framework for audio-visual personalization, addressing critical limitations in existing generative pipelines for personalized media. Prior systems typically decompose the task into separate stages: first generating video and subsequently synthesizing audio (or vice versa), which inherently decouples the two modalities. This design imposes a hard constraint: the audio model can only condition on the reference signal and the transcript, ignoring scene content and new context descriptions. Consequently, cascaded approaches are incapable of generating environment sounds or altering speaking style according to a prompt, resulting in outputs where speaker identity and prompt intention are weakly entangled.

ID-LoRA proposes a unified, joint audio-video diffusion model that takes as input a reference audio clip, a first-frame image of the target, and a free-form text prompt. These inputs allow simultaneous personalization of both visual appearance and vocal identity, with fine-grained control over speaking style and acoustic scene attributes. The core technical novelty is the adaptation of the LTX-2 backbone using parameter-efficient in-context LoRA, enabling zero-shot transfer of subject identity and robust prompt adherence across new environments and interactions.

Figure 1: ID-LoRA performs unified audio-visual personalization, preserving vocal identity and visual likeness while generating synchronized, prompt-controlled video and audio content.

Model Architecture and Technical Innovations

ID-LoRA is architected around a dual-stream diffusion transformer (DiT) backbone, specifically LTX-2, which fuses spatiotemporal video and temporal audio latents in a shared cross-modal latent space. The IC-LoRA framework is generalized here to handle multimodal identity transfer: audio and video latents from the reference and generation targets are concatenated, enabling the transformer backbone to attend jointly over both modalities.

A pivotal challenge in joint attention is distinguishing reference tokens from target generation tokens during the attention process. ID-LoRA resolves this by introducing negative temporal positions for reference tokens in RoPE positional encoding space, which guarantees robust demarcation without disrupting temporal structure within either block. This explicit separation is essential for disambiguating the reference signal and preventing information leakage during denoising.

Inference is further enhanced by “identity guidance,” a classifier-free guidance variant. This mechanism propagates and amplifies identity-specific cues (timbre, cadence) from the reference audio during generation, via difference extrapolation between conditioned and unconditioned model outputs, directly targeting identity retention without leaking prompt-driven environment cues into the reference stream.

Figure 2: The ID-LoRA pipeline encodes the reference audio and first frame into latents, separates token roles using negative RoPE positions, and performs unified generation with identity guidance.

Evaluation and Empirical Performance

ID-LoRA is benchmarked against a suite of cascaded pipelines (CosyVoice 3.0 + WAN2.2, VoiceCraft + WAN2.2, ElevenLabs + WAN2.2) as well as Kling 2.6 Pro, a commercial unified audio-video generation model. Datasets include CelebV-HQ (with easy and hard splits for varying intra- and cross-video challenges) and TalkVid for broad speaker and acoustic coverage.

ID-LoRA exhibits consistent, statistically significant improvements in speaker similarity and lip synchronization compared to all cascaded baselines. Notably:

• On the challenging cross-video split, speaker similarity improves by 24% over Kling, with the performance gap growing as reference-target conditions diverge.
• Lip synchronization (LSE-D, LSE-C) and audio prompt adherence (CLAP) scores are highest for ID-LoRA, confirming that unified joint modeling is necessary for robust audio-visual alignment.
• Face similarity is competitive but reveals a confound: methods that suppress animation boost metric scores, underpinning the necessity for complementary perceptual evaluations.

Human studies (9 AMT annotators per item) confirm these findings: ID-LoRA is preferred over Kling 2.6 Pro by 73% for voice similarity and 65% for speaking style, and by even larger margins over ElevenLabs-based cascaded pipelines.

Figure 3: Preference rates in human A/B studies demonstrate that annotators consistently favor ID-LoRA for voice similarity and speech style over Kling 2.6 Pro.

Automatic metrics are complemented with a rigorous MOS study focusing on physical environment sound interaction. Here, ID-LoRA's joint modeling enables accurate synthesis of diegetic sounds keyed to on-screen events, substantially outperforming Kling 2.6 Pro in scenario fidelity and reducing inter-annotator variance.

Figure 4: ID-LoRA scores higher on 8/10 environment sound interaction scenarios in MOS studies, validating its unified inductive bias for grounded sound synthesis.

Ablations and Analysis

Ablation experiments clarify the contribution of each architectural decision:

• Negative temporal positions are critical—fallback to standard PE significantly degrades synchronization and speech intelligibility (WER increases from 0.113 to 0.252).
• Identity guidance is central to speaker identity preservation. Without it, speaker similarity drops from 0.477 to 0.438.
• Hyperparameter sensitivity studies on identity guidance scale confirm an optimal tradeoff curve, with increased similarity at moderate scales and diminishing returns/overfitting for extreme extrapolation.

Figure 5: Qualitative examples for the MOS test: varying environment sounds are reconstructed controlling for physical scene and speaker, demonstrating high fidelity and prompt control.

Theoretical Implications and Future Directions

By coupling modalities in a shared conditional generative backbone, ID-LoRA accentuates the theoretical advantage of unified architectures for multimodal alignment and cross-modal transfer. The introduction of negative temporal positions represents an extensible technique for sequence segmentation in transformer-based models. The classifier-free guidance variant further generalizes the concept of prompt influence to arbitrary identity representations, potentially benefitting other conditional generative tasks.

In practice, ID-LoRA's ability to synthesize both environment and interaction-driven sounds—beyond mere speech cloning—portends models that can reason about diegetic consistency and scene-level semantics, approaching more holistic “world simulation” objectives. Future work is likely to extend in-context personalization to complex scenarios: compositional, multi-person audiovisual synthesis, cross-lingual transfer, and fine-grained control of non-speech vocal attributes.

Conclusion

ID-LoRA advances the state-of-the-art in audio-visual personalization by enabling text-prompted, jointly synthesized video and audio from minimal subject references. The adoption of in-context LoRA, negative RoPE positioning, and identity-guided inference allows for robust, prompt-controllable content creation that surpasses both open and proprietary baselines in most automatic and subjective metrics. The model’s architecture provides a principled route for further innovation in large-scale, multimodal generation, specifically in settings demanding both cross-modal synchronization and subject identity preservation.

Figure 6: In A/B studies, annotators select the more similar voice; ID-LoRA's unified attention mechanism yields outputs that are perceived as closer to the reference.