LumosX: Relate Any Identities with Their Attributes for Personalized Video Generation

Abstract: Recent advances in diffusion models have significantly improved text-to-video generation, enabling personalized content creation with fine-grained control over both foreground and background elements. However, precise face-attribute alignment across subjects remains challenging, as existing methods lack explicit mechanisms to ensure intra-group consistency. Addressing this gap requires both explicit modeling strategies and face-attribute-aware data resources. We therefore propose LumosX, a framework that advances both data and model design. On the data side, a tailored collection pipeline orchestrates captions and visual cues from independent videos, while multimodal LLMs (MLLMs) infer and assign subject-specific dependencies. These extracted relational priors impose a finer-grained structure that amplifies the expressive control of personalized video generation and enables the construction of a comprehensive benchmark. On the modeling side, Relational Self-Attention and Relational Cross-Attention intertwine position-aware embeddings with refined attention dynamics to inscribe explicit subject-attribute dependencies, enforcing disciplined intra-group cohesion and amplifying the separation between distinct subject clusters. Comprehensive evaluations on our benchmark demonstrate that LumosX achieves state-of-the-art performance in fine-grained, identity-consistent, and semantically aligned personalized multi-subject video generation. Code and models are available at https://jiazheng-xing.github.io/lumosx-home/.

• The paper introduces an explicit relational modeling framework to ensure strict face–attribute alignment in multi-subject text-to-video generation.
• It employs a novel data curation pipeline and relational attention modules (RSA and RCA) to achieve superior identity preservation and subject consistency compared to prior methods.
• Quantitative and human evaluations demonstrate robust state-of-the-art performance, especially in maintaining subject-attribute bindings as the number of subjects increases.

LumosX: Explicit Relational Modeling for Personalized Multi-Subject Video Generation

Introduction and Problem Formulation

LumosX introduces an explicit relational modeling framework to address the challenge of personalized multi-subject text-to-video (T2V) generation, particularly focusing on accurate face–attribute alignment and robust control over subject–attribute relationships. Current SOTA T2V systems (e.g., Diffusion Transformers/DiTs such as Wan2.1) achieve strong photorealism but frequently fail in scenarios involving multiple interacting subjects, often producing attribute entanglement, ambiguous face–attribute bindings, or degraded subject consistency due to an ill-posed mapping between user input references and video content. LumosX systematically solves these weaknesses by integrating (i) a data curation pipeline that yields explicit face–attribute bindings and relational priors, and (ii) relational attention modules that inject these dependencies into the generative process.

Figure 1: LumosX provides flexible, fine-grained multi-subject video generation, supporting diverse personalized foregrounds and backgrounds with strict face–attribute mapping.

Data Construction for Relational Multi-Subject T2V

Effective multi-subject face–attribute control is fundamentally bottlenecked by the lack of public datasets with explicit entity–attribute mappings. The LumosX pipeline first constructs such a dataset by curating raw multi-person videos via a three-stage process:

1. Semantic Caption and Detection: VILA generates dense captions; YOLOv9 extracts human subject locations.
2. Entity–Attribute Resolution: Qwen2.5-VL parses captions, resolving subject nouns to specific individuals and extracting (subject, attribute) tuples, leveraging vision priors to disambiguate co-referent entities.
3. Condition Image Derivation: Face crops and subject masks are extracted via detection, segmentation tools (SAM, GroundingDINO), and high-fidelity backgrounds are recovered using FLUX inpainting after dilating and removing subject/entity masks.

   Figure 2: Pipeline for dataset construction—captioning, entity retrieval/matching, and per-entity/background image processing yield explicit subject–attribute bindings and visual condition images.

The resulting dataset covers up to three subjects per video and is subject to heavy filtering for quality and consistency, yielding 1.57M curated samples and benchmarks that probe both single- and multi-subject controllability with ground-truth reference mapping.

Model Architecture: Relational Attention Mechanisms

Built atop Wan2.1 DiT, LumosX replaces generic positional and attention strategies with explicit relational modules to inscribe subject–attribute dependencies at the core of the model's attention operations.

Relational Self-Attention (RSA)

RSA incorporates:

• Relational Rotary Position Embedding (R2PE): Assigns group-preserving position indices to visual tokens—face and their attribute tokens within a subject share indices, enforcing group structure at the positional embedding level and supporting extrapolation to increased subject counts.
• Causal Self-Attention Mask (CSAM): Structures attention to only allow interactions:
  • Within each conditional branch (face–attribute combinations),
  • From the denoising tokens to reference tokens (unidirectional),
  • No leakage between subject groups.
  • 

  • Figure 3: Architecture overview—reference images are encoded, concatenated with denoising tokens, and processed with relational self- and cross-attention modules enabling strict subject–attribute association throughout generation.

Relational Cross-Attention (RCA)

RCA features:

• Multilevel Cross-Attention Mask (MCAM): Imposes a graded mask on the query–key cross-attention similarity matrix, defining strong (+1) correlation for matching subject and textual entity tokens, weak (−1) correlation for unrelated subject groups, and neutral (0) otherwise.
• Dynamic Scalability: Control hyperparameter $r$ tunes the strength of MCAM, and a computationally-optimized scaling mechanism minimizes performance overhead despite PyTorch’s lack of native numeric mask support.

  Figure 4: CSAM (left) and MCAM (right) attention masks schematically enforce intra-group and semantic correspondence constraints (example: enforcing association of subject 1’s face, “red shirt,” and “glasses”).

Experimental Results and Analysis

Benchmark and Metrics

LumosX is evaluated under two principal tasks:

• Identity-Consistent Generation: Using only facial references, measuring identity similarity (ArcSim/CurSim, ViCLIP-T).
• Subject-Consistent Generation: Using face, attribute, object and background references jointly, with metrics covering full-video semantics (ViCLIP-T, ViCLIP-V, Dynamics) and subject-region accuracy (CLIP-T/I, DINO-I, ArcSim, CurSim).

Quantitative Results

LumosX consistently achieves SOTA performance:

• Identity Preservation: Outperforms ConsisID and Concat-ID on single-face scenarios (ArcSim: 0.542 vs. 0.467/0.458); outperforms (or equals) SkyReels-A2 and Phantom on multi-face benchmarks (ArcSim: 0.510 vs. 0.382/0.508).
• Subject-Attribute Consistency: Surpasses strong baselines on subject-region metrics (e.g., ArcSim: 0.454 vs. 0.271/0.444 for SkyReels-A2, Phantom; DINO-I: 0.261 vs. 0.192/0.216).
• End-to-End Subject Differentiation: As the number of subjects increases, LumosX’s margin over baselines widens, validating the effectiveness of explicit relational modeling versus simple concatenation-based approaches.

  Figure 5: Qualitative comparison—LumosX maintains strict identity and subject–attribute alignment, while prior methods exhibit frequent face–attribute mismatches in multi-person scenarios.

Ablation Studies

Component-wise ablations reveal:

• R2PE is critical for mitigating inter-subject confusion (ArcSim increase +0.05 over base).
• CSAM further reduces cross-subject interference, improving both CLIP-T and ArcSim.
• MCAM, with $r=0.5$, optimizes face-attribute binding strength.
• Replacing FlashAttention with MagiAttention (for masking) has negligible computational overhead due to optimized implementation.
• Background inpainting realism is essential—FLUX outperforms Stable-Diffusion-2, as shown both quantitatively (FID) and by downstream effect on video generation quality.

  Figure 6: Visual effectiveness of individual model components—progressive improvements in face–attribute alignment from R2PE, CSAM, to full MCAM.

  

  Figure 7: High-quality, artifact-free video backgrounds generated via FLUX inpainting, demonstrating the necessity of realistic background conditioning.

Human Evaluation

A user study (N=30) confirms LumosX's clear preference in face–attribute alignment, face similarity, video naturalness, and prompt adherence compared to state-of-the-art baselines.

Figure 8: Human subject preference—LumosX dominates all evaluation dimensions in user study for multi-subject personalized video generation.

Scalability and Generalization

• R2PE’s group-based positional encoding supports scaling to subject counts well beyond training distributions with robust identity preservation—although practical limits are bounded by DiT/T2V base model capacity and dataset diversity.
• Relational modules are architecture-agnostic and transferable to other DiT-style (including multi-tower) backbones.
• Performance robustly generalizes to settings with only text-based attribute control and on public personalization (e.g., MSRVTT) and out-of-distribution online datasets.

Implications and Future Directions

LumosX's explicit relational modeling strategy constitutes a paradigm shift from implicit, text-only conditioning to a principled, entity-resolved framework for compositional video synthesis. Its advances are particularly salient for applications requiring multi-agent consistency, hierarchical control, or high-fidelity, user-driven personalization in media, digital avatars, and e-commerce.

Practically, further extension to four or more subjects is shown to be tractable with no evidence of sharp performance drop, though continued improvement will require both model scaling (larger DiT backbones) and longer-range positional encoding strategies, such as NTK-aware RoPE or expanded high-quality datasets for such configurations. Integration of dynamic, motion-aware constraints at both the dataset and modeling level is a promising avenue to ensure multi-subject semantic and spatiotemporal coherence in complex interactive scenes.

Conclusion

LumosX demonstrates that explicit, position- and group-aware relational attention is fundamentally necessary for high-fidelity, controllable multi-subject video generation. Through a custom-curated dataset and integrated R2PE, CSAM, and MCAM modules, LumosX achieves SOTA subject-attribute consistency, robust generalization across subjects and settings, and forms a strong foundation for the next generation of compositional, personalized video synthesis frameworks (2603.20192).