D-Rex : Diffusion Rendering for Relightable Expressive Avatars
Abstract: We present D-Rex, a person-specific framework for photorealistic, relightable, expressive, and animatable full-body human avatars with free-viewpoint rendering. Existing methods for relightable full-body avatars rely on explicit 3D intrinsic decomposition with analytic reflectance models, which require accurate geometry registration and careful optimization to capture realistic light transport effects. This tight coupling of relighting with avatar modeling has hindered expressiveness: to our knowledge, no existing method demonstrates strong facial animation alongside relighting, limiting applicability in telepresence, gaming, and virtual production. We propose to decouple relighting entirely from avatar modeling by treating it as an image-space post-process: a learned translation from flat-lit, albedo-like renderings to a target HDR illumination. To this end, we leverage the strong generative prior of a pre-trained video diffusion relighting model, fine-tuned via LoRA on paired flat-lit and relit frames captured in a light stage. The flat-lit driving frames are produced by an independent expressive full-body avatar framework trained under white-light conditions, requiring no modification to support relighting, making D-Rex directly applicable to any white-light avatar system. We demonstrate that D-Rex enables view- and temporally consistent relighting while faithfully preserving expressive motion and fine-grained facial detail, outperforming physically-based relightable avatar baselines. Project page is https://vcai.mpi-inf.mpg.de/projects/DRex/
Summary
- The paper demonstrates that decoupling relighting from avatar construction via a learned image-space translation yields photorealistic, relightable avatars.
- It leverages LoRA fine-tuning on a video diffusion prior, achieving strong temporal, view, and illumination consistency with improved PSNR, LPIPS, and SSIM metrics.
- The study provides practical insights on multi-illumination capture, avatar conditioning, and trade-offs in scalability versus color fidelity through comprehensive ablation studies.
Diffusion Rendering for Relightable Expressive Avatars: An Expert Review
Motivation and Background
The "D-Rex: Diffusion Rendering for Relightable Expressive Avatars" (2604.27871) introduces a paradigm shift in the modeling of photorealistic, relightable, animatable human avatars for free-viewpoint rendering. The longstanding challenge in avatar research has been jointly achieving realistic relighting, expressive facial animation, and dynamic clothing within a unified framework. Traditional approaches rely on explicit 3D intrinsic decomposition and analytic reflectance modeling, which require accurate geometry registration and are tightly coupled to avatar modeling. This coupling restricts expressiveness and complicates optimization, particularly for realistic light transport effects and robust expression control.
D-Rex addresses these limitations by decoupling relighting from avatar construction. Relighting is formulated as an image-space translation: a learned mapping from albedo-like renderings, generated by a white-light avatar system, to photorealistic images under arbitrary target high dynamic range (HDR) illumination, leveraging generative video diffusion priors.
Figure 1: D-Rex renders relightable avatars by translating albedo-like frames (driven by pose, facial expression, view, and HDR map) to photorealistic outputs via fine-tuned video diffusion.
Methodology
Data Acquisition and Preprocessing
D-Rex relies on a person-specific, multi-illumination capture protocol, acquiring synchronized paired flat-lit and HDR-illuminated frames across multiple viewpoints and expressions. High-resolution geometry and expression parameters are derived via Captury, Sapiens foundation model, skinned template meshes, and commercial 3D scanning methods. Segmentation, pose refinement, and expression regression (FAN, Mediapipe) ensure accurate avatar conditioning for subsequent pipeline stages.
Figure 2: Illustration of the multi-camera spherical dome capture setup and dataset diversity.
Albedo-Driven Expressive Avatar
The expressive avatar component utilizes EVA [junkawitsch2025eva], employing a deformable template and disentangled UV Gaussian appearance layer to produce albedo-like renderings with arbitrary pose/expression/viewpoint control. Training incorporates high-resolution crops and modified loss functions to mitigate background artifacts from segmentation imprecision.
Diffusion Relighting Module
The core innovation lies in the adaptation of Cosmos DiffusionRenderer [liang2025DiffusionRenderer], a large-scale video diffusion model, fine-tuned via LoRA only on forward-rendering weights using flat-lit to relit frame pairs. The base color channel of albedo renderings is used as input, with ancillary G-Buffer channels discarded. HDR environment maps are encoded and concatenated for conditioning. The model is trained to minimize reconstruction loss on noisy latents, preserving temporal and spatial consistency within 57-frame video chunks. Empirically, LoRA-based fine-tuning on even single-image frame pairs is sufficient to retain longitudinal consistency in output videos.
Figure 3: D-Rex pipeline overview—expressive avatar rendering and diffusion relighting decoupled for flexible and scalable avatar production.
Experimental Evaluation
Qualitative and Quantitative Analysis
D-Rex is quantitatively benchmarked against adapted baselines:
- MeshAvatar (PBR): Traditional relightable template-based avatar with explicit BRDF modeling.
- MeshAvatar + DiffusionRenderer: Flat-lit MeshAvatar outputs post-processed via diffusion.
- EVA + IC-Light: Zero-shot in-the-wild diffusion-based relighting method.
- D-Rex (Ours): EVA-driven input, LoRA-finetuned diffusion relighting.
Across several metrics (PSNR, LPIPS, SSIM), D-Rex consistently outperforms all baselines in novel view/motion/light settings. Notably, D-Rex achieves PSNR 26.3–28.7, LPIPS 0.065–0.094, and SSIM 0.937–0.953, surpassing physically-based approaches by a substantial margin, particularly on expressive faces and loose clothing.
Figure 4: Qualitative comparison against constructed baselines—D-Rex preserves facial details and achieves realistic relighting beyond MeshAvatar and IC-Light.
Consistency and Generalization Studies
D-Rex demonstrates strong temporal, view, and illumination consistency within inference chunks. Ablations reveal that full model fine-tuning and LoRA adaptation from pretrained priors both converge to similar performance, whereas training from random initialization fails to recover realistic relighting. Zero-shot diffusion yields poor results due to domain gap.
Figure 5: View and illumination consistency across inference runs, validating robustness of D-Rex's fine-tuned diffusion relighting.
Generalization experiments indicate plausible relighting across subjects even when not subject-specific, albeit with visible color shifts. This opens avenues for scalable, general human relighting given sufficiently diverse training data.
Figure 6: Cross-subject training—non subject-specific models yield reasonable results but are limited by color fidelity.
Ablation Studies
- Background Masking: Masking out backgrounds during training marginally improves color accuracy (PSNR/LPIPS/SSIM), but training without masks is more scalable (0s preprocessing).
- Enhancement Model: Training the diffusion module for enhancement (from EVA to relit) can correct artifacts but induces more hallucinations, particularly in untracked regions (e.g., gaze).
- Prior Importance: LoRA-based adaptation with pretrained diffusion priors attains competitive results with significant GPU efficiency versus full model retraining.
Figure 7: Ablations across training strategies—only fine-tuned models (LoRA or full) recover convincing relighting.
Figure 8: Removing background elements demonstrates improvement in color fidelity; unmasked training remains perceptually robust.
Limitations and Implications
The principal limitation of D-Rex is its dependency on light stage capture for paired flat-lit/relit supervision, constraining accessibility. Additionally, fine-tuned diffusion models require minimal subject motion between paired frames for stable inference; artifacts from misalignment manifest as subject drift or blurred regions, predominantly in hands and dynamic clothing.
Figure 9: Artifacts from misaligned frame pairs—motion-induced misalignment degrades output quality, particularly for high-motion subjects.
Despite these constraints, D-Rex demonstrates that diffusion-based, decoupled relighting can robustly bypass physically-based decomposition, achieving superior expressivity and visual fidelity. This modular architecture is conducive to integration with any albedo-white-light avatar pipeline, scaling the production of relightable avatars for telepresence, virtual production, and immersive VR/AR.
Future Directions
Large-scale data collection, e.g., leveraging datasets such as HumanOLAT [teufelgera2025HumanOLAT], can enable generalization across subjects, reducing the necessity for person-specific fine-tuning. Integration with emerging frame alignment techniques [digitalbipack2025yu] and more flexible generative diffusion frameworks is likely to further mitigate misalignment artifacts and broaden accessibility. Enhanced cross-domain priors may also facilitate training on in-the-wild data, supporting practical deployment in unconstrained environments.
Conclusion
D-Rex advances state-of-the-art photorealistic avatar relighting by fully decoupling avatar modeling and relighting as a learned image-space translation, leveraging fine-tuned generative video diffusion priors. The framework achieves high numerical performance and perceptual realism, reliably preserving expressiveness and temporal/view consistency. By scaling data acquisition and refining diffusion adaptation protocols, D-Rex sets the stage for generalizable, accessible relightable avatars, with significant implications for the virtual human rendering pipeline and downstream applications.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What is this paper about?
This paper introduces D‑Rex, a way to make super-realistic digital people (avatars) that can:
- move their whole bodies,
- show detailed facial expressions,
- be viewed from any camera angle,
- and look correct under different kinds of lighting (like morning sun, indoor lamps, neon signs).
The big idea: instead of doing complicated physics to simulate light on a 3D human, D‑Rex treats lighting as a smart “photo filter” step applied to a simple, neutral-looking avatar image. This makes the system simpler, faster to train, and better at faces and clothes.
What questions are the researchers asking?
They focus on three easy-to-understand goals:
- Can we change the lighting on a moving, expressive full‑body avatar so it still looks real from any camera angle?
- Can we keep the face and small details (like eyes, hair, and cloth folds) looking stable over time, frame after frame?
- Can we do this without relying on tricky, fragile physics models that need perfect 3D geometry?
How did they do it?
They split the problem into two steps, like a two-layer cake:
- Make a “flat-lit” avatar that looks like it’s under plain white light (no dramatic shadows).
- Add realistic lighting on top using a powerful video model that has learned how light behaves in the real world.
The data they collect
They filmed real people inside a “light stage” (a dome with many cameras and hundreds of lights all around). This let them capture pairs of frames:
- one under plain white light (flat-lit),
- one under a specific, complex lighting (like an HDR environment map — a 360° picture of light in a scene).
They did this for four different people showing lots of body motion and facial expressions.
Step 1: Build the base avatar (the “flat-lit” version)
They used an existing avatar system (called EVA) that can:
- take body pose and facial expression as input,
- and output a clean, flat-lit render of the person from any camera angle.
Think of this as a coloring-book version of the person: the right colors and details, but with simple lighting.
Step 2: Teach a relighting “translator” (the diffusion model)
They trained a video diffusion model (a kind of smart AI that learns to improve and transform images over time) to convert the flat-lit avatar images into relit images that match any target lighting.
To do this efficiently, they:
- started from a strong, pre-trained relighting model (it already “knows” a lot about how light looks),
- fine-tuned it with LoRA, a lightweight “plug-in” that tweaks the model using relatively little data and compute,
- used the captured pairs (flat-lit → relit) so the model learns the translation.
In simple terms: the model learns, “When the flat-lit image looks like this, and the lighting should be like that, here’s how the final image should look.”
Helpful mini‑glossary
- HDR environment map: a 360° “light bubble” of the world that tells you where light comes from and how bright it is.
- Albedo (or flat-lit): the basic color/texture of a surface without shadows or shiny highlights.
- Diffusion model: an AI that starts with noisy images and learns to clean them up into realistic results (like a master photo restorer).
- LoRA: a compact way to fine‑tune big models by adding small adapter layers instead of changing the whole model.
- Light stage: a special studio with many lights and cameras to capture people under controlled lighting.
What did they find?
- Realistic and stable results: D‑Rex produces photorealistic videos where lighting, camera views, and motion stay consistent across frames.
- Detailed expressions: It preserves fine details in faces and clothing (something many older relighting methods struggle with).
- Better than physics‑based baselines: Compared to methods that try to compute exact light physics on 3D meshes, D‑Rex looks more realistic and is more robust, especially for expressive faces and loose clothing.
- Efficient training: With LoRA fine-tuning and only paired flat-lit/relit images, the model learns to relight full-body avatars without re-engineering the avatar itself.
- Works across time: Even when trained on single image pairs, the video model keeps things steady over short video chunks, and simple blending smooths longer clips.
Why this matters: Stable, believable lighting is crucial for telepresence (video calls with lifelike avatars), games, movies, and VR/AR, where the same character needs to look good in many scenes and angles.
What are the implications?
- Plug-and-play relighting: Any avatar system that can produce a flat-lit render could use D‑Rex as a relighting “add-on.” You don’t have to rebuild the avatar.
- Less reliance on perfect 3D: Because relighting happens in image space, you avoid the fragile step of estimating exact materials and geometry.
- Scales with data: The method learns from examples. With more diverse training pairs, future models might handle many people without separate fine-tuning.
Limitations and what’s next
- Needs special capture right now: Getting the paired flat-lit/relit data is easiest in a light stage, which not everyone has.
- Sensitive to motion misalignment: The flat-lit and relit frames should be closely matched in time; big shifts can confuse training.
- Future directions: Gather larger, more varied datasets and aim for a general model that works on new people without person‑specific training.
In short: D‑Rex shows that treating lighting as a smart, learned “post‑process” on top of a simple avatar makes lifelike, expressive, relightable digital humans much more practical.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a concise list of missing pieces, uncertainties, and unexplored directions that emerge from the paper. Each point is phrased to be concrete and actionable for future work.
- Person-specific requirement and light-stage dependency: How to remove or reduce reliance on subject-specific, multi-illumination light-stage captures (e.g., via unpaired/weakly paired data, synthetic augmentation, or domain adaptation) while preserving quality and consistency.
- Sample efficiency and data scaling: What is the minimal number of views, frames, and lighting conditions needed per subject for high-quality LoRA fine-tuning; how LoRA rank/placement and dataset size trade off against quality and consistency.
- Generalization beyond four subjects: Effectiveness across diverse skin tones, hair types, accessories, and clothing materials (e.g., shiny fabrics, translucent materials), including fairness and robustness analyses.
- Long-range temporal and multi-view consistency without blending: Eliminating cross-inference variation (currently mitigated by 32-frame blending) and guaranteeing stable long videos and simultaneous multi-camera outputs with a single consistent latent trajectory.
- Sensitivity to WL–RL misalignment: Robust training under motion-induced misalignments between flat-lit and relit frames (e.g., using optical-flow alignment, latent warping, or motion-conditioned fine-tuning), and quantifying tolerance to inter-frame shifts.
- HDR correctness and photometric fidelity: The environment maps are tone-mapped to LDR for conditioning; fidelity to true HDR intensities and exposure control is unquantified. Explore true HDR conditioning/output and evaluate energy-conserving, intensity-scaled responses.
- Physical controllability and relighting faithfulness: Beyond PSNR/LPIPS/SSIM, quantify how well directional cues, shadow placement/softness, and highlight behavior match the specified HDRI/OLAT (e.g., light-direction error, shadow mask IoU, highlight spec alignment).
- Input representation limitations: Only the base color channel is provided to the forward renderer; normals/roughness/metalness are set to zero. Investigate conditioning on EVA-derived normals, depth, or roughness proxies to improve view consistency, specular control, and 3D awareness.
- Entanglement between relighting and enhancement: The diffusion model sometimes “enhances” details (e.g., faces, hands), risking identity drift. Develop constraints/losses (e.g., identity, feature-matching, color constancy) to decouple lighting changes from appearance edits.
- Real-time feasibility: Inference latency, throughput, and scaling to multi-view rendering are not reported. Explore acceleration (e.g., distillation, consistency models, low-latency schedulers) for telepresence and interactive use.
- Dynamic illumination over time: Performance under time-varying HDRIs or moving light sources is not evaluated. Assess temporal stability and responsiveness under smoothly or abruptly changing lighting.
- Scene interaction and shadow casting onto backgrounds: The approach focuses on subject relighting and does not model subject–environment interactions (e.g., ground shadows, interreflections). Extend to joint subject–scene relighting and consistent compositing.
- Robustness to segmentation/matting errors: Although background masking improves color fidelity, preprocessing is costly and the method remains sensitive to mask quality (especially under underexposed frames). Develop fast, robust matting or segmentation-in-the-loop training.
- Eye gaze and micro-expressions: EVA lacks gaze tracking and the enhancement variant hallucinated eyes. Integrate gaze estimation/capture and refine facial micro-expression modeling to avoid artifacts in close-ups.
- Extreme motions, occlusions, and fast dynamics: Stability under rapid motion, self-occlusions (e.g., hands over face), and strong motion blur is not quantified; current training prefers slow movement. Establish benchmarks and augment training for these cases.
- Multi-view training and 3D consistency constraints: Fine-tuning used single-image frame pairs; multi-view/multiframe training with geometric priors (e.g., epipolar constraints, 3D Gaussians, depth-conditioned diffusion) might further improve cross-view coherence.
- Interpretable lighting control: While HDRIs/OLATs are supported, the method does not expose disentangled controls (e.g., key/rim/fill intensity). Investigate decomposition into controllable bases (e.g., OLAT basis learning) for user-friendly lighting edits.
- Photometric calibration pipeline: The mapping from physical LED intensities/exposure to the tone-mapped conditioning used by the diffusion model is not detailed. Provide calibration procedures and evaluate their impact on relighting accuracy.
- Identity preservation under strong expressions/clothing dynamics: Quantify identity and texture stability (e.g., face embedding similarity) across extreme expressions, poses, and garment motions; identify failure modes.
- General relighting without per-subject fine-tuning: The conclusion hints at training on large-scale multi-illumination data for subject-agnostic relighting. The required scale, diversity, and architecture changes (e.g., class/subject conditioning) remain open.
- Multi-person and interaction scenarios: The approach is single-subject; handling multiple people, mutual occlusions, and inter-person light transport is unexplored.
- Failure case taxonomy and benchmarks: Beyond select visuals and metrics, a systematic benchmark for relightable avatars (with ground-truth lighting parameters, dynamic lights, and diverse materials) would enable rigorous comparison and diagnosis.
Practical Applications
Practical Applications of D-Rex (Diffusion Rendering for Relightable Expressive Avatars)
D-Rex introduces a person-specific, diffusion-based, image-space relighting pipeline that decouples relighting from avatar modeling. It leverages LoRA fine-tuning of a pre-trained video diffusion relighting model on paired flat-lit/relit frames, enabling photorealistic, view- and temporally consistent full-body avatars with expressive faces and dynamic clothing. Below are actionable applications organized by deployment horizon.
Immediate Applications
- Media & Entertainment — Virtual Production and VFX
- Use: Post-process relighting of digital doubles with expressive faces for film, TV, and commercials; “shoot once under white light” and relight to match different scenes, times of day, or set extensions.
- Tools/Workflows: Plugin or bridge for Unreal/Unity/Nuke (flat-lit avatar pass in, HDRI out); “Relightable Digital Double” service that takes EVA or any white-light avatar, plus HDR environment maps; shot-based batch relighting with 57-frame chunks and linear blending.
- Dependencies/Assumptions: Light-stage paired capture per actor; white-light avatar (e.g., EVA or equivalent) trained from flat-lit frames; LoRA fine-tuning of a pre-trained video diffusion renderer; HDR environment maps; high-end GPUs; licensing for DiffusionRenderer and avatar tech; not real-time.
- Gaming — Cinematics, Trailers, and Live-Ops Assets
- Use: Rapid relighting of character performances for marketing trailers and in-game cinematics, maintaining facial expressiveness and clothing detail.
- Tools/Workflows: Offline pipeline that ingests flat-lit cinematic passes and applies consistent HDRI relighting; asset libraries of HDRIs per franchise/location.
- Dependencies/Assumptions: Person- or character-specific fine-tuning; studio GPU capacity; not suitable for runtime in-engine use yet.
- Advertising & Marketing — Campaign Localization
- Use: Generate region-specific versions by adapting lighting to local contexts (e.g., “golden hour” vs. overcast city), minimizing reshoots while preserving expression and wardrobe details.
- Tools/Workflows: Cloud API for “flat-lit-to-HDRI” relighting; batch processing of talent digital doubles.
- Dependencies/Assumptions: Consent and rights to create/use a digital double; HDRI libraries; controlled capture.
- Telepresence for Creators and VTubing (Studio-Grade)
- Use: High-end avatar content creation for pre-recorded streams, music videos, and shorts with consistent relighting across camera cuts and takes.
- Tools/Workflows: Creator studio pipeline where a talent is captured once, then driven by tracked pose/expression and relit per scene.
- Dependencies/Assumptions: Access to capture stage; offline rendering; motion/expression drivers; compute budget.
- Previsualization and Lighting Design
- Use: DP/cinematographer tools to audition HDRIs on a performer’s digital double before lighting the set; compare lighting plans while preserving expressive facial cues.
- Tools/Workflows: Lighting pre-vis dashboard with HDRI library, digital double library, and camera path previews.
- Dependencies/Assumptions: Per-talent fine-tuning; non-real-time previews; alignment between virtual and physical HDR measurements.
- Academic Research — Generative Relighting and Inverse Rendering
- Use: Benchmarking and ablation studies on relighting without explicit intrinsic decomposition; studying LoRA vs. full-finetune tradeoffs; evaluating temporal/view consistency.
- Tools/Workflows: Reproducible pipelines for flat-lit-to-relit translation, dataset curation protocols, and evaluation under novel view/motion/light.
- Dependencies/Assumptions: Access to multi-illumination captures or partnerships with light-stage facilities; compliance with data-use permissions.
- Synthetic Data Generation — Vision under Varied Illumination
- Use: Produce photorealistic human sequences across lighting conditions for training or stress-testing pose, re-identification, or tracking models; augment edge cases (specular highlights, backlighting).
- Tools/Workflows: Data engine that varies HDRIs, poses, expressions, and viewpoints; automatic annotation pass-through from driving avatar.
- Dependencies/Assumptions: Ethical review; domain gap mitigation; disclosure when synthetic data is used.
- Museums and Cultural Heritage — Digital Performers and Docents
- Use: Pre-rendered guided tours with lifelike avatars whose lighting matches exhibit environments.
- Tools/Workflows: Offline content batches tailored to exhibit HDR captures.
- Dependencies/Assumptions: Consent/rights; on-site HDR capture; offline render timeline.
- Robotics and HRI Simulation
- Use: Generate diverse, physically plausible human visuals under challenging lighting to simulate robot perception in homes, hospitals, or factories.
- Tools/Workflows: Scenario generators that pair human motion scripts with environment HDRIs; integration with sim stacks (e.g., Isaac Sim).
- Dependencies/Assumptions: Offline generation; careful domain adaptation to sensor characteristics.
- Policy Pilots — Provenance and Disclosure
- Use: Test operationalization of provenance (watermarking, C2PA) in pipelines that generate photorealistic avatars; evaluate consent management for digital doubles.
- Tools/Workflows: Reference workflows with embedded provenance metadata and audit logs.
- Dependencies/Assumptions: Organizational buy-in; legal and ethical review; collaboration with standards bodies.
Long-Term Applications
- General-Purpose, Non–Person-Specific Relighting
- Use: “Relighting-as-a-Service” that works for arbitrary humans without per-person fine-tuning, enabling widespread deployment in content tools and consumer apps.
- Tools/Workflows: Foundation relighting model trained on large-scale multi-illumination datasets; lightweight personalization only if needed.
- Dependencies/Assumptions: Significant data scaling (incl. dynamic multi-illumination human datasets), robust generalization, and bias/coverage audits.
- Real-Time Telepresence and XR Collaboration
- Use: Live, relightable avatars in conferencing and AR/VR platforms that match lighting to the participant’s environment while preserving expressive nuance.
- Tools/Workflows: On-device or edge/cloud inference with low-latency streaming; compact diffusion variants or distillation; HDRI estimation from device sensors.
- Dependencies/Assumptions: Model compression/distillation; efficient HDR environment capture; network QoS; privacy-preserving pipelines.
- In-Engine Runtime for Games and Interactive Media
- Use: Physically plausible, temporally stable relighting of player-specific avatars at runtime (day-night cycles, weather changes) with expressive faces and garments.
- Tools/Workflows: Engine plugins (Unreal/Unity) with GPU-accelerated inference, caching, and LOD strategies; hybrid neural + PBR fallback.
- Dependencies/Assumptions: Significant inference speedups and memory optimizations; interoperability with engine shading pipelines.
- Consumer-Grade Capture and Personalization
- Use: Smartphone or webcam-based capture to create personal relightable avatars for social, fitness, or education apps, without requiring a light stage.
- Tools/Workflows: Few-shot or self-supervised pipelines; commodity HDR estimation; guided capture UX; auto-alignment and motion stabilization.
- Dependencies/Assumptions: Advances in alignment under motion, self-calibration, and robustness to household lighting noise.
- E-Commerce and Fashion — Environment-Aware Virtual Try-On
- Use: Try-on experiences where garments on personal avatars are relit to match the shopper’s home, store, or outdoor scene; consistent product appearance across contexts.
- Tools/Workflows: Retail SDKs for relightable avatars; HDRI capture from device cameras; garment-specific appearance priors.
- Dependencies/Assumptions: Robust generalization to varied fabrics and accessories; fit/physics coupling; privacy, consent, and IP for user avatars.
- Healthcare and Training Simulations
- Use: High-fidelity, expressive patient/clinician avatars in VR training with lighting that mirrors real clinical environments; exposure therapy with realistic visuals.
- Tools/Workflows: Healthcare XR modules with controlled lighting scenarios; integration with standardized curricula.
- Dependencies/Assumptions: Clinical validation; accessibility and safety guidelines; strict privacy controls.
- Standards, Governance, and Regulation
- Use: Industry-wide frameworks for consent, ownership, and usage of relightable digital doubles; labeling and watermarking mandates for synthetic content; energy/transparency reporting.
- Tools/Workflows: Contract templates for talent digital doubles; provenance standards (C2PA), watermarking, and audit toolkits.
- Dependencies/Assumptions: Multistakeholder coordination; legal harmonization across jurisdictions; enforcement mechanisms.
- Security and Forensics
- Use: Detection tools trained to identify diffusion-based relighting artifacts and provenance breaks; risk assessments for misuse (deepfakes).
- Tools/Workflows: Forensic classifiers co-developed with pipeline access; red team/blue team evaluations on relighted content.
- Dependencies/Assumptions: Access to training artifacts and model fingerprints; evolving adversarial tactics.
- Data Ecosystems and Marketplaces
- Use: Curated HDRI libraries tied to scene semantics; open multi-illumination human datasets; LoRA “relighting heads” marketplaces for specific styles/environments.
- Tools/Workflows: Dataset governance and licensing models; bias and coverage dashboards; interoperable environment map formats.
- Dependencies/Assumptions: Sustainable data stewardship; incentives for contributors; ethical review of content rights.
- Green AI and Compute Efficiency
- Use: Policies and tooling to reduce the carbon footprint of training/fine-tuning/inference (e.g., LoRA sharing, distillation, scheduling).
- Tools/Workflows: Energy dashboards; model cards reporting compute/energy; cloud credits tied to efficiency targets.
- Dependencies/Assumptions: Vendor support; accepted measurement standards; user demand for sustainability.
Notes on feasibility and assumptions common across applications:
- Current pipeline is person-specific and offline: requires paired flat-lit/relit captures (light stage), a trained white-light avatar (e.g., EVA), and LoRA fine-tuning of a pre-trained video diffusion renderer. Inference is chunked (57 frames) with overlap blending.
- High-quality results depend on small motion drift between paired frames and good segmentation/masks; background masking improves color fidelity but increases preprocessing time.
- Hardware requirements include multi-GPU training and at least one high-end GPU for LoRA fine-tuning and inference; licensing constraints may apply to the pre-trained diffusion renderer and avatar frameworks.
- Ethical, legal, and privacy considerations are central: consent for digital doubles, content provenance/watermarking, and guardrails against misuse are necessary for responsible deployment.
Glossary
- A-pose: A standardized reference pose with arms angled outward used for scanning and rigging human models. "a high-detail template mesh $V_{\text{Templ.}$ of the subject in A-pose via a commercial scanner setup"
- albedo: The view- and lighting-independent base color of a surface, used here as “albedo-like” flat-lit appearance for relighting. "a learned translation from flat-lit, albedo-like renderings to a target HDR illumination."
- analytic BRDF: A closed-form bidirectional reflectance distribution function used in physically-based rendering to model material reflectance. "perform physically-based rendering with an analytic BRDF."
- area light: A light source modeled with finite area producing soft shadows and more realistic illumination than a point light. "relit under a single directional area light."
- denoising function: The learned network module in diffusion models that removes noise from latent variables during sampling. "the denoising function employed by the model"
- DiT (Diffusion Transformer): A transformer-based architecture used for diffusion models to process images or videos. "two DiT-based modules"
- diffusion relighting: Using diffusion models to translate images/videos from one lighting condition to another. "a pre-trained video diffusion relighting model"
- domain gap: A mismatch between training and deployment data distributions that degrades performance without adaptation. "Due to the domain gap, without fine-tuning the diffusion model is unable to perform realistic relighting on the full-body human."
- environment map (HDR): A spherical illumination representation capturing surrounding light, often high dynamic range, used to light virtual scenes. "a High Dynamic Range (HDR) environment map"
- EVA (Expressive Virtual Avatars): A person-specific, controllable avatar framework used to produce expressive, albedo-like renders. "D-Rex adopts Expressive Virtual Avatars (EVA)"
- FLAME: A parametric head/face model whose expression parameters drive facial deformations. "FLAME expressions"
- Fresnel reflectance: The angle-dependent reflectance at a surface interface that increases at grazing angles. "Fresnel reflectance and specular highlights"
- G-Buffer: A set of per-pixel material and geometric buffers (e.g., albedo, normal) used for deferred rendering and learned rendering. "G-Buffers (albedo, metalness, roughness, and normal)"
- HDR (High Dynamic Range): Imaging that captures a wide luminance range, enabling realistic lighting and reflections. "High Dynamic Range (HDR)"
- HDRI (High Dynamic Range Imaging): Using HDR images for environment lighting in rendering. "Results for direct HDRI relighting"
- image-space relighting: Relighting that operates directly on rendered or captured images rather than 3D reflectance components. "via image-space relighting"
- intrinsic decomposition: Factorizing images into components like albedo and shading to separate material and illumination. "explicit 3D intrinsic decomposition"
- latent representations: Compressed feature vectors (often from a VAE) that encode images or videos for diffusion. "encoded to their corresponding latent representations"
- light stage: A dome-like capture rig with many controllable lights and cameras used to acquire multi-illumination data. "captured in a light stage."
- linear blending: Overlapping and averaging adjacent video chunks to smooth transitions and reduce temporal variation. "render perceptually convincing long-form videos via linear blending using an overlap of $32$ frames."
- LoRA (Low-Rank Adaptation): A parameter-efficient fine-tuning technique that injects low-rank weight updates into a frozen model. "fine-tuned via LoRA"
- LPIPS (Learned Perceptual Image Patch Similarity): A perceptual image quality metric based on deep features. "we provide PSNR, SSIM, and LPIPS"
- metalness: A PBR material parameter indicating how metallic a surface is, affecting reflectance behavior. "estimating albedo, roughness, and metalness"
- multi-view stereo reconstruction: Recovering 3D geometry from multiple calibrated images. "multi-view stereo reconstruction software"
- neural rendering: Techniques that leverage neural networks to synthesize images from 3D/scene representations or features. "The integration of neural rendering techniques"
- OLAT (One-Light-At-a-Time): A capture/relighting protocol that illuminates the subject with one light at a time for compositing. "One-Light-At-a-Time (OLAT)"
- PBR (physically-based rendering): Rendering that models light transport and material behavior using physically grounded models. "achieve high-quality PBR-based relighting"
- PSNR (Peak Signal-to-Noise Ratio): A pixel-wise fidelity metric comparing reconstructions to ground truth in decibels. "we provide PSNR, SSIM, and LPIPS"
- Reinhard tone-mapping: A global tone-mapping operator that compresses HDR intensities to low dynamic range. "tone-mapped to low dynamic range using Reinhard tone-mapping."
- relighting: The process of changing an image/video’s illumination while preserving scene content. "to decouple relighting entirely from avatar modeling"
- roughness: A PBR parameter controlling microfacet distribution and the sharpness of specular highlights. "estimating albedo, roughness, and metalness"
- SMPL-H: A human body model (with hand articulation) used for pose estimation and tracking. "The required SMPL-H poses are optimized using EasyMocap"
- specular highlights: Bright reflections of light sources on shiny surfaces due to specular reflection. "specular highlights"
- splatting: Rendering technique that projects and blends point- or Gaussian-based primitives onto the image plane. "anchored to the deformable template and splatted."
- SSIM (Structural Similarity Index Measure): An image quality metric assessing structural similarity between images. "we provide PSNR, SSIM, and LPIPS"
- template mesh: A canonical, rigged 3D mesh of a subject used as the base for deformation and appearance. "template meshes"
- temporal coherence: Consistency of appearance over time in video sequences. "to synthesize photorealistic and temporally coherent relit videos."
- UV Gaussian: A Gaussian-based appearance representation defined in UV space for neural rendering. "a disentangled UV Gaussian appearance layer"
- UV space: A 2D parameterization of a 3D surface used to store textures and per-surface attributes. "predict Gaussian attributes in UV space"
- VAE (Variational Autoencoder): A generative model that learns latent encodings and reconstructions via a variational objective. "the DRCosmos VAE encoder"
- view consistency: Maintaining consistent appearance across different camera viewpoints. "do not demonstrate view consistency."
Collections
Sign up for free to add this paper to one or more collections.
