Papers
Topics
Authors
Recent
2000 character limit reached

LumiSculpt: A Consistency Lighting Control Network for Video Generation (2410.22979v1)

Published 30 Oct 2024 in cs.CV

Abstract: Lighting plays a pivotal role in ensuring the naturalness of video generation, significantly influencing the aesthetic quality of the generated content. However, due to the deep coupling between lighting and the temporal features of videos, it remains challenging to disentangle and model independent and coherent lighting attributes, limiting the ability to control lighting in video generation. In this paper, inspired by the established controllable T2I models, we propose LumiSculpt, which, for the first time, enables precise and consistent lighting control in T2V generation models.LumiSculpt equips the video generation with strong interactive capabilities, allowing the input of custom lighting reference image sequences. Furthermore, the core learnable plug-and-play module of LumiSculpt facilitates remarkable control over lighting intensity, position, and trajectory in latent video diffusion models based on the advanced DiT backbone.Additionally, to effectively train LumiSculpt and address the issue of insufficient lighting data, we construct LumiHuman, a new lightweight and flexible dataset for portrait lighting of images and videos. Experimental results demonstrate that LumiSculpt achieves precise and high-quality lighting control in video generation.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (40)
  1. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. Pix2Video: Video editing using image diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  23206–23217, 2023.
  3. StableVideo: Text-driven consistency-aware diffusion video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  23040–23050, 2023.
  4. VideoCrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512, 2023.
  5. Acquiring the reflectance field of a human face. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’00, pp.  145–156, USA, 2000. ACM Press/Addison-Wesley Publishing Co.
  6. TaleCrafter: Interactive story visualization with multiple characters. In ACM SIGGRAPH Asia Conference Proceedings, pp.  101:1–101:10, 2023.
  7. AnimateDiff: Animate your personalized text-to-image diffusion models without specific tuning. In International Conference on Learning Representations (ICLR), 2024.
  8. CameraCtrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101, 2024.
  9. Animate-A-Story: Storytelling with retrieval-augmented video generation. arXiv preprint arXiv:2307.06940, 2023.
  10. Imagen Video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022a.
  11. Video diffusion models. In International Conference on Learning Representations (ICLR), 2022b.
  12. CogVideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022.
  13. Towards high fidelity face relighting with realistic shadows. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  14719–14728, 2021.
  14. Face relighting with geometrically consistent shadows. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  4217–4226, 2022.
  15. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision, pp.  1501–1510, 2017.
  16. Neural style transfer: A review. IEEE Transactions on Visualization and Computer Graphics, 26(11):3365–3385, nov 2020. ISSN 1941-0506.
  17. Switchlight: Co-design of physics-driven architecture and pre-training framework for human portrait relighting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  25096–25106, 2024.
  18. Open-sora-plan, April 2024. URL https://doi.org/10.5281/zenodo.10948109.
  19. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pp.  12888–12900. PMLR, 2022.
  20. LightPainter: interactive portrait relighting with freehand scribble. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  195–205, 2023.
  21. Holo-Relighting: Controllable volumetric portrait relighting from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  4263–4273, 2024.
  22. MetaHuman, 2023. URL https://www.unrealengine.com/en-US/metahuman.
  23. Learning physics-guided face relighting under directional light. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  5124–5133, 2020.
  24. Total Relighting: learning to relight portraits for background replacement. ACM Transactions on Graphics, 40(4):43:–43:21, 2021.
  25. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pp.  8748–8763, 2021.
  26. Relightful harmonization: Lighting-aware portrait background replacement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  6452–6462, 2024.
  27. Single image portrait relighting. ACM Transactions on Graphics, 38(4):79:1–79:12, 2019.
  28. ModelScope text-to-video technical report. arXiv preprint arXiv:2308.06571, 2023a.
  29. VideoComposer: Compositional video synthesis with motion controllability. In Advances in Neural Information Processing Systems (NeurIPS), 2023b.
  30. SunStage: Portrait reconstruction and relighting using the sun as a light stage. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  20792–20802, 2023c.
  31. Single image portrait relighting via explicit multiple reflectance channel modeling. ACM Transactions on Graphics, 39(6):220:1–220:13, 2020.
  32. DreamVideo: Composing your dream videos with customized subject and motion. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  6537–6549, 2024.
  33. Tune-a-Video: One-shot tuning of image diffusion models for text-to-video generation. In IEEE/CVF International Conference on Computer Vision (ICCV), pp.  7623–7633, 2023.
  34. Retinex-diffusion: On controlling illumination conditions in diffusion models via retinex theory. arXiv preprint arXiv:2407.20785, 2024.
  35. Learning to relight portrait images via a virtual light stage and synthetic-to-real adaptation. ACM Transactions on Graphics, 41(6):23:1–231:21, 2022.
  36. Dilightnet: Fine-grained lighting control for diffusion-based image generation. In ACM SIGGRAPH 2024 Conference Papers, pp.  1–12, 2024.
  37. Ic-light github page, 2024.
  38. Neural light transport for relighting and view synthesis. ACM Transactions on Graphics, 40(1):9:1–9:17, 2021.
  39. MotionCrafter: One-shot motion customization of diffusion models. arXiv preprint arXiv:2312.05288, 2023.
  40. MotionDirector: Motion customization of text-to-video diffusion models. arXiv preprint arXiv:2310.08465, 2023.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Slide Deck Streamline Icon: https://streamlinehq.com

Whiteboard

Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
Reddit Logo Streamline Icon: https://streamlinehq.com

Reddit

  1. [2410.22979] LumiSculpt: A Consistency Lighting Control Network for Video Generation (1 point, 0 comments)