Papers
Topics
Authors
Recent
Search
2000 character limit reached

FashionFlow: Leveraging Diffusion Models for Dynamic Fashion Video Synthesis from Static Imagery

Published 29 Sep 2023 in cs.CV and cs.AI | (2310.00106v2)

Abstract: Our study introduces a new image-to-video generator called FashionFlow to generate fashion videos. By utilising a diffusion model, we are able to create short videos from still fashion images. Our approach involves developing and connecting relevant components with the diffusion model, which results in the creation of high-fidelity videos that are aligned with the conditional image. The components include the use of pseudo-3D convolutional layers to generate videos efficiently. VAE and CLIP encoders capture vital characteristics from still images to condition the diffusion model at a global level. Our research demonstrates a successful synthesis of fashion videos featuring models posing from various angles, showcasing the fit and appearance of the garment. Our findings hold great promise for improving and enhancing the shopping experience for the online fashion industry.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. “Augmented reality platforms for virtual fitting rooms” In The International Journal of Multimedia & Its Appliications 4.4 Academy & Industry Research Collaboration Center (AIRCC), 2012, pp. 35
  2. “Fashion meets computer vision: A survey” In ACM Computing Surveys (CSUR) 54.4 ACM New York, NY, USA, 2021, pp. 1–41
  3. “Viton: An image-based virtual try-on network” In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7543–7552
  4. “Toward characteristic-preserving image-based virtual try-on network” In Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 589–604
  5. “Texturegan: Controlling deep image synthesis with texture patches” In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8456–8465
  6. Tero Karras, Samuli Laine and Timo Aila “A style-based generator architecture for generative adversarial networks” In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 4401–4410
  7. “Analyzing and improving the image quality of stylegan” In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 8110–8119
  8. “Generative adversarial nets” In Advances in neural information processing systems 27, 2014
  9. “Conditional generative adversarial nets” In arXiv preprint arXiv:1411.1784, 2014
  10. “Mocogan: Decomposing motion and content for video generation” In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 1526–1535
  11. Carl Vondrick, Hamed Pirsiavash and Antonio Torralba “Generating videos with scene dynamics” In Advances in neural information processing systems 29, 2016
  12. Ivan Skorokhodov, Sergey Tulyakov and Mohamed Elhoseiny “Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 3626–3636
  13. “Storygan: A sequential conditional gan for story visualization” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 6329–6338
  14. “Mocycle-gan: Unpaired video-to-video translation” In Proceedings of the 27th ACM international conference on multimedia, 2019, pp. 647–655
  15. “Recycle-gan: Unsupervised video retargeting” In Proceedings of the European conference on computer vision (ECCV), 2018, pp. 119–135
  16. “Imaginator: Conditional spatio-temporal gan for video generation” In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020, pp. 1160–1169
  17. “Training generative adversarial networks with limited data” In Advances in neural information processing systems 33, 2020, pp. 12104–12114
  18. “A closer look at spatiotemporal convolutions for action recognition” In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2018, pp. 6450–6459
  19. “Diffusion models in vision: A survey” In IEEE Transactions on Pattern Analysis and Machine Intelligence IEEE, 2023
  20. Jonathan Ho, Ajay Jain and Pieter Abbeel “Denoising diffusion probabilistic models” In Advances in neural information processing systems 33, 2020, pp. 6840–6851
  21. “Generative modeling by estimating gradients of the data distribution” In Advances in neural information processing systems 32, 2019
  22. “Score-based generative modeling through stochastic differential equations” In arXiv preprint arXiv:2011.13456, 2020
  23. “Diffusion models beat gans on image synthesis” In Advances in Neural Information Processing Systems 34, 2021, pp. 8780–8794
  24. “High-resolution image synthesis with latent diffusion models” In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10684–10695
  25. “Palette: Image-to-image diffusion models” In ACM SIGGRAPH 2022 Conference Proceedings, 2022, pp. 1–10
  26. “Make-a-video: Text-to-video generation without text-video data” In arXiv preprint arXiv:2209.14792, 2022
  27. “Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation” In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 7623–7633
  28. “Imagen video: High definition video generation with diffusion models” In arXiv preprint arXiv:2210.02303, 2022
  29. “VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 10209–10218
  30. “Phenaki: Variable length video generation from open domain textual description” In arXiv preprint arXiv:2210.02399, 2022
  31. “Align your latents: High-resolution video synthesis with latent diffusion models” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 22563–22575
  32. “Preserve your own correlation: A noise prior for video diffusion models” In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 22930–22941
  33. “Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning” In arXiv preprint arXiv:2311.10709, 2023
  34. “Structure and content-guided video synthesis with diffusion models” In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 7346–7356
  35. “Magicvideo: Efficient video generation with latent diffusion models” In arXiv preprint arXiv:2211.11018, 2022
  36. “Stable video diffusion: Scaling latent video diffusion models to large datasets” In arXiv preprint arXiv:2311.15127, 2023
  37. “MagicVideo-V2: Multi-Stage High-Aesthetic Video Generation” In arXiv preprint arXiv:2401.04468, 2024
  38. “Dreampose: Fashion image-to-video synthesis via stable diffusion” In arXiv preprint arXiv:2304.06025, 2023
  39. “Video diffusion models” In arXiv:2204.03458, 2022
  40. “Beautyglow: On-demand makeup transfer framework with reversible generative network” In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, 2019, pp. 10042–10050
  41. “Pose guided person image generation” In Advances in neural information processing systems 30, 2017
  42. “Towards multi-pose guided virtual try-on network” In Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 9026–9035
  43. Luisa F Polanía and Satyajit Gupte “Learning fashion compatibility across apparel categories for outfit recommendation” In 2019 IEEE International Conference on Image Processing (ICIP), 2019, pp. 4489–4493 IEEE
  44. “Image-based recommendations on styles and substitutes” In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, 2015, pp. 43–52
  45. “Fashion recommendation systems, models and methods: A review” In Informatics 8.3, 2021, pp. 49 MDPI
  46. “Fw-gan: Flow-navigated warping gan for video virtual try-on” In Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1161–1170
  47. “ShineOn: Illuminating design choices for practical video-based virtual clothing try-on” In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021, pp. 191–200
  48. “Mv-ton: Memory-based video virtual try-on network” In Proceedings of the 29th ACM International Conference on Multimedia, 2021, pp. 908–916
  49. “Clothformer: Taming video virtual try-on in all module” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 10799–10808
  50. “Difffashion: Reference-based fashion design with structure-aware transfer by diffusion models” In arXiv preprint arXiv:2302.06826, 2023
  51. “Person image synthesis via denoising diffusion model” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 5968–5976
  52. “Attention is all you need” In Advances in neural information processing systems 30, 2017
  53. “Very deep convolutional networks for large-scale image recognition” In arXiv preprint arXiv:1409.1556, 2014
  54. Alex Krizhevsky, Ilya Sutskever and Geoffrey E Hinton “Imagenet classification with deep convolutional neural networks” In Advances in neural information processing systems 25, 2012
  55. “Going deeper with convolutions” In Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1–9
  56. François Chollet “Xception: Deep learning with depthwise separable convolutions” In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1251–1258
  57. “Learning transferable visual models from natural language supervision” In International conference on machine learning, 2021, pp. 8748–8763 PMLR
  58. “Attention mechanisms in computer vision: A survey” In Computational visual media 8.3 Springer, 2022, pp. 331–368
  59. “Dwnet: Dense warp-based network for pose-guided human video generation” In arXiv preprint arXiv:1910.09139, 2019
  60. Alexander Quinn Nichol and Prafulla Dhariwal “Improved denoising diffusion probabilistic models” In International Conference on Machine Learning, 2021, pp. 8162–8171 PMLR
  61. “Decoupled weight decay regularization” In arXiv preprint arXiv:1711.05101, 2017
  62. “Stochastic image-to-video synthesis using cinns” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 3742–3753
  63. “Understanding object dynamics for interactive image-to-video synthesis” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 5171–5181
  64. “Liquid warping gan: A unified framework for human motion imitation, appearance transfer and novel view synthesis” In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 5904–5913
  65. “FVD: A new metric for video generation”, 2019
  66. “Towards accurate generative models of video: A new metric & challenges” In arXiv preprint arXiv:1812.01717, 2018
  67. “Gans trained by a two time-scale update rule converge to a local nash equilibrium” In Advances in neural information processing systems 30, 2017
  68. “Quo vadis, action recognition? a new model and the kinetics dataset” In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6299–6308
  69. Aram Davtyan “GitHub - Araachie/frechet_video_distance-pytorch-: Frechet Video Distance metric implemented on PyTorch — github.com” [Accessed 18-01-2024], 2020 URL: https://github.com/Araachie/frechet_video_distance-pytorch-
  70. “Learnable gated temporal shift module for deep video inpainting” In arXiv preprint arXiv:1907.01131, 2019
  71. “Improved techniques for training gans” In Advances in neural information processing systems 29, 2016
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up