Papers
Topics
Authors
Recent
Search
2000 character limit reached

MaSkel: A Model for Human Whole-body X-rays Generation from Human Masking Images

Published 13 Apr 2024 in eess.IV, cs.CV, and cs.LG | (2404.09000v1)

Abstract: The human whole-body X-rays could offer a valuable reference for various applications, including medical diagnostics, digital animation modeling, and ergonomic design. The traditional method of obtaining X-ray information requires the use of CT (Computed Tomography) scan machines, which emit potentially harmful radiation. Thus it faces a significant limitation for realistic applications because it lacks adaptability and safety. In our work, We proposed a new method to directly generate the 2D human whole-body X-rays from the human masking images. The predicted images will be similar to the real ones with the same image style and anatomic structure. We employed a data-driven strategy. By leveraging advanced generative techniques, our model MaSkel(Masking image to Skeleton X-rays) could generate a high-quality X-ray image from a human masking image without the need for invasive and harmful radiation exposure, which not only provides a new path to generate highly anatomic and customized data but also reduces health risks. To our knowledge, our model MaSkel is the first work for predicting whole-body X-rays. In this paper, we did two parts of the work. The first one is to solve the data limitation problem, the diffusion-based techniques are utilized to make a data augmentation, which provides two synthetic datasets for preliminary pretraining. Then we designed a two-stage training strategy to train MaSkel. At last, we make qualitative and quantitative evaluations of the generated X-rays. In addition, we invite some professional doctors to assess our predicted data. These evaluations demonstrate the MaSkel's superior ability to generate anatomic X-rays from human masking images. The related code and links of the dataset are available at https://github.com/2022yingjie/MaSkel.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. A survey on variational autoencoders from a green AI perspective. SN Computer Science 2, 4 (2021), 301.
  2. Ilvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.02938 (2021).
  3. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).
  4. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 12873–12883.
  5. Generative diffusion prior for unified image restoration and enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9935–9946.
  6. Creating and animating subject-specific anatomical models. In Computer Graphics Forum, Vol. 29. Wiley Online Library, 2340–2351.
  7. Generative adversarial networks. Commun. ACM 63, 11 (2020), 139–144.
  8. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 16000–16009.
  9. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
  10. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 (2022).
  11. Denoising diffusion probabilistic models. Advances in neural information processing systems 33 (2020), 6840–6851.
  12. Jonathan Ho and Tim Salimans. 2022. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022).
  13. A survey on GANs for computer vision: Recent research, analysis and taxonomy. Computer Science Review 48 (2023), 100553.
  14. Recurrent neural network-based semantic variational autoencoder for sequence-to-sequence learning. Information Sciences 490 (2019), 59–73.
  15. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14. Springer, 694–711.
  16. Reconstructing personalized anatomical models for physics-based body animation. ACM Transactions on Graphics (TOG) 35, 6 (2016), 1–13.
  17. From Skin to Skeleton: Towards Biomechanically Accurate 3D Digital Humans. ACM Transactions on Graphics (TOG) 42, 6 (2023), 1–12.
  18. OSSO: Obtaining skeletal shape from outside. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 20492–20501.
  19. Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).
  20. Piano: A parametric hand bone model from magnetic resonance imaging. arXiv preprint arXiv:2106.10893 (2021).
  21. Nimble: a non-rigid hand model with bones and muscles. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1–16.
  22. SMPL: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2. 851–866.
  23. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems 35 (2022), 5775–5787.
  24. A statistical shape model of the skull developed from a south african population. In 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, 333–336.
  25. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 (2021).
  26. Alexander Quinn Nichol and Prafulla Dhariwal. 2021. Improved denoising diffusion probabilistic models. In International conference on machine learning. PMLR, 8162–8171.
  27. Perception-oriented single image super-resolution using optimal objective estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1725–1735.
  28. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2536–2544.
  29. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10975–10985.
  30. A 3D face model for pose and illumination invariant face recognition. In 2009 sixth IEEE international conference on advanced video and signal based surveillance. Ieee, 296–301.
  31. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems 32 (2019).
  32. Real-time flying object detection with YOLOv8. arXiv preprint arXiv:2305.09972 (2023).
  33. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10684–10695.
  34. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 22500–22510.
  35. Image super-resolution via iterative refinement. IEEE transactions on pattern analysis and machine intelligence 45, 4 (2022), 4713–4726.
  36. Computational bodybuilding: Anatomically-based modeling of human bodies. ACM Transactions on Graphics (TOG) 34, 4 (2015), 1–12.
  37. UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS medicine 12, 3 (2015), e1001779.
  38. Generating synthetic x-ray images of a person from the surface geometry. In Proceedings of the IEEE conference on computer vision and pattern recognition. 9059–9067.
  39. Recovering 3d human mesh from monocular images: A survey. IEEE transactions on pattern analysis and machine intelligence (2023).
  40. Neural discrete representation learning. Advances in neural information processing systems 30 (2017).
  41. Deep high-resolution representation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence 43, 10 (2020), 3349–3364.
  42. Exploiting diffusion prior for real-world image super-resolution. arXiv preprint arXiv:2305.07015 (2023).
  43. High-resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference on computer vision and pattern recognition. 8798–8807.
  44. Econ: Explicit clothed humans optimized via normal integration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 512–523.
  45. Icon: Implicit clothed humans obtained from normals. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 13286–13296.
  46. Diffusion models: A comprehensive survey of methods and applications. Comput. Surveys 56, 4 (2023), 1–39.
  47. Local implicit normalizing flow for arbitrary-scale image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1776–1785.
  48. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18456–18466.
  49. Zongsheng Yue and Chen Change Loy. 2022. Difface: Blind face restoration with diffused error contraction. arXiv preprint arXiv:2212.06512 (2022).
  50. Efficient Diffusion Model for Image Restoration by Residual Shifting. arXiv preprint arXiv:2403.07319 (2024).
  51. Pymaf-x: Towards well-aligned full-body model regression from monocular images. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).
  52. Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 11446–11456.
  53. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 3836–3847.
  54. Adaptable anatomical models for realistic bone motion reconstruction. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 459–471.
  55. Silvia Zuffi and Michael J Black. 2015. The stitched puppet: A graphical model of 3d human shape and pose. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3537–3546.
  56. Lions and tigers and bears: Capturing non-rigid, 3d, articulated shape from images. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 3955–3963.
  57. 3D menagerie: Modeling the 3D shape and pose of animals. In Proceedings of the IEEE conference on computer vision and pattern recognition. 6365–6373.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up