Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

LPSNet: End-to-End Human Pose and Shape Estimation with Lensless Imaging (2404.01941v3)

Published 2 Apr 2024 in cs.CV

Abstract: Human pose and shape (HPS) estimation with lensless imaging is not only beneficial to privacy protection but also can be used in covert surveillance scenarios due to the small size and simple structure of this device. However, this task presents significant challenges due to the inherent ambiguity of the captured measurements and lacks effective methods for directly estimating human pose and shape from lensless data. In this paper, we propose the first end-to-end framework to recover 3D human poses and shapes from lensless measurements to our knowledge. We specifically design a multi-scale lensless feature decoder to decode the lensless measurements through the optically encoded mask for efficient feature extraction. We also propose a double-head auxiliary supervision mechanism to improve the estimation accuracy of human limb ends. Besides, we establish a lensless imaging system and verify the effectiveness of our method on various datasets acquired by our lensless imaging system.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. Single-Frame 3D Fluorescence Microscopy with Ultraminiature Lensless FlatScope. Sci Adv., 3(12):e1701548, 2017.
  2. DiffuserCam: Lensless Single-exposure 3D Imaging. Optica, 5(1):1–9, 2018.
  3. Exploiting Temporal Context for 3D Human Pose Estimation in the ild. In IEEE Conf. Comput. Vis. Pattern Recog., pages 3395–3404, 2019.
  4. Flatcam: Thin, Lensless Cameras using Coded Aperture and Computation. IEEE Trans Comput Imaging., 3(3):384–397, 2016.
  5. Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image. In Eur. Conf. Comput. Vis. Springer International Publishing, 2016.
  6. Lensless Imaging: A Computational Renaissance. IEEE Signal Process Mag., 33(5):23–35, 2016.
  7. Phlatcam: Designed Phase-Mask Based Thin Lensless Camera. IEEE Trans. Pattern Anal. Mach. Intell., 42(7):1618–1629, 2020.
  8. Cross-attention of disentangled modalities for 3d human mesh recovery with transformers. In Eur. Conf. Comput. Vis., pages 342–359. Springer, 2022.
  9. Imagenet: A large-scale hierarchical image database. In IEEE Conf. Comput. Vis. Pattern Recog. Workshops (CVPR Workshops).
  10. Hierarchical Kinematic Human Mesh Recovery. In Eur. Conf. Comput. Vis., pages 768–784. Springer, 2020.
  11. Humans in 4D: Reconstructing and Tracking Humans with Transformers. In Int. Conf. Comput. Vis., pages 14783–14794, 2023.
  12. Densepose: Dense Human Pose Estimation in the Wild. In Int. Conf. Comput. Vis., pages 7297–7306, 2018.
  13. Exemplar Fine-Tuning for 3D Human Model Fitting Towards In-the-Wild 3D Human Pose Estimation. In Int. Conf. 3D. Vis., pages 42–52. IEEE, 2021.
  14. End-to-end Recovery of Human Shape and Pose. In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
  15. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  16. PARE: Part Attention Regressor for 3D Human Body Estimation. In Int. Conf. Comput. Vis., pages 11127–11137, 2021.
  17. Learning to Reconstruct 3D Human Pose and Shape Via Model-fitting in the Loop. In Int. Conf. Comput. Vis., pages 2252–2261, 2019a.
  18. Convolutional Mesh Regression for Single-Image Human Shape Reconstruction. In IEEE Conf. Comput. Vis. Pattern Recog., pages 4501–4510, 2019b.
  19. DiffuserCam: Diffuser-Based Lensless Cameras. In Computational Optical Sensing and Imaging, pages CTu3B–2. Optica Publishing Group, 2017.
  20. Unite the People: Closing the Loop Between 3D and 2D Human Representations. In IEEE Conf. Comput. Vis. Pattern Recog., 2017.
  21. SMPLy Benchmarking 3D Human Pose Estimation in the wild. In Int. Conf. 3D. Vis., pages 301–310. IEEE, 2020.
  22. SimCC: A Simple Coordinate Classification Perspective for Human Pose Estimation. In Eur. Conf. Comput. Vis., pages 89–106. Springer, 2022.
  23. End-to-end human pose and mesh reconstruction with transformers. In IEEE Conf. Comput. Vis. Pattern Recog., pages 1954–1963, 2021a.
  24. Mesh Graphormer. In Int. Conf. Comput. Vis., pages 12939–12948, 2021b.
  25. Lensless Imaging and Sensing. Annual review of biomedical engineering, 18:77–102, 2016.
  26. Expressive Body Capture: 3D Hands, Face, and Body from a Single Image. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
  27. Human Mesh Recovery from Multiple Shots. In IEEE Conf. Comput. Vis. Pattern Recog., pages 1485–1495, 2022.
  28. Robust Lensless Image Reconstruction Via PSF Estimation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 403–412, 2021.
  29. HuMoR: 3D Human Motion Model for Robust Pose Estimation. In Int. Conf. Comput. Vis., 2021.
  30. Lensless Ultra-miniature CMOS Computational Imagers and Sensors. Proc. Sensorcomm, pages 186–190, 2013.
  31. Optical, mathematical, and computational foundations of lensless ultra-miniature diffractive imagers and sensors. International Journal on Advances in Systems and Measurements, 7(3):4, 2014.
  32. Deep High-resolution Representation Learning for Human Pose Estimation. In IEEE Conf. Comput. Vis. Pattern Recog., pages 5693–5703, 2019.
  33. Pose-NDF: Modeling Human Pose Manifolds with Neural Distance Fields. In Eur. Conf. Comput. Vis., 2022.
  34. Learning 3D Human Shape and Pose from Dense Body Parts. IEEE Trans. Pattern Anal. Mach. Intell., 44(5):2610–2627, 2020.
  35. Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop. In Int. Conf. Comput. Vis., pages 11446–11456, 2021.
  36. Predicting 3D Human Dynamics from Video. In Int. Conf. Comput. Vis., pages 7114–7123, 2019.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now