Papers
Topics
Authors
Recent
Search
2000 character limit reached

ROBUSfT: Robust Real-Time Shape-from-Template, a C++ Library

Published 10 Jan 2023 in cs.CV | (2301.04037v3)

Abstract: Tracking the 3D shape of a deforming object using only monocular 2D vision is a challenging problem. This is because one should (i) infer the 3D shape from a 2D image, which is a severely underconstrained problem, and (ii) implement the whole solution pipeline in real-time. The pipeline typically requires feature detection and matching, mismatch filtering, 3D shape inference and feature tracking algorithms. We propose ROBUSfT, a conventional pipeline based on a template containing the object's rest shape, texturemap and deformation law. ROBUSfT is ready-to-use, wide-baseline, capable of handling large deformations, fast up to 30 fps, free of training, and robust against partial occlusions and discontinuity in video frames. It outperforms the state-of-the-art methods in challenging datasets. ROBUSfT is implemented as a publicly available C++ library and we provide a tutorial on how to use it in https://github.com/mrshetab/ROBUSfT

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. J. Pilet, V. Lepetit, and P. Fua, “Fast non-rigid surface detection, registration and realistic augmentation,” International Journal of Computer Vision, vol. 76, no. 2, pp. 109–122, 2008.
  2. N. Haouchine, J. Dequidt, M.-O. Berger, and S. Cotin, “Single view augmentation of 3D elastic objects,” in 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 229–236, IEEE, 2014.
  3. M. Hu, G. P. Penney, D. Rueckert, P. J. Edwards, F. Bello, R. Casula, M. Figl, and D. J. Hawkes, “Non-rigid reconstruction of the beating heart surface for minimally invasive cardiac surgery,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 34–42, Springer, 2009.
  4. T. Collins, B. Compte, and A. Bartoli, “Deformable shape-from-motion in laparoscopy using a rigid sliding window.,” in MIUA, pp. 173–178, 2011.
  5. A. Malti, A. Bartoli, and T. Collins, “Template-based conformal shape-from-motion-and-shading for laparoscopy,” in International Conference on Information Processing in Computer-Assisted Interventions, pp. 1–10, Springer, 2012.
  6. T. Collins, A. Bartoli, N. Bourdel, and M. Canis, “Robust, real-time, dense and deformable 3D organ tracking in laparoscopic videos,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 404–412, Springer, 2016.
  7. J. Lamarca, S. Parashar, A. Bartoli, and J. Montiel, “DefSLAM: Tracking and mapping of deforming scenes from monocular sequences,” IEEE Transactions on Robotics, vol. 37, no. 1, pp. 291–303, 2020.
  8. Y. Li, Y. Wang, M. Case, S.-F. Chang, and P. K. Allen, “Real-time pose estimation of deformable objects using a volumetric approach,” in 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1046–1052, IEEE, 2014.
  9. B. Frank, C. Stachniss, R. Schmedding, M. Teschner, and W. Burgard, “Learning object deformation models for robot motion planning,” Robotics and Autonomous Systems, vol. 62, no. 8, pp. 1153–1174, 2014.
  10. M. Aranda, J. A. Corrales Ramon, Y. Mezouar, A. Bartoli, and E. Özgür, “Monocular visual shape tracking and servoing for isometrically deforming objects,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7542–7549, 2020.
  11. M. Salzmann and P. Fua, “Reconstructing sharply folding surfaces: A convex formulation,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1054–1061, IEEE, 2009.
  12. M. Salzmann, F. Moreno-Noguer, V. Lepetit, and P. Fua, “Closed-form solution to non-rigid 3D surface registration,” in European Conference on Computer Vision, pp. 581–594, Springer, 2008.
  13. M. Perriollat, R. Hartley, and A. Bartoli, “Monocular template-based reconstruction of inextensible surfaces,” International Journal of Computer Vision, vol. 95, no. 2, pp. 124–137, 2011.
  14. A. Bartoli, Y. Gérard, F. Chadebecq, T. Collins, and D. Pizarro, “Shape-from-template,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 10, pp. 2099–2118, 2015.
  15. A. Chhatkuli, D. Pizarro, A. Bartoli, and T. Collins, “A stable analytical framework for isometric shape-from-template by surface integration,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 5, pp. 833–850, 2016.
  16. M. Famouri, A. Bartoli, and Z. Azimifar, “Fast shape-from-template using local features,” Machine Vision and Applications, vol. 29, no. 1, pp. 73–93, 2018.
  17. F. Brunet, A. Bartoli, and R. I. Hartley, “Monocular template-based 3D surface reconstruction: Convex inextensible and nonconvex isometric methods,” Computer Vision and Image Understanding, vol. 125, pp. 138–154, 2014.
  18. E. Özgür and A. Bartoli, “Particle-SfT: A provably-convergent, fast shape-from-template algorithm,” International Journal of Computer Vision, vol. 123, no. 2, pp. 184–205, 2017.
  19. M. Salzmann and P. Fua, “Linear local models for monocular reconstruction of deformable surfaces,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 5, pp. 931–944, 2010.
  20. J. Östlund, A. Varol, D. T. Ngo, and P. Fua, “Laplacian meshes for monocular 3D shape recovery,” in European Conference on Computer Vision, pp. 412–425, Springer, 2012.
  21. D. T. Ngo, J. Östlund, and P. Fua, “Template-based monocular 3D shape recovery using Laplacian meshes,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 1, pp. 172–187, 2015.
  22. T. Collins and A. Bartoli, “[POSTER] Realtime shape-from-template: System and applications,” in 2015 IEEE International Symposium on Mixed and Augmented Reality, pp. 116–119, IEEE, 2015.
  23. Q. Liu-Yin, R. Yu, L. Agapito, A. Fitzgibbon, and C. Russell, “Better together: Joint reasoning for non-rigid 3D reconstruction with specularities and shading,” arXiv preprint arXiv:1708.01654, 2017.
  24. A. Pumarola, A. Agudo, L. Porzi, A. Sanfeliu, V. Lepetit, and F. Moreno-Noguer, “Geometry-aware network for non-rigid shape prediction from a single view,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4681–4690, 2018.
  25. V. Golyanik, S. Shimada, K. Varanasi, and D. Stricker, “HDM-Net: Monocular non-rigid 3D reconstruction with learned deformation model,” in International Conference on Virtual Reality and Augmented Reality, pp. 51–72, Springer, 2018.
  26. S. Shimada, V. Golyanik, C. Theobalt, and D. Stricker, “IsMo-GAN: Adversarial learning for monocular non-rigid 3D reconstruction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 0–0, 2019.
  27. D. Fuentes-Jimenez, D. Pizarro, D. Casillas-Pérez, T. Collins, and A. Bartoli, “Deep shape-from-template: Single-image quasi-isometric deformable registration and reconstruction,” Image and Vision Computing, vol. 127, p. 104531, 2022.
  28. D. Fuentes-Jimenez, D. Pizarro, D. Casillas-Perez, T. Collins, and A. Bartoli, “Texture-generic deep shape-from-template,” IEEE Access, vol. 9, pp. 75211–75230, 2021.
  29. D. Pizarro and A. Bartoli, “Feature-based deformable surface detection with self-occlusion reasoning,” International Journal of Computer Vision, vol. 97, no. 1, pp. 54–70, 2012.
  30. T. Collins, P. Mesejo, and A. Bartoli, “An analysis of errors in graph-based keypoint matching and proposed solutions,” in European Conference on Computer Vision, pp. 138–153, Springer, 2014.
  31. Q.-H. Tran, T.-J. Chin, G. Carneiro, M. S. Brown, and D. Suter, “In defence of RANSAC for outlier rejection in deformable registration,” in European Conference on Computer Vision, pp. 274–287, Springer, 2012.
  32. O. Sorkine and M. Alexa, “As-rigid-as-possible surface modeling,” in Symposium on Geometry processing, vol. 4, pp. 109–116, 2007.
  33. Agisoft, “Agisoft PhotoScan.” https://www.agisoft.com.
  34. D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” International Journal of Computer Vision, vol. 60, no. 2, pp. 91–110, 2004.
  35. C. Griwodz, L. Calvet, and P. Halvorsen, “Popsift: A faithful SIFT implementation for real-time applications,” in Proceedings of the 9th ACM Multimedia Systems Conference, MMSys ’18, (New York, NY, USA), pp. 415–420, ACM, 2018.
  36. F. L. Bookstein, “Principal warps: Thin-plate splines and the decomposition of deformations,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 11, no. 6, pp. 567–585, 1989.
  37. D. Rueckert, L. I. Sonoda, C. Hayes, D. L. Hill, M. O. Leach, and D. J. Hawkes, “Nonrigid registration using free-form deformations: application to breast MR images,” IEEE Transactions on Medical Imaging, vol. 18, no. 8, pp. 712–721, 1999.
  38. A. Varol, A. Shaji, M. Salzmann, and P. Fua, “Monocular 3D reconstruction of locally textured surfaces,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 6, pp. 1118–1130, 2012.
  39. M. Müller, B. Heidelberger, M. Hennix, and J. Ratcliff, “Position based dynamics,” Journal of Visual Communication and Image Representation, vol. 18, no. 2, pp. 109–118, 2007.
  40. J. Bender, M. Müller, M. A. Otaduy, M. Teschner, and M. Macklin, “A survey on position-based simulation methods in computer graphics,” in Computer Graphics Forum, vol. 33, pp. 228–251, Wiley Online Library, 2014.
  41. A. Chhatkuli, D. Pizarro, and A. Bartoli, “Stable template-based isometric 3D reconstruction in all imaging conditions by linear least-squares,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 708–715, 2014.
  42. M. Shetab-Bushehri, M. Aranda, Y. Mezouar, and E. Özgür, “As-rigid-as-possible shape servoing,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 3898–3905, 2022.
Citations (4)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up