Papers
Topics
Authors
Recent
Search
2000 character limit reached

SURESTEP: An Uncertainty-Aware Trajectory Optimization Framework to Enhance Visual Tool Tracking for Robust Surgical Automation

Published 29 Mar 2024 in cs.RO | (2404.00123v1)

Abstract: Inaccurate tool localization is one of the main reasons for failures in automating surgical tasks. Imprecise robot kinematics and noisy observations caused by the poor visual acuity of an endoscopic camera make tool tracking challenging. Previous works in surgical automation adopt environment-specific setups or hard-coded strategies instead of explicitly considering motion and observation uncertainty of tool tracking in their policies. In this work, we present SURESTEP, an uncertainty-aware trajectory optimization framework for robust surgical automation. We model the uncertainty of tool tracking with the components motivated by the sources of noise in typical surgical scenes. Using a Gaussian assumption to propagate our uncertainty models through a given tool trajectory, SURESTEP provides a general framework that minimizes the upper bound on the entropy of the final estimated tool distribution. We compare SURESTEP with a baseline method on a real-world suture needle regrasping task under challenging environmental conditions, such as poor lighting and a moving endoscopic camera. The results over 60 regrasps on the da Vinci Research Kit (dVRK) demonstrate that our optimized trajectories significantly outperform the un-optimized baseline.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. G. Hubens, H. Coveliers, L. Balliu, M. Ruppert, and W. Vaneerdeweg, “A performance study comparing manual and robotically assisted laparoscopic surgery using the da vinci system,” Surgical Endoscopy and other interventional techniques, vol. 17, no. 10, pp. 1595–1599, 2003.
  2. J. A. Khubchandani, A. M. Ingraham, V. T. Daniel, D. Ayturk, C. I. Kiefe, and H. P. Santry, “Geographic diffusion and implementation of acute care surgery: an uneven solution to the national emergency general surgery crisis,” JAMA surgery, vol. 153, no. 2, pp. 150–159, 2018.
  3. P. Kazanzides, Z. Chen, A. Deguet, G. S. Fischer, R. H. Taylor, and S. P. DiMaio, “An open-source research kit for the da vinci® surgical system,” in 2014 IEEE international conference on robotics and automation (ICRA).   IEEE, 2014, pp. 6434–6439.
  4. S. Iyer, T. Looi, and J. Drake, “A single arm, single camera system for automated suturing,” in 2013 IEEE International Conference on Robotics and Automation.   IEEE, 2013, pp. 239–244.
  5. S. Sen, A. Garg, D. V. Gealy, S. McKinley, Y. Jen, and K. Goldberg, “Automating multi-throw multilateral surgical suturing with a mechanical needle guide and sequential convex optimization,” in 2016 IEEE international conference on robotics and automation (ICRA).   IEEE, 2016, pp. 4178–4185.
  6. S. A. Pedram, C. Shin, P. W. Ferguson, J. Ma, E. P. Dutson, and J. Rosen, “Autonomous suturing framework and quantification using a cable-driven surgical robot,” IEEE Transactions on Robotics, vol. 37, no. 2, pp. 404–417, 2020.
  7. K. L. Schwaner, I. Iturrate, J. K. Andersen, P. T. Jensen, and T. R. Savarimuthu, “Autonomous bi-manual surgical suturing based on skills learned from demonstration,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2021, pp. 4017–4024.
  8. F. Richter, S. Shen, F. Liu, J. Huang, E. K. Funk, R. K. Orosco, and M. C. Yip, “Autonomous robotic suction to clear the surgical field for hemostasis using image-based blood flow detection,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 1383–1390, 2021.
  9. J. Huang, F. Liu, F. Richter, and M. C. Yip, “Model-predictive control of blood suction for surgical hemostasis using differentiable fluid simulations,” in 2021 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2021, pp. 12 380–12 386.
  10. K.-H. Oh, L. Borgioli, M. Zefran, L. Chen, and P. C. Giulianotti, “A framework for automated dissection along tissue boundary,” arXiv preprint arXiv:2310.09669, 2023.
  11. R. Hao, O. Özgüner, and M. C. Çavuşoğlu, “Vision-based surgical tool pose estimation for the da vinci® robotic surgical system,” in 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS).   IEEE, 2018, pp. 1298–1305.
  12. J. Lu, A. Jayakumari, F. Richter, Y. Li, and M. C. Yip, “Super deep: A surgical perception framework for robotic tissue manipulation using deep learning for feature extraction,” in 2021 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2021, pp. 4783–4789.
  13. F. Richter, J. Lu, R. K. Orosco, and M. C. Yip, “Robotic tool tracking under partially visible kinematic chain: A unified approach,” IEEE Transactions on Robotics, vol. 38, no. 3, pp. 1653–1670, 2021.
  14. C. D’Ambrosia, F. Richter, Z.-Y. Chiu, N. Shinde, F. Liu, H. I. Christensen, and M. C. Yip, “Robust surgical tool tracking with pixel-based probabilities for projected geometric primitives,” arXiv preprint arXiv:2403.04971, 2024.
  15. C. D’Ettorre, G. Dwyer, X. Du, F. Chadebecq, F. Vasconcelos, E. De Momi, and D. Stoyanov, “Automated pick-up of suturing needles for robotic surgical assistance,” in 2018 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2018, pp. 1370–1377.
  16. Z.-Y. Chiu, F. Richter, E. K. Funk, R. K. Orosco, and M. C. Yip, “Bimanual regrasping for suture needles using reinforcement learning for rapid motion planning,” in 2021 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2021, pp. 7737–7743.
  17. A. Wilcox, J. Kerr, B. Thananjeyan, J. Ichnowski, M. Hwang, S. Paradis, D. Fer, and K. Goldberg, “Learning to localize, grasp, and hand over unmodified surgical needles,” in 2022 International Conference on Robotics and Automation (ICRA).   IEEE, 2022, pp. 9637–9643.
  18. M. Miyasaka, M. Haghighipanah, Y. Li, J. Matheson, A. Lewis, and B. Hannaford, “Modeling cable-driven robot with hysteresis and cable–pulley network friction,” IEEE/ASME Transactions on Mechatronics, vol. 25, no. 2, pp. 1095–1104, 2020.
  19. M. Hwang, B. Thananjeyan, S. Paradis, D. Seita, J. Ichnowski, D. Fer, T. Low, and K. Goldberg, “Efficiently calibrating cable-driven surgical robots with rgbd fiducial sensing and recurrent neural networks,” IEEE Robotics and Automation Letters, vol. 5, no. 4, pp. 5937–5944, 2020.
  20. S. Eslamian, L. A. Reisner, B. W. King, and A. K. Pandya, “Towards the implementation of an autonomous camera algorithm on the da vinci platform,” in Medicine Meets Virtual Reality 22.   IOS Press, 2016, pp. 118–123.
  21. J. J. Ji, S. Krishnan, V. Patel, D. Fer, and K. Goldberg, “Learning 2d surgical camera motion from demonstrations,” in 2018 IEEE 14th International Conference on Automation Science and Engineering (CASE).   IEEE, 2018, pp. 35–42.
  22. R. Moccia and F. Ficuciello, “Autonomous endoscope control algorithm with visibility and joint limits avoidance constraints for da vinci research kit robot,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 776–781.
  23. F. Zhong, Y. Wang, Z. Wang, and Y.-H. Liu, “Dual-arm robotic needle insertion with active tissue deformation for autonomous suturing,” IEEE Robotics and Automation Letters, vol. 4, no. 3, pp. 2669–2676, 2019.
  24. O. Özgüner, T. Shkurti, S. Lu, W. Newman, and M. C. Çavuşoğlu, “Visually guided needle driving and pull for autonomous suturing,” in 2021 IEEE 17th International Conference on Automation Science and Engineering (CASE).   IEEE, 2021, pp. 242–248.
  25. K. Fozilov, J. Colan, K. Sekiyama, and Y. Hasegawa, “Towards autonomous robotic minimally invasive surgery: A hybrid framework combining task-motion planning and dynamic behavior trees,” IEEE Access, 2023.
  26. K. Dharmarajan, W. Panitch, M. Jiang, K. Srinivas, B. Shi, Y. Avigal, H. Huang, T. Low, D. Fer, and K. Goldberg, “Automating vascular shunt insertion with the dvrk surgical robot,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 6781–6788.
  27. J. Van Den Berg, P. Abbeel, and K. Goldberg, “Lqg-mp: Optimized path planning for robots with motion uncertainty and imperfect state information,” International Journal of Robotics Research, vol. 30, no. 7, pp. 895–913, 2011.
  28. J. Van Den Berg, S. Patil, and R. Alterovitz, “Motion planning under uncertainty using iterative local optimization in belief space,” The International Journal of Robotics Research, vol. 31, no. 11, pp. 1263–1278, 2012.
  29. Z. Sunberg and M. Kochenderfer, “Online algorithms for pomdps with continuous state, action, and observation spaces,” in Proceedings of the International Conference on Automated Planning and Scheduling, vol. 28, 2018, pp. 259–263.
  30. S. Huang, N. M. Kwok, G. Dissanayake, Q. P. Ha, and G. Fang, “Multi-step look-ahead trajectory planning in slam: Possibility and necessity,” in Proceedings of the 2005 IEEE international conference on robotics and automation.   IEEE, 2005, pp. 1091–1096.
  31. J. Clemens, T. Reineking, and T. Kluth, “An evidential approach to slam, path planning, and active exploration,” International Journal of Approximate Reasoning, vol. 73, pp. 1–26, 2016.
  32. S. Ali, Y. Jonmohamadi, Y. Takeda, J. Roberts, R. Crawford, and A. K. Pandey, “Supervised scene illumination control in stereo arthroscopes for robot assisted minimally invasive surgery,” IEEE Sensors Journal, vol. 21, no. 10, pp. 11 577–11 587, 2020.
  33. S. Eslamian, L. A. Reisner, and A. K. Pandya, “Development and evaluation of an autonomous camera control algorithm on the da vinci surgical system,” The International Journal of Medical Robotics and Computer Assisted Surgery, vol. 16, no. 2, p. e2036, 2020.
  34. Y. Ou, S. Zargarzadeh, and M. Tavakoli, “Robot learning incorporating human interventions in the real world for autonomous surgical endoscopic camera control,” Journal of Medical Robotics Research, vol. 8, no. 03n04, p. 2340004, 2023.
  35. H.-C. Lin, M. M. Marinho, and K. Harada, “Autonomous field-of-view adjustment using adaptive kinematic constrained control with robot-held microscopic camera feedback,” arXiv preprint arXiv:2309.10287, 2023.
  36. A. Reiter, P. K. Allen, and T. Zhao, “Feature classification for tracking articulated surgical tools,” in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2012: 15th International Conference, Nice, France, October 1-5, 2012, Proceedings, Part II 15.   Springer, 2012, pp. 592–600.
  37. M. Ye, L. Zhang, S. Giannarou, and G.-Z. Yang, “Real-time 3d tracking of articulated tools for robotic surgery,” in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part I 19.   Springer, 2016, pp. 386–394.
  38. Y. Li, F. Richter, J. Lu, E. K. Funk, R. K. Orosco, J. Zhu, and M. C. Yip, “Super: A surgical perception framework for endoscopic tissue manipulation with surgical robotics,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 2294–2301, 2020.
  39. R. Platt Jr, R. Tedrake, L. P. Kaelbling, and T. Lozano-Perez, “Belief space planning assuming maximum likelihood observations.” in Robotics: Science and Systems, vol. 2, 2010.
  40. Z.-Y. Chiu, A. Z. Liao, F. Richter, B. Johnson, and M. C. Yip, “Markerless suture needle 6d pose tracking with robust uncertainty estimation for autonomous minimally invasive robotic surgery,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2022, pp. 5286–5292.
  41. Z.-Y. Chiu, F. Richter, and M. C. Yip, “Real-time constrained 6d object-pose tracking of an in-hand suture needle for minimally invasive robotic surgery,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 4761–4767.
  42. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems, vol. 32, 2019.
  43. C. Wang, D. Gao, K. Xu, J. Geng, Y. Hu, Y. Qiu, B. Li, F. Yang, B. Moon, A. Pandey et al., “Pypose: A library for robot learning with physics-based optimization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 22 024–22 034.
  44. D. C. Liu and J. Nocedal, “On the limited memory bfgs method for large scale optimization,” Mathematical programming, vol. 45, no. 1, pp. 503–528, 1989.
  45. S. Bansal, M. Chen, S. Herbert, and C. J. Tomlin, “Hamilton-jacobi reachability: A brief overview and recent advances,” 2017.
  46. A. K. Akametalu, J. F. Fisac, J. H. Gillula, S. Kaynama, M. N. Zeilinger, and C. J. Tomlin, “Reachability-based safe learning with gaussian processes,” in 53rd IEEE Conference on Decision and Control, 2014, pp. 1424–1431.
  47. H. K. Cheng, S. W. Oh, B. Price, J.-Y. Lee, and A. Schwing, “Putting the object back into video object segmentation,” arXiv preprint arXiv:2310.12982, 2023.
  48. A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo et al., “Segment anything,” arXiv preprint arXiv:2304.02643, 2023.
  49. K. Omote, H. Feussner, A. Ungeheuer, K. Arbter, G.-Q. Wei, J. R. Siewert, and G. Hirzinger, “Self-guided robotic camera control for laparoscopic surgery compared with human camera control,” The American journal of surgery, vol. 177, no. 4, pp. 321–324, 1999.
  50. Y. Zhi, N. Das, and M. Yip, “Diffco: Autodifferentiable proxy collision detection with multiclass labels for safety-aware trajectory optimization,” IEEE Transactions on Robotics, vol. 38, no. 5, pp. 2668–2685, 2022.
  51. I. Greenberg, N. Yannay, and S. Mannor, “Optimization or architecture: How to hack kalman filtering,” 2023.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 1 tweet with 1 like about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free