Papers
Topics
Authors
Recent
2000 character limit reached

Differentiable Uncalibrated Imaging (2211.10525v3)

Published 18 Nov 2022 in eess.IV and cs.LG

Abstract: We propose a differentiable imaging framework to address uncertainty in measurement coordinates such as sensor locations and projection angles. We formulate the problem as measurement interpolation at unknown nodes supervised through the forward operator. To solve it we apply implicit neural networks, also known as neural fields, which are naturally differentiable with respect to the input coordinates. We also develop differentiable spline interpolators which perform as well as neural networks, require less time to optimize and have well-understood properties. Differentiability is key as it allows us to jointly fit a measurement representation, optimize over the uncertain measurement coordinates, and perform image reconstruction which in turn ensures consistent calibration. We apply our approach to 2D and 3D computed tomography, and show that it produces improved reconstructions compared to baselines that do not account for the lack of calibration. The flexibility of the proposed framework makes it easy to extend to almost arbitrary imaging problems.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention.   Springer, 2015, pp. 234–241.
  2. K. H. Jin, M. T. McCann, E. Froustey, and M. Unser, “Deep convolutional neural network for inverse problems in imaging,” IEEE Transactions on Image Processing, vol. 26, no. 9, pp. 4509–4522, 2017.
  3. M. T. McCann, K. H. Jin, and M. Unser, “Convolutional neural networks for inverse problems in imaging: A review,” IEEE Signal Processing Magazine, vol. 34, no. 6, pp. 85–95, 2017.
  4. G. Ongie, A. Jalal, C. A. Metzler, R. G. Baraniuk, A. G. Dimakis, and R. Willett, “Deep learning techniques for inverse problems in imaging,” IEEE Journal on Selected Areas in Information Theory, vol. 1, no. 1, pp. 39–56, 2020.
  5. S. Antholzer, M. Haltmeier, and J. Schwab, “Deep learning for photoacoustic tomography from sparse data,” Inverse problems in science and engineering, vol. 27, no. 7, pp. 987–1005, 2019.
  6. K. Kothari, S. Gupta, M. v. de Hoop, and I. Dokmanic, “Random mesh projectors for inverse problems,” in International Conference on Learning Representations, 2019.
  7. D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Deep image prior,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 9446–9454.
  8. K. Gregor and Y. LeCun, “Learning fast approximations of sparse coding,” in Proceedings of the 27th international conference on international conference on machine learning, 2010, pp. 399–406.
  9. J. Adler and O. Öktem, “Learned primal-dual reconstruction,” IEEE transactions on medical imaging, vol. 37, no. 6, pp. 1322–1332, 2018.
  10. H. Gupta, K. H. Jin, H. Q. Nguyen, M. T. McCann, and M. Unser, “Cnn-based projected gradient descent for consistent ct image reconstruction,” IEEE transactions on medical imaging, vol. 37, no. 6, pp. 1440–1453, 2018.
  11. J. Rick Chang, C.-L. Li, B. Poczos, B. Vijaya Kumar, and A. C. Sankaranarayanan, “One network to solve them all–solving linear inverse problems using deep projection models,” in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 5888–5897.
  12. D. Gilton, G. Ongie, and R. Willett, “Neumann networks for linear inverse problems in imaging,” IEEE Transactions on Computational Imaging, vol. 6, pp. 328–343, 2019.
  13. K. Zhang, Y. Li, W. Zuo, L. Zhang, L. Van Gool, and R. Timofte, “Plug-and-play image restoration with deep denoiser prior,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
  14. A. Bora, A. Jalal, E. Price, and A. G. Dimakis, “Compressed sensing using generative models,” in International Conference on Machine Learning.   PMLR, 2017, pp. 537–546.
  15. K. Kothari, A. Khorashadizadeh, M. de Hoop, and I. Dokmanić, “Trumpets: Injective flows for inference and inverse problems,” in Uncertainty in Artificial Intelligence.   PMLR, 2021, pp. 1269–1278.
  16. Y. Song, L. Shen, L. Xing, and S. Ermon, “Solving inverse problems in medical imaging with score-based generative models,” arXiv preprint arXiv:2111.08005, 2021.
  17. D. Gilton, G. Ongie, and R. Willett, “Model adaptation for inverse problems in imaging,” IEEE Transactions on Computational Imaging, vol. 7, pp. 661–674, 2021.
  18. A. Gossard and P. Weiss, “Training adaptive reconstruction networks for inverse problems,” arXiv preprint arXiv:2202.11342, 2022.
  19. L. Piegl, “On nurbs: a survey,” IEEE Computer Graphics and Applications, vol. 11, no. 01, pp. 55–71, 1991.
  20. E. Cohen, T. Lyche, and R. Riesenfeld, “Discrete b-splines and subdivision techniques in computer-aided geometric design and computer graphics,” Computer graphics and image processing, vol. 14, no. 2, pp. 87–111, 1980.
  21. A. D. Prasad, A. Balu, H. Shah, S. Sarkar, C. Hegde, and A. Krishnamurthy, “Nurbs-diff: A differentiable programming module for nurbs,” Computer-Aided Design, vol. 146, p. 103199, 2022.
  22. Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, F. Tombari, J. Tompkin, V. Sitzmann, and S. Sridhar, “Neural fields in visual computing and beyond,” in Computer Graphics Forum, vol. 41, no. 2.   Wiley Online Library, 2022, pp. 641–676.
  23. J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove, “Deepsdf: Learning continuous signed distance functions for shape representation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 165–174.
  24. B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” in European conference on computer vision.   Springer, 2020, pp. 405–421.
  25. V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein, “Implicit neural representations with periodic activation functions,” Advances in Neural Information Processing Systems, vol. 33, pp. 7462–7473, 2020.
  26. A. W. Reed, H. Kim, R. Anirudh, K. A. Mohan, K. Champley, J. Kang, and S. Jayasuriya, “Dynamic ct reconstruction from limited views with implicit neural representations and parametric motion fields,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, pp. 2258–2268.
  27. L. Lozenski, M. Anastasio, and U. Villa, “Neural fields for dynamic imaging,” in Medical Imaging 2022: Physics of Medical Imaging, vol. 12031.   SPIE, 2022, pp. 231–238.
  28. Y. Sun, J. Liu, M. Xie, B. Wohlberg, and U. S. Kamilov, “Coil: Coordinate-based internal learning for tomographic imaging,” IEEE Transactions on Computational Imaging, vol. 7, pp. 1400–1412, 2021.
  29. N. Chen, L. Cao, T.-C. Poon, B. Lee, and E. Y. Lam, “Differentiable imaging: A new tool for computational optical imaging,” Advanced Physics Research, 2023.
  30. M. Du, Y. S. Nashed, S. Kandel, D. Gürsoy, and C. Jacobsen, “Three dimensions, two microscopes, one code: Automatic differentiation for x-ray nanotomography beyond the depth of focus limit,” Science advances, vol. 6, no. 13, p. eaay3700, 2020.
  31. M. Du, S. Kandel, J. Deng, X. Huang, A. Demortiere, T. T. Nguyen, R. Tucoulou, V. De Andrade, Q. Jin, and C. Jacobsen, “Adorym: A multi-platform generic x-ray image reconstruction framework based on automatic differentiation,” Optics express, vol. 29, no. 7, pp. 10 000–10 035, 2021.
  32. B. Triggs, P. F. McLauchlan, R. I. Hartley, and A. W. Fitzgibbon, “Bundle adjustment—a modern synthesis,” in Vision Algorithms: Theory and Practice: International Workshop on Vision Algorithms Corfu, Greece, September 21–22, 1999 Proceedings.   Springer, 2000, pp. 298–372.
  33. T. F. Chan and C.-K. Wong, “Total variation blind deconvolution,” IEEE transactions on Image Processing, vol. 7, no. 3, pp. 370–375, 1998.
  34. N. A. B. Riis, Y. Dong, and P. C. Hansen, “Computed tomography reconstruction with uncertain view angles by iteratively updated model discrepancy,” Journal of Mathematical Imaging and Vision, vol. 63, no. 2, pp. 133–143, 2021.
  35. S. Basu and Y. Bresler, “Uniqueness of tomography with unknown view angles,” IEEE Transactions on Image Processing, vol. 9, no. 6, pp. 1094–1106, 2000.
  36. R. R. Coifman, Y. Shkolnisky, F. J. Sigworth, and A. Singer, “Graph laplacian tomography from unknown random projections,” IEEE Transactions on Image Processing, vol. 17, no. 10, pp. 1891–1899, 2008.
  37. T. Bendory, A. Bartesaghi, and A. Singer, “Single-particle cryo-electron microscopy: Mathematical theory, computational challenges, and opportunities,” IEEE signal processing magazine, vol. 37, no. 2, pp. 58–76, 2020.
  38. G. H. Golub and C. F. Van Loan, “An analysis of the total least squares problem,” SIAM journal on numerical analysis, vol. 17, no. 6, pp. 883–893, 1980.
  39. I. Markovsky and S. Van Huffel, “Overview of total least-squares methods,” Signal processing, vol. 87, no. 10, pp. 2283–2302, 2007.
  40. S. Gupta and I. Dokmanić, “Total least squares phase retrieval,” IEEE Transactions on Signal Processing, vol. 70, pp. 536–549, 2021.
  41. J. Adler, H. Kohr, and O. Öktem, “Operator discretization library (odl),” Zenodo, 2017.
  42. Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3d u-net: learning dense volumetric segmentation from sparse annotation,” in International conference on medical image computing and computer-assisted intervention.   Springer, 2016, pp. 424–432.
  43. J. Leuschner, M. Schmidt, D. O. Baguer, and P. Maaß, “The lodopab-ct dataset: A benchmark dataset for low-dose ct reconstruction methods,” arXiv preprint arXiv:1910.01113, 2019.
  44. S. Lunz, A. Hauptmann, T. Tarvainen, C.-B. Schonlieb, and S. Arridge, “On learned operator correction in inverse problems,” SIAM Journal on Imaging Sciences, vol. 14, no. 1, pp. 92–127, 2021.
  45. D. Tegunov and P. Cramer, “Real-time cryo-electron microscopy data preprocessing with warp,” Nature methods, vol. 16, no. 11, pp. 1146–1152, 2019.
  46. K. Naydenova, P. Jia, and C. J. Russo, “Cryo-em with sub–1 å specimen movement,” Science, vol. 370, no. 6513, pp. 223–226, 2020.
  47. J.-J. Fernandez and S. Li, “Tomoalign: A novel approach to correcting sample motion and 3d ctf in cryoet,” Journal of Structural Biology, vol. 213, no. 4, p. 107778, 2021.
  48. B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, Y. Burren, N. Porz, J. Slotboom, R. Wiest et al., “The multimodal brain tumor image segmentation benchmark (brats),” IEEE transactions on medical imaging, vol. 34, no. 10, pp. 1993–2024, 2014.
  49. S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. S. Kirby, J. B. Freymann, K. Farahani, and C. Davatzikos, “Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features,” Scientific data, vol. 4, no. 1, pp. 1–13, 2017.
  50. S. Bakas, M. Reyes, A. Jakab, S. Bauer, M. Rempfler, A. Crimi, R. T. Shinohara, C. Berger, S. M. Ha, M. Rozycki et al., “Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge,” arXiv preprint arXiv:1811.02629, 2018.
  51. M. Buda, A. Saha, and M. A. Mazurowski, “Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by a deep learning algorithm,” Computers in Biology and Medicine, vol. 109, 2019.
  52. J. L. Prince and A. S. Willsky, “Constrained sinogram restoration for limited-angle tomography,” Optical Engineering, vol. 29, no. 5, pp. 535–544, 1990.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up