PINQI: An End-to-End Physics-Informed Approach to Learned Quantitative MRI Reconstruction (2306.11023v2)
Abstract: Quantitative Magnetic Resonance Imaging (qMRI) enables the reproducible measurement of biophysical parameters in tissue. The challenge lies in solving a nonlinear, ill-posed inverse problem to obtain the desired tissue parameter maps from acquired raw data. While various learned and non-learned approaches have been proposed, the existing learned methods fail to fully exploit the prior knowledge about the underlying MR physics, i.e. the signal model and the acquisition model. In this paper, we propose PINQI, a novel qMRI reconstruction method that integrates the knowledge about the signal, acquisition model, and learned regularization into a single end-to-end trainable neural network. Our approach is based on unrolled alternating optimization, utilizing differentiable optimization blocks to solve inner linear and non-linear optimization tasks, as well as convolutional layers for regularization of the intermediate qualitative images and parameter maps. This design enables PINQI to leverage the advantages of both the signal model and learned regularization. We evaluate the performance of our proposed network by comparing it with recently published approaches in the context of highly undersampled $T_1$-mapping, using both a simulated brain dataset, as well as real scanner data acquired from a physical phantom and in-vivo data from healthy volunteers. The results demonstrate the superiority of our proposed solution over existing methods and highlight the effectiveness of our method in real-world scenarios.
- K. P. Pruessmann, M. Weiger, M. B. Scheidegger, and P. Boesiger, “SENSE: Sensitivity encoding for fast MRI,” Magnetic Resonance in Medicine, vol. 42, no. 5, pp. 952–962, 1999. 10.1002/(SICI)1522-2594(199911)42:5¡952::AID-MRM16¿3.0.CO;2-S
- M. Lustig, D. L. Donoho, J. M. Santos, and J. M. Pauly, “Compressed Sensing MRI,” IEEE Signal Processing Magazine, vol. 25, no. 2, pp. 72–82, 2008. 10.1109/MSP.2007.914728
- M. Doneva, P. Börnert, H. Eggers, C. Stehning, J. Sénégas, and A. Mertins, “Compressed sensing reconstruction for magnetic resonance parameter mapping,” Magnetic Resonance in Medicine, vol. 64, no. 4, pp. 1114–1120, 2010. 10.1002/mrm.22483
- X. Wang, V. Roeloffs, J. Klosowski, Z. Tan, D. Voit, M. Uecker, and J. Frahm, “Model-based T1 mapping with sparsity constraints using single-shot inversion-recovery radial FLASH,” Magnetic Resonance in Medicine, vol. 79, no. 2, pp. 730–740, 2018. 10.1002/mrm.26726
- B. Zhao, F. Lam, and Z. P. Liang, “Model-based MR parameter mapping with sparsity constraints: Parameter estimation and performance bounds,” IEEE Transactions on Medical Imaging, vol. 33, no. 9, pp. 1832–1844, 2014. 10.1109/TMI.2014.2322815
- K. T. Block, M. Uecker, and J. Frahm, “Undersampled radial MRI with multiple coils. Iterative image reconstruction using a total variation constraint,” Magnetic Resonance in Medicine, vol. 57, no. 6, pp. 1086–1098, 2007. 10.1002/mrm.21236
- J. Tran-Gia, D. Stäb, T. Wech, D. Hahn, and H. Köstler, “Model-based Acceleration of Parameter mapping (MAP) for saturation prepared radially acquired data,” Magnetic Resonance in Medicine, vol. 70, no. 6, pp. 1524–1534, 2013. 10.1002/mrm.24600
- K. M. Becker, J. Schulz-Menger, T. Schaeffter, and C. Kolbitsch, “Simultaneous high-resolution cardiac T1 mapping and cine imaging using model-based iterative image reconstruction,” Magnetic Resonance in Medicine, vol. 81, no. 2, pp. 1080–1091, 2019. 10.1002/mrm.27474
- A. Sriram, J. Zbontar, T. Murrell, A. Defazio, C. L. Zitnick, N. Yakubova, F. Knoll, and P. Johnson, “End-to-End Variational Networks for Accelerated MRI Reconstruction,” Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. Lecture Notes in Computer Science, vol. 12262 LNCS, pp. 64–73, 2020. 10.1007/978-3-030-59713-9_7
- H. K. Aggarwal, M. P. Mani, and M. Jacob, “MoDL: Model-Based Deep Learning Architecture for Inverse Problems,” IEEE Transactions on Medical Imaging, vol. 38, no. 2, pp. 394–405, 2019. 10.1109/TMI.2018.2865356
- Y. Jun, H. Shin, T. Eo, T. Kim, and D. Hwang, “Deep model-based magnetic resonance parameter mapping network (DOPAMINE) for fast T1 mapping using variable flip angle method,” Medical Image Analysis, vol. 70, p. 102017, 2021. 10.1016/j.media.2021.102017
- H. Jeelani, Y. Yang, R. Zhou, C. M. Kramer, M. Salerno, and D. S. Weller, “A Myocardial T1-Mapping Framework with Recurrent and U-Net Convolutional Neural Networks,” Proceedings - International Symposium on Biomedical Imaging, vol. 2020-April, pp. 1941–1944, 2020. 10.1109/ISBI45749.2020.9098459
- X. Xu, W. Gan, S. V. Kothapalli, D. A. Yablonskiy, and U. S. Kamilov, “CoRRECT: A Deep Unfolding Framework for Motion-Corrected Quantitative R2* Mapping,” 2022. 10.48550/arXiv.2210.06330
- F. Liu, L. Feng, and R. Kijowski, “MANTIS: Model-Augmented Neural neTwork with Incoherent k-space Sampling for efficient MR parameter mapping,” Magnetic Resonance in Medicine, vol. 82, no. 1, pp. 174–188, 2019. 10.1002/mrm.27707
- R. Guo, H. El-Rewaidy, S. Assana, X. Cai, A. Amyar, K. Chow, X. Bi, T. Yankama, J. Cirillo, P. Pierce, B. Goddu, L. Ngo, and R. Nezafat, “Accelerated cardiac T1 mapping in four heartbeats with inline MyoMapNet: a deep learning-based T1 estimation approach,” Journal of Cardiovascular Magnetic Resonance, vol. 24, no. 1, pp. 1–15, 2022. 10.1186/s12968-021-00834-0
- O. Cohen, B. Zhu, and M. S. Rosen, “MR fingerprinting Deep RecOnstruction NEtwork (DRONE),” Magnetic Resonance in Medicine, vol. 80, no. 3, pp. 885–894, 2018. 10.1002/mrm.27198
- J. Schlemper, J. Caballero, J. V. Hajnal, A. N. Price, and D. Rueckert, “A Deep Cascade of Convolutional Neural Networks for Dynamic MR Image Reconstruction,” IEEE Transactions on Medical Imaging, vol. 37, no. 2, pp. 491–503, 2018. 10.1007/978-3-319-59050-951
- C. Qin, J. Schlemper, J. Caballero, A. N. Price, J. V. Hajnal, and D. Rueckert, “Convolutional recurrent neural networks for dynamic MR image reconstruction,” IEEE Transactions on Medical Imaging, vol. 38, no. 1, pp. 280–290, 2019. 10.1109/TMI.2018.2863670
- Y. Yang, J. Sun, H. Li, and Z. Xu, “Deep ADMM-Net for compressive sensing MRI,” Advances in Neural Information Processing Systems, vol. 29, 2016. 10.5555/3157096.3157098
- K. Chow, J. A. Flewitt, J. D. Green, J. J. Pagano, M. G. Friedrich, and R. B. Thompson, “Saturation recovery single-shot acquisition (SASHA) for myocardial T 1 mapping,” Magnetic Resonance in Medicine, vol. 71, no. 6, pp. 2082–2095, 2014. 10.1002/mrm.24878
- J. Z. Bojorquez, S. Bricq, C. Acquitter, F. Brunotte, P. M. Walker, and A. Lalande, “What are normal relaxation times of tissues at 3 T?” Magnetic Resonance Imaging, vol. 35, pp. 69–80, 2017. 10.1016/j.mri.2016.08.021
- D. C. Liu and J. Nocedal, “On the limited memory BFGS method for large scale optimization,” Mathematical Programming, vol. 45, no. 1-3, pp. 503–528, 8 1989. 10.1007/BF01589116
- D. Ma, V. Gulani, N. Seiberlich, K. Liu, J. L. Sunshine, J. L. Duerk, and M. A. Griswold, “Magnetic resonance fingerprinting,” Nature, vol. 495, no. 7440, pp. 187–192, 3 2013. 10.1038/nature11971
- H. Li, M. Yang, J. H. Kim, C. Zhang, R. Liu, P. Huang, D. Liang, X. Zhang, X. Li, and L. Ying, “SuperMAP: Deep ultrafast MR relaxometry with joint spatiotemporal undersampling,” Magnetic Resonance in Medicine, vol. 89, no. 1, pp. 64–76, 2023. 10.1002/mrm.29411
- O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9351, pp. 234–241, 2015. 10.1007/978-3-319-24574-4_28
- K. Hammernik, T. Kustner, B. Yaman, Z. Huang, D. Rueckert, F. Knoll, and M. Akcakaya, “Physics-Driven Deep Learning for Computational Magnetic Resonance Imaging: Combining physics and machine learning for improved medical imaging,” IEEE Signal Processing Magazine, vol. 40, no. 1, pp. 98–114, 2023. 10.1109/msp.2022.3215288
- D. Chen, M. E. Davies, and M. Golbabaee, “Deep Unrolling for Magnetic Resonance Fingerprinting,” Proceedings - International Symposium on Biomedical Imaging, no. 2, 2022. 10.1109/ISBI52829.2022.9761475
- A. Kofler, M. Haltmeier, T. Schaeffter, and C. Kolbitsch, “An end-to-end-trainable iterative network architecture for accelerated radial multi-coil 2d cine mr image reconstruction,” Medical Physics, vol. 48, no. 5, pp. 2412–2425, 2021.
- B. Amos and J. Z. Kolter, “OptNet: Differentiable optimization as a layer in neural networks,” 34th International Conference on Machine Learning, ICML 2017, vol. 1, pp. 179–191, 2017. 10.48550/arXiv.1703.0044
- A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and J. Zico Kolter, “Differentiable convex optimization layers,” Advances in Neural Information Processing Systems, vol. 32, no. NeurIPS, 2019. 10.48550/arXiv.1910.12430
- Z. Lv, F. Dellaert, J. M. Rehg, and A. Geiger, “Taking a deeper look at the inverse compositional algorithm,” Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2019-June, pp. 4576–4585, 2019. 10.1109/CVPR.2019.00471
- K. Hammernik, T. Küstner, and D. Rueckert, “Machine Learning for MRI Reconstruction,” in Magnetic Resonance Image Reconstruction, C. Prieto, M. I. Doneva, and M. Akcakaya, Eds. Elsevier, 2022, ch. 11, pp. 281–317.
- S. Gould, B. Fernando, A. Cherian, P. Anderson, R. S. Cruz, and E. Guo, “On differentiating parameterized argmin and argmax problems with application to bi-level optimization,” 2016.
- O. de Oliveira, “The implicit and the inverse function theorems: Easy proofs,” Real Analysis Exchange, vol. 39, no. 1, pp. 207–218, 2013. 10.14321/realanalexch.39.1.0207
- F. Pedregosa, “Hyperparameter optimization with approximate gradient,” 33rd International Conference on Machine Learning, ICML 2016, vol. 2, pp. 1150–1159, 2016. 10.48550/arXiv.1602.02355
- M. V. Afonso, J. M. Bioucas-Dias, and M. A. Figueiredo, “Fast image recovery using variable splitting and constrained optimization,” IEEE Transactions on Image Processing, vol. 19, no. 9, pp. 2345–2356, 2010. 10.1109/TIP.2010.2047910
- K. Kreutz-Delgado, “The complex gradient operator and the CR-calculus,” arXiv preprint, 2009. 10.48550/arXiv.0906.4835
- B. Aubert-Broche, M. Griffin, G. B. Pike, A. C. Evans, and D. L. Collins, “Twenty new digital brain phantoms for creation of validation image data bases,” IEEE Transactions on Medical Imaging, vol. 25, no. 11, pp. 1410–1416, 2006. 10.1109/TMI.2006.883453
- M. J. Muckley, R. Stern, T. Murrell, and F. Knoll, “mrisensesim.py,” TorchKbNufft: A High-Level, Hardware-Agnostic Non-Uniform Fast Fourier Transform, ISMRM Workshop on Data Sampling & Image Reconstruction. Online github.com/mmuckley/torchkbnufft/tree/v0.3.4/torchkbnufft/mri.
- D. O. Walsh, A. F. Gmitro, and M. W. Marcellin, “Adaptive reconstruction of phased array MR imagery,” Magnetic Resonance in Medicine, vol. 43, no. 5, pp. 682–690, 2000. 10.1002/(SICI)1522-2594(200005)43:5¡682::AID-MRM10¿3.0.CO;2-G
- S. J. Inati, M. S. Hansen, and P. Kellman, “A Fast Optimal Method for Coil Sensitivity Estimation and Adaptive Coil Combination for Complex Images,” Proceedings of the 22nd Annual Meeting of ISMRM, 2014.
- M. Uecker, P. Lai, M. J. Murphy, P. Virtue, M. Elad, J. M. Pauly, S. S. Vasanawala, and M. Lustig, “ESPIRiT–an eigenvalue approach to autocalibrating parallel MRI: where SENSE meets GRAPPA,” Magnetic Resonance in Medicine, vol. 71, no. 3, pp. 990–1001, 2014. 10.1002/mrm.24751
- K. J. Layton, S. Kroboth, F. Jia, S. Littin, H. Yu, J. Leupold, J.-F. Nielsen, T. Stöcker, and M. Zaitsev, “Pulseq: A rapid and hardware-independent pulse sequence prototyping framework,” Magnetic Resonance in Medicine, vol. 77, no. 4, pp. 1544–1552, 2017. 10.1002/mrm.26235
- B. Zoph and Q. V. Le, “Searching for activation functions,” 6th International Conference on Learning Representations, ICLR 2018 - Workshop Track Proceedings, no. 1, pp. 1–12, 2018. 10.48550/arXiv.1710.05941
- Y. Wu and K. He, “Group Normalization,” International Journal of Computer Vision, vol. 128, no. 3, pp. 742–755, 2020. 10.1007/s11263-019-01198-w
- Z. Qiu, T. Yao, and T. Mei, “Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks,” Proceedings of the IEEE International Conference on Computer Vision, vol. 2017-Octob, pp. 5534–5542, 10 2017. 10.1109/ICCV.2017.590
- E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville, “FiLM: Visual reasoning with a general conditioning layer,” 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, pp. 3942–3951, 2018. 10.1609/aaai.v32i1.11671
- F. F. Zimmermann and A. Kofler, “NoSENSE: Learned unrolled cardiac MRI reconstruction without explicit sensitivity maps,” International Workshop on Statistical Atlases and Computational Models of the Heart (STACOM), 2023. 10.48550/arXiv.2309.15608
- T. Bachlechner, B. P. Majumder, H. Mao, G. Cottrell, and J. McAuley, “ReZero is All You Need: Fast Convergence at Large Depth,” 37th Conference on Uncertainty in Artificial Intelligence, UAI 2021, no. 1, pp. 1352–1361, 2021. 10.48550/arXiv./2003.04887
- B. Zhou and S. Kevin Zhou, “Dudornet: Learning a dual-domain recurrent network for fast MRI reconstruction with deep T1 prior,” Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 4272–4281, 2020. 10.1109/CVPR42600.2020.00433
- L. Wang, C.-Y. Lee, Z. Tu, and S. Lazebnik, “Training deeper convolutional networks with deep supervision,” 2015.
- D. P. Kingma and J. L. Ba, “Adam: A method for stochastic optimization,” 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, pp. 1–15, 2015. 10.48550/arXiv.412.6980
- T. He, Z. Zhang, H. Zhang, Z. Zhang, J. Xie, and M. Li, “Bag of tricks for image classification with convolutional neural networks,” Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2019-June, pp. 558–567, 2019. 10.48550/arXiv.1812.01187
- I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” 7th International Conference on Learning Representations, ICLR 2019, 2019. 10.48550/arXiv.1711.05101
- Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing, vol. 13, no. 4, pp. 600–612, 2004.
- X. Zhang, Q. Duchemin, K. Liu*, C. Gultekin, S. Flassbeck, C. Fernandez-Granda, and J. Assländer, “Cramér–Rao bound-informed training of neural networks for quantitative MRI,” Magnetic Resonance in Medicine, vol. 88, no. 1, pp. 436–448, 2022. 10.1002/mrm.29206
- A. Kendall, Y. Gal, and R. Cipolla, “Multi-task learning using uncertainty to weigh losses for scene geometry and semantics,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018. 10.48550/arXiv.1705.07115 pp. 7482–7491.
- D. Gilton, G. Ongie, and R. Willett, “Model Adaptation for Inverse Problems in Imaging,” IEEE Transactions on Computational Imaging, vol. 7, no. 2, pp. 661–674, 2021. 10.1109/TCI.2021.3094714
- K. Hammernik, J. Schlemper, C. Qin, J. Duan, R. M. Summers, and D. Rueckert, “Systematic evaluation of iterative deep neural networks for fast parallel mri reconstruction with sensitivity-weighted coil combination,” Magnetic Resonance in Medicine, vol. 86, no. 4, pp. 1859–1872, 2021. 10.1002/mrm.28827
- F. F. Zimmermann, A. Kofler, C. Kolbitsch, and P. Schuenke, “Semi-Supervised Learning for Spatially Regularized Quantitative MRI Reconstruction - Application to Simultaneous T1, B0, B1 Mapping,” 2023, 1166, ISMRM Annual Meeting.
- Y. Jun, J. Cho, X. Wang, M. Gee, P. E. Grant, B. Bilgic, and B. Gagoski, “SSL-QALAS: Self-Supervised Learning for rapid multiparameter estimation in quantitative MRI using 3D-QALAS,” Magnetic Resonance in Medicine, vol. 90, no. 5, pp. 2019–2032, 2023. 10.1002/mrm.29786