Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
175 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Physics-Informed Computer Vision: A Review and Perspectives (2305.18035v3)

Published 29 May 2023 in eess.IV and cs.CV

Abstract: The incorporation of physical information in machine learning frameworks is opening and transforming many application domains. Here the learning process is augmented through the induction of fundamental knowledge and governing physical laws. In this work, we explore their utility for computer vision tasks in interpreting and understanding visual data. We present a systematic literature review of more than 250 papers on formulation and approaches to computer vision tasks guided by physical laws. We begin by decomposing the popular computer vision pipeline into a taxonomy of stages and investigate approaches to incorporate governing physical equations in each stage. Existing approaches in computer vision tasks are analyzed with regard to what governing physical processes are modeled and formulated, and how they are incorporated, i.e. modification of input data (observation bias), modification of network architectures (inductive bias), and modification of training losses (learning bias). The taxonomy offers a unified view of the application of the physics-informed capability, highlighting where physics-informed learning has been conducted and where the gaps and opportunities are. Finally, we highlight open problems and challenges to inform future research. While still in its early days, the study of physics-informed computer vision has the promise to develop better computer vision models that can improve physical plausibility, accuracy, data efficiency, and generalization in increasingly realistic applications.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (148)
  1. Real-world super-resolution of face-images from surveillance cameras. IET Image Processing 16, 2 (2022), 442–452.
  2. The FEniCS project version 1.5. Archive of Numerical Software 3, 100 (2015).
  3. Applications of generative adversarial networks (gans): An updated review. Archives of Computational Methods in Engineering 28 (2021), 525–552.
  4. Physics-Informed Attention Temporal Convolutional Network for EEG-Based Motor Imagery Classification. IEEE Transactions on Industrial Informatics 19, 2 (2022), 2249–2258.
  5. A deep journey into super-resolution: A survey. Comput. Surveys 53, 3 (2020), 1–34.
  6. Rajat Arora. 2022. PhySRNet: Physics informed super-resolution network for application in computational solid mechanics. arXiv preprint arXiv:2206.15457 (2022).
  7. Rajat Arora and Ankit Shrivastava. 2022. Spatio-Temporal Super-Resolution of Dynamical Systems using Physics-Informed Deep-Learning. arXiv preprint arXiv:2212.04457 (2022).
  8. Computer vision and IoT-based sensors in flood monitoring and mapping: A systematic review. Sensors 19, 22 (2019), 5012.
  9. A physics-informed road user safety field theory for traffic safety assessments applying artificial intelligence-based video analytics. Analytic Methods in Accident Research 37 (2023), 100252.
  10. Big self-supervised models advance medical image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, Piscataway, NJ, USA, 3478–3488.
  11. Structural health monitoring. Vol. 90. John Wiley & Sons.
  12. Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI. Inf. Fusion 58, C (jun 2020), 82–115. https://doi.org/10.1016/j.inffus.2019.12.012
  13. Thermal image reconstruction using deep learning. IEEE Access 8 (2020), 126839–126858.
  14. PIDLNet: A physics-induced deep learning network for characterization of crowd videos. In IEEE International Conference on Advanced Video and Signal Based Surveillance. IEEE, 1–8.
  15. High-resolution single-photon imaging with physics-informed deep learning. Nature Communications 14, 1 (2023), 5902.
  16. Using physics-informed enhanced super-resolution generative adversarial networks for subfilter modeling in turbulent reactive flows. Combustion Institute 38, 2 (2021), 2617–2625.
  17. Acquisition-invariant brain MRI segmentation with informative uncertainties. Medical Image Analysis 92 (2024), 103058.
  18. Physics-informed brain MRI segmentation. In International Workshop on Simulation and Synthesis in Medical Imaging. 100–109.
  19. Predicting the temporal dynamics of turbulent channels through deep learning. International Journal of Heat and Fluid Flow 96 (2022), 109010.
  20. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine learning 3, 1 (2011), 1–122.
  21. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 (2018).
  22. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine 34, 4 (2017), 18–42.
  23. Personalising left-ventricular biophysical models of the heart using parametric physics-informed neural networks. Medical Image Analysis 71 (2021), 102066.
  24. Reduced-order modeling of blood flow for noninvasive functional evaluation of coronary artery disease. Biomechanics and modeling in mechanobiology 18 (2019), 1867–1881.
  25. Zachary Burns and Zhaowei Liu. 2023. Untrained, physics-informed neural networks for structured illumination microscopy. Optics Express 31, 5 (2023), 8714–8724.
  26. Physics-informed neural networks (PINNs) for fluid mechanics: A review. Acta Mechanica Sinica (2022), 1–12.
  27. Flow over an espresso cup: inferring 3-D velocity and pressure fields from tomographic background oriented Schlieren via physics-informed neural networks. Journal of Fluid Mechanics 915 (2021), A102.
  28. DeepUrbanDownscale: A physics informed deep learning framework for high-resolution urban surface temperature estimation via 3D point clouds. International Journal of Applied Earth Observation and Geoinformation 106 (2022), 102650.
  29. Aug-nerf: Training stronger neural radiance fields with triple-level physically-grounded augmentations. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 15191–15202.
  30. A deep neural network surrogate modeling benchmark for temperature field prediction of heat source layout. Science China Physics, Mechanics & Astronomy 64, 11 (2021), 1.
  31. Yuyao Chen and Luca Dal Negro. 2022. Physics-informed neural networks for imaging and parameter retrieval of photonic nanostructures from near-field data. APL Photonics 7, 1 (2022), 010802.
  32. Generative adversarial networks in medical image augmentation: a review. Computers in Biology and Medicine (2022), 105382.
  33. Physics-informed generative neural network: an application to troposphere temperature prediction. Environmental Research Letters 16, 6 (2021), 065003.
  34. Grounding physical concepts of objects and events through dynamic visual reasoning. arXiv preprint arXiv:2103.16564 (2021).
  35. Physics-informed deep learning for T2-deblurred superresolution turbo spin echo MRI. Magnetic Resonance in Medicine 90, 6 (2023), 2362–2374.
  36. Fashion meets computer vision: A survey. Comput. Surveys 54, 4 (2021), 1–41.
  37. Physics informed neural fields for smoke reconstruction with sparse data. ACM Transactions on Graphics 41, 4 (2022), 1–14.
  38. Learning temporal coherence via self-supervision for GAN-based video generation. ACM Transactions on Graphics 39, 4 (2020), 75–1.
  39. 3D U-Net: learning dense volumetric segmentation from sparse annotation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI). Springer, 424–432.
  40. Gauge equivariant convolutional networks and the icosahedral CNN. In International Conference on Machine Learning (ICML). PMLR, 1321–1330.
  41. Lagrangian neural networks. arXiv preprint arXiv:2003.04630 (2020).
  42. Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What’s next. arXiv preprint arXiv:2201.05624 (2022).
  43. Deep-learning-based image reconstruction and enhancement in optical microscopy. IEEE 108, 1 (2019), 30–50.
  44. Physics-informed neural networks for gravity currents reconstruction from limited data. Physics of Fluids 35, 2 (2023).
  45. Imagenet: A large-scale hierarchical image database. In IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 248–255.
  46. On the interplay between physical and content priors in deep learning for computational imaging. Optics Express 28, 16 (2020), 24152–24170.
  47. Learning vortex dynamics for fluid inference and prediction. arXiv preprint arXiv:2301.11494 (2023).
  48. Dynamic visual reasoning by learning differentiable physics models from video and language. Advances In Neural Information Processing Systems 34 (2021), 887–899.
  49. A Machine Learning and Computer Vision Approach to Geomagnetic Storm Forecasting. arXiv preprint arXiv:2204.05780 (2022).
  50. Physics-informed machine learning and mechanistic modeling of additive manufacturing to reduce defects. Applied Materials Today 24 (2021), 101123.
  51. Physics-Informed Deep Learning for Motion-Corrected Reconstruction of Quantitative Brain MRI. arXiv preprint arXiv:2403.08298 (2024).
  52. Hamidreza Eivazi and Ricardo Vinuesa. 2022. Physics-informed deep-learning applications to experimental fluid mechanics. arXiv preprint arXiv:2203.15402 (2022).
  53. Mohamed Elgendy. 2020. Deep learning for vision systems. Simon and Schuster.
  54. Computer vision and machine learning for medical image analysis: recent advances, challenges, and way forward. Artificial Intelligence Surgery 2 (2022).
  55. Physics-informed autoencoders for Lyapunov-stable fluid flow prediction. arXiv preprint arXiv:1905.10866 (2019).
  56. Meshfreeflownet: A physics-constrained deep continuous space-time super-resolution framework. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–15.
  57. Deep learning-enabled medical computer vision. NPJ Digital Medicine 4, 1 (2021), 5.
  58. End-to-end physics-informed representation learning from and for satellite ocean remote sensing data. In Intenational Society for Photogrammetry and Remote Sensing Congress.
  59. FedDrive: generalizing federated learning to semantic segmentation in autonomous driving. In IEEE/RSJ International Conference on Intelligent Robots and Systems. 11504–11511.
  60. Human face super-resolution on poor quality surveillance video footage. Neural Computing and Applications 33 (2021), 13505–13523.
  61. Super-resolution and denoising of 4D-flow MRI using physics-informed deep neural nets. Computer Methods and Programs in Biomedicine 197 (2020), 105729.
  62. Deep learning in computed tomography super resolution using multi-modality data training. Medical Physics (2023).
  63. GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification. Neurocomputing 321 (2018), 321–331.
  64. Super-resolution and denoising of fluid flow using physics-informed convolutional neural networks without high-resolution labels. Physics of Fluids 33, 7 (2021), 073603.
  65. Differentiable dynamics for articulated 3d human motion reconstruction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 13190–13200.
  66. Trajectory optimization for physics-based reconstruction of 3d human pose from monocular video. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 13106–13115.
  67. Estimating permeability of 3D micro-CT images by physics-informed CNNs based on DNS. arXiv preprint arXiv:2109.01818 (2021).
  68. DART: A 3D model for remote sensing images and radiative budget of earth surfaces. Modeling and simulation in engineering 2 (2012).
  69. On the self-similarity of line segments in decaying homogeneous isotropic turbulence. Computers & Fluids 180 (2019), 206–217.
  70. Multimodal multi-head convolutional attention with various kernel sizes for medical image super-resolution. In IEEE/CVF Winter Conference on Applications of Computer Vision. 2195–2205.
  71. PET image reconstruction using deep image prior. IEEE Transactions on Medical Imaging 38, 7 (2018), 1655–1665.
  72. Low photon count phase retrieval using deep learning. Physical review letters 121, 24 (2018), 243902.
  73. Hamiltonian neural networks. Advances in neural information processing systems 32 (2019).
  74. A review on 2D instance segmentation based on deep neural networks. Image and Vision Computing (2022), 104401.
  75. MedSRGAN: medical images super-resolution using generative adversarial networks. Multimedia Tools and Applications 79 (2020), 21815–21840.
  76. Furkan Guc and YangQuan Chen. 2021. Fault Cause Assignment with Physics Informed Transfer Learning. IFAC-PapersOnLine 54, 20 (2021), 53–58.
  77. Improved training of wasserstein gans. Advances in neural information processing systems (NeuRIPS) 30 (2017).
  78. Dynamic imaging through random perturbed fibers via physics-informed learning. Optics & Laser Technology 158 (2023), 108923.
  79. xbd: A dataset for assessing building damage from satellite imagery. arXiv preprint arXiv:1911.09296 (2019).
  80. Accelerating stereo image simulation for automotive applications using neural stereo super resolution. IEEE Transactions on Intelligent Transportation Systems (2023).
  81. MRI-MECH: mechanics-informed MRI to estimate esophageal health. Frontiers in Physiology 14 (2023), 1195067.
  82. Boumediene Hamzi and Houman Owhadi. 2021. Learning dynamical systems from data: a simple cross-validation perspective, part I: parametric kernel flows. Physica D: Nonlinear Phenomena 421 (2021), 132817.
  83. Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications. arXiv preprint arXiv:2211.08064 (2022).
  84. Arlen W Harbaugh. 2005. MODFLOW-2005, the US Geological Survey modular ground-water model: the ground-water flow process. Vol. 6. US Department of the Interior, US Geological Survey Reston, VA, USA.
  85. Unetr: Transformers for 3d medical image segmentation. In IEEE/CVF Winter conference on Applications of Computer Vision (WACV). 574–584.
  86. Mask r-cnn. In IEEE International Conference on Computer Vision (ICCV). 2961–2969.
  87. Deep residual learning for image recognition. In IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR). 770–778.
  88. EP-PINNs: Cardiac electrophysiology characterisation using physics-informed neural networks. Frontiers in Cardiovascular Medicine 8 (2022), 2179.
  89. The ERA5 global reanalysis. Quarterly Journal of the Royal Meteorological Society 146, 730 (2020), 1999–2049.
  90. Denoising diffusion probabilistic models. Advances in neural information processing systems (NeuRIPS) 33 (2020), 6840–6851.
  91. Accurate prediction of melt electrowritten laydown patterns from simple geometrical considerations. Advanced Materials Technologies 5, 12 (2020), 2000772.
  92. Case studies: prognostics and health management (PHM). Engineering Design under Uncertainty and Health Prognostics (2019), 303–342.
  93. Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capture. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 6417–6426.
  94. Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop on faces in’Real-Life’Images: detection, alignment, and recognition. EECV, Marseille, France.
  95. Auggan: Cross domain adaptation with gan-based data augmentation. In European Conference on Computer Vision (ECCV). Springer, Munich, Germany, 718–731.
  96. EEG-TCNet: An accurate temporal convolutional network for embedded motor-imagery brain–machine interfaces. In IEEE International Conference on Systems, Man, and Cybernetics. IEEE, Ontario, 2958–2965.
  97. Optical non-line-of-sight physics-based 3d human pose estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Seattle, Washington, 7013–7022.
  98. OpenFOAM: A C++ library for complex physics simulations. In International workshop on coupled methods in numerical dynamics, Vol. 1000. 1–20.
  99. Physics-informed detection and segmentation of type II solar radio bursts. In British Machine Vision Virtual Conference (BMVC). British Machine Vision Association (BMVA), London, UK.
  100. The affine particle-in-cell method. ACM Transactions on Graphics 34, 4 (2015), 1–10.
  101. Semantic segmentation based stereo visual servoing of nonholonomic mobile robot in intelligent manufacturing environment. Expert Systems with Applications 190 (2022), 116203.
  102. A review and meta-analysis of generative adversarial networks and their applications in remote sensing. International Journal of Applied Earth Observation and Geoinformation 108 (2022), 102734.
  103. Soyi Jung and Joongheon Kim. 2021. Adaptive and stabilized real-time super-resolution control for UAV-assisted smart harbor surveillance platforms. Journal of Real-Time Image Processing 18 (2021), 1815–1825.
  104. Elasticity imaging using physics-informed neural networks: Spatial discovery of elastic modulus and Poisson’s ratio. Acta biomaterialia 155 (2023), 400–409.
  105. Physics-informed machine learning. Nature Reviews Physics 3, 6 (2021), 422–440.
  106. A point-cloud deep learning framework for prediction of fluid flow fields on irregular geometries. Physics of Fluids 33, 2 (2021), 027104.
  107. Physics-informed machine learning: case studies for weather and climate modelling. Philosophical Transactions of the Royal Society A 379 (2021), 1–36.
  108. MRI image synthesis for fluid-attenuated inversion recovery and diffusion-weighted images with deep learning. Physical and Engineering Sciences in Medicine 46, 1 (2023), 313–323.
  109. Physics-Informed CNNs for Super-Resolution of Sparse Observations on Dynamical Systems. arXiv preprint arXiv:2210.17319 (2022).
  110. Exploiting Spatial Dimensions of Latent in GAN for Real-Time Image Editing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Nashville, TN, USA, 852–861.
  111. Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering 358 (2020), 112623.
  112. Usgs spectral library version 7 data: Us geological survey data release. United States Geological Survey (USGS): Reston, VA, USA (2017).
  113. NeRF-VAE: A Geometry Aware 3D Scene Generative Model. In International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139). PMLR, Online, 5742–5752.
  114. Imagenet classification with deep convolutional neural networks. Commun. ACM 60, 6 (2017), 84–90.
  115. Full-field structural monitoring using event cameras and physics-informed sparse identification. Mechanical Systems and Signal Processing 145 (2020), 106905.
  116. EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces. Journal of neural engineering 15, 5 (2018), 056013.
  117. Yann LeCun. 1998. The MNIST database of handwritten digits. http://yann. lecun. com/exdb/mnist/ (1998).
  118. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks 3361, 10 (1995), 1995.
  119. Photo-realistic single image super-resolution using a generative adversarial network. In IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Honolulu, HI, 4681–4690.
  120. Super resolution label-free dark-field microscopy by deep learning. Nanoscale (2024).
  121. Learning-based super-resolution interpolation for sub-Nyquist sampled laser speckles. Photonics Research 11, 4 (2023), 631–642.
  122. Matthew Li and Christopher McComb. 2022. Using physics-informed generative adversarial networks to perform super-resolution for multiphase fluid simulations. Journal of Computing and Information Science in Engineering 22, 4 (2022), 044501.
  123. Xinye Li and Ding Chen. 2022. A survey on deep learning-based panoptic segmentation. Digital Signal Processing 120 (2022), 103283.
  124. PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification. In International Conference on Learning Representations (ICLR).
  125. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895 (2020).
  126. Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. Journal of Fluid Mechanics 807 (2016), 155–166.
  127. Hyperspectral remote sensing imagery generation from RGB images based on joint discrimination. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 14 (2021), 7624–7636.
  128. Physics-Informed Hyperspectral Remote Sensing Image Synthesis With Deep Conditional Generative Adversarial Networks. IEEE Transactions on Geoscience and Remote Sensing 60 (2022), 1–15.
  129. Anatomy-aided deep learning for medical image segmentation: a review. Physics in Medicine & Biology 66, 11 (2021), 11TR01.
  130. Po-Yu Liu and Edmund Y Lam. 2018. Image reconstruction using deep learning. arXiv preprint arXiv:1809.10410 (2018).
  131. Semi-supervised medical image classification with relation-driven self-ensembling model. IEEE Transactions on Medical Imaging 39, 11 (2020), 3429–3440.
  132. Image-adaptive YOLO for object detection in adverse weather conditions. In AAAI Conference on Artificial Intelligence, Vol. 36. 1792–1800.
  133. Walking on thin air: Environment-free physics-based markerless motion capture. In 15th Conference on Computer and Robot Vision. IEEE, 158–165.
  134. Fully convolutional networks for semantic segmentation. In IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR). 3431–3440.
  135. WarpPINN: Cine-MR image registration with physics-informed neural networks. Medical Image Analysis 89 (2023), 102925.
  136. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence 3, 3 (2021), 218–229.
  137. Physics-informed gans for coastal flood visualization. arXiv preprint arXiv:2010.08103 (2020).
  138. Pan-GAN: An unsupervised pan-sharpening method for remote sensing image fusion. Information Fusion 62 (2020), 110–120.
  139. Learning neural constitutive laws from motion observations for generalizable pde dynamics. In International Conference on Machine Learning. PMLR, 23279–23300.
  140. Risp: Rendering-invariant state predictor with differentiable simulation and rendering for cross-domain parameter estimation. arXiv preprint arXiv:2205.05678 (2022).
  141. Stéphane Mallat. 2016. Understanding deep convolutional networks. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 374, 2065 (2016), 20150203.
  142. Physics Informed Synthetic Image Generation for Deep Learning-Based Detection of Wrinkles and Folds. Journal of Computing and Information Science in Engineering 23, 3 (2023), 030903.
  143. Bagan: Data augmentation with balancing gan. arXiv preprint arXiv:1803.09655 (2018).
  144. Physics-Informed Spatiotemporal Deep Learning for Emulating Coupled Dynamical Systems.. In AAAI Spring Symposium: MLPS.
  145. When Physics Meets Machine Learning: A Survey of Physics-Informed Machine Learning. arXiv preprint arXiv:2203.16797 (2022).
  146. Learning functional priors and posteriors from data and physics. J. Comput. Phys. 457 (2022), 111073.
  147. Review the state-of-the-art technologies of semantic segmentation based on deep learning. Neurocomputing 493 (2022), 626–646.
  148. Estimating density, velocity, and pressure fields in supersonic flows using physics-informed BOS. Experiments in Fluids 64, 1 (2023), 14.
Citations (13)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets