Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Inferring Hybrid Neural Fluid Fields from Videos (2312.06561v1)

Published 11 Dec 2023 in cs.CV and cs.GR

Abstract: We study recovering fluid density and velocity from sparse multiview videos. Existing neural dynamic reconstruction methods predominantly rely on optical flows; therefore, they cannot accurately estimate the density and uncover the underlying velocity due to the inherent visual ambiguities of fluid velocity, as fluids are often shapeless and lack stable visual features. The challenge is further pronounced by the turbulent nature of fluid flows, which calls for properly designed fluid velocity representations. To address these challenges, we propose hybrid neural fluid fields (HyFluid), a neural approach to jointly infer fluid density and velocity fields. Specifically, to deal with visual ambiguities of fluid velocity, we introduce a set of physics-based losses that enforce inferring a physically plausible velocity field, which is divergence-free and drives the transport of density. To deal with the turbulent nature of fluid velocity, we design a hybrid neural velocity representation that includes a base neural velocity field that captures most irrotational energy and a vortex particle-based velocity that models residual turbulent velocity. We show that our method enables recovering vortical flow details. Our approach opens up possibilities for various learning and reconstruction applications centered around 3D incompressible flow, including fluid re-simulation and editing, future prediction, and neural dynamic scene composition. Project website: https://kovenyu.com/HyFluid/

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Particle image velocimetry. Cambridge university press, 2011.
  2. A parallel multigrid preconditioned conjugate gradient algorithm for groundwater flow simulations. Nuclear science and engineering, 1996.
  3. The quiet revolution of numerical weather prediction. Nature, 2015.
  4. Dennis M Bushnell and KJ Moore. Drag reduction in nature. Annual review of fluid mechanics, 1991.
  5. Physics informed neural fields for smoke reconstruction with sparse data. ACM TOG, 2022.
  6. Vortex methods: theory and practice, volume 8. Cambridge university press, 2000.
  7. Learning vortex dynamics for fluid inference and prediction. In ICLR, 2023.
  8. Neural radiance flow for 4d view synthesis and video processing. In ICCV, 2021.
  9. Coupled fluid density and motion from single views. CGF, 2018.
  10. Scalarflow: a large-scale volumetric data set of real-world scalar transport flows for computer animation and machine learning. ACM TOG, 2019.
  11. Tomographic particle image velocimetry. Experiments in fluids, 2006.
  12. Visual simulation of smoke. In SIGGRAPH, 2001.
  13. Global transport for fluid reconstruction with learned self-supervision. In CVPR, 2021.
  14. Turbulence: the legacy of an kolmogorov. Physics Today, 1996.
  15. From capture to simulation: connecting forward and inverse problems in fluids. ACM TOG, 2014.
  16. Compressive structured light for recovering inhomogeneous participating media. IEEE TPAMI, 2012.
  17. Neurofluid: Fluid dynamics grounding with particle-driven neural radiance fields. In ICML, 2022.
  18. Acquisition of time-varying participating media. ACM TOG, 2005.
  19. Taichi: a language for high-performance computation on spatially sparse data structures. ACM TOG, 2019.
  20. Reconstructing gas flows using light-path approximation. In CVPR, 2013.
  21. Ray tracing volume densities. SIGGRAPH, 1984.
  22. Deep fluids: A generative network for parameterized fluid simulations. In CGF, 2019.
  23. Differentiable physics simulation of dynamics-augmented neural objects. IEEE RA-L, 2023.
  24. Neural 3d video synthesis from multi-view video. In CVPR, 2022.
  25. Pac-nerf: Physics augmented continuum neural radiance fields for geometry-agnostic system identification. In ICLR, 2023.
  26. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, 2021.
  27. Inferring fluid dynamics via inverse rendering. arXiv:2304.04446, 2023.
  28. Nelson Max. Optical models for direct volume rendering. IEEE TVCG, 1995.
  29. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019.
  30. Nerf: Representing scenes as neural radiance fields for view synthesis. arXiv:2003.08934, 2020.
  31. Instant neural graphics primitives with a multiresolution hash encoding. ACM TOG, 2022.
  32. Fluid volume modeling from sparse multi-view images by appearance transfer. ACM TOG, 2015.
  33. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
  34. Nerfies: Deformable neural radiance fields. In CVPR, 2021a.
  35. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv:2106.13228, 2021b.
  36. Automatic differentiation in pytorch. In NeurIPS Workshops, 2017.
  37. Scalable fluid simulation using anisotropic turbulence particles. In SIGGRAPH Asia, 2010.
  38. Stephen B Pope. Turbulent flows. Cambridge university press, 2000.
  39. Guaranteed conservation of momentum for learning particle-based fluid dynamics. In NeurIPS, 2022.
  40. D-nerf: Neural radiance fields for dynamic scenes. In CVPR, 2021.
  41. On the spectral bias of neural networks. In ICML, 2019.
  42. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 2019.
  43. Object scene representation transformer. NeurIPS, 2022.
  44. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, 2023.
  45. A vortex particle method for smoke, water and explosions. In SIGGRAPH, 2005.
  46. An unconditionally stable maccormack method. Journal of Scientific Computing, 2008.
  47. Scene representation networks: Continuous 3d-structure-aware neural scene representations. In NeurIPS, 2019.
  48. Implicit neural representations with periodic activation functions. NeurIPS, 2020.
  49. Fourier features let networks learn high frequency functions in low dimensional domains. In NeurIPS, 2020.
  50. Multi-object manipulation via object-centric neural scattering functions. In CVPR, 2023.
  51. Accelerating eulerian fluid simulation with convolutional networks. In ICML, 2017.
  52. Lagrangian fluid simulation with continuous convolutions. In ICLR, 2019.
  53. Three-dimensional scene flow. In ICCV, 1999.
  54. Latent space physics: Towards learning the temporal evolution of fluid flow. In CGF, 2019.
  55. Space-time neural irradiance fields for free-viewpoint video. In CVPR, 2021.
  56. Unsupervised discovery of object radiance fields. ICLR, 2022.
  57. Learning object-centric neural scattering functions for free-viewpoint relighting and scene composition. TMLR, 2023.
  58. Tomofluid: Reconstructing dynamic fluid from sparse view videos. In CVPR, 2020.
  59. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In CVPR, 2021.
  60. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
Citations (8)
View on Semantic Scholar

Summary

  • The paper presents HyFluid to accurately model fluid velocity and density fields by merging sparse visual data with physics-based losses.
  • It employs a hybrid representation that combines a neural base velocity field with a vortex particle model to capture large-scale and turbulent flows.
  • High-fidelity reconstructions enable fluid re-simulation, future predictions, and integration into dynamic neural scenes.

Understanding Hybrid Neural Fluid Dynamics

Deciphering Fluid Properties from Video

Fluids are omnipresent in nature, yet their dynamic nature poses significant challenges in capturing and understanding their density and velocity fields. Conventional approaches predominantly utilize optical flows but fail to yield accurate results due to the indistinct visual traits of fluids. Addressing these issues, a unique neural approach called hybrid neural fluid fields (HyFluid) has been introduced. HyFluid infers the velocity and density fields of fluids by leveraging visual and physical data from sparse multiview videos.

Tackling the Ambiguity of Fluid Motion

One of the key challenges in interpreting fluid dynamics from video is the intrinsic visual ambiguity. Fluids lack solid color and distinct features that aid tracking, especially in regions where the flow is smooth and the appearance changes minimally. To overcome this, HyFluid employs physics-based losses, stemming from the principles of fluid dynamics captured by Navier-Stokes equations, which enable the modeling of physically plausible, divergence-free velocity fields that interact with the fluid density fields.

Innovations in Velocity Representation

HyFluid introduces a hybrid approach to represent fluid velocity. Blending a neural base velocity field with a vortex particle model, the system breaks down the velocity into two components:

  1. The base velocity field manages most of the irrotational energy, capturing the larger flow motions.
  2. The residual vorticity-driven velocity handles the finer, turbulent flow details that are challenging to depict using standard neural representations.

These components work in conjunction, allowing HyFluid to capture a wide range of fluid motion with varying degrees of turbulence.

Practical Applications and Findings

This innovative approach to understanding fluid fields has profound implications across several domains, such as:

  • Enabling the re-simulation and editing of fluid dynamics.
  • Promising future predictions of fluid movements.
  • Integrating inferred fluid dynamics into dynamic neural scenes.

HyFluid demonstrates the ability to provide high-fidelity reconstructions of fluid density and velocity, surpassing existing methods while upholding the unique turbulent features and complex dynamics of the fluid.

Bridging the Gap in Fluid Dynamics Inference

In conclusion, HyFluid serves to bridge the gap between neural dynamic reconstruction methods and the intricacies of fluid dynamics inference. It presents a comprehensive solution by incorporating physical constraints, a detailed representation of turbulent velocities, and the utilization of visual cues from complex multiview video data to recover accurate and richly detailed fluid fields.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube