Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Preconditioners for the Stochastic Training of Neural Fields (2402.08784v2)

Published 13 Feb 2024 in cs.CV and cs.LG

Abstract: Neural fields encode continuous multidimensional signals as neural networks, enabling diverse applications in computer vision, robotics, and geometry. While Adam is effective for stochastic optimization, it often requires long training times. To address this, we explore alternative optimization techniques to accelerate training without sacrificing accuracy. Traditional second-order methods like L-BFGS are unsuitable for stochastic settings. We propose a theoretical framework for training neural fields with curvature-aware diagonal preconditioners, demonstrating their effectiveness across tasks such as image reconstruction, shape modeling, and Neural Radiance Fields (NeRF).

Definition Search Book Streamline Icon: https://streamlinehq.com
References (69)
  1. Augmenting implicit neural shape representations with explicit deformation fields. arXiv preprint arXiv:2108.08931, 2021.
  2. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167–175, 2003.
  3. An estimator for the diagonal of a matrix. Applied numerical mathematics, 57(11-12):1214–1229, 2007.
  4. Optimization methods for large-scale machine learning. SIAM review, 60(2):223–311, 2018.
  5. Andrew M Bradley. Algorithms for the equilibration of matrices and their application to limited-memory quasi-newton methods. Technical report, STANFORD UNIV CA, 2010.
  6. Matrix-free approximate equilibration. arXiv preprint arXiv:1110.2805, 2011.
  7. Charles George Broyden. The convergence of a class of double-rank minimization algorithms 1. general considerations. IMA Journal of Applied Mathematics, 6(1):76–90, 1970.
  8. Tensorf: Tensorial radiance fields. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII, pages 333–350. Springer, 2022a.
  9. Factor fields: A unified framework for neural fields and beyond. arXiv preprint arXiv:2302.01226, 2023a.
  10. Fully body visual self-modeling of robot morphologies. Science Robotics, 7(68):eabn1944, 2022b.
  11. Dehazenerf: Multiple image haze removal and 3d shape reconstruction using neural radiance fields. arXiv preprint arXiv:2303.11364, 2023b.
  12. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8628–8638, 2021.
  13. Neural motion fields: Encoding grasp trajectories as implicit value functions. arXiv preprint arXiv:2206.14854, 2022c.
  14. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5939–5948, 2019.
  15. Gaussian activated neural radiance fields for high fidelity reconstruction and pose estimation. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII, pages 264–280. Springer, 2022.
  16. Equilibrated adaptive learning rates for non-convex optimization. Advances in neural information processing systems, 28, 2015.
  17. Nasa neural articulated shape approximation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, pages 612–628, Cham, 2020. Springer International Publishing.
  18. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
  19. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5501–5510, 2022.
  20. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  21. Donald Goldfarb. A family of variable-metric methods derived by variational means. Mathematics of computation, 24(109):23–26, 1970.
  22. Anne Greenbaum. Iterative methods for solving linear systems. SIAM, 1997.
  23. A simple estimate of the condition number of a linear system. The College Mathematics Journal, 26(1):2–5, 1995.
  24. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pages 1842–1850. PMLR, 2018.
  25. Sape: Spatially-adaptive progressive encoding for neural optimization. Advances in Neural Information Processing Systems, 34:8820–8832, 2021.
  26. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  27. Xi-Lin Li. Preconditioned stochastic gradient descent. IEEE transactions on neural networks and learning systems, 29(5):1454–1466, 2017.
  28. 3d neural scene representations for visuomotor control. In Conference on Robot Learning, pages 112–123. PMLR, 2022.
  29. Barf: Bundle-adjusting neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5741–5751, 2021.
  30. Iteration complexity analysis of multi-block admm for a family of convex minimization without strong convexity. Journal of Scientific Computing, 69:52–81, 2016.
  31. Bacon: Band-limited coordinate networks for multiscale scene representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16252–16262, 2022.
  32. Andy Lomas. Synthesis of abstract dynamic quasiperiodic 3d forms using sirens. Proceedings of EVA London 2022, pages 08–12, 2022.
  33. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pages 2408–2417. PMLR, 2015.
  34. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 38(4):1–14, 2019.
  35. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021.
  36. The implicit bias of minima stability: A view from function space. Advances in Neural Information Processing Systems, 34:17749–17761, 2021.
  37. Numerical optimization. Springer, 1999.
  38. Learning neural radiance fields from multi-view geometry. arXiv preprint arXiv:2210.13041, 2022.
  39. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  40. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865–5874, 2021.
  41. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318–10327, 2021.
  42. Optimal diagonal preconditioning. arXiv preprint arXiv:2209.00809, 2022.
  43. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301–5310. PMLR, 2019.
  44. S. Ramasinghe and S. Lucey. Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPs. In ECCV, 2022.
  45. Derf: Decomposed radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14153–14161, June 2021.
  46. Hogwild!: A lock-free approach to parallelizing stochastic gradient descent. Advances in neural information processing systems, 24, 2011.
  47. KiloNeRF: Speeding Up Neural Radiance Fields With Thousands of Tiny MLPs. In ICCV, 2021.
  48. Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747, 2016.
  49. W Rudin. Functional analysis tata mcgraw, 1973.
  50. Yousef Saad. Iterative methods for sparse linear systems. SIAM, 2003.
  51. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  52. Miner: Multiscale implicit neural representation. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIII, pages 318–333. Springer, 2022.
  53. Wire: Wavelet implicit neural representations. arXiv preprint arXiv:2301.05187, 2023.
  54. Curvature-aware training for coordinate networks. arXiv preprint arXiv:2305.08552, 2023.
  55. Implicit Neural Representations with Periodic Activation Functions. In NIPS, 2020.
  56. Scene representation networks: Continuous 3d-structure-aware neural scene representations. Advances in Neural Information Processing Systems, 32, 2019.
  57. Adversarial generation of continuous images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10753–10764, 2021.
  58. Measure theory, integration, and hilbert spaces, 2005.
  59. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5459–5469, 2022.
  60. Coil: Coordinate-based internal learning for imaging inverse problems. arXiv preprint arXiv:2102.05181, 2021.
  61. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537–7547, 2020.
  62. Learning implicit priors for motion optimization. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7672–7679. IEEE, 2022.
  63. Nerf–: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064, 2021a.
  64. NeRF−⁣−--- -: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064, 2021b.
  65. Sinerf: Sinusoidal neural radiance fields for joint pose estimation and scene reconstruction. arXiv preprint arXiv:2210.04553, 2022.
  66. Adahessian: An adaptive second order optimizer for machine learning. In proceedings of the AAAI conference on artificial intelligence, volume 35, pages 10665–10673, 2021.
  67. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4578–4587, June 2021.
  68. Trading positional complexity vs deepness in coordinate networks. In Computer Vision – ECCV 2022, pages 144–160, Cham, 2022. Springer Nature Switzerland.
  69. Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12786–12796, 2022.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now