Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Equivariant Single View Pose Prediction Via Induced and Restricted Representations (2307.03704v1)

Published 7 Jul 2023 in cs.CV, cs.LG, and math.GR

Abstract: Learning about the three-dimensional world from two-dimensional images is a fundamental problem in computer vision. An ideal neural network architecture for such tasks would leverage the fact that objects can be rotated and translated in three dimensions to make predictions about novel images. However, imposing SO(3)-equivariance on two-dimensional inputs is difficult because the group of three-dimensional rotations does not have a natural action on the two-dimensional plane. Specifically, it is possible that an element of SO(3) will rotate an image out of plane. We show that an algorithm that learns a three-dimensional representation of the world from two dimensional images must satisfy certain geometric consistency properties which we formulate as SO(2)-equivariance constraints. We use the induced and restricted representations of SO(2) on SO(3) to construct and classify architectures which satisfy these geometric consistency constraints. We prove that any architecture which respects said consistency constraints can be realized as an instance of our construction. We show that three previously proposed neural architectures for 3D pose prediction are special cases of our construction. We propose a new algorithm that is a learnable generalization of previously considered methods. We test our architecture on three pose predictions task and achieve SOTA results on both the PASCAL3D+ and SYMSOL pose estimation tasks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010.
  2. Multiple View Geometry in Computer Vision. Cambridge University Press, 2 edition, 2004. doi: 10.1017/CBO9780511811685.
  3. A survey of structure from motion, 2017. URL https://arxiv.org/abs/1701.08493.
  4. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges, 2021. URL https://arxiv.org/abs/2104.13478.
  5. Steerable cnns. axriv, 2016a. doi: 10.48550/ARXIV.1612.08498. URL https://arxiv.org/abs/1612.08498.
  6. On the generalization of equivariance and convolution in neural networks to the action of compact groups, 2018. URL https://arxiv.org/abs/1802.03690.
  7. Intertwiners between induced representations (with applications to the theory of equivariant neural networks), 2018a. URL https://arxiv.org/abs/1803.10743.
  8. A wigner-eckart theorem for group equivariant convolution kernels, 2020. URL https://arxiv.org/abs/2010.10952.
  9. Group equivariant convolutional networks. axriv, 2016b. doi: 10.48550/ARXIV.1602.07576. URL https://arxiv.org/abs/1602.07576.
  10. Image to icosahedral projection for SO⁢(3)SO3\mathrm{SO}(3)roman_SO ( 3 ) object reasoning from single-view images, 2022. URL https://arxiv.org/abs/2207.08925.
  11. Equivariant multi-view networks, 2019a. URL https://arxiv.org/abs/1904.00993.
  12. Image to sphere: Learning equivariant features for efficient pose prediction. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=_2bDpAtr7PI.
  13. Beyond pascal: A benchmark for 3d object detection in the wild. In IEEE Winter Conference on Applications of Computer Vision, pages 75–82, 2014. doi: 10.1109/WACV.2014.6836101.
  14. Implicit-pdf: Non-parametric representation of probability distributions on the rotation manifold, 2022.
  15. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995, 1995.
  16. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155, 2018.
  17. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  18. Tensor field networks: Rotation- and translation-equivariant neural networks for 3d point clouds, 2018.
  19. The surprising effectiveness of equivariant models in domains with latent symmetry, 2022. URL https://arxiv.org/abs/2211.09231.
  20. General e⁢(2)𝑒2e(2)italic_e ( 2 )-equivariant steerable cnns, 2021.
  21. 3d steerable cnns: Learning rotationally equivariant features in volumetric data, 2018a.
  22. Spherical cnns, 2018b.
  23. Explorations in homeomorphic variational auto-encoding, 2018.
  24. Learning symmetric embeddings for equivariant world models. arXiv preprint arXiv:2204.11371, 2022.
  25. Cross-domain 3d equivariant image embeddings. In International Conference on Machine Learning, pages 1812–1822. PMLR, 2019b.
  26. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):1231–1237, 2013.
  27. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint arXiv:1711.00199, 2017.
  28. Reconstructing continuous distributions of 3d protein structure from cryo-em images, 2020.
  29. Viewpoints and keypoints, 2015.
  30. A mixed classification-regression framework for 3d pose estimation from 2d images, 2018.
  31. On the continuity of rotation representations in neural networks, 2020.
  32. Romain Brégier. Deep regression on manifolds: A 3d rotation case study, 2021.
  33. Spherical regression: Learning viewpoints, surface normals and 3d rotations on n-spheres, 2019.
  34. Deep bingham networks: Dealing with uncertainty and ambiguity in pose estimation, 2020.
  35. Deep directional statistics: Pose estimation with uncertainty quantification, 2018.
  36. A laplace-inspired distribution on SO(3) for probabilistic rotation estimation. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=Mvetq8DO05O.
  37. Harmonic Analysis on Finite Groups: Representation Theory, Gelfand Pairs and Markov Chains. Cambridge Studies in Advanced Mathematics. Cambridge University Press, 2008. doi: 10.1017/CBO9780511619823.
  38. Learning steerable filters for rotation equivariant cnns, 2018b.
  39. Nonlinearities in steerable so(2)-equivariant cnns, 2021.
  40. Gauge equivariant mesh cnns: Anisotropic convolutions on geometric graphs, 2021.
  41. A functional approach to rotation equivariant non-linearities for tensor field networks. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13169–13178, 2021. doi: 10.1109/CVPR46437.2021.01297.
  42. Unified fourier-based kernel and nonlinearity design for equivariant networks on homogeneous spaces, 2022.
  43. e3nn: Euclidean neural networks, 2022.
  44. Tom Leinster. Basic category theory, 2016.
  45. Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views, 2015.
  46. Pytorch: An imperative style, high-performance deep learning library, 2019.
  47. Probabilistic regression with huber distributions, 2021.
  48. Deep orientation uncertainty learning based on a bingham loss. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=ryloogSKDS.
  49. On the symmetries of the synchronization problem in cryo-EM: Multi-frequency vector diffusion maps on the projective plane. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=owDcdLGgEm.
  50. Invariant slot attention: Object discovery with slot-centric reference frames, 2023.
  51. Object scene representation transformer, 2022.
  52. A. Zee. Group Theory in a Nutshell for Physicists. In a Nutshell. Princeton University Press, 2016. ISBN 9780691162690. URL https://books.google.com/books?id=FWkujgEACAAJ.
  53. J. P. Serre. Groupes finis, 2005. URL https://arxiv.org/abs/math/0503154.
  54. Induced representations and Mackey theory, page 399–425. Cambridge Studies in Advanced Mathematics. Cambridge University Press, 2018. doi: 10.1017/9781316856383.012.
  55. 3d shapenets: A deep representation for volumetric shapes, 2015.
  56. Sparse steerable convolutions: An efficient learning of SE(3)-equivariant features for estimation and tracking of object poses in 3d space. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=Fa-w-10s7YQ.
  57. Steerable partial differential operators for equivariant neural networks, 2022.
  58. Deconstructing self-supervised monocular reconstruction: The design decisions that matter, 2022.
  59. Monocular depth estimation using diffusion models, 2023.
  60. Dense depth estimation in monocular endoscopy with self-supervised learning methods, 2019.
  61. Photometric single-view dense 3d reconstruction in endoscopy, 2022.
  62. M4depth: Monocular depth estimation for autonomous vehicles in unseen environments, 2022.
  63. Reducing so(3) convolutions to so(2) for efficient equivariant gnns, 2023.
  64. Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):359–366, 1989. ISSN 0893-6080. doi: https://doi.org/10.1016/0893-6080(89)90020-8. URL https://www.sciencedirect.com/science/article/pii/0893608089900208.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now