2000 character limit reached
Semantic Attention Flow Fields for Monocular Dynamic Scene Decomposition (2303.01526v2)
Published 2 Mar 2023 in cs.CV
Abstract: From video, we reconstruct a neural volume that captures time-varying color, density, scene flow, semantics, and attention information. The semantics and attention let us identify salient foreground objects separately from the background across spacetime. To mitigate low resolution semantic and attention features, we compute pyramids that trade detail with whole-image context. After optimization, we perform a saliency-aware clustering to decompose the scene. To evaluate real-world scenes, we annotate object masks in the NVIDIA Dynamic Scene and DyCheck datasets. We demonstrate that this method can decompose dynamic scenes in an unsupervised way with competitive performance to a supervised method, and that it improves foreground/background segmentation over recent static/dynamic split methods. Project Webpage: https://visual.cs.brown.edu/saff
- Deep vit features as dense visual descriptors. arXiv preprint arXiv:2112.05814, 2021.
- Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855–5864, 2021.
- Baking in the feature: Accelerating volumetric segmentation by rendering feature maps. ArXiv, abs/2209.12744, 2022.
- Monet: Unsupervised scene decomposition and representation. ArXiv, abs/1901.11390, 2019.
- The 2019 davis challenge on vos: Unsupervised multi-object segmentation. arXiv:1905.00737, 2019.
- Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
- SAVi++: Towards end-to-end object-centric learning from real-world videos. In Advances in Neural Information Processing Systems, 2022.
- Nerf-sos: Any-view self-supervised object segmentation on complex scenes. ArXiv, abs/2209.08776, 2022.
- Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation. In International Conference on 3D Vision (3DV), 2022.
- Monocular dynamic view synthesis: A reality check. In NeurIPS, 2022.
- Kubric: A scalable dataset generator. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3739–3751, 2022.
- Multi-object representation learning with iterative variational inference. In ICML, 2019.
- Revealing occlusions with 4d neural fields. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3001–3011, 2022.
- Simone: View-invariant, temporally-abstracted object representations via unsupervised video decomposition. ArXiv, abs/2106.03849, 2021.
- Conditional Object-Centric Learning from Video. In International Conference on Learning Representations (ICLR), 2022.
- Decomposing nerf for editing via feature field distillation. In Advances in Neural Information Processing Systems, volume 35, 2022.
- Panoptic neural fields: A semantic object-aware neural scene representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12871–12881, June 2022.
- Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
- Video instance segmentation with a propose-reduce paradigm. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1719–1728, 2021.
- Object-centric learning with slot attention. arXiv preprint arXiv:2006.15055, 2020.
- Feature-realistic neural fusion for real-time, open set scene understanding. ArXiv, abs/2210.03043, 2022.
- Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In CVPR, 2022.
- Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
- Unsupervised layered image decomposition into object prototypes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8640–8650, October 2021.
- Giraffe: Representing scenes as compositional generative neural feature fields. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
- Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. ACM Trans. Graph., 40(6), dec 2021.
- Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3), 2022.
- Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
- Bridging the gap to real-world object-centric learning. ArXiv, abs/2209.14860, 2022.
- Unsupervised salient object detection with spectral cluster voting. In CVPRW, 2022.
- Unsupervised discovery and composition of object light fields. ArXiv, abs/2205.03923, 2022.
- Decomposing 3d scenes into objects via unsupervised volume segmentation. ArXiv, abs/2104.01148, 2021.
- Scalability in perception for autonomous driving: Waymo open dataset. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
- Raft: Recurrent all-pairs field transforms for optical flow. In European conference on computer vision, pages 402–419. Springer, 2020.
- Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 12939–12950, 2021.
- Neural Feature Fusion Fields: 3D distillation of self-supervised 2D image representations. In Proceedings of the International Conference on 3D Vision (3DV), 2022.
- NeuralDiff: Segmenting 3D objects that move in egocentric videos. In Proceedings of the International Conference on 3D Vision (3DV), 2021.
- Entity abstraction in visual model-based reinforcement learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 1439–1456. PMLR, 30 Oct–01 Nov 2020.
- Self-supervised transformers for unsupervised object discovery using normalized cut. In Conference on Computer Vision and Pattern Recognition, 2022.
- D2nerf: Self-supervised decoupling of dynamic and static objects from a monocular video. ArXiv, abs/2205.15838, 2022.
- Neural fields in visual computing and beyond. Computer Graphics Forum, 2022.
- Learning object-compositional neural radiance field for editable scene rendering. In International Conference on Computer Vision (ICCV), October 2021.
- Video instance segmentation. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2019.
- Deformable sprites for unsupervised video decomposition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
- Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
- Unsupervised discovery of object radiance fields. In International Conference on Learning Representations, 2022.
- In-place scene labelling and understanding with implicit scene representation. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.