2000 character limit reached
Amodal Completion via Progressive Mixed Context Diffusion (2312.15540v1)
Published 24 Dec 2023 in cs.CV
Abstract: Our brain can effortlessly recognize objects even when partially hidden from view. Seeing the visible of the hidden is called amodal completion; however, this task remains a challenge for generative AI despite rapid progress. We propose to sidestep many of the difficulties of existing approaches, which typically involve a two-step process of predicting amodal masks and then generating pixels. Our method involves thinking outside the box, literally! We go outside the object bounding box to use its context to guide a pre-trained diffusion inpainting model, and then progressively grow the occluded object and trim the extra background. We overcome two technical challenges: 1) how to be free of unwanted co-occurrence bias, which tends to regenerate similar occluders, and 2) how to judge if an amodal completion has succeeded. Our amodal completion method exhibits improved photorealistic completion results compared to existing approaches in numerous successful completion cases. And the best part? It doesn't require any special training or fine-tuning of models.
- Amodal intra-class instance segmentation: New dataset and benchmark, 2023.
- Oconet: Image extrapolation by object completion. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2307–2317, 2021.
- Monet: Unsupervised scene decomposition and representation, 2019.
- Training-free layout control with cross-attention guidance, 2023.
- Inout: Diverse image outpainting via gan inversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11431–11440, 2022.
- Object-driven multi-layer scene decomposition from a single image, 2019.
- Diffusion models beat gans on image synthesis, 2021.
- Segan: Segmenting and generating the invisible, 2018.
- Genesis: Generative scene inference and sampling with object-centric latent representations, 2020.
- Training-free structured diffusion guidance for compositional text-to-image synthesis, 2023.
- Dreamsim: Learning new dimensions of human visual similarity using synthetic data. arXiv:2306.09344, 2023.
- Multi-object representation learning with iterative variational inference, 2020.
- Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
- Denoising diffusion probabilistic models, 2020.
- Sail-vos: Semantic amodal instance level video object segmentation-a synthetic dataset and baselines. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3105–3115, 2019.
- Segment anything, 2023.
- Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://storage.googleapis.com/openimages/web/index.html, 2017.
- The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 2020.
- Instance-wise occlusion and depth orders in natural scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21210–21221, 2022.
- Contextual outpainting with object-level contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11451–11460, 2022.
- Amodal instance segmentation. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 677–693. Springer, 2016.
- Controllable and progressive image extrapolation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2140–2149, 2021.
- Microsoft coco: Common objects in context, 2015.
- Variational amodal object completion. In Advances in Neural Information Processing Systems, pages 16246–16257. Curran Associates, Inc., 2020.
- Zero-1-to-3: Zero-shot one image to 3d object, 2023a.
- Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2023b.
- S. Lloyd. Least squares quantization in pcm. IEEE Transactions on Information Theory, 28(2):129–137, 1982.
- Sdedit: Guided image synthesis and editing with stochastic differential equations, 2022.
- Dragondiffusion: Enabling drag-style manipulation on diffusion models, 2023.
- Improved denoising diffusion probabilistic models, 2021.
- Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022.
- Counterfactual image networks. 2018.
- How to make a pizza: Learning a compositional layer-based gan model, 2019.
- Localizing object-level shape variations with text-to-image diffusion models, 2023.
- Amodal instance segmentation with kins dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3014–3023, 2019.
- Learning transferable visual models from natural language supervision, 2021.
- Hierarchical text-conditional image generation with clip latents, 2022.
- High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
- U-net: Convolutional networks for biomedical image segmentation, 2015.
- Photorealistic text-to-image diffusion models with deep language understanding, 2022.
- Deep unsupervised learning using nonequilibrium thermodynamics, 2015.
- Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
- Resolution-robust large mask inpainting with fourier convolutions, 2021.
- Emergent correspondence from image diffusion, 2023.
- Robert van Lier. Investigating global effects in visual occlusion: from a partly occluded square to the back of a tree-trunk. Acta Psychologica, 102:203–220, 1999.
- Attention is all you need. Advances in neural information processing systems, 30, 2017.
- Structure-guided image outpainting, 2022.
- Wide-context semantic image extrapolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
- Image outpainting: Hallucinating beyond the image. IEEE Access, 8:173576–173583, 2020.
- Visualizing the invisible: Occluded vehicle segmentation and recovery, 2019.
- Paint by example: Exemplar-based image editing with diffusion models, 2022.
- Inst-inpaint: Instructing to remove objects with diffusion models, 2023.
- Temporal properties of amodal completion: Influences of knowledge. Vision research, 145, 2018.
- What does stable diffusion know about the 3d scene? arXiv preprint arXiv:2310.06836, 2023.
- Self-supervised scene de-occlusion, 2020.
- A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. 2023a.
- Adding conditional control to text-to-image diffusion models, 2023.
- Perceptual artifacts localization for image synthesis tasks, 2023b.
- The unreasonable effectiveness of deep features as a perceptual metric, 2018.
- Visiting the invisible: Layer-by-layer completed scene decomposition, 2021.
- Human de-occlusion: Invisible perception and recovery for humans. In Computer Vision and Pattern Recognition (CVPR), 2021.
- Semantic amodal segmentation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.