Finding NeMO: A Geometry-Aware Representation of Template Views for Few-Shot Perception

Abstract: We present Neural Memory Object (NeMO), a novel object-centric representation that can be used to detect, segment and estimate the 6DoF pose of objects unseen during training using RGB images. Our method consists of an encoder that requires only a few RGB template views depicting an object to generate a sparse object-like point cloud using a learned UDF containing semantic and geometric information. Next, a decoder takes the object encoding together with a query image to generate a variety of dense predictions. Through extensive experiments, we show that our method can be used for few-shot object perception without requiring any camera-specific parameters or retraining on target data. Our proposed concept of outsourcing object information in a NeMO and using a single network for multiple perception tasks enhances interaction with novel objects, improving scalability and efficiency by enabling quick object onboarding without retraining or extensive pre-processing. We report competitive and state-of-the-art results on various datasets and perception tasks of the BOP benchmark, demonstrating the versatility of our approach. https://github.com/DLR-RM/nemo

• The paper introduces Neural Memory Objects (NeMO), a novel representation that decouples object features from network weights to enhance few-shot perception.
• It employs a Vision Transformer-based encoder with a Geometric Mapping Block to effectively fuse 2D image data with 3D spatial information.
• Experiments on benchmarks like HOPEv2 and HANDAL demonstrate NeMO's efficiency in object detection, pose estimation, and adaptability to unseen objects.

Detailed Analysis of "Finding NeMO: A Geometry-Aware Representation of Template Views for Few-Shot Perception"

Introduction

The paper "Finding NeMO: A Geometry-Aware Representation of Template Views for Few-Shot Perception" introduces Neural Memory Objects (NeMO), a novel object-centric representation aimed at augmenting object detection, segmentation, and pose estimation in scenarios involving previously unseen objects. It leverages RGB images to generate sparse object-like point clouds through a learned Unsigned Distance Field (UDF) without the requirement for camera-specific parameters or retraining on new data. NeMO's potential to operate both in model-free and model-based scenarios sets it apart in the domain of few-shot perception.

Figure 1: Overview of the NeMO approach, showing the utilization of a multi-view encoder to generate object-centric geometric encoding.

Methodology

The NeMO framework is built upon an encoder-decoder architecture. The encoder processes a set of RGB images to create a structured, sparse NeMO representation encapsulating semantic and geometric features. Unlike conventional methods that rely on latent vector compression, NeMO maintains an interpretable point cloud abstraction, allowing flexible modifications such as translation and rotation.

NeMO Encoder

The encoder uses Vision Transformers (ViTs) to extract image features, combining these with sampled 3D points through multiple transformer decoder blocks. A key innovation here is the Geometric Mapping Block, which enhances image features with 3D spatial information, effectively bridging the gap between the 2D RGB inputs and the 3D world representation.

Figure 2: Geometric Mapping Block, fusing updated image features with pre-processed geometric features.

NeMO Representation

NeMO amalgamates surface point estimation and feature processing into a unified structure, facilitating object-agnostic interaction. It offers scalability with a memory footprint that remains invariant to the number of input views. This results in improved adaptability and efficiency in real-world applications, with the encoder supporting incremental refinement.

NeMO Decoder

Upon receiving a query image, the decoder leverages the NeMO representation to produce dense predictions necessary for object perception tasks. This procedure is robustly designed to enable downstream processing, thereby extracting multiple features essential for applications like amodal segmentation and pose estimation.

Experiments

The research validates NeMO through extensive experimentation, comparing its efficacy against existing state-of-the-art methods across several benchmarks including the BOP challenge datasets. Notably, NeMO excels in few-shot perception without retracting model weights, demonstrating improved scalability and performance.

Figure 3: Qualitative example of model-free few-shot detection and pose estimation illustrating NeMO's capability in cluttered scenes.

Model-Free Perception

In the model-free setup, NeMO outperforms existing methods in detecting and estimating poses of unseen objects, showcasing its adaptability in real-world scenarios where CAD models are unavailable. The paper reports competitive results on the HOPEv2 and HANDAL datasets, underscoring the utility of amodal segmentation predictions in object detection.

Model-Based Perception

Despite NeMO's design primarily for model-free applications, it also achieves competitive results in model-based settings using CAD models. This versatility is evident in scenarios where rendered images aid in template generation, further proving its potential in crossover applications.

Figure 4: Object surface reconstruction and camera pose estimation, illustrating NeMO's capabilities in dynamic scenarios.

Implications and Future Work

The NeMO framework provides substantial implications for the future of few-shot perception, particularly in advancing object recognition tasks without reliance on exhaustive 3D modeling. Its capacity to decouple object knowledge from model weights paves the way for rapid adaptation and onboarding of novel objects, enhancing interaction capabilities in fields like robotics and autonomous systems.

Future research should focus on optimizing NeMO for symmetric objects and refining its handling of textureless surfaces. Additionally, expanding the synthetic training dataset to include a broader range of object types will improve its robustness in diverse environmental conditions. Given its current trajectory, NeMO is poised to transform perception paradigms, enabling efficient onboarding and recognition in dynamic, cluttered scenes.

Figure 5: Analysis of memory consumption versus number of template images, highlighting NeMO's efficient use of resources.

Conclusion

The introduction of Neural Memory Objects through the "Finding NeMO" framework marks a significant step forward in few-shot perception. By creating a geometry-aware representation that decouples object-specific information from neural network weights, NeMO enables scalable, efficient interaction with new objects across both model-free and model-based domains. This research sets the stage for future developments in AI, potentially leading to more intuitive and adaptable perception systems.

This paper advocates for decoupling visual and geometric data from model weights, indicating a direction for future research to enhance AI's adaptability to novel scenarios without extensive retraining, thereby revolutionizing the landscape of object-centric perception tasks.