MemPose: Category-level Object Pose Estimation with Memory

Abstract: In the pursuit of robust and generalizable category-level object pose estimation, most existing methods adopt parametric formulations that learn effective representations from data, yet they primarily encode category-level patterns into fixed shape priors or static parameter weights, which limits their scalability to highly diverse instances. In this paper, we rethink category-level pose estimation from a memory-centric perspective and present MemPose, a memory-augmented framework that explicitly incorporates category-level geometric memory into the pose estimation pipeline. We introduce an external memory buffer that stores and dynamically updates structural representations from previously observed instances, enabling the model to leverage accumulated experience to support current perception. Extensive experiments on four challenging benchmarks (REAL275, CAMERA25, Housecat6D and Wild6D) demonstrate the superiority of our proposed method over previous state-of-the-art approaches.

• The paper introduces a dynamic memory buffer that accumulates historical geometric features to enhance category-level object pose prediction.
• It combines RGB-D inputs with semantic and geometric encoders, using similarity-based merging and attention for robust keypoint estimation.
• Extensive experiments on benchmarks such as REAL275 and Housecat6D demonstrate significant performance gains over static, prior-based models.

Memory-Augmented Category-Level Pose Estimation: MemPose

Motivation and Limitations of Existing Approaches

Category-level object pose estimation (COPE) targets 9-DoF pose prediction for arbitrary objects within predefined categories, eschewing reliance on instance-specific CAD models. Prior approaches encode category-level information either via static shape priors or embed it in fixed network parameters. Shape priors, often constructed as average shapes, cannot accommodate the structural variety encountered across category members and are non-adaptive to new data. Parametric models, while powerful, suffer from static parameters, lacking a dynamic mechanism to accumulate and exploit historical geometric experiences. This bottleneck restricts scalability and generalization across diverse object instances.

MemPose proposes a unified, memory-centric architecture for COPE, inspired by human analogical and memory-driven reasoning. It introduces an explicit geometric memory buffer capable of storing and updating structural representations from previous observations. This memory explicitly accumulates analogs of encountered geometries, providing retrieval and fusion mechanisms for pose prediction on new inputs.

Figure 1: Overview contrasting static prior-based pipelines (a) with MemPose's dynamic, memory-augmented system (b).

Methodology: Architecture and Memory Integration

MemPose operates on RGB-D inputs processed via MaskRCNN-derived segmentation, yielding cropped images and point clouds. Point features are extracted using PointNet++; semantic RGB features from DINOv2 (ViT-S/14). Local keypoints are detected via instance-adaptive attention between learnable category embeddings and combined features, producing discriminative keypoint-level representations.

The core innovation is the memory module—a category-aware buffer $\mathcal{M}^{(c)} \in \mathbb{R}^{L \times C}$ for each category $c$. Memory entries are initialized as empty and populated with pooled local features during training. Upon capacity, a similarity-based token merge mechanism updates entries: new features are averaged with the most similar memory entries, optionally perturbed by Gumbel noise or selected from top-$k$ matches. This ensures persistent, geometry-aware, non-redundant updates.

For retrieval and fusion, global features formed by geometric aggregation act as queries; memory entries constitute the keys and values in an attention-based readout. Adaptive gating guarantees robust integration, controlling the relative weight between current global and retrieved memory features. The fused representation enables pose and size regression through learned correspondences with NOCS coordinates.

Figure 2: MemPose pipeline: semantic-geometric encoding, dynamic memory formation, and joint pose/size estimation via memory-augmented features.

Loss Formulation

Training is supervised with a composite loss: keypoint pose loss ($\mathcal{L}_{pose}$, L2-norm), diversity loss enforcing spatial distribution among keypoints, object-aware Chamfer distance for surface proximity, NOCS coordinate projection match (SmoothL1), and partial reconstruction loss (Chamfer distance between reconstructed and observed geometry). A set of hyperparameters weigh each term, balancing keypoint fidelity, geometric reconstruction, and pose estimation.

Experimental Evaluation

MemPose was evaluated across four COPE benchmarks: REAL275, CAMERA25, Housecat6D, and Wild6D. The metrics encompass mAP for 3D IoU under varying thresholds and combined rotation/translation measures (n°m cm).

On REAL275, MemPose achieves the best results across all metrics, surpassing both prior-based and recent prior-free models. Specifically, it outperforms GCE-Pose by +2.9% 5°2 cm, +2.6% 5°5 cm; exceeds SpherePose by +1.7% 5°2 cm, +2.1% 10°2 cm. On Housecat6D, MemPose improves $IoU_{50}$ by +5.4% and 10°2cm by +3.3% over GCE-Pose, and markedly outperforms AG-Pose and SpherePose.

CAMERA25 results further confirm MemPose's efficacy, with +2.2% in 10°2cm over SpherePose, and +1.9% mIoU$_{75}$ over AG-Pose. On Wild6D, it delivers +4.3% IoU$_{75}$ improvement, with an average +2.5% gain on n°m cm thresholds.

Qualitative comparisons demonstrate superior pose prediction and robustness to shape diversity and occlusion. The ablation studies dissect the memory module, evidencing that: storing local (not global) features as memory entries yields strongest performance; warm-up initialization is crucial, similarity-based merging and gating maximize retrieval utility, and memory size is a key tradeoff (96 entries as optimal). Visualization of memory attention scores during inference reveals retrieval of strongly pose-relevant category instances.

Memory Update and Efficiency

The memory buffer is non-parametric and does not participate in backpropagation. The architecture incurs negligible parameter overhead relative to baselines like AG-Pose (223M vs 225M params with DINOv2, equal training time, comparable inference FPS). Replacing the memory module with MLPs of the same size confirms improvement is due to dynamic memory, not parameter scale.

In inference, memory can be either frozen (standard evaluations) or updated online (test-time adaptation), with the latter yielding further gains (e.g., REAL275: +5.6% 5°2 cm, +3.7% 5°5 cm vs AG-Pose). This positions MemPose for agentic, continual learning scenarios.

Practical and Theoretical Implications

MemPose demonstrates that a dynamically updatable geometric memory buffer both improves pose accuracy and enables scalable, adaptive generalization across category-level structural diversity. The paradigm bridges parametric perception (feature extraction) and non-parametric memory (experience accumulation), analogous to human perceptual reasoning. Future directions include integration with larger-scale datasets, application to open-set categories, and further exploration of lifelong adaptive memory mechanisms.

Conclusion

MemPose introduces a robust, memory-augmented pipeline for category-level object pose estimation, unifying parametric learning and dynamic geometric memory. Extensive empirical results validate the superiority of memory-centric COPE, marking a shift towards architectures capable of analogical, experience-driven prediction. This methodology unlocks new directions in scalable 3D vision, robotic interaction, and agentic learning, anchoring future advances in adaptive perception frameworks.