Image-based localization using LSTMs for structured feature correlation

Abstract: In this work we propose a new CNN+LSTM architecture for camera pose regression for indoor and outdoor scenes. CNNs allow us to learn suitable feature representations for localization that are robust against motion blur and illumination changes. We make use of LSTM units on the CNN output, which play the role of a structured dimensionality reduction on the feature vector, leading to drastic improvements in localization performance. We provide extensive quantitative comparison of CNN-based and SIFT-based localization methods, showing the weaknesses and strengths of each. Furthermore, we present a new large-scale indoor dataset with accurate ground truth from a laser scanner. Experimental results on both indoor and outdoor public datasets show our method outperforms existing deep architectures, and can localize images in hard conditions, e.g., in the presence of mostly textureless surfaces, where classic SIFT-based methods fail.

• The paper presents a novel CNN+LSTM approach that reduces localization errors by 32-37% compared to PoseNet.
• It employs LSTM layers for structured dimensionality reduction of CNN features to enhance camera pose regression.
• Experimental results on datasets such as Cambridge Landmarks, 7Scenes, and TUM-LSI validate its robustness against motion blur and illumination changes.

Image-based Localization using LSTMs for Structured Feature Correlation

In this paper, the authors present a novel approach for camera pose regression utilizing a CNN+LSTM architecture for both indoor and outdoor scenes. The central premise is to enhance localization by learning feature representations that are robust against challenging conditions such as motion blur and illumination changes. The proposed method leverages CNNs for feature extraction followed by LSTM units, which serve as structured dimensionality reduction tools that improve localization performance through effective feature correlation.

Highlights and Methodology

The research introduces a system that addresses the limitations of both traditional SIFT-based localization and previous neural-based approaches like PoseNet. Key elements of the methodology include:

1. CNN Feature Extraction: The architecture builds upon a pre-trained GoogLeNet model, modified to output a 2048-dimensional feature vector which represents the image for localization purposes.
2. LSTM Dimensionality Reduction: The use of LSTMs after the convolutional layers allows for a structured reduction of the feature vector's dimensionality. By reshaping the feature vector and applying LSTMs in multiple directions, the network efficiently identifies and uses the most relevant feature correlations for pose estimation.
3. End-to-End Learning: Direct camera pose regression from RGB images is performed without the need for precomputed feature maps or complex model building, differing from traditional approaches that rely on SIFT descriptors and 3D models.
4. Quantitative Comparisons: The study presents a robust comparison with state-of-the-art methods, documenting improved performance over PoseNet with a 32-37% reduction in localization errors. It also contrasts CNN-based approaches with traditional SIFT-based methods, highlighting scenarios where neural-based methods show promise.

Experimental Results

The paper provides extensive experimental results on several datasets, demonstrating the robustness and applicability of the approach:

• Cambridge Landmarks and 7Scenes: The approach delivers significant improvements over existing CNN architectures like PoseNet, although it does not yet match the precision of SIFT-based methods, which remain more accurate in well-textured outdoor scenes.
• New Indoor Dataset (TUM-LSI): The authors introduce a challenging dataset that includes large textureless areas and repetitive structures. The CNN+LSTM architecture outperforms PoseNet significantly and shows the practical utility of deep learning approaches in scenarios where traditional methods fail.

Implications and Future Directions

This research indicates that CNN+LSTM architectures can bridge the gap between traditional feature-based localization and modern deep learning approaches, especially in environments where classic methods are prone to fail. Practical implications include:

• Robustness in Varied Conditions: The model's robustness against motion blur and poor texturing can benefit applications such as autonomous vehicle navigation and augmented reality.
• Dataset Advancements: By introducing the TUM-LSI dataset, the paper provides a new benchmark that challenges where texture and structure-based failures are prevalent, promoting further innovation in this area.
• Future Research: The observed complementary nature between SIFT and CNN-based methods suggests potential for hybrid approaches that intelligently leverage the strengths of both methodologies.

In conclusion, while traditional SIFT-based methods still outperform CNN-based solutions in many standard scenarios, this paper provides important contributions that showcase the potential of neural networks in tackling difficult localization problems. The continued refinement of neural approaches, guided by insights from these findings, remains a promising direction for advancing the field of image-based localization.