LaSeR Framework for 2D Visual Localization

• LaSeR Framework is an image-based Monte Carlo Localization system that uses geometrically structured latent space rendering and dynamic feature coding to align 2D pose hypotheses with indoor scenes.
• It employs a dual-branch architecture, combining a 2D PointNet-encoded map branch with a ResNet-50 image branch to aggregate viewing ray features efficiently.
• The system achieves state-of-the-art performance with real-time inference rates (over 10 kHz) on benchmarks like ZInD and Structured3D, ensuring robust indoor localization.

The LaSeR Framework denotes an image-based Monte Carlo Localization (MCL) system for 2D visual localization, predicated on the integration of geometrically structured latent space rendering and dynamic, view-dependent feature coding. The framework is engineered to efficiently and accurately align 2D pose hypotheses with panoramic or perspective scene observations in large indoor environments. LaSeR introduces latent space rendering, eschewing explicit synthesis of intermediate visual modalities by directly aggregating viewing ray features into a pose-parameterized latent feature space. This representation and inference strategy enables high-throughput sampling and comparison, with reported rendering rates exceeding 10 kHz and state-of-the-art localization performance on prominent benchmarks including ZInD and Structured3D (Min et al., 2022).

1. System Architecture and Branch Decomposition

LaSeR operates via a dual-branch architectural decomposition, converging at a measurement model compatible with MCL-based inference.

• Map Branch: 2D floor plans are discretized into rasterized point clouds around occupancy boundaries (architectural elements such as walls). Each sampled point is processed with a 2D PointNet encoder, resulting in a rendering codebook that contains precomputed, geometry-aware feature embeddings for each map point. This codebook architecture is central to endowing each scene element with feature codes parameterized by prospective view geometry.
• Image Branch: Scene observations—panorama (equirectangular 360°) or perspective images—are processed with a ResNet-50 encoder, producing high-dimensional feature maps. These maps are partitioned and reassembled into circular feature vectors, preserving the full angular context of the input and facilitating view alignment.

During inference, candidate poses (particles) are generated, and for each, a latent scene representation is rendered online. The similarity between this synthesized feature and the image-derived latent feature is computed and used as a likelihood in the MCL loop. An additional prediction refinement network, consisting of 1D convolutional and fully-connected layers, regresses residual corrections to the pose estimate.

2. Latent Space Rendering and View-Dependent Feature Coding

The framework is distinguished by its direct synthesis of latent feature embeddings at arbitrary hypothesized poses on the map—termed latent space rendering—as opposed to explicit rendering of visual modalities (e.g., layouts, depth) followed by CNN encoding.

• For each pose hypothesis, view-dependent “ray” features are synthesized by dynamically aggregating across visible map points, based on pose-to-point geometry (distance and incident angle between viewing ray and surface normal).
• The latent codes thus capture fine-grain photogeometric variations (such as shading, occlusions, and foreshortening) implicitly, via interpolation within the codebook, parameterized by computed geometry metrics.

This design enables high render rates, as code generation is fully decoupled from costly image rendering and runs above 10 kHz. The elimination of intermediate visual domains also tightens the correspondence between geometry and recognition cues in the embedding space.

3. Rendering Codebook Structure and Feature Aggregation

The rendering codebook implemented in LaSeR contains for each map point two discretized sets of feature codes:

• Distance Codebook: Codes $V_d$ encode features as a function of the Euclidean distance $d = ||t_i - \hat{t}||$ from the pose hypothesis to each point.
• Incident-Angle Codebook: Codes $V_\psi$ encode features by the angle $$\psi = \arctan2(||(t_i - \hat{t}) \times n_i||, (t_i - \hat{t}) \cdot n_i)$$ between the incoming ray and the surface normal.

At inference, the required codebooks are queried and interpolated based on the actual geometry, constructing a view descriptor. Features corresponding to each visible ray are projected into angular sectors covering the $[0,2\pi)$ domain, resulting in a latent scene representation partitioned into $V$ circular segments.

A schematic summary:

Codebook Type	Parameterized by	Interpolation at Runtime
Distance	$d$	Query or interpolate over distances
Incident-Angle	$\psi$	Query or interpolate over angles

This decoupling of encoding (offline, costly) and runtime rendering (online, efficient) fundamentally reduces computation and supports fine-grained geometric generalization.

4. Geometric Structure and Metric Learning in Latent Space

Latent features in LaSeR are explicitly organized as ordered, circular feature vectors:

$F = \{ f^{\alpha} \mid \alpha = 0,1,...,V-1 \}$

Each segment $f^{\alpha}$ encodes a direction bin of $2\pi/V$ radians, maintaining topological continuity across the angular domain. This structure is preserved for both map renderings and image encodings, allowing angle- and rotation-aware metric comparisons.

Metric and context learning objectives are used to align rendered and observed features in this space:

• Triplet Loss: Encourages query features $F_I$ from ground-truth pose to be closer to hypothesized renderings $F^+$ than negatives $F^-$:

$$\mathcal{L}_{\text{triplet}} = 2 \cdot \max \left\{ S(F_I, F^+) - S(F_I, F^-) + 0.5, 0 \right\}$$

where $S(F_i,F_j)$ is a normalized cosine similarity: $$S(F_i, F_j) = \frac{\sum_{\alpha=0}^{V-1} \cos(f_i^{\alpha}, f_j^{\alpha})}{2V} + 0.5$$.

• Context Loss: Ensures that the mean-of-segments representation is coherent, promoting robust matching even with coarse or partial geometric overlap.

This geometric structuring allows efficient search over rotational alignment during localization, with the optimal rotation $\theta_{\text{opm}}$ maximizing $S(F_I, R(F_t, \theta))$ via circular shift.

5. Measurement Model, Inference, and Runtime Considerations

In the MCL pipeline, the likelihood of an image observation $I$ given a pose $p$ is modeled as

$$P(I \mid p) = A \cdot S(F_I, R(F_{t,\theta}, \theta))$$

where $F_{t,\theta}$ is the rendered codebook feature at $(t, \theta)$ and $A$ is a normalization constant. Inference proceeds by sampling pose hypotheses, rendering features at each, and computing the rotationally-maximized similarity to the query.

The process is highly efficient: due to the codebook scheme, latent rendering for thousands of pose samples is possible at $\approx$10,000 fps, with the bottleneck shifted to image CNN encoding and final pose refinement.

6. Benchmark Performance and Empirical Results

LaSeR demonstrates superior performance against established methods on real-world and synthetic indoor datasets.

• Accuracy: Median translation and rotation errors are significantly reduced relative to PfNet, LaLaLoc, and classical MCL variants.
• Recall: High recall rates under strict spatial (e.g., 1m) and angular (e.g., 30°) thresholds.
• Speed: Sampling and rendering rates exceeding 10 kHz, enabling real-time or near real-time operation.

Reported datasets include:

Dataset	Domain	Attributes
ZInD	Real homes	~1,575 houses, tens of thousands of panoramas, 2D floor maps
Structured3D	Synthetic	Thousands of layouts, various furnishing, rendered imagery

On both, LaSeR consistently yields lower localization errors and higher recall, while operating orders of magnitude faster in sampling than preceding approaches.

7. Practical Applications and Limitations

LaSeR is designed primarily for indoor localization based on 2D architectural plans and rich visual input, enabling robust navigation for robotics and augmented reality applications without reliance on depth sensors or highly accurate odometry.

Key practical features include:

• Support for both panoramic and perspective images, with perspective-to-equirectangular conversion for directional feature alignment.
• Modular architecture, allowing for adaptation to different feature backbones and codebook parameterizations.
• Real-time inference capability, facilitating scalable deployment in large environments.

Noted limitations include potential performance sensitivity to map quality (e.g., occluded or ambiguous architectural features), and dependence on the sufficiency of codebook sampling and training coverage for generalization to previously unseen geometries.

---

LaSeR provides a comprehensive and computationally efficient solution to the problem of 2D visual localization with metric learning in a geometrically structured latent space, advancing the state of both robustness and speed in image-based MCL contexts (Min et al., 2022).