TopoBenchmarkX: Benchmarking Topological Mapping

• TopoBenchmarkX is an open-source framework that provides a rigorous, reproducible benchmarking protocol for evaluating topological mapping approaches in robotics.
• It quantitatively measures dataset ambiguity and perceptual aliasing while using localization accuracy as a surrogate for topological consistency.
• The framework integrates six standardized baselines, spanning classical and deep-learned methods, to diagnose performance under controlled ambiguity conditions.

TopoBenchmarkX is an open-source benchmarking framework for the systematic, quantifiable evaluation of topological mapping approaches in robotics, with a particular focus on perceptual aliasing and topological consistency. It provides a rigorous evaluation protocol, curated benchmark datasets spanning diverse environments with calibrated ambiguity levels, and standardized baselines covering both classical and deep-learned mapping algorithms. By addressing the lack of unified metrics, datasets, and protocols in topological representation research, TopoBenchmarkX enables reproducible and fair comparison of algorithms, serving as a critical resource for academic and industrial research in navigation and localization.

1. Foundations: Topological Consistency and Localization Accuracy

TopoBenchmarkX formalizes the core property of topological mapping as topological consistency at a physical route scale $d$ with tolerance $\epsilon$. In this context, a topological map is mathematically represented as a graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$, where $\mathcal{V}$ denotes discrete "place" nodes and $\mathcal{E}$ are the navigable edges. Consistency at scale $d$ with tolerance $\epsilon$ requires two conditions:

• Edge Precision: For all $u,v\in\mathcal{V}$, if $\mathrm{dist}_G(u,v)\leq n$ then $\mathrm{dist}_R(u,v)\leq d$.
• Policy-Conditioned Edge Recall: For all $(u,v)\in\Omega_P(d)$, if $\mathrm{dist}_R(u,v)\leq d$ then $\mathrm{dist}_G(u,v)\leq n$.

Here, $\mathrm{dist}_G(\cdot,\cdot)$ is the hop distance in $\mathcal{G}$; $\mathrm{dist}_R(\cdot,\cdot)$ is the ground-truth geodesic distance, and $\Omega_P(d)$ is the set of node pairs that the mapping update policy $P$ could connect at scale $d$. The hop budget $n$ is defined as $n = \max(1, \lfloor \epsilon d / \mu_e \rfloor)$, with $\mu_e$ as the median mapped route length.

Crucially, TopoBenchmarkX proves that localization accuracy—the fraction of time steps where the predicted node $\hat{v}_t$ satisfies $\mathrm{dist}_R(\hat{v}_t, v_t^*) \leq d$ (with $v_t^*$ as ground-truth)—serves as an exact surrogate for topological consistency under mild assumptions (specifically, that new edges have route length $\leq \kappa \mu_e$ and $\epsilon \leq 1/\kappa$) (Wang et al., 5 Oct 2025). This makes localization accuracy the central metric of the benchmark.

2. Quantification of Perceptual Aliasing and Dataset Ambiguity

TopoBenchmarkX advances field methodology by introducing the first quantitative measure of dataset ambiguity (perceptual aliasing), which has historically impeded reproducible evaluation. Perceptual aliasing is treated as an intrinsic dataset property, systematically stratified in the evaluation protocol.

A sequence-based protocol is employed: an agent traverses a mapping sequence $S_m = \{z_m^1, ..., z_m^N\}$; short test subsequences $S_t = \{z_t^1, ..., z_t^L\}$ sample revisits or novel routes. Each $z_t^i$ is aligned to a mapping frame by ground truth $\pi(i)\in\{1...N\}\cup\{\emptyset\}$.

Given a similarity function $\mathrm{sim}(\cdot,\cdot)$ (provided by a visual place recognition model BOQ), sequence similarity is defined as

$$\mathrm{Sim}(S_t, S_m^{[u]}) = \frac{1}{L} \sum_{i=1}^L \mathrm{sim}(z_t^i, z_m^{j_u + i - 1})$$

where $S_m^{[u]}$ is a candidate subsequence starting at $u$. With similarity threshold $\tau$ and ambiguity ratio $\alpha$, each $S_t$ is categorized:

• Ambiguous + Positive (A+P): revisit; distractor subsequence nearly as similar as true one ($$\max_{u\neq\pi(S_t)} \mathrm{Sim}/\mathrm{Sim} \geq \alpha$$).
• Positive Only (P.O.): revisit; true match clearly dominates ($< \alpha$).
• Ambiguous Only (A.O.): novel; mapped subsequence(s) spuriously similar ($\max_u \mathrm{Sim} \geq \tau$).

This triage allows for precise diagnosis of performance breakdown due to aliasing—phenomena that are not detected by conventional precision/recall loop-closure metrics (Wang et al., 5 Oct 2025).

3. Curated Benchmark Dataset: Structure and Scope

The TopoBenchmarkX dataset aggregates 25 distinct “maps” drawn from six publicly available sources:

• OpenLORIS (5 indoor environments, diverse lighting and scene changes)
• Oxford RobotCar (1 outdoor route, multiple seasons/weather/traffic)
• Rawseeds (1 mixed indoor/outdoor trajectory)
• Habitat-Sim (16 simulated indoor scenes with engineered ambiguities)
• RELLIS-3D (1 off-road, varied vegetation/terrain)
• ROVER (1 outdoor multi-season route)

For each map, a mapping sequence ($100$s–$1000$s of frames) and dozens of stratified test subsequences (length $L=10$–$20$) are extracted and labeled per ambiguity class, yielding 51 A+P, 384 P.O., and 194 A.O. test cases (Wang et al., 5 Oct 2025). The benchmark thereby spans both easy (P.O.) and hard (A+P, A.O.) conditions at controlled, pre-calibrated difficulty levels.

4. Baseline Implementations and Supported Mapping Frameworks

Six SLAM-free mapping strategies are re-implemented in a consistent framework with unified visual place recognition descriptors (MegaLoc, ResNet-VLAD D=4096):

• Classical Appearance-only Baselines:
  • FAB-MAP 2.0: Bag-of-words on SURF features, Chow–Liu dependency tree, RANSAC vetting.
  • RatSLAM: Continuous pose-cell filter, experience graph, template matching.
• Deep-learned VPR-based Methods:
  • Greedy Matching (GM): Highest-score node if $\mathrm{sim} > \tau$.
  • Sequence Matching – Median (SM-Med): Median aggregated similarity over a window.
  • Sequence Matching – All (SM-All): All similarities must exceed $\tau$ within the window.
  • Probabilistic Belief Update (PBU): Bayesian filtering over nodes with policy-informed motion prior ($w_u=1$ hop, $\alpha=0.9$, $\beta=0.1$), and exponential likelihood mapping $g(\mathrm{sim})=\exp(10\cdot\mathrm{sim})$.

No fine-tuning of the underlying VPR (MegaLoc) is performed, ensuring uniform experimental conditions. All key parameters (window width, hop thresholds, etc.) are explicitly set and reproducible from open-source code (Wang et al., 5 Oct 2025).

5. Standardized Evaluation Protocol and Metrics

Evaluation in TopoBenchmarkX enforces rigor and reproducibility via a protocol that covers:

• Splitting each map into mapping and test subsequences, all ground-truth aligned.
• Incrementally building the topological graph online as sequences are traversed.
• Class-wise reporting of localization accuracy:
  • $L_{\text{A+P}}$: fraction of A+P cases with correct localization (within $d$ meters route distance)
  • $L_{\text{P.O.}}, L_{\text{A.O.}}$ similarly for other classes
• Summary metric: Balanced Localization Accuracy (BLA)

$$\text{BLA} = (L_{\text{A+P}} \cdot L_{\text{P.O.}} \cdot L_{\text{A.O.}})^{1/3}$$

with Jeffreys prior smoothing (add 0.5 successes, +1 trial).

• Two principal reporting regimes:
  • $L_{\text{A.O.}}@\rho$: Fix $\tau$ so that A.O. case accuracy $\approx \rho$ ($\rho=0.90, 0.99$) on held-out validation.
  • $\text{BLA}_{\max}$: $\tau$ chosen to maximize BLA.
• Complete codebase (Python, NumPy, SciPy, PyTorch) for data regeneration and metric computation (Wang et al., 5 Oct 2025).

6. Experimental Results, Insights, and Limitations

Experiments using TopoBenchmarkX reveal that overall performance degrades sharply under perceptual aliasing. When tuned for high safety (e.g., $L_{\text{A.O.}}@90$), revisit accuracy collapses to near zero for all methods, irrespective of baseline sophistication (Table III, (Wang et al., 5 Oct 2025)). Deep-learned approaches (GM, SM, PBU) outperform FAB-MAP and RatSLAM on revisit accuracy in low-ambiguity settings but fail catastrophically on ambiguous revisits when the false positive rate is controlled.

Sequence matching yields only marginal gains and is highly sensitive to window size. Belief filtering offers benefit only if the motion prior fits the data exactly—a condition rarely met in practice due to variable node spacing. No method can reliably disambiguate visually similar places; increasing the matching threshold $\tau$ reduces false positives at the expense of missing legitimate revisits, thus preventing loop closure.

A practical implication is that existing loop-closure metrics conceal the frequency and nature of these critical failures, highlighting the necessity for the nuanced, case-based protocol introduced by TopoBenchmarkX.

7. Availability and Impact

All datasets, baseline implementations, and evaluation tools are distributed under an open-source license at https://github.com/your-repo/TopoBenchmarkX. The benchmark constitutes a principled standard: it provides the first quantitative ambiguity measures, explicitly separates aliases and revisits by case type, and offers difficulty-controlled test scenarios. By uncovering the limitations of current SLAM-free mappers even in well-studied environments, it underscores the urgent need for advances in robust disambiguation (e.g., algorithms leveraging semantics, metric cues, or multimodal data) (Wang et al., 5 Oct 2025).

TopoBenchmarkX is positioned as both a diagnostic tool and a baseline platform, fostering rigorous, reproducible research in topological mapping under perceptual aliasing.