Event-LAB: Unified Benchmark for Neuromorphic Localization

• Event-LAB is a standardized framework for evaluating event-based neuromorphic localization methods, unifying datasets, algorithms, and metrics.
• It streamlines experimental workflows by integrating automated data formatting, baseline execution, and metric computation into a single command-line interface.
• Its modular design supports diverse pipelines like VPR and SLAM while revealing performance sensitivity to parameters such as event count and time window.

Event-LAB is a unified and standardized framework for evaluating neuromorphic localization methods across multiple datasets and algorithms. It was introduced in response to a rapid expansion of event-based localization research, described as a tenfold increase in the cumulative total number of published papers on this topic over the past 10 years, alongside a corresponding growth in code dependencies and data formats that makes comparisons difficult and cumbersome to implement reliably. Event-LAB is implemented using the Pixi package and dependency manager, provides a single command-line installation and invocation for combinations of localization methods and datasets, and demonstrates its capabilities through two common event-based localization pipelines: Visual Place Recognition (VPR) and Simultaneous Localization and Mapping (SLAM) (Hines et al., 18 Sep 2025).

1. Motivation and problem setting

Event-LAB was designed to address fragmentation in event-based localization evaluation. The stated problem is not the absence of methods, but the difficulty of comparing them fairly and reproducibly when codebases, package dependencies, and dataset formats diverge across implementations. In this setting, Event-LAB aims to enable reproducible, consistent benchmarking, streamline workflows for researchers, and facilitate community-driven development (Hines et al., 18 Sep 2025).

The framework’s standardization goal is operational as well as methodological. It provides a unified evaluation interface in which baseline methods and diverse datasets are run through a common pipeline, using standardized inputs, outputs, and evaluation metrics. It also performs automatic data formatting and conversion so that, regardless of the dataset’s original format, evaluation can proceed under uniform conditions. This emphasis on comparability is central to the framework’s purpose, and the paper explicitly links it to fair comparison of methodologies with consistent event image generation parameters.

A common misunderstanding in rapidly growing benchmark ecosystems is that reported scores are directly comparable once methods are nominally evaluated on the same dataset. Event-LAB directly challenges that assumption by showing that parameters controlling event collection counts and window sizes for frame generation can induce large variations in performance. The framework therefore treats evaluation setup, not only model execution, as part of the benchmarked system (Hines et al., 18 Sep 2025).

2. Software architecture and execution workflow

Event-LAB is implemented with the Pixi environment manager. The framework uses Pixi to provide cross-platform support on Linux, Mac, and Windows, bit-for-bit reproducibility via environment lockfiles, and integration with multiple package ecosystems, including conda-forge, PyPI, and ROS via RoboStack. The result is a single-command setup and execution model for method–dataset combinations (Hines et al., 18 Sep 2025).

A representative invocation is:

$TP$7

The command format is:

$TP$8

The execution flow described in the paper is similarly standardized. A single command triggers data download, standardization, event frame generation, baseline cloning, and metric computation. Figure 1 is summarized as:

$TP$9

This workflow integrates dataset preparation and method execution rather than treating them as separate manual stages. The framework formats datasets, and if needed converts them, to a common .hdf5 structure. It supports both simple event-count frames and complex reconstructed frames, for example using E2VID, with configurable event window controls.

3. Modularity, interfaces, and extensibility

Event-LAB organizes baseline methods as classes inheriting from EventBaseline. These classes expose standardized interfaces for configuring options, data formatting, execution, result parsing, and cleanup. Baseline-specific YAML configuration files are used to adapt models, backbones, and parameters (Hines et al., 18 Sep 2025).

The interface outline given in the paper is:

$FP$0

Dataset handling is equally modular. Event-LAB supports automatic downloading and formatting from remote sources based on the user’s specification, and exposes configurable parameters including event count, time window, reconstruction method, and related frame-generation settings. This design makes dataset preparation part of the benchmark interface rather than a method-specific preprocessing script.

For large experiment sweeps, the framework includes YAML-based batch mode. A batch configuration generates scripts to run a series of experiments across multiple datasets, time windows, baselines, and related parameters in sequence, while accumulating results in comprehensive spreadsheets. The example provided in the paper specifies a dataset, reference sequence, multiple queries, several event counts, a frame generator, a frame accumulator, and multiple baselines. This batching capability is important because the framework’s empirical analysis depends on systematic variation of evaluation parameters rather than isolated runs (Hines et al., 18 Sep 2025).

4. Pipelines, datasets, and evaluation protocol

Event-LAB demonstrates two localization pipelines: VPR and SLAM. The paper lists the following implemented methods and datasets.

Pipeline	Methods included	Datasets
VPR	LENS; VPR-evaluation-methods (MixVPR, CosPlace, EigenPlaces, etc.); Sparse-Event-VPR; EventVLAD; Ensemble-Event-VPR	NSAVP; Fast-and-Slow; QCR-Event-VPR; Brisbane-Event-VPR
SLAM	Ultimate-SLAM (U-SLAM); PL-EVIO	Event-Camera Dataset and Simulator

For VPR, Event-LAB automatically creates pseudo ground-truth in the form of binary matrices for reference–query matches, using configurable temporal or spatial tolerances tailored to each dataset. The standardized evaluation metrics are Recall@K for $K=1,5,10$ and PR-AUC. Precision and Recall are defined as

$$\text{Precision} = \frac{TP}{TP + FP} \qquad \text{Recall} = \frac{TP}{GTP}$$

where $TP$ is true positives, $FP$ is false positives, and $GTP$ is ground truth positives (Hines et al., 18 Sep 2025).

For SLAM, the framework supports evaluation through metrics reported in the paper’s figures, including trajectory error (RMSE-ATE) for several scenes. The role of SLAM in Event-LAB is therefore not limited to launching third-party systems; it extends the same standardization logic used for VPR to a second, distinct localization regime.

The protocol’s significance lies in the combination of common input formatting, common invocation, and common metrics. Event-LAB’s claim is not merely that multiple baselines can be run from a shared launcher, but that their data preparation, ground-truth generation, and output analysis are harmonized sufficiently to support systematic comparison (Hines et al., 18 Sep 2025).

5. Parameter sensitivity and empirical findings

A principal result of Event-LAB is that localization performance is highly sensitive to event count and time window. The framework was used to visualize and analyze results across multiple methods and datasets, revealing an association between parameters that control event collection counts and window sizes for frame generation and large variations in performance (Hines et al., 18 Sep 2025).

The paper reports that baseline methods developed for longer event durations, such as LENS and Sparse-Event-VPR, underperformed with short frames such as $33$ ms windows but significantly improved with longer windows, up to $1000$ ms. It also reports that methods using image reconstruction, including E2VID with CosPlace, EigenPlaces, and MixVPR, consistently outperformed event count frame-based approaches, especially in Recall@1 and PR-AUC.

The Fast-Slow dataset example in Table 2 illustrates the trend numerically. For LENS, Recall@1 rises from $0.05$ at $33$ ms to $0.16$ at $$\text{Precision} = \frac{TP}{TP + FP} \qquad \text{Recall} = \frac{TP}{GTP}$$0 ms, $$\text{Precision} = \frac{TP}{TP + FP} \qquad \text{Recall} = \frac{TP}{GTP}$$1 at $$\text{Precision} = \frac{TP}{TP + FP} \qquad \text{Recall} = \frac{TP}{GTP}$$2 ms, and $$\text{Precision} = \frac{TP}{TP + FP} \qquad \text{Recall} = \frac{TP}{GTP}$$3 at $$\text{Precision} = \frac{TP}{TP + FP} \qquad \text{Recall} = \frac{TP}{GTP}$$4 ms, with mean $$\text{Precision} = \frac{TP}{TP + FP} \qquad \text{Recall} = \frac{TP}{GTP}$$5 and PR-AUC $$\text{Precision} = \frac{TP}{TP + FP} \qquad \text{Recall} = \frac{TP}{GTP}$$6. For MixVPR, Recall@1 rises from $$\text{Precision} = \frac{TP}{TP + FP} \qquad \text{Recall} = \frac{TP}{GTP}$$7 at $$\text{Precision} = \frac{TP}{TP + FP} \qquad \text{Recall} = \frac{TP}{GTP}$$8 ms to $$\text{Precision} = \frac{TP}{TP + FP} \qquad \text{Recall} = \frac{TP}{GTP}$$9 at $TP$0 ms, $TP$1 at $TP$2 ms, and $TP$3 at $TP$4 ms, with mean $TP$5 and PR-AUC $TP$6. The paper states that increasing the time window improves Recall@1 for nearly every method (Hines et al., 18 Sep 2025).

To compare across windows, the paper introduces a winner-takes-all grouping strategy. Under this strategy, if within a broader time bin a sufficient fraction of small-window frames are correctly matched, the entire bin is counted as correct. This relaxes overly strict matching criteria. Figure 2 is reported to show that the strategy, with relaxed percentage requirements, substantially boosts Recall@1 and can surpass results for long-window baselines.

These findings are methodologically significant because they shift attention from raw benchmark numbers to the conditions under which those numbers are produced. Event-LAB’s results and analysis are presented as evidence for the importance of fairly comparing methodologies with consistent event image generation parameters (Hines et al., 18 Sep 2025).

6. Reproducibility, operational utility, and research significance

Event-LAB’s broader significance lies in standardized evaluation as infrastructure. The framework is presented as enabling fair, rapid, and reliable benchmarking by unifying datasets, methods, and metrics, and by lowering the barrier to rigorous comparison. It is also described as facilitating discovery, since systematic batch runs across parameter ranges expose the sensitivity of methods to experimental choices and identify when and why a method succeeds or fails (Hines et al., 18 Sep 2025).

The framework’s operational contribution is the reduction of engineering overhead. From installation to execution, results can be obtained with a single command, minimizing errors and discrepancies in experimental setup. The paper further reports setup times ranging from 1 min to under an hour for very large cases. This does not alter the underlying localization methods, but it changes the practical cost of reproducing and extending evaluations across multiple methods and datasets.

Event-LAB also emphasizes extensibility. Its modular design is intended to make it easy to add new baselines or datasets, and the paper presents this as support for transparent, cumulative progress in the neuromorphic localization community. A plausible implication is that the framework’s main scientific value is not only in reproducing published baselines, but in making parameter-controlled, cross-dataset comparison routine rather than exceptional.

Within event-based localization research, Event-LAB therefore occupies the role of a benchmarking and orchestration framework rather than a new localization model. Its central contribution is to standardize how several event-based localization methodologies are installed, configured, executed, and analyzed across multiple datasets, while making explicit that preprocessing choices such as event count, time window, and reconstruction method are part of the experimental variable set rather than incidental implementation details (Hines et al., 18 Sep 2025).