NeuroBench: Neuromorphic Benchmark Framework

• NeuroBench is a standardized, open, and extensible benchmarking framework for neuromorphic computing that evaluates both algorithmic and hardware-dependent performance.
• It employs a dual-track methodology with rigorous metrics such as accuracy, latency, and energy-per-inference to facilitate fair and reproducible comparisons.
• The platform drives community engagement and reproducible research, enabling effective progress in brain-inspired, energy-efficient AI technologies.

NeuroBench Benchmark

NeuroBench is a standardized, open, and extensible benchmarking framework that serves as a focal point for performance measurement and comparison in neuromorphic computing, neural inference hardware, and computational neuroscience. Designed by an open, cross-disciplinary community, it addresses the evaluation needs of both hardware-independent (algorithmic) and hardware-dependent (systems) approaches, encapsulating quantitative measures of efficiency, scalability, and domain relevance across diverse task domains and device substrates (Yik et al., 2023). Multiple related benchmarks and sub-suites with the “NeuroBench” label have also emerged in spiking neural network (SNN) hardware (Ke et al., 2024), neural inference circuits (Nikonov et al., 2019), and computational neuroscience simulation (Albers et al., 2021), each adapted to their particular technical challenges and scientific objectives.

1. Rationale and Historical Context

Neuromorphic computing seeks to emulate brain-inspired information processing with the aim of achieving significant advances in energy efficiency, latency, and resilience over von Neumann machines. Progress in this domain has been hindered by the absence of consensus benchmarks, leading to fragmentation: comparison is impeded by idiosyncratic problem definitions, a lack of shared evaluation methodology, and incompatible performance metrics. Previous efforts—often limited in scope or outdated by rapid evolution in the field—have failed to provide actionable, reproducible, and extensible platforms for tracking the impact of algorithmic and hardware innovations (Yik et al., 2023, Ke et al., 2024, Nikonov et al., 2019). NeuroBench directly addresses these deficits, drawing explicit analogy to MLPerf in the conventional AI community.

2. Framework Design and Methodology

NeuroBench adopts a dual-track structure:

• Algorithm (hardware-independent) track: Assesses models and algorithms for task performance (e.g., accuracy, $R^2$, mean average precision) and complexity metrics (model size, effective synaptic operations, activation/connection sparsity) on conventional compute platforms.
• System (hardware-dependent) track: Measures end-to-end inference performance—including wall-clock latency, throughput, power consumption, and energy per inference—on live neuromorphic hardware such as Loihi, SpiNNaker, application-specific mixed-signal ASICs, and hardware-accelerated SNN platforms (Yik et al., 2023, Ke et al., 2024).

Open-source harnesses (PyTorch, snnTorch, SpikingJelly for algorithm track; planned ONNX/NIR/Lava-based backends for system track) ensure reproducibility. All evaluation scripts, data loaders, and post-processing routines are versioned and transparent. Community engagement is institutionalized via open workshops, ratified versioning, and public leaderboards.

3. Task Domains and Canonical Workloads

NeuroBench v1.0 and its derivatives encompass well-defined tasks representative of major application axes:

Task Category	Data Modality	Core Metric(s)
Keyword Class-Incremental Learning	Audio	Accuracy after each session (FSCIL, MSWC)
Event Camera Object Detection	Event-based Vision	COCO mAP at IoU 0.5–0.95
Nonhuman Primate Motor Prediction	Neural timeseries	Coefficient of determination ($R^2$)
Chaotic Function Forecasting	Time series	sMAPE
Acoustic Scene Classification	Audio (DCASE 2020)	Classification accuracy

Tasks adhere to real-world relevance (e.g., few-shot learning, real-time classification), cover both spike-driven and analog modalities, and control for confounding by using public datasets with well-established splits (Yik et al., 2023, Ke et al., 2024).

4. Metrics and Measurement Procedures

Both static and dynamic performance metrics are supported, rigorously defined to ensure comparability. Principal measures include:

Algorithm Track

• Correctness: Task-specific accuracy, $R^2$, mAP, sMAPE.
• Model footprint: $F = \sum_i p_i + \sum_j q_j$ (bytes).
• Connection sparsity: $S_c = \sum_l m_l / \sum_l n_l$.
• Activation sparsity: $$S_a = (\sum_{l,t,i}z_{l,t}^i) / (\sum_{l,t,i}N_{l,t}^i)$$.
• Effective operation count: Nonzero MACs or ACs per execution.

System Track

• Latency: $T = 1/f$, with $f$ as execution rate.
• Throughput: Inferences per second in batched or streaming mode.
• Average power: $P$, measured in situ during inference.
• Energy per inference: $E = P/f$.
• Scenario metrics: Time/energy to solution at specific quality thresholds.

All device- and workload-dependent metrics are normalized to facilitate fair trade-off analyses (performance vs. energy, accuracy vs. latency) (Yik et al., 2023, Nikonov et al., 2019).

5. Reference Baselines and Comparative Analyses

Initial NeuroBench releases establish cross-modal and cross-architecture comparisons. For example:

• Keyword Incremental Learning: SNN base accuracy 93.5%, ANN 97.1%; SNN session-averaged accuracy drops faster but with $R^2$0 less dense MACs (spike-based computation).
• Event Camera Detection: Hybrid ANN-SNN achieves mAP 0.271 (vs. ConvLSTM-SSD RED 0.429), but with an order of magnitude less memory and operation count.
• Physical Inference Circuits: MEME (ME-ME) spintronic ANN yields $R^2$1 ps, $R^2$2 aJ, $R^2$3 mm$R^2$4 for LeNet, outperforming FeFET and SRAM solutions on energy (Nikonov et al., 2019).
• Real Hardware: XyloAudio 2 SNN achieves 80% accuracy on DCASE 2020 at 84 ms latency and 57.6 µJ/inference, demonstrating <100 µJ audio inference at real-time (Ke et al., 2024).

6. Extensibility, Evolution, and Future Directions

Open governance and a modular architecture enable NeuroBench to incorporate new domains (IMU, robotics, tactile sensors), support for emerging hardware (photonic, RRAM, 2D materials), and continual updates to evaluation protocols. Performance-mode and efficiency-mode benchmarks are permitted, but require both power and quality reporting. Priority is given to evolving the system track with richer real-time, multitask, optimization, and continuous-learning scenarios.

Future expansions will extend metrics to analog and asynchronous computation, robustness and quantization tolerance, and full-stack I/O and data-transfer inclusion. Annual leaderboard events and GitHub-based proposal ratifications ensure community relevance and scientific transparency (Yik et al., 2023).

7. Significance and Impact

NeuroBench provides the neuromorphic community with a transparent, reproducible, and evolving platform for quantifying technological progress across hardware and algorithmic frontiers. It plays a crucial role in closing the evaluation gap, enabling apples-to-apples comparisons, and informing both device–algorithm co-design and system-level optimizations. Widespread adoption of NeuroBench and its task-specialized derivatives is fundamental for guiding the development and deployment of next-generation, brain-inspired AI hardware and methodologies (Yik et al., 2023, Nikonov et al., 2019, Ke et al., 2024, Albers et al., 2021).