NeuroBench: Neuromorphic Benchmark Framework

• NeuroBench is a standardized benchmarking framework for neuromorphic computing, enabling rigorous and comparable evaluation across diverse algorithms and hardware.
• It features a dual-track architecture, with one track for hardware-agnostic algorithm assessment and another for hardware-dependent system metrics such as accuracy, latency, and power consumption.
• Its inclusive, actionable, and iterative design promotes transparent reporting, reproducibility, and community-driven evolution in brain-inspired AI.

NeuroBench is a standardized benchmarking framework for evaluating neuromorphic computing algorithms and systems, designed as an inclusive, actionable, and iterative suite to foster progress and enable rigorous comparison across the advancing landscape of brain-inspired AI hardware and software. Developed through open collaboration between academic and industrial researchers, NeuroBench introduces a dual-track architecture supporting both hardware-agnostic algorithm development and hardware-specific system evaluation, enabling comparability and reproducibility across a wide array of models and platforms (Yik et al., 2023).

1. Foundational Principles and Framework Architecture

NeuroBench is constructed on three core principles:

• Inclusiveness: The benchmarks impose minimal constraints on solution modality, supporting artificial neural networks (ANNs), spiking neural networks (SNNs), reservoir computing models, mixed-signal circuits, and in-memory processing alike. Metrics and tasks are defined at the task level to enable hierarchical and cross-modal evaluation.
• Actionability: An open-source benchmarking harness offers standardized data loaders, metric calculators, and launcher scripts, minimizing setup overhead and allowing new models or hardware backends to be incorporated with minimal boilerplate.
• Iterativity: Version-controlled benchmark releases and community-driven governance ensure the suite evolves alongside advances in neuromorphic research, maintaining relevance and encouraging broad adoption.

The framework is organized into two complementary tracks:

• Algorithm Track (Hardware-Independent): Focused on platform-agnostic evaluation of algorithms via correctness, complexity, and workload metrics.
• System Track (Hardware-Dependent): Dedicated to end-to-end assessment of neuromorphic hardware, incorporating latency, throughput, power, and energy measurements including data movement and pre/post-processing overheads.

2. Benchmark Tasks, Input/Output Specifications, and Metrics

Algorithm Track Benchmarks (v1.0):

Task	Dataset	Correctness Metric	Domain
Keyword FSCIL	MSWC audio keywords (1s clips)	Accuracy	Few-shot continual learning
Event Camera Detection	Prophesee 1MP automotive events	COCO mAP	High-dynamic-range vision
NHP Motor Prediction	Primate reaching (cortical recordings)	$R^2$	Sensorimotor regression
Chaotic Prediction	Mackey-Glass (τ=17–30) time series	sMAPE	Temporal forecasting

Input/Output Specifications are strictly prescribed per task, e.g., spike-train versus MFCC encoding for audio, event time-surface representation for cameras, and 2D continuous velocity output for motor decoding.

Performance Metrics (hardware-independent):

• Correctness: Accuracy, $R^2$, and sMAPE computed via standard formulae.
• Complexity: Model footprint (bytes), connection sparsity, activation sparsity, dense versus effective synaptic operation counts, and execution rate (Hz).
• Workload: Synaptic operations are partitioned as dense (all possible MACs) and effective (nonzero weight × nonzero activation MACs for ANNs, binary ACs for SNNs).

System Track Metrics (hardware-dependent):

• Correctness (as above)
• Performance: Throughput (samples/s), latency (ms)
• Efficiency: Measured average power (W), energy per inference or per task (J)

3. Baseline Results and Comparative Analyses

NeuroBench provides standardized baseline comparisons across ANNs, SNNs, and ESNs for each core task. Selected v1.0 algorithm track table results:

Model	Correctness (Accuracy/$R^2$/sMAPE)	Footprint (B)	Exec Rate (Hz)	Conn Sparsity	Act Sparsity	Dense SynOps	Eff_MACs	Eff_ACs
ANN	97.1%/0.581/13.4%	6.0e6	1–250	0.0	0.68–0.78	2.6e7	7.9e6	0
SNN	93.5%/0.580/14.8%	1.4e7	200–250	0.0	0.78–0.998	3.4e6	0	4e2–5e5

SNNs and ESNs in multiple tasks achieve correctness (accuracy or $R^2$) approaching or matching standard ANNs but with substantially smaller footprints and reduced effective operation counts. Notably, activation sparsity (e.g., up to 99.8% for SNNs) translates to compute savings only in the absence of upstream normalization.

Key comparative findings:

• Hybrid ANN/SNN models may not fully exploit accumulation-only hardware if ANN MACs are dominant.
• ESN reservoir sparsity yields dramatic reductions in effective operation counts compared to dense LSTM baselines.

4. Benchmarking Guidelines, Reporting, and Reproducibility

Standardized procedures are enforced to ensure experimental rigor and result reproducibility:

• Dataset Preprocessing: Benchmarks require fixed train/validation/test splits, download scripts, and standardized encoding (e.g., MFCC, time surfaces) built into the harness.
• Network Architecture Documentation: Layer types, time constants, threshold/reset dynamics, and other biophysical parameters for spiking models must be precisely declared.
• Analytic Reporting: Static metrics (e.g., parameter counts, sparsities) must be derived analytically from model definition.
• Reproducibility Best Practices: Random seed fixation, documented data shuffling, open-source code publishing (PyTorch, snnTorch, SpikingJelly), harness-based metric automation, and transparent hyperparameter search methodologies are required.

5. System Track: Task Classes, Scenarios, and Harness Implementation

System Track Task Scenarios:

• Offline/server (batched throughput, latency)
• Single-stream (batch=1, latency focus)
• Real-time (continuous deadline tracking, e.g., ≥90% in-stride)
• Optimization (time/energy to achieve solution quality, e.g., in QUBO optimization)

Benchmarks (v1.0 examples):

• Acoustic Scene Classification (real-time DCASE, power and latency measured)
• QUBO Optimization (time/energy to a binary quadratic optimality threshold across varying graph instances)

Harness Design:

• Top-level API selectable for algorithm versus system tracks.
• ONNX/NIR-based model IRs enable cross-platform deployment.
• Modular hardware-specific backends directly interface with data I/O, energy/latency metrology, and deployment scripts.

6. Open Governance, Extension, and Community Contribution

NeuroBench supports a formal proposal→discussion→review→release cycle for new benchmarks, tasks, and hardware backends. Task submission requires dataset definitions, I/O mapping, and correctness metric specification, assessed for coverage, interest, and harness compatibility. Accepted contributions become part of versioned benchmark releases and are integrated into the public leaderboard.

7. Future Directions and Outstanding Challenges

Identified priority areas for benchmark evolution include:

• Continuous-Time and Analog Extensions: Defining operations and rate metrics for event-driven or analog-domain neuromorphic execution.
• Closed-Loop and Embodied Evaluation: Introducing online control, robotic, and sensorimotor loop benchmarks for direct assessment of latency-adaptive, energy-constrained intelligence.
• Emerging Hardware Modalities: Expanding support for memristive crossbars, photonic neuromorphic processors, spintronics, and heterogeneous SoC platforms, with attention to platform-specific performance and robustness metrics.
• Standardized Power and Energy Reporting: Refining real-time energy protocols for board-level measurements, establishing robust methods for scaling up to system-level (rack/cluster) evaluation.

In sum, NeuroBench supplies a unified “lingua franca” for quantitative evaluation and comparative co-design across diverse neuromorphic approaches, thereby enabling rapid, rigorous progress toward scalable, energy-efficient, and real-time brain-inspired computation (Yik et al., 2023).