Stereo Spike Cameras: Neuromorphic 3D Vision

• Stereo spike cameras are high-speed, neuromorphic sensors that generate asynchronous binary spikes from pixel-level luminance integration, capturing rich scene texture and motion.
• Adaptive coding techniques, including temporal partitioning and inter-/intra-pixel prediction, enable compression ratios up to 140× while maintaining high fidelity in reconstructed images.
• By analyzing temporal disparities between synchronized spike streams, these systems achieve precise depth estimation and support robust 3D reconstruction for dynamic environments.

A stereo spike camera system comprises two temporally precise, neuromorphic sensors—each independently integrating luminance at the pixel level and emitting discrete, asynchronous “spikes” when a preset threshold is crossed. Unlike conventional frame-based or event-based (DVS) stereo vision, stereo spike cameras produce high-frequency, binary output streams that preserve scene texture and high-speed motion fidelity. Recent years have seen the development of dedicated algorithms, datasets, and neuromorphic architectures tailored for depth estimation, 3D reconstruction, and visuomotor applications using stereo spike streams. This article provides a comprehensive overview of the technology, models, coding strategies, depth inference, 3D understanding, and future prospects associated with stereo spike cameras.

1. Principles of Spike Camera Operation and Stereoscopic Extension

Spike cameras are bio-inspired sensors that asynchronously accumulate pixel-wise intensity until a dispatch threshold is reached, at which point a 0-1 “spike” is emitted and the accumulator resets. Formally, for each pixel, the device generates a spike whenever

$\int_{t_0}^{t} I(x,y,s)\,ds \geq \Theta,$

where $I(x, y, s)$ is the scene luminance and $\Theta$ is a fixed threshold. This mechanism produces a stream of asynchronous binary events per pixel, distributed at rates up to tens of kHz (Dong et al., 2019).

In a stereo configuration, two such devices, rigidly mounted with a known baseline, simultaneously capture spatially offset spike streams (left and right cameras). Each spike camera independently samples the scene; spatial disparities between synchronized spike trains encode depth.

Distinctive aspects compared to frame-based and event-based (DVS) stereo systems include:

• Frame-free, microsecond-level sampling with high dynamic range and minimal motion blur, extending the effective operating domain to rapid or HDR scenes (Li et al., 2022, Risi et al., 2021).
• Unlike DVS sensors, which fire events only on log-brightness changes and miss static texture, spike cameras produce continuous spikes even for stationary regions, enabling full-scene texture recovery (Dong et al., 2019).
• The disparity between corresponding pixel spike streams across the left and right cameras encodes depth while retaining the temporal precision and edge sensitivity characteristic of neuromorphic sensing.

2. Efficient Spike Stream Representation and Compression

A primary technical challenge in stereo spike imaging is the enormous bandwidth resulting from high-frequency, high-resolution spike streams. Naive transmission or storage is prohibitive.

An efficient coding methodology integrates several key components (Dong et al., 2019):

• Adaptive Temporal Partitioning: Each spike train is adaptively segmented by modeling the distribution of inter-spike intervals (ISIs) as gamma-distributed, exploiting regions of locally constant statistics (stationarity).
• Intra-/Inter-Pixel Prediction: Within and across pixels, prediction modes estimate spike timing using local statistics (mean for stationary, motion-compensated reference for dynamic regions) and spatial information from nearby pixel spike trains.
• Quantization and Intensity-aware Coding: Quantization step sizes are dynamically computed per ISI duration to ensure perceptual uniformity. Inter-spike interval differences are mapped to underlying intensity differences, which are the relevant measure for both spatial texture and subsequent image recovery.
• Entropy Coding: Residuals are contextually and adaptively entropy encoded.

This unified framework achieves compression ratios of 67–140× on the PKU-Spike dataset (up to 40,000 Hz, multi-second recordings) with PSNR > 47 dB and SSIM ≈ 0.96 in reconstructed textures, demonstrating that spike data can be stored and transmitted efficiently for downstream analysis without significant fidelity loss (Dong et al., 2019).

A plausible implication is that similar strategies will be necessary in stereo setups, where redundancy between the two views can be further exploited by cross-view predictive models or joint entropy coding.

3. Depth Estimation from Stereo Spike Streams

Depth reconstruction leverages the temporal disparity between corresponding spike streams in the stereo pair:

• ISI-Based Intensity Distance: Since the reciprocal of ISI approximates local intensity, comparing ISI sequences across left/right pixel pairs allows for robust matching even under variable lighting (Dong et al., 2019). The intensity-based spike train distance,

$$D(fs_1, fs_2) = \frac{1}{K}\sum_i |\Delta I_i(fs_1) - \Delta I_i(fs_2)|,$$

can be used to select correspondences or supervise learning-based matching.

• Neuro-Inspired SNNs: Several models (Risi et al., 2021, Rançon et al., 2021, Gao et al., 26 May 2025) use brain-inspired SNNs to process spike or event streams. Coincidence detectors or encoder-decoder spiking networks infer disparity by integrating spatio-temporal activity. Recurrent spiking neural network (RSNN) modules iteratively refine depth, mirroring temporal integration in visual cortex.
• Correlation Volumes and End-to-End Models: Architectures such as SpikeStereoNet (Gao et al., 26 May 2025) compute multiscale correlation volumes by feature extraction from left/right spike streams, followed by RSNN-based disparity refinement and upsampling. These networks can be trained and evaluated on custom synthetic and real-world stereo spike datasets, producing sharp, accurate depth maps even in high-speed or textureless scenes.

Certain approaches introduce domain-specific innovations:

• Uncertainty-Guided Fusion: Fusing monocular and stereo branches, weighting each pixel’s prediction by learned spatial uncertainty, addresses the problem of unreliable stereo matching over distant or low-texture regions and improves overall performance (Li et al., 2022).
• Ray Density Fusion: For event-based stereo (closely related to spike streams), depth is inferred by back-projecting rays from asynchronous events and fusing their density in the 3D scene, bypassing explicit event matching (Ghosh et al., 2022).

Experiments across benchmarks such as MVSEC, CitySpike20K, and custom synthetic datasets show that spike-based stereo architectures achieve high accuracy across a range of lighting, speed, and scene complexity conditions.

4. 3D Reconstruction and Scene Understanding

Beyond pixelwise depth, stereo spike cameras enable advanced 3D scene inference:

• Neural Field Approaches: Recent works (Guo et al., 25 Mar 2024, Chen et al., 15 Nov 2024, Dai et al., 10 Apr 2024, Dai et al., 23 May 2025) adapt NeRF and 3D Gaussian Splatting (3DGS) pipelines to handle spike streams directly, using spike-aware renderers, temporal masking, and hybrid reconstruction loss terms (e.g., “texture from spike” loss). End-to-end joint optimization of image recovery, pose correction, and 3D radiance field fitting (USP-Gaussian) mitigates the cascading error common to sequential pipelines (Chen et al., 15 Nov 2024).
• Latent Fusion of Multimodal Inputs: Generative frameworks such as SpikeGen (Dai et al., 23 May 2025) operate in the latent space, fusing compressed representations of spike and RGB data with stochastic modality dropout, handling spatially sparse spike input while leveraging RGB spatial priors. This approach generalizes naturally to the stereo setting, combining left/right spike streams and leveraging disparity for richer 3D representations.
• Robotic Grasping Without 3D Reconstruction: The SpikeGrasp system (Gao et al., 12 Oct 2025) directly infers 6-DoF grasp poses from fused stereo spike streams using a recurrent spiking visual pathway network, bypassing explicit 3D point cloud construction and demonstrating superior performance and data efficiency in cluttered/textureless scenes.

These methods efficiently exploit the unique coding and temporal characteristics of spike streams to recover high-fidelity geometry and object pose directly applicable in robotics, navigation, and automated manipulation.

5. Datasets and Real-World Evaluation

Advancement in stereo spike camera research has been propelled by the release of large-scale, high-fidelity datasets:

Dataset	Sensor Type	Main Features
PKU-Spike	Spike camera	6 sequences, 40,000 Hz, diverse scenes for compression tests
CitySpike20K	Simulated spike	20k pairs, 1024×768, day/night, ground-truth depth, urban scenes
MVSEC	DAVIS event cams	Indoor/outdoor stereo, LIDAR GT, vehicle-mounted
RS-3D	Spike + RGB	Real spike/RGB pairs, time-synced, for NVS benchmarking
SpikeStereoNet	Synthetic/Real	150 synthetic scenes, >2000 real indoor, stereo spike/depth pairs
SpikeGrasp	Synthetic	Blender, domain randomization, stereo spike streams, 6-DoF grasps

Extensive evaluations show that spike-based systems outperform conventional vision methods in scenarios involving rapid motion, HDR, and low-texture environments. Metrics such as RMSE, Percentage of Correct Disparities (PCD), PSNR, SSIM, and task-specific success rates are reported. Data efficiency—maintaining high accuracy with less labeled training data—was also demonstrated for neuromorphic architectures (Gao et al., 26 May 2025, Gao et al., 12 Oct 2025).

6. Implementation, Hardware Considerations, and Practical Applications

Stereo spike camera systems and their processing frameworks are optimized for deployment on neuromorphic hardware:

• RSNN-based models and stateless convolutional SNNs enable integration with chips such as Intel Loihi, IBM TrueNorth, and custom analog/digital LIF platforms, offering low power, low latency, and real-time capability (Gao et al., 26 May 2025, Risi et al., 2021, Rançon et al., 2021).
• Efficient coding and compression (Dong et al., 2019) are essential for managing the data deluge typical of synchronous multi-view systems at kHz rates; entropy/quantization schemes and inter-view redundancy exploitation mitigate bandwidth bottlenecks.
• Application fields include autonomous driving (robust sensor fusion for rapid navigation), robotics (real-time 3D scene reconstruction and manipulation), SLAM/odometry (direct spatio-temporal mapping and tracking (Niu et al., 12 Oct 2024)), and NVS (enhanced deblurred synthesis under motion (Guo et al., 25 Mar 2024, Dai et al., 10 Apr 2024))—all benefiting from the unique synergy of spike-based sampling and binocular depth perception.

7. Limitations and Prospective Research Directions

While stereo spike camera systems offer compelling advantages in high-speed, HDR, and challenging visual scenarios, open problems and directions remain:

• Bridging domain gaps between synthetic and real data—through improved noise modeling, domain adaptation strategies, and expanded real-world benchmarks—is necessary for robust deployment (Chen et al., 8 Jan 2025, Gao et al., 26 May 2025).
• System calibration (timing, spatial alignment, and inter-camera disparity) and synchronized spike stream fusion are nontrivial in practice, especially as resolution and speed increase.
• Future directions include multimodal integration—combining spike, RGB, and event information (Dai et al., 23 May 2025), extended neurally inspired dynamics (plasticity, adaptation), efficient pose estimation refinement under unconstrained motion (Chen et al., 15 Nov 2024), and leveraging spike streams for real-time, energy-efficient processing in edge devices.

Overall, stereo spike camera technology has established itself as a versatile, biologically inspired approach for temporally precise, robust 3D perception, with ongoing research closing the gap to mature real-world systems attuned to the demands of high-speed, dynamic, and information-rich environments.