Event Topology-Based Visual Microphone

• The paper presents non-contact visual microphones that reconstruct sound by converting asynchronous event camera data into audio waveforms using explicit and learned topological methods.
• It employs diverse formulations including Mapper-based clustering, graph-neighbor flow estimation, and attention-driven spatial aggregation to capture micro-vibrations.
• Applications span from speech eavesdropping to structural health monitoring, though challenges remain in illumination sensitivity and recovering out-of-plane motions.

An event topology-based visual microphone is a non-contact vibration-sensing system that reconstructs sound or vibration from the asynchronous event stream of an event camera. In the recent literature, this designation covers at least three technically distinct formulations: learned spatial-temporal modeling over event voxels and speckle nodes, streaming optical-flow recovery from event-induced speckle motion, and explicit topological data analysis of the event cloud via Mapper and HDBSCAN. Across these formulations, the common premise is that sound-induced micro-vibrations produce structured changes in brightness or speckle position that can be converted into an acoustic or vibratory waveform without contact transduction (Yin et al., 3 Apr 2025, Cai et al., 4 Jul 2025, Niwa et al., 20 Oct 2025).

1. Event sensing and vibration encoding

The basic signal emitted by the sensor is an event stream $E=\{e_k\}_{k=1}^K$, where each event is represented either as $e_k=(u_k,t_k,p_k)$ with $u_k=(x_k,y_k)$, or equivalently as $e_i=(x_i,y_i,t_i,p_i)$. The polarity $p\in\{+1,-1\}$ is triggered when the change in log-brightness crosses a contrast threshold. In the EvMic formulation, polarity is defined by threshold crossings of $L(u,t)$ with threshold $C$; in Event2Audio, an event is emitted as soon as $(\log I(x,y,t)-\log I(x,y,t_0))\cdot p>\epsilon$, with the equivalent finite-difference condition $|\Delta \log I(x,y,t)|>\epsilon$ (Yin et al., 3 Apr 2025, Cai et al., 4 Jul 2025).

Two optical regimes appear in the cited work. Active systems amplify vibration visibility with coherent illumination. EvMic uses a 5 mW laser-matrix that projects a grid of small laser points onto the object surface, thereby enhancing local gradients $|\nabla L|$ so that tiny vibrations cause brightness changes large enough to trigger events. Event2Audio uses a low-power coherent laser at $e_k=(u_k,t_k,p_k)$0, $e_k=(u_k,t_k,p_k)$1, focused onto a small spot on a vibrating surface; the rough surface scatters the beam into a granular speckle pattern, and small surface tilts $e_k=(u_k,t_k,p_k)$2 produce a nearly pure lateral shift $e_k=(u_k,t_k,p_k)$3 of the entire speckle pattern when the lens is slightly defocused. By contrast, the topology-based amplitude-and-frequency reconstruction framework is explicitly passive and reconstructs vibrations directly from raw event streams without external illumination (Yin et al., 3 Apr 2025, Cai et al., 4 Jul 2025, Niwa et al., 20 Oct 2025).

The physical encoding model is correspondingly different but related. EvMic writes the event-generation relation as $e_k=(u_k,t_k,p_k)$4, and states that the event rate at pixel $e_k=(u_k,t_k,p_k)$5 is proportional to the temporal derivative of log-brightness, $e_k=(u_k,t_k,p_k)$6. Event2Audio instead emphasizes that lateral speckle translation sweeps bright and dark granules across pixels, producing local intensity ramps and asynchronous events. The passive topology-based method treats the filtered event set directly as a point cloud in $e_k=(u_k,t_k,p_k)$7 (Yin et al., 3 Apr 2025, Cai et al., 4 Jul 2025, Niwa et al., 20 Oct 2025).

2. Topology as graph structure, learned relation, and topological data analysis

In this literature, “topology” does not denote a single canonical representation. The most explicit topological construction appears in the passive framework, which embeds filtered events into the point cloud

$e_k=(u_k,t_k,p_k)$8

and applies a simplified Mapper pipeline. The filter function is one-dimensional,

$e_k=(u_k,t_k,p_k)$9

chosen to align with the dominant vibration axis. The range $u_k=(x_k,y_k)$0 is subdivided into overlapping intervals $u_k=(x_k,y_k)$1, and the pull-back sets $u_k=(x_k,y_k)$2 form an overlapping cover of the event cloud. Within each $u_k=(x_k,y_k)$3, HDBSCAN extracts dense clusters $u_k=(x_k,y_k)$4, retaining only those whose persistence $u_k=(x_k,y_k)$5 exceeds a threshold $u_k=(x_k,y_k)$6 (Niwa et al., 20 Oct 2025).

A second interpretation appears in Event2Audio. That system states that it does not explicitly build a large graph, but the underlying event set $u_k=(x_k,y_k)$7 in $u_k=(x_k,y_k)$8 can equivalently be viewed as a graph $u_k=(x_k,y_k)$9, in which each event node connects to nearest polarity-matching neighbors within a 3D window $e_i=(x_i,y_i,t_i,p_i)$0 and $e_i=(x_i,y_i,t_i,p_i)$1. In that view, flow computation corresponds to estimating local derivatives on the graph, and optional denoising can be written as

$e_i=(x_i,y_i,t_i,p_i)$2

Offline mode can apply 3D kernel convolutions or graph Laplacian regularization, whereas real-time mode skips this smoothing for speed (Cai et al., 4 Jul 2025).

A third interpretation appears in EvMic. That system also does not build an explicit graph Laplacian, but it treats the $e_i=(x_i,y_i,t_i,p_i)$3 detected speckles as nodes and applies multi-head self-attention across them in its Spatial Aggregation Block. At each time $e_i=(x_i,y_i,t_i,p_i)$4, with $e_i=(x_i,y_i,t_i,p_i)$5, the block computes

$e_i=(x_i,y_i,t_i,p_i)$6

followed by

$e_i=(x_i,y_i,t_i,p_i)$7

This induces a fully connected, learned relational structure over patch centers rather than a hand-specified topological complex (Yin et al., 3 Apr 2025).

This suggests that, in event-based visual microphones, topology spans at least three levels: explicit topological summarization of event clouds, implicit neighborhood graphs over asynchronous events, and learned attention-defined connectivity over spatially localized vibration carriers.

3. Reconstruction pipelines

The passive topology-based pipeline begins with aggressive event preprocessing. Only positive events $e_i=(x_i,y_i,t_i,p_i)$8 are retained, and the Delbrück background activity filter is applied with a temporal window $e_i=(x_i,y_i,t_i,p_i)$9; an event is discarded unless there exists at least one spatial neighbor event within $p\in\{+1,-1\}$0 in time. After Mapper-based covering and HDBSCAN clustering, each surviving cluster $p\in\{+1,-1\}$1 is reduced to a centroid

$p\in\{+1,-1\}$2

Sorting centroids by $p\in\{+1,-1\}$3 defines a discrete 3D trajectory. Because only vertical motion is of interest, the trajectory is projected by $p\in\{+1,-1\}$4, producing a nonuniformly sampled waveform that is then resampled to a uniform grid at $p\in\{+1,-1\}$5 via linear interpolation. Amplitude is estimated as half the peak-to-peak span, and frequency is obtained from the dominant peak of the DFT magnitude. The same pipeline is applied independently to multiple ROIs for multi-source recovery (Niwa et al., 20 Oct 2025).

Event2Audio follows a streaming analytical route rather than a learned one. For each new event $p\in\{+1,-1\}$6, the method searches the four cardinal neighbors at radius $p\in\{+1,-1\}$7 for the most recent polarity-matching event, computes horizontal and vertical flow by

$p\in\{+1,-1\}$8

and then quantizes time into uniform bins $p\in\{+1,-1\}$9 with $L(u,t)$0, equivalent to $L(u,t)$1. Aggregation yields dense traces

$L(u,t)$2

These are projected to a scalar channel $L(u,t)$3, integrated into relative displacement $L(u,t)$4, high-pass filtered with a Butterworth filter, and denoised by spectral subtraction. Under small-angle, small-motion assumptions, the paper states $L(u,t)$5, but in practice it omits explicit deconvolution by $L(u,t)$6 and treats $L(u,t)$7, after DC removal, as the recovered audio waveform (Cai et al., 4 Jul 2025).

EvMic represents the event stream as a 4D voxel tensor $L(u,t)$8 over temporal bins of width $L(u,t)$9, with separate channels for the two polarities. Small patches $C$0 are cropped around detected speckle centers. A ResNet-18 backbone with 3D submanifold sparse convolutions exploits the fact that only $C$1 of voxels are nonzero, producing feature tensors $C$2, or $C$3 across all speckles. The Spatial Aggregation Block models inter-speckle relations, after which a Mamba module performs long-range temporal modeling as a linear state-space model: $C$4 with $C$5 and

$C$6

The output $C$7 is the reconstructed audio waveform per speckle, followed if desired by a final $C$8 weighted sum over speckles (Yin et al., 3 Apr 2025).

4. Supervision, losses, and implementation regimes

EvMic is the most fully supervised of the three systems. Its training data are generated by a simulation pipeline in Blender, where an audio waveform $C$9 sampled at $(\log I(x,y,t)-\log I(x,y,t_0))\cdot p>\epsilon$0 drives an object displacement $(\log I(x,y,t)-\log I(x,y,t_0))\cdot p>\epsilon$1 along a random 3D direction, with $(\log I(x,y,t)-\log I(x,y,t_0))\cdot p>\epsilon$2. The scene is rendered at $(\log I(x,y,t)-\log I(x,y,t_0))\cdot p>\epsilon$3, resolution $(\log I(x,y,t)-\log I(x,y,t_0))\cdot p>\epsilon$4, and converted to synthetic events with the V2E event simulator. To bridge sim-to-real, real speckle patches are extracted from accumulated event frames, tiled on a black background, animated with the same audio-driven motion, and re-simulated; these vibrating speckles are used to fine-tune the Spatial Aggregation Block. Training uses PyTorch on a single NVIDIA RTX 4090 with SGD, learning rate $(\log I(x,y,t)-\log I(x,y,t_0))\cdot p>\epsilon$5, and batch size $(\log I(x,y,t)-\log I(x,y,t_0))\cdot p>\epsilon$6. The system pre-trains “SPConv + Mamba” for $(\log I(x,y,t)-\log I(x,y,t_0))\cdot p>\epsilon$7 iterations and fine-tunes the full network, including SAB, for another $(\log I(x,y,t)-\log I(x,y,t_0))\cdot p>\epsilon$8 iterations. Input streams are $(\log I(x,y,t)-\log I(x,y,t_0))\cdot p>\epsilon$9 long and discretized into $|\Delta \log I(x,y,t)|>\epsilon$0 bins, giving $|\Delta \log I(x,y,t)|>\epsilon$1 temporal resolution. The synthetic corpus contains $|\Delta \log I(x,y,t)|>\epsilon$2 training segments from TIMIT speech plus random MIDI, and speckle fine-tuning uses approximately $|\Delta \log I(x,y,t)|>\epsilon$3 synthetic speckle-only sequences. The total loss is

$|\Delta \log I(x,y,t)|>\epsilon$4

where $|\Delta \log I(x,y,t)|>\epsilon$5 is the scale-invariant SNR loss and $|\Delta \log I(x,y,t)|>\epsilon$6 is a multi-scale spectral reconstruction loss over Mel-spectrogram windows $|\Delta \log I(x,y,t)|>\epsilon$7 with $|\Delta \log I(x,y,t)|>\epsilon$8 (Yin et al., 3 Apr 2025).

Event2Audio is optimized for speed rather than supervised learning. Its reported hardware includes a Prophesee Metavision EVK3-HD at $|\Delta \log I(x,y,t)|>\epsilon$9 pixels, a $|\nabla L|$0 achromatic doublet, and approximately $|\nabla L|$1 defocus beyond focus to magnify the speckle pattern. The bias or contrast threshold is left at factory defaults, $|\nabla L|$2–$|\nabla L|$3 log-intensity units, and the typical event rate during audio sensing is $|\nabla L|$4–$|\nabla L|$5. The same paper reports temporal quantization $|\nabla L|$6, equivalent to $|\nabla L|$7 flow sampling, end-to-end latency of approximately $|\nabla L|$8–$|\nabla L|$9 in real-time mode on a modern CPU, and offline processing time of approximately $e_k=(u_k,t_k,p_k)$00 for $e_k=(u_k,t_k,p_k)$01 of audio (Cai et al., 4 Jul 2025).

The passive topology-based system is configured around a SilkyEvCam HD (EVK3) with an $e_k=(u_k,t_k,p_k)$02 lens at a distance of $e_k=(u_k,t_k,p_k)$03, with imaging scale approximately $e_k=(u_k,t_k,p_k)$04. Its target is an aluminum plate of $e_k=(u_k,t_k,p_k)$05, thickness $e_k=(u_k,t_k,p_k)$06, driven vertically by a mechanical wave driver. Ground truth is provided by a Laser Doppler Vibrometer (Polytec-500-3D-HV-Xtra) recorded at $e_k=(u_k,t_k,p_k)$07. The ROI is chosen as $e_k=(u_k,t_k,p_k)$08 pixels centered on the highest event-density pixel. HDBSCAN is configured with $e_k=(u_k,t_k,p_k)$09, $e_k=(u_k,t_k,p_k)$10, and a persistence threshold $e_k=(u_k,t_k,p_k)$11 selected to reject noise clusters (Niwa et al., 20 Oct 2025).

5. Empirical performance

EvMic reports synthetic-test metrics in terms of SNR and STOI. Against EvPhase and RGBPhase, the method improves SNR by approximately $e_k=(u_k,t_k,p_k)$12 and STOI by approximately $e_k=(u_k,t_k,p_k)$13 over EvPhase on the synthetic test set. On real-world data, spectrogram alignment against a microphone reference shows that the method recovers the fundamental and harmonics more faithfully. After post-processing with FullSubNet-Plus for speech, classical spectral subtraction for non-speech, and a final Butterworth high-pass to remove DC drift, the result is $e_k=(u_k,t_k,p_k)$14 reconstructed waveforms that are playable in real time. The deployment blueprint further reports that a $e_k=(u_k,t_k,p_k)$15 event buffer, $e_k=(u_k,t_k,p_k)$16-bin voxelization, and $e_k=(u_k,t_k,p_k)$17 speckle patches yield approximately $e_k=(u_k,t_k,p_k)$18 CPU+GPU inference time (Yin et al., 3 Apr 2025).

Event2Audio evaluates speech quality using PESQ, STOI, Mel-cepstral distortion, and log spectral distance. For speaker-membrane recovery in real time, the passive event baseline of Niwa ’23 records PESQ $e_k=(u_k,t_k,p_k)$19, STOI $e_k=(u_k,t_k,p_k)$20, MCD $e_k=(u_k,t_k,p_k)$21, and LSD $e_k=(u_k,t_k,p_k)$22, whereas the proposed method reports PESQ $e_k=(u_k,t_k,p_k)$23, STOI $e_k=(u_k,t_k,p_k)$24, MCD $e_k=(u_k,t_k,p_k)$25, and LSD $e_k=(u_k,t_k,p_k)$26. In offline mode, high-speed frame-based recovery by Niwa ’23 records PESQ $e_k=(u_k,t_k,p_k)$27, STOI $e_k=(u_k,t_k,p_k)$28, MCD $e_k=(u_k,t_k,p_k)$29, LSD $e_k=(u_k,t_k,p_k)$30, and processing time measured in hours, whereas the event-based method reports PESQ $e_k=(u_k,t_k,p_k)$31, STOI $e_k=(u_k,t_k,p_k)$32, MCD $e_k=(u_k,t_k,p_k)$33, LSD $e_k=(u_k,t_k,p_k)$34, and time $e_k=(u_k,t_k,p_k)$35. For chip-bag recovery, Howard [2023] in real time reports PESQ $e_k=(u_k,t_k,p_k)$36, STOI $e_k=(u_k,t_k,p_k)$37, MCD $e_k=(u_k,t_k,p_k)$38, LSD $e_k=(u_k,t_k,p_k)$39, while the event-based method reports PESQ $e_k=(u_k,t_k,p_k)$40, STOI $e_k=(u_k,t_k,p_k)$41, MCD $e_k=(u_k,t_k,p_k)$42, LSD $e_k=(u_k,t_k,p_k)$43. In offline comparison, Sheinin [2022] reports PESQ $e_k=(u_k,t_k,p_k)$44, STOI $e_k=(u_k,t_k,p_k)$45, MCD $e_k=(u_k,t_k,p_k)$46, LSD $e_k=(u_k,t_k,p_k)$47, and $e_k=(u_k,t_k,p_k)$48, while the event-based method reports PESQ $e_k=(u_k,t_k,p_k)$49, STOI $e_k=(u_k,t_k,p_k)$50, MCD $e_k=(u_k,t_k,p_k)$51, LSD $e_k=(u_k,t_k,p_k)$52, and $e_k=(u_k,t_k,p_k)$53 (Cai et al., 4 Jul 2025).

The passive topology-based system evaluates waveform fidelity with Normalized Cross-Correlation and spectral agreement with Median Vector Angle error. For the increasing-$e_k=(u_k,t_k,p_k)$54 case, NCC is $e_k=(u_k,t_k,p_k)$55 for Abe et al. (frame, 2014), $e_k=(u_k,t_k,p_k)$56 for Niwa et al. (event phase, 2023), and $e_k=(u_k,t_k,p_k)$57 for the topology-based method; the corresponding MVA errors are $e_k=(u_k,t_k,p_k)$58, $e_k=(u_k,t_k,p_k)$59, and $e_k=(u_k,t_k,p_k)$60. For decreasing-$e_k=(u_k,t_k,p_k)$61, NCC is $e_k=(u_k,t_k,p_k)$62, $e_k=(u_k,t_k,p_k)$63, and $e_k=(u_k,t_k,p_k)$64, and MVA is $e_k=(u_k,t_k,p_k)$65, $e_k=(u_k,t_k,p_k)$66, and $e_k=(u_k,t_k,p_k)$67. For the speech case “Mary had a little lamb,” NCC is $e_k=(u_k,t_k,p_k)$68, $e_k=(u_k,t_k,p_k)$69, and $e_k=(u_k,t_k,p_k)$70, while MVA is $e_k=(u_k,t_k,p_k)$71, $e_k=(u_k,t_k,p_k)$72, and $e_k=(u_k,t_k,p_k)$73. The same paper also reports a two-speaker experiment in which the recovered spectra showed distinct peaks at $e_k=(u_k,t_k,p_k)$74 and $e_k=(u_k,t_k,p_k)$75, confirming effective separation without cross-talk (Niwa et al., 20 Oct 2025).

6. Scope, applications, and common misconceptions

A common misconception is that event topology necessarily refers to explicit topological data analysis. The cited work shows otherwise. In the passive framework, topology is implemented directly through Mapper covers, overlapping pull-backs, and HDBSCAN persistence. In Event2Audio, topology is implicit in local event neighborhoods and optional graph-based smoothing. In EvMic, topology is learned through attention over speckle nodes. The term therefore describes a family of structural treatments of event data rather than a single algorithmic primitive (Yin et al., 3 Apr 2025, Cai et al., 4 Jul 2025, Niwa et al., 20 Oct 2025).

Another misconception is that visual microphones automatically recover calibrated acoustic pressure. Event2Audio states the linear relation $e_k=(u_k,t_k,p_k)$76 under small-angle, small-motion assumptions, but then omits explicit deconvolution by $e_k=(u_k,t_k,p_k)$77 in practice and treats the integrated displacement trace, after DC removal, as the recovered audio waveform. By contrast, the topology-based passive system is framed as amplitude-and-frequency reconstruction of a physical vibration signal, with amplitude estimated from peak-to-peak span and frequency from the dominant DFT peak. EvMic is optimized for sound recovery quality rather than explicit physical calibration of displacement amplitude (Cai et al., 4 Jul 2025, Niwa et al., 20 Oct 2025, Yin et al., 3 Apr 2025).

The application range given in the literature is broad. EvMic lists non-contact speech eavesdropping through windows, structural health monitoring, acoustic holography and source localization with large field of view, and audio-visual scene understanding in low-light or extreme dynamics. Event2Audio demonstrates recovery even for multiple simultaneous sources and in the presence of environmental distortions. The passive topology-based system likewise reports simultaneous recovery of multiple sound sources from a single event stream. A plausible implication is that event-based visual microphones occupy a boundary zone between audio reconstruction, remote vibrometry, and structured event-stream analysis (Yin et al., 3 Apr 2025, Cai et al., 4 Jul 2025, Niwa et al., 20 Oct 2025).

7. Limitations and directions for further development

The cited papers identify different bottlenecks. EvMic emphasizes the sim-to-real gap between event simulator physics and camera noise, along with sensitivity to laser power and ambient light: excessive brightness or insufficient illumination causes missing events or saturation. It also proposes extending topology with graph convolution over speckle nodes, using multi-view or multi-camera arrays for 3D vibration and better source separation, and integrating generative priors such as diffusion or score models to hallucinate high frequencies (Yin et al., 3 Apr 2025).

The passive topology-based framework identifies a more geometric limitation: because it relies on in-plane pixel displacements, vibrations along the optical axis produce negligible event topology and cannot be recovered. It also assumes vertical alignment of the principal vibration axis, although the paper notes that any planar orientation can be accommodated by a simple rotational pre-warp of the event cloud. Proposed extensions include stereo event-camera rigs for depth and out-of-plane motion, automatic selection of Mapper cover parameters $e_k=(u_k,t_k,p_k)$78 via persistence-based criteria, and spectral regularization that enforces known harmonic structure (Niwa et al., 20 Oct 2025).

Event2Audio frames its method as real-time-capable active optical vibration sensing. Its design decisions reflect a persistent systems-level trade-off: denoising by graph or kernel smoothing can improve robustness offline, but the real-time mode skips that smoothing for speed. This suggests that future event topology-based visual microphones are likely to continue balancing explicit structure, computational latency, illumination strategy, and calibration fidelity rather than converging immediately to a single dominant architecture (Cai et al., 4 Jul 2025).