---
title: Event Topology-Based Visual Microphone
url: https://www.emergentmind.com/topics/event-topology-based-visual-microphone
type: topic
---

# Event Topology-Based Visual Microphone

An event topology-based visual microphone is a non-contact vibration-sensing system that reconstructs sound or vibration from the asynchronous event stream of an event camera. In the recent literature, this designation covers at least three technically distinct formulations: learned spatial-temporal modeling over event voxels and speckle nodes, streaming optical-flow recovery from event-induced speckle motion, and explicit topological data analysis of the event cloud via Mapper and HDBSCAN. Across these formulations, the common premise is that sound-induced micro-vibrations produce structured changes in brightness or speckle position that can be converted into an acoustic or vibratory waveform without contact transduction [2504.02402] [2507.03273] [2510.17092].

## 1. Event sensing and vibration encoding

The basic signal emitted by the sensor is an event stream \(E=\{e_k\}_{k=1}^K\), where each event is represented either as \(e_k=(u_k,t_k,p_k)\) with \(u_k=(x_k,y_k)\), or equivalently as \(e_i=(x_i,y_i,t_i,p_i)\). The polarity \(p\in\{+1,-1\}\) is triggered when the change in log-brightness crosses a contrast threshold. In the EvMic formulation, polarity is defined by threshold crossings of \(L(u,t)\) with threshold \(C\); in Event2Audio, an event is emitted as soon as \((\log I(x,y,t)-\log I(x,y,t_0))\cdot p>\epsilon\), with the equivalent finite-difference condition \(|\Delta \log I(x,y,t)|>\epsilon\) [2504.02402] [2507.03273].

Two optical regimes appear in the cited work. Active systems amplify vibration visibility with coherent illumination. EvMic uses a 5 mW laser-matrix that projects a grid of small laser points onto the object surface, thereby enhancing local gradients \(|\nabla L|\) so that tiny vibrations cause brightness changes large enough to trigger events. Event2Audio uses a low-power coherent laser at \(532\ \mathrm{nm}\), \(4.5\ \mathrm{mW}\), focused onto a small spot on a vibrating surface; the rough surface scatters the beam into a granular speckle pattern, and small surface tilts \(\Delta \theta(t)\) produce a nearly pure lateral shift \(\Delta x(t),\Delta y(t)\) of the entire speckle pattern when the lens is slightly defocused. By contrast, the topology-based amplitude-and-frequency reconstruction framework is explicitly passive and reconstructs vibrations directly from raw event streams without external illumination [2504.02402] [2507.03273] [2510.17092].

The physical encoding model is correspondingly different but related. EvMic writes the event-generation relation as \(p_k C \approx -L'\cdot(\alpha(q_t-q_{t-1}))\), and states that the event rate at pixel \(u\) is proportional to the temporal derivative of log-brightness, \(\lambda(u,t)\propto |dL(u,t)/dt|/C \approx |L'(u)\cdot v(t)|/C\). Event2Audio instead emphasizes that lateral speckle translation sweeps bright and dark granules across pixels, producing local intensity ramps and asynchronous events. The passive topology-based method treats the filtered event set directly as a point cloud in \(\mathbb{R}^3\) [2504.02402] [2507.03273] [2510.17092].

## 2. Topology as graph structure, learned relation, and topological data analysis

In this literature, “topology” does not denote a single canonical representation. The most explicit topological construction appears in the passive framework, which embeds filtered events into the point cloud
\[
X=\{(x_i,y_i,t_i)\mid e_i \text{ passed filtering}\}\subset \mathbb{R}^3
\]
and applies a simplified Mapper pipeline. The filter function is one-dimensional,
\[
f(x,y,t)=y,
\]
chosen to align with the dominant vibration axis. The range \([y_{\min},y_{\max}]\) is subdivided into overlapping intervals \(U_j=[a_j,b_j]\), and the pull-back sets \(X_j=f^{-1}(U_j)\) form an overlapping cover of the event cloud. Within each \(X_j\), HDBSCAN extracts dense clusters \(C_j^\ell\), retaining only those whose persistence \(\pi(C)\) exceeds a threshold \(\tau\) [2510.17092].

A second interpretation appears in Event2Audio. That system states that it does not explicitly build a large graph, but the underlying event set \(E\) in \(\mathbb{R}^3\) can equivalently be viewed as a graph \(G=(E,\mathcal{E})\), in which each event node connects to nearest polarity-matching neighbors within a 3D window \(|\Delta x|,|\Delta y|\le r\) and \(|\Delta t|\le \Delta \tau\). In that view, flow computation corresponds to estimating local derivatives on the graph, and optional denoising can be written as
\[
v_{\mathrm{smoothed}}(e_i)=\sum_{j\in \mathcal{N}(i)} w_{ij}\,v(e_j),
\qquad
w_{ij}\propto \exp\!\left(-\|\Delta x_{ij}\|^2/\sigma_x^2-|\Delta t_{ij}|^2/\sigma_t^2\right).
\]
Offline mode can apply 3D kernel convolutions or graph Laplacian regularization, whereas real-time mode skips this smoothing for speed [2507.03273].

A third interpretation appears in EvMic. That system also does not build an explicit graph Laplacian, but it treats the \(N\) detected speckles as nodes and applies multi-head self-attention across them in its Spatial Aggregation Block. At each time \(t\), with \(f_t\in \mathbb{R}^{N\times C}\), the block computes
\[
Q_t=F_q(f_t),\quad K_t=F_k(f_t),\quad V_t=F_v(f_t),
\]
followed by
\[
A_t=\mathrm{SoftMax}(Q_tK_t^\top/\sqrt{d})\in \mathbb{R}^{N\times N},\quad
g_t=A_tV_t\in \mathbb{R}^{N\times d}.
\]
This induces a fully connected, learned relational structure over patch centers rather than a hand-specified topological complex [2504.02402].

This suggests that, in event-based visual microphones, topology spans at least three levels: explicit topological summarization of event clouds, implicit neighborhood graphs over asynchronous events, and learned attention-defined connectivity over spatially localized vibration carriers.

## 3. Reconstruction pipelines

The passive topology-based pipeline begins with aggressive event preprocessing. Only positive events \((p_i=+1)\) are retained, and the Delbrück background activity filter is applied with a temporal window \(T=200\ \mu\mathrm{s}\); an event is discarded unless there exists at least one spatial neighbor event within \(\pm T\) in time. After Mapper-based covering and HDBSCAN clustering, each surviving cluster \(C_j^\ell\) is reduced to a centroid
\[
v_j^\ell=(t_j^\ell,x_j^\ell,y_j^\ell)
=
\frac{1}{|C_j^\ell|}
\sum_{(x_i,y_i,t_i)\in C_j^\ell}(t_i,x_i,y_i).
\]
Sorting centroids by \(t_j^\ell\) defines a discrete 3D trajectory. Because only vertical motion is of interest, the trajectory is projected by \(P(t,x,y)=(t,y)\), producing a nonuniformly sampled waveform that is then resampled to a uniform grid at \(16\,\mathrm{kHz}\) via linear interpolation. Amplitude is estimated as half the peak-to-peak span, and frequency is obtained from the dominant peak of the DFT magnitude. The same pipeline is applied independently to multiple ROIs for multi-source recovery [2510.17092].

Event2Audio follows a streaming analytical route rather than a learned one. For each new event \(e_i=(x_i,y_i,t_i,p_i)\), the method searches the four cardinal neighbors at radius \(r=7\ \mathrm{px}\) for the most recent polarity-matching event, computes horizontal and vertical flow by
\[
v_{x_i}=\frac{x^h-x_i}{t^h-t_i},
\qquad
v_{y_i}=\frac{y^v-y_i}{t^v-t_i},
\]
and then quantizes time into uniform bins \(T_k=[k\Delta T,(k+1)\Delta T)\) with \(\Delta T=10\ \mu\mathrm{s}\), equivalent to \(100\ \mathrm{kHz}\). Aggregation yields dense traces
\[
V_x(k)=\frac{1}{N_k}\sum_{i\in T_k}v_{x_i},
\qquad
V_y(k)=\frac{1}{N_k}\sum_{i\in T_k}v_{y_i}.
\]
These are projected to a scalar channel \(V_{\mathrm{proj}}(t)=\alpha_xV_x(t)+\alpha_yV_y(t)\), integrated into relative displacement \(D(t)=\int_0^t V_{\mathrm{proj}}(\tau)d\tau\), high-pass filtered with a Butterworth filter, and denoised by spectral subtraction. Under small-angle, small-motion assumptions, the paper states \(\hat p(t)=H\star D(t)\), but in practice it omits explicit deconvolution by \(H\) and treats \(D(t)\), after DC removal, as the recovered audio waveform [2507.03273].

EvMic represents the event stream as a 4D voxel tensor \(V\in\mathbb{R}^{2\times T\times H\times W}\) over temporal bins of width \(\Delta T=T_{\mathrm{total}}/T\), with separate channels for the two polarities. Small patches \(P_i\in \mathbb{R}^{2\times T\times pw_i\times ph_i}\) are cropped around detected speckle centers. A ResNet-18 backbone with 3D submanifold sparse convolutions exploits the fact that only \(\sim 1\%\) of voxels are nonzero, producing feature tensors \(f_i\in\mathbb{R}^{T\times C}\), or \(f\in\mathbb{R}^{N\times T\times C}\) across all speckles. The Spatial Aggregation Block models inter-speckle relations, after which a Mamba module performs long-range temporal modeling as a linear state-space model:
\[
h_t=\bar A h_{t-1}+\bar B g_t,\qquad o_t=Ch_t,
\]
with \(\bar A=\exp(\Delta A)\) and
\[
\bar B=(\Delta A)^{-1}(\exp(\Delta A)-I)\Delta B.
\]
The output \(o\in\mathbb{R}^T\) is the reconstructed audio waveform per speckle, followed if desired by a final \(1\times N\) weighted sum over speckles [2504.02402].

## 4. Supervision, losses, and implementation regimes

EvMic is the most fully supervised of the three systems. Its training data are generated by a simulation pipeline in Blender, where an audio waveform \(q(t)\) sampled at \(f_s\) drives an object displacement \(\Delta X(t)=\mathrm{gain}\cdot q(t)\cdot \hat v\) along a random 3D direction, with \(\mathrm{gain}\in[0.5,2.0]\). The scene is rendered at \(10\ \mathrm{kFPS}\), resolution \(640\times 360\), and converted to synthetic events with the V2E event simulator. To bridge sim-to-real, real speckle patches are extracted from accumulated event frames, tiled on a black background, animated with the same audio-driven motion, and re-simulated; these vibrating speckles are used to fine-tune the Spatial Aggregation Block. Training uses PyTorch on a single NVIDIA RTX 4090 with SGD, learning rate \(10^{-4}\), and batch size \(1\). The system pre-trains “SPConv + Mamba” for \(200\mathrm{k}\) iterations and fine-tunes the full network, including SAB, for another \(200\mathrm{k}\) iterations. Input streams are \(0.5\ \mathrm{s}\) long and discretized into \(T=2000\) bins, giving \(4\ \mathrm{kHz}\) temporal resolution. The synthetic corpus contains \(10\mathrm{k}\) training segments from TIMIT speech plus random MIDI, and speckle fine-tuning uses approximately \(2\mathrm{k}\) synthetic speckle-only sequences. The total loss is
\[
L_{\mathrm{total}}=L_{\mathrm{sisnr}}+\beta L_{\mathrm{spec}},\qquad \beta=10^{-4},
\]
where \(L_{\mathrm{sisnr}}\) is the scale-invariant SNR loss and \(L_{\mathrm{spec}}\) is a multi-scale spectral reconstruction loss over Mel-spectrogram windows \(s\in\{2^6,\dots,2^9\}\) with \(\alpha_s=\sqrt{s/2}\) [2504.02402].

Event2Audio is optimized for speed rather than supervised learning. Its reported hardware includes a Prophesee Metavision EVK3-HD at \(1280\times 720\) pixels, a \(75\ \mathrm{mm}\ f/4.5\) achromatic doublet, and approximately \(2\ \mathrm{mm}\) defocus beyond focus to magnify the speckle pattern. The bias or contrast threshold is left at factory defaults, \(\epsilon\approx 0.2\)–\(0.3\) log-intensity units, and the typical event rate during audio sensing is \(50\)–\(200\ \mathrm{kEv/s}\). The same paper reports temporal quantization \(\Delta T=10\ \mu\mathrm{s}\), equivalent to \(100\ \mathrm{kHz}\) flow sampling, end-to-end latency of approximately \(10\)–\(20\ \mathrm{ms}\) in real-time mode on a modern CPU, and offline processing time of approximately \(1.5\ \mathrm{min}\) for \(10\ \mathrm{s}\) of audio [2507.03273].

The passive topology-based system is configured around a SilkyEvCam HD (EVK3) with an \(8\ \mathrm{mm}\) lens at a distance of \(20\ \mathrm{cm}\), with imaging scale approximately \(0.12\ \mathrm{mm/pixel}\). Its target is an aluminum plate of \(10\ \mathrm{mm}\times 100\ \mathrm{mm}\), thickness \(0.3\ \mathrm{mm}\), driven vertically by a mechanical wave driver. Ground truth is provided by a Laser Doppler Vibrometer (Polytec-500-3D-HV-Xtra) recorded at \(20\ \mathrm{kHz}\). The ROI is chosen as \(5\times 25\) pixels centered on the highest event-density pixel. HDBSCAN is configured with \(\mathrm{min\_cluster\_size}=30\), \(\mathrm{min\_samples}=30\), and a persistence threshold \(\tau\) selected to reject noise clusters [2510.17092].

## 5. Empirical performance

EvMic reports synthetic-test metrics in terms of SNR and STOI. Against EvPhase and RGBPhase, the method improves SNR by approximately \(1.3\ \mathrm{dB}\) and STOI by approximately \(0.10\) over EvPhase on the synthetic test set. On real-world data, spectrogram alignment against a microphone reference shows that the method recovers the fundamental and harmonics more faithfully. After post-processing with FullSubNet-Plus for speech, classical spectral subtraction for non-speech, and a final Butterworth high-pass to remove DC drift, the result is \(8\ \mathrm{kHz}\) reconstructed waveforms that are playable in real time. The deployment blueprint further reports that a \(0.5\ \mathrm{s}\) event buffer, \(2000\)-bin voxelization, and \(8\) speckle patches yield approximately \(30\ \mathrm{ms}\) CPU+GPU inference time [2504.02402].

Event2Audio evaluates speech quality using PESQ, STOI, Mel-cepstral distortion, and log spectral distance. For speaker-membrane recovery in real time, the passive event baseline of Niwa ’23 records PESQ \(1.05\), STOI \(0.52\), MCD \(17.28\), and LSD \(2.50\), whereas the proposed method reports PESQ \(1.26\), STOI \(0.67\), MCD \(5.87\), and LSD \(2.50\). In offline mode, high-speed frame-based recovery by Niwa ’23 records PESQ \(1.16\), STOI \(0.45\), MCD \(17.87\), LSD \(3.50\), and processing time measured in hours, whereas the event-based method reports PESQ \(1.49\), STOI \(0.74\), MCD \(5.57\), LSD \(1.90\), and time \(1.5\ \mathrm{min}\). For chip-bag recovery, Howard [2023] in real time reports PESQ \(1.04\), STOI \(0.15\), MCD \(26.10\), LSD \(3.26\), while the event-based method reports PESQ \(1.40\), STOI \(0.79\), MCD \(14.13\), LSD \(2.41\). In offline comparison, Sheinin [2022] reports PESQ \(1.39\), STOI \(0.73\), MCD \(12.79\), LSD \(2.20\), and \(54\ \mathrm{min}\), while the event-based method reports PESQ \(1.60\), STOI \(0.81\), MCD \(12.56\), LSD \(1.72\), and \(1.5\ \mathrm{min}\) [2507.03273].

The passive topology-based system evaluates waveform fidelity with Normalized Cross-Correlation and spectral agreement with Median Vector Angle error. For the increasing-\(100\ \mathrm{Hz}\) case, NCC is \(0.23\) for Abe et al. (frame, 2014), \(0.23\) for Niwa et al. (event phase, 2023), and \(0.97\) for the topology-based method; the corresponding MVA errors are \(1.32\), \(1.18\), and \(0.22\ \mathrm{rad}\). For decreasing-\(100\ \mathrm{Hz}\), NCC is \(0.34\), \(0.29\), and \(0.85\), and MVA is \(1.34\), \(1.13\), and \(0.33\ \mathrm{rad}\). For the speech case “Mary had a little lamb,” NCC is \(0.08\), \(0.08\), and \(0.46\), while MVA is \(1.54\), \(1.14\), and \(0.53\ \mathrm{rad}\). The same paper also reports a two-speaker experiment in which the recovered spectra showed distinct peaks at \(100\ \mathrm{Hz}\) and \(120\ \mathrm{Hz}\), confirming effective separation without cross-talk [2510.17092].

## 6. Scope, applications, and common misconceptions

A common misconception is that event topology necessarily refers to explicit topological data analysis. The cited work shows otherwise. In the passive framework, topology is implemented directly through Mapper covers, overlapping pull-backs, and HDBSCAN persistence. In Event2Audio, topology is implicit in local event neighborhoods and optional graph-based smoothing. In EvMic, topology is learned through attention over speckle nodes. The term therefore describes a family of structural treatments of event data rather than a single algorithmic primitive [2504.02402] [2507.03273] [2510.17092].

Another misconception is that visual microphones automatically recover calibrated acoustic pressure. Event2Audio states the linear relation \(\hat p(t)=H\star D(t)\) under small-angle, small-motion assumptions, but then omits explicit deconvolution by \(H\) in practice and treats the integrated displacement trace, after DC removal, as the recovered audio waveform. By contrast, the topology-based passive system is framed as amplitude-and-frequency reconstruction of a physical vibration signal, with amplitude estimated from peak-to-peak span and frequency from the dominant DFT peak. EvMic is optimized for sound recovery quality rather than explicit physical calibration of displacement amplitude [2507.03273] [2510.17092] [2504.02402].

The application range given in the literature is broad. EvMic lists non-contact speech eavesdropping through windows, structural health monitoring, acoustic holography and source localization with large field of view, and audio-visual scene understanding in low-light or extreme dynamics. Event2Audio demonstrates recovery even for multiple simultaneous sources and in the presence of environmental distortions. The passive topology-based system likewise reports simultaneous recovery of multiple sound sources from a single event stream. A plausible implication is that event-based visual microphones occupy a boundary zone between audio reconstruction, remote vibrometry, and structured event-stream analysis [2504.02402] [2507.03273] [2510.17092].

## 7. Limitations and directions for further development

The cited papers identify different bottlenecks. EvMic emphasizes the sim-to-real gap between event simulator physics and camera noise, along with sensitivity to laser power and ambient light: excessive brightness or insufficient illumination causes missing events or saturation. It also proposes extending topology with graph convolution over speckle nodes, using multi-view or multi-camera arrays for 3D vibration and better source separation, and integrating generative priors such as diffusion or score models to hallucinate high frequencies [2504.02402].

The passive topology-based framework identifies a more geometric limitation: because it relies on in-plane pixel displacements, vibrations along the optical axis produce negligible event topology and cannot be recovered. It also assumes vertical alignment of the principal vibration axis, although the paper notes that any planar orientation can be accommodated by a simple rotational pre-warp of the event cloud. Proposed extensions include stereo event-camera rigs for depth and out-of-plane motion, automatic selection of Mapper cover parameters \((N,\ell,\delta)\) via persistence-based criteria, and spectral regularization that enforces known harmonic structure [2510.17092].

Event2Audio frames its method as real-time-capable active optical vibration sensing. Its design decisions reflect a persistent systems-level trade-off: denoising by graph or kernel smoothing can improve robustness offline, but the real-time mode skips that smoothing for speed. This suggests that future event topology-based visual microphones are likely to continue balancing explicit structure, computational latency, illumination strategy, and calibration fidelity rather than converging immediately to a single dominant architecture [2507.03273].

Source: https://www.emergentmind.com/topics/event-topology-based-visual-microphone