EgoTrigger: Audio-Driven Sensing for Smart Glasses

• EgoTrigger is an audio-based sensing system that activates camera capture only during memory-relevant hand-object interactions, reducing continuous power and bandwidth use.
• It employs a binary audio classifier with fixed and hysteresis thresholds to convert fluctuating audio cues into stable, trigger-based camera activations.
• Empirical results demonstrate substantial power and bitrate reductions while preserving episodic memory question answering performance.

EgoTrigger is an audio-driven sensing strategy for all-day smart glasses that uses microphone signals to selectively activate power-intensive cameras for human memory enhancement. Rather than keeping the camera on continuously, it listens continuously for hand-object interaction (HOI) audio cues—such as the sound of a drawer opening or a medication bottle being opened—and captures images only when those cues indicate a memory-relevant moment. The method is motivated by the energy, bandwidth, and thermal constraints of all-day smart glasses, and is evaluated as a trigger-based front end for episodic-memory question answering on egocentric video (Paruchuri et al., 3 Aug 2025).

1. Concept and problem formulation

EgoTrigger addresses a specific systems bottleneck in memory-assistive smart glasses: continuous camera capture is expensive in power and bandwidth, while all-day glasses have small batteries and limited onboard compute. Even when downstream visual understanding is offloaded, the system still incurs substantial cost from image sensor operation, image compression or ISP, and wireless transmission. EgoTrigger therefore treats audio as the always-available low-power modality and vision as the expensive modality that should be activated only when context warrants it (Paruchuri et al., 3 Aug 2025).

The operational premise is that many memory-relevant events are HOI moments. The target cues include opening a drawer, opening a medication bottle, handling objects, typing, placing or picking up items, washing hands, and opening or closing objects. The paper ties these cues directly to episodic-memory use cases such as recalling where keys were left, whether a stove was turned off, whether medication was taken, and how routine tasks were completed (Paruchuri et al., 3 Aug 2025).

A common misconception is to treat EgoTrigger as a full multimodal reasoning architecture. The paper instead defines it as a context-aware sensor-management strategy: audio determines when images should be captured, and the value of that strategy is assessed by how much downstream memory-QA utility is preserved under reduced visual sensing. This suggests that the distinctive contribution lies in when the camera is activated, not in replacing the downstream QA model itself (Paruchuri et al., 3 Aug 2025).

2. Triggering mechanism and system design

EgoTrigger uses a binary audio classifier $f_\theta$ with classes $C_0$ for no HOI and $C_1$ for HOI present. The system processes a continuous audio stream $X$ with a sliding window,

$x_i = X[i \cdot w_h : i \cdot w_h + w_d],$

where $w_d$ is the window duration and $w_h$ is the hop size. The paper gives example values of $w_d = 4$ seconds and $w_h = 2$ seconds. The classifier outputs $P(C_1 \mid x_i) = f_\theta(x_i)$, and image capture is triggered when that probability exceeds a threshold such as $C_0$0 (Paruchuri et al., 3 Aug 2025).

The paper describes two trigger-control strategies. In the Fixed OFF duration mode, once triggered, the camera remains active for a fixed interval $C_0$1, with an example of $C_0$2 second. In the Hysteresis mode, the system uses separate thresholds $C_0$3, with example values $C_0$4 and $C_0$5, to avoid rapid toggling from noisy probability fluctuations. In both cases, the purpose is to convert fluctuating audio evidence into stable camera-on intervals suitable for wearable deployment (Paruchuri et al., 3 Aug 2025).

This trigger design is event-centered rather than uniformly sampled. The resulting captured visual stream is therefore sparse but semantically biased toward interaction moments. A plausible implication is that EgoTrigger is best understood as a front-end scheduler for memory capture, not as a generic video subsampling rule.

3. Audio model, training pipeline, and deployment footprint

The trigger model is built by transfer learning from YAMNet, a lightweight pretrained audio event model. YAMNet provides 1024-dimensional embeddings, and the final YAMNet classifier is replaced with a custom 4-layer dense head described as

$C_0$6

with ReLU activations and dropout rates of $C_0$7, $C_0$8, $C_0$9, and $C_1$0 after the hidden layers. The model is trained with binary cross-entropy for HOI detection (Paruchuri et al., 3 Aug 2025).

Audio preprocessing converts clips to mono, resamples them to $C_1$1 kHz, and amplitude-normalizes them to $C_1$2. Training uses AdamW with learning rate $C_1$3, weight decay $C_1$4, batch size $C_1$5, and $C_1$6 epochs. The paper compares class-weighted training, SMOTE oversampling, and random undersampling, ultimately selecting the SMOTE-based detector for the main experiments because it gives the best overall result on the HOI classification task (Paruchuri et al., 3 Aug 2025).

For deployment, the trained model is converted to TensorFlow Lite (LiteRT). The reported model size decreases from $C_1$7 MB to $C_1$8 MB, an $C_1$9 reduction. This deployment result is central to the all-day-glasses framing: the trigger mechanism is intended to be lightweight enough for continuous use, while the expensive visual sensing remains intermittent (Paruchuri et al., 3 Aug 2025).

4. Benchmarking context and the HME-QA dataset

A major contribution associated with EgoTrigger is HME-QA, the Human Memory Enhancement Question-Answer dataset. The paper introduces it because QA-Ego4D, while useful, contains a large fraction of test videos without audio tracks; the paper states that nearly $X$0 of QA-Ego4D test videos lack audio, which makes them unsuitable for evaluating audio-triggered capture. HME-QA is therefore designed to match the audio-centric memory-enhancement setting more closely (Paruchuri et al., 3 Aug 2025).

HME-QA is curated from Ego4D episodic-memory videos by filtering an initial set of $X$1 full-length videos, verifying audio with FFMPEG, retaining $X$2 valid videos, generating candidate QA pairs with Gemini 2.0 Flash, and then having an expert annotator review and discard poor pairs. The resulting dataset contains $X$3 human-validated QA pairs over $X$4 full-length videos, with videos averaging about $X$5 minutes and some reaching $X$6 minutes (Paruchuri et al., 3 Aug 2025).

The paper also filters QA-Ego4D for fair comparison in the audio-triggered setting, reducing the original test set of $X$7 QA pairs across $X$8 videos to $X$9 QA pairs across $x_i = X[i \cdot w_h : i \cdot w_h + w_d],$0 videos. This makes HME-QA complementary to QA-Ego4D: QA-Ego4D remains the larger standard benchmark, whereas HME-QA is smaller, curated, audio-guaranteed, and explicitly HOI-centric (Paruchuri et al., 3 Aug 2025).

5. Empirical performance, robustness, and energy–utility tradeoff

On HOI classification, the paper reports that the original class-weighted model achieves $x_i = X[i \cdot w_h : i \cdot w_h + w_d],$1 F1 $x_i = X[i \cdot w_h : i \cdot w_h + w_d],$2, $x_i = X[i \cdot w_h : i \cdot w_h + w_d],$3 F1 $x_i = X[i \cdot w_h : i \cdot w_h + w_d],$4, and weighted-average F1 $x_i = X[i \cdot w_h : i \cdot w_h + w_d],$5. With SMOTE oversampling, the best overall result becomes $x_i = X[i \cdot w_h : i \cdot w_h + w_d],$6 F1 $x_i = X[i \cdot w_h : i \cdot w_h + w_d],$7, $x_i = X[i \cdot w_h : i \cdot w_h + w_d],$8 F1 $x_i = X[i \cdot w_h : i \cdot w_h + w_d],$9, and weighted-average F1 $w_d$0. Random undersampling yields $w_d$1 F1 $w_d$2, $w_d$3 F1 $w_d$4, and weighted-average F1 $w_d$5. The threshold analysis further reports that class-$w_d$6 F1 stays at or above $w_d$7 for $w_d$8, indicating that the trigger can be tuned over a useful precision–recall range without collapsing HOI sensitivity (Paruchuri et al., 3 Aug 2025).

Robustness measurements quantify both false positives and noise sensitivity. The false-positive rate is $w_d$9 on ESC-50 environmental sounds and $w_h$0 on Ego4DSounds non-HOI or speech-containing clips. Under additive white noise, weighted F1 drops from $w_h$1 when clean to $w_h$2 at low noise $w_h$3 and $w_h$4 at high noise $w_h$5. On $w_h$6 full-length HME-QA videos, the system averages $w_h$7 false positives per minute; the paper characterizes this as workable for standby-style gating while noting that noisier environments remain an open challenge (Paruchuri et al., 3 Aug 2025).

The downstream question is whether reduced capture preserves memory-QA performance. On HME-QA, the paper reports Full continuous capture at $w_h$8, Decimated $w_h$9 FPS capture at $w_d = 4$0, ET-1s at $w_d = 4$1, and ET-Hyst. at $w_d = 4$2. On filtered QA-Ego4D, the corresponding numbers are $w_d = 4$3, $w_d = 4$4, $w_d = 4$5, and $w_d = 4$6. EgoTrigger therefore remains much closer to full capture than naive decimation while using substantially fewer frames (Paruchuri et al., 3 Aug 2025).

The principal efficiency result is frame reduction. ET-1s uses $w_d = 4$7 fewer frames on HME-QA and $w_d = 4$8 fewer frames on QA-Ego4D. The bitrate analysis reports, on HME-QA, $w_d = 4$9 Mbps for Full, $w_h = 2$0 Mbps for ET-1s, and $w_h = 2$1 Mbps for Decimated; on QA-Ego4D, it reports $w_h = 2$2 Mbps, $w_h = 2$3 Mbps, and $w_h = 2$4 Mbps, respectively. Prototype power measurements on a Raspberry Pi 5 with USB camera and USB microphone show $w_h = 2$5 W when idle, $w_h = 2$6 W for continuous capture, $w_h = 2$7 W for continuous capture plus Wi-Fi transmission, $w_h = 2$8 W for $w_h = 2$9 FPS decimation, and $P(C_1 \mid x_i) = f_\theta(x_i)$0 W for EgoTrigger ET-1s. The paper summarizes this as a $P(C_1 \mid x_i) = f_\theta(x_i)$1 power reduction relative to continuous capture in the prototype and a roughly halved visual bitrate with much of the QA utility preserved (Paruchuri et al., 3 Aug 2025).

6. Position within egocentric assistance research

EgoTrigger belongs to a broader line of egocentric assistance research in which sensing, query formation, and reasoning are shaped by human behavior rather than by arbitrary clip sampling. In "EgoEverything," questions are generated around gaze-attended objects, with gaze treated as a proxy for human attention and a recall delay sampled after the attended moment. The paper does not define a literal trigger-event benchmark, but it directly supports an EgoTrigger-style setting in which a behavior signal determines what later becomes query-worthy in augmented reality (Tang et al., 9 Apr 2026).

Adjacent work clarifies what EgoTrigger does and does not solve. "EgoEnv" argues that many egocentric queries cannot be answered from the visible clip alone and require a persistent environment representation predictive of local surroundings, including unseen directions (Nagarajan et al., 2022). "Visual Intention Grounding for Egocentric Assistants" shows that assistants must often infer the object implied by a user’s need rather than ground an explicitly named object, making intention inference and affordance reasoning central to egocentric assistance (Sun et al., 18 Apr 2025). "EgoInteract" provides synthetic egocentric episodes with precise start and end times, contact states, hand-object associations, and gaze-primed anticipation clips, supplying several trigger-like temporal signals for interaction onset and next-active-object prediction (Leonardi et al., 18 May 2026).

Taken together, these works indicate that triggering in egocentric systems can be instantiated through different modalities and purposes: audio-gated sensing in EgoTrigger, gaze-conditioned query formation in EgoEverything, environment memory in EgoEnv, implicit intention reasoning in EgoIntention, and temporal onset supervision in EgoInteract. This suggests that EgoTrigger is best situated as a systems-level contribution to energy-efficient sensing within a larger research program on behavior-aware, memory-oriented egocentric assistants.