Human Memory Enhancement QA Benchmark

• HME-QA is an episodic-memory QA benchmark that evaluates selective sensing of memory-relevant events from egocentric video.
• It employs audio-driven trigger systems like EgoTrigger to selectively capture key hand-object interactions and reduce resource usage.
• Empirical results demonstrate that smart capture strategies nearly match full continuous capture performance while cutting down on energy and storage costs.

Human Memory Enhancement Question-Answer (HME-QA) is an egocentric episodic-memory question-answering benchmark introduced to evaluate whether selective sensing can preserve the evidence needed for later recall in all-day smart glasses systems (Paruchuri et al., 3 Aug 2025). In the benchmark’s intended use case, a wearer later asks questions such as where an object was left or whether an action was completed, while the sensing system must balance recall utility against the energy cost of continuous video capture. HME-QA therefore occupies a specific place within memory-augmentation research: it is not primarily a retrieval benchmark over an already curated memory bank, but a benchmark for whether memory-relevant moments can be captured selectively, especially around hand-object interactions with distinctive audio cues.

1. Definition and conceptual scope

HME-QA was introduced together with EgoTrigger, an audio-driven camera-gating system for human memory enhancement on smart glasses (Paruchuri et al., 3 Aug 2025). The motivating claim is that future all-day smart glasses should do more than continuously record: they should record selectively in ways that preserve the moments most useful for later recall. In this framing, the bottleneck is not only multimodal reasoning but sensing itself, because continuous video capture is expensive in battery and transmission cost, and large multimodal models are unrealistic to run locally on glasses.

The benchmark is explicitly tied to episodic-memory-style assistance. The target questions concern object locations, object states, completed actions, interactions, personal experiences, and habitual routines when inferable. This emphasis distinguishes HME-QA from generic egocentric QA, because the questions are written from the wearer’s perspective and are meant to resemble what someone would plausibly want to remember later (Paruchuri et al., 3 Aug 2025).

Within the broader literature, HME-QA addresses a different stage of the memory pipeline from systems that assume memories have already been stored. “Encode-Store-Retrieve” converts egocentric video into language, stores those textual traces in a vector database, and answers questions over encoded experience (Shen et al., 2023). “Memory-QA” formalizes recall questions over previously stored multimodal memories $M_i=(I_i,C_i,T_i,L_i)$ and studies augmentation, retrieval, and generation over those saved items (Jiang et al., 22 Sep 2025). HME-QA instead asks whether the relevant moments can be sensed efficiently enough in the first place. This suggests that HME-QA isolates the acquisition problem inside a larger memory-enhancement stack.

2. Dataset construction and annotation protocol

HME-QA is a curated subset of Ego4D episodic-memory videos rather than a broad random sample (Paruchuri et al., 3 Aug 2025). The source selection began with 82 full-length videos that had complete narrations and moments annotations. FFMPEG was then used to verify that the videos actually contained usable audio, reducing the source pool to 50 valid videos. These 50 videos form the final source pool for HME-QA; on average they are about 12 minutes long, with some extending to 30 minutes.

Candidate question-answer pairs were generated with Gemini 2.0 Flash from video content plus metadata, specifically narrations and moments annotations. The prompting targeted memory-recall questions centered on the camera wearer’s interactions. An expert annotator with egocentric-video experience then reviewed all generated pairs and discarded low-quality, ambiguous, irrelevant, or factually incorrect items. This human-in-the-loop filtering produced the final dataset of 340 human-annotated QA pairs (Paruchuri et al., 3 Aug 2025).

Each QA pair is timestamped with a start and end time in the source video. The timestamps are essential because the downstream use case is selective sensing: a capture policy can be evaluated by whether it preserved the visual evidence in the annotated interval well enough for a QA model to answer correctly. The paper emphasizes that HME-QA is curated not only for QA quality but also for the presence of meaningful audio, making it suitable for studying audio-driven triggering rather than generic egocentric perception (Paruchuri et al., 3 Aug 2025).

3. Task structure and memory content

The HME-QA task is episodic-memory question answering over full-length first-person videos (Paruchuri et al., 3 Aug 2025). Questions are phrased from the wearer’s perspective and ask about events that are plausible targets of later recall. The appendix prompt template explicitly requests questions about “object placements,” “object state,” “key interactions,” “task completion/confirmation,” “personal experiences,” and “habitual actions/locations.” Answers are concise factual responses addressed to the user, such as “you left the keys on the table” or “yes, you turned off the lights.”

A defining property of HME-QA is its bias toward hand-object interactions (HOIs). The underlying hypothesis is that HOI moments—opening a drawer, handling a bottle, placing an item somewhere—are especially memory-relevant, and that those moments often have distinctive audio cues (Paruchuri et al., 3 Aug 2025). The benchmark therefore supports questions grounded in both vision and audio, but with a deliberate emphasis on embodied interaction with objects.

This task design has methodological consequences. It does not assume that every frame has equal value for later recall. Instead, it operationalizes the idea that memory-relevant evidence is sparse, interaction-centric, and often marked by acoustic events. In that respect, HME-QA differs from benchmarks where the full visual archive is already available to the retriever or answer generator. It tests whether selective sensing policies can preserve the semantically critical intervals rather than merely subsample the stream uniformly.

4. EgoTrigger and audio-driven selective capture

EgoTrigger is the sensing policy evaluated on HME-QA (Paruchuri et al., 3 Aug 2025). It continuously processes microphone audio with a lightweight classifier and activates the camera only when the system predicts that a hand-object interaction is occurring. The classifier uses YAMNet embeddings, with the final head replaced by a custom four-layer dense network. In the described implementation, YAMNet yields 1024-dimensional embeddings, which are passed through a dense stack with dimensions $$1024 \rightarrow 256 \rightarrow 384 \rightarrow 192 \rightarrow 384 \rightarrow 2$$, using ReLU and dropout.

The model outputs class probabilities for $C_0$ (no HOI) and $C_1$ (HOI) through softmax: $$P(C_k \mid x_i) = \frac{e^{z_k}}{e^{z_0} + e^{z_1}}, \quad k \in \{0,1\}.$$ Audio is processed with a sliding window of duration $w_d=4$ seconds and hop $w_h=2$ seconds: $x_i = X[i \cdot w_h : i \cdot w_h + w_d].$ A trigger is fired when $P(C_1 \mid x_i) \ge \tau$.

The paper describes two post-trigger ON/OFF strategies. One uses a fixed OFF duration $T_{\text{fixed}}$, exemplified with 1 second. The other uses hysteresis with separate thresholds $$1024 \rightarrow 256 \rightarrow 384 \rightarrow 192 \rightarrow 384 \rightarrow 2$$0, reducing rapid toggling (Paruchuri et al., 3 Aug 2025). HME-QA is the downstream benchmark used to test whether these trigger policies preserve enough evidence for episodic-memory QA.

Training for the HOI trigger uses Ego4DSounds, a separate dataset derived from Ego4D audio-video clips with narrations. Roughly 14,000 audio clips were sampled for training and about 2,300 for testing. Gemini 1.5 Pro labeled each clip as HOI or non-HOI by jointly inspecting the audio waveform and narration. Because the classes are imbalanced, the study used class weights and also evaluated SMOTE oversampling and random undersampling. The best classifier for later experiments was the SMOTE-trained model, which achieved a weighted F1 of 0.86 on the test set, with $$1024 \rightarrow 256 \rightarrow 384 \rightarrow 192 \rightarrow 384 \rightarrow 2$$1 F1 improving from 0.15 under the original weighted baseline to 0.54 under SMOTE, while $$1024 \rightarrow 256 \rightarrow 384 \rightarrow 192 \rightarrow 384 \rightarrow 2$$2 remained strong at 0.90 (Paruchuri et al., 3 Aug 2025).

5. Evaluation methodology and empirical results

Evaluation on HME-QA uses an LLM-as-a-judge protocol with Gemini 1.5 Pro (Paruchuri et al., 3 Aug 2025). For each processed video and question, Gemini generates an answer and then judges correctness against the ground-truth answer on a 0–5 scale. Scores of 3–5 count as correct and 0–2 as incorrect. The main reported metric is answer correctness percentage.

Four video-sampling strategies are compared: Full continuous capture, naive Decimated capture at one frame every five seconds ($$1024 \rightarrow 256 \rightarrow 384 \rightarrow 192 \rightarrow 384 \rightarrow 2$$3 FPS), EgoTrigger with fixed 1-second OFF duration (ET-1s), and EgoTrigger with hysteresis (ET-Hyst.). On HME-QA, Full capture obtains 77.3% correctness, ET-1s obtains 75.7%, ET-Hyst. obtains 74.7%, and Decimated capture falls to 71.9% (Paruchuri et al., 3 Aug 2025).

Strategy	HME-QA correctness	HME-QA frame reduction vs. Full
Full capture	77.3%	0%
ET-1s	75.7%	54.39%
ET-Hyst.	74.7%	27.88%
Decimated	71.9%	not reported as frame reduction

The same pattern appears on QA-Ego4D: Full capture is 41.08%, ET-1s is 40.1%, ET-Hyst. is 39.2%, and Decimated is 34.6% (Paruchuri et al., 3 Aug 2025). ET-1s is therefore the strongest trigger variant, remaining within about 1–2 percentage points of Full while substantially reducing frames.

The efficiency analysis is central to the benchmark’s significance. On HME-QA, ET-1s reduces frames by 54.39% relative to Full, while ET-Hyst. reduces them by 27.88%. These savings translate into estimated bitrate reductions on HME-QA from 5.47 Mbps for Full to 2.50 Mbps for ET-1s and 3.96 Mbps for ET-Hyst.; Decimated capture is 1.13 Mbps (Paruchuri et al., 3 Aug 2025). A prototype power measurement on a Raspberry Pi 5 with a USB camera and microphone reports 4.97 W for continuous capture, 5.51 W for continuous capture plus Wi-Fi transmission, 3.02 W for Decimated 0.2 FPS, and 4.11 W for EgoTrigger ET-1s, corresponding to a 17.3% power reduction relative to continuous capture. The paper’s interpretation is that selective capture can reduce camera-on time and transmission load without collapsing memory-QA performance.

6. Position within the memory-enhancement QA landscape

HME-QA belongs to a broader family of systems that model personal or long-term memory for question answering, but it emphasizes a different failure mode from most retrieval-centric work. The surrounding literature spans language-encoded egocentric archives, multimodal personal memories, associative retrieval over dialogue histories, and personalized referential memory across heterogeneous sources.

Benchmark or system	Memory substrate	Central challenge
HME-QA (Paruchuri et al., 3 Aug 2025)	Full-length egocentric video with guaranteed audio	Selective sensing for episodic-memory QA
Memory-QA (Jiang et al., 22 Sep 2025)	Snapshot image, invocation command, timestamp, location	Temporal vagueness, location anchoring, multi-memory aggregation
ATM-Bench (Mei et al., 2 Mar 2026)	Images, videos, emails with metadata	Personalized references, multi-evidence reasoning, conflicting evidence
AssoMem (Zhang et al., 12 Oct 2025)	Large evolving dialogue and meeting memory banks	Similarity-dense retrieval and usefulness-aware ranking

“Memory-QA” studies recall questions over multimodal saved memories and proposes Pensieve, which integrates memory-specific augmentation, time- and location-aware multi-signal retrieval, and multi-memory QA fine-tuning (Jiang et al., 22 Sep 2025). Its benchmark contains 5,800 images with timestamp and location metadata, and the task explicitly includes temporal vagueness, location anchoring, and multi-memory aggregation. HME-QA is narrower in modality but more specific about continuous egocentric capture and energy-aware sensing.

ATM-Bench extends the scope further to multimodal, multi-source personalized referential Memory QA over approximately four years of privacy-preserving personal memory data drawn from images, videos, and emails (Mei et al., 2 Mar 2026). It introduces challenges such as Personalized Referential Reasoning, Location Awareness, Memory Update Over Time, Multi-Evidence Composition, and Abstention. The paper reports poor performance under 20% accuracy on the ATM-Bench-Hard set, indicating that even when memory artifacts are already stored, realistic long-term personalized recall remains difficult.

AssoMem addresses a different bottleneck: retrieval from large, evolving memory banks of dialogue sessions, meeting notes, or similar personal records (Zhang et al., 12 Oct 2025). It argues that relevance alone is insufficient in similarity-dense repositories and introduces an associative memory graph with importance-aware ranking and temporal alignment. This suggests that a complete HME-QA system would require not only selective acquisition of memory-relevant moments but also sophisticated downstream retrieval and evidence ranking once those moments have been stored.

Earlier egocentric memory-augmentation work also provides complementary context. “Encode-Store-Retrieve” converts egocentric perception into language, stores it in Chroma, and answers questions with GPT-4 over retrieved textual traces, achieving a BLEU score of 8.3 on QA-Ego4D (Shen et al., 2023). Relative to that line of work, HME-QA is less about how to encode and query already captured experience and more about how to decide what should be captured at all.

7. Limitations and open directions

The limitations of HME-QA arise directly from the assumptions of EgoTrigger (Paruchuri et al., 3 Aug 2025). First, the system depends on audible HOIs, so it can miss silent interactions, purely visual events, or interactions masked by noise. This means that not all memory-relevant events are well served by an audio-only trigger. Second, evaluation is offline and based on simulated capture by blacking out frames rather than on-device deployment; the paper explicitly states that true on-device latency and power still need hardware validation.

Third, the trigger model was trained with automatically generated HOI labels and a specific audio/video annotation pipeline, so generalization to new environments, speakers, or devices is uncertain (Paruchuri et al., 3 Aug 2025). Fourth, false positives remain nontrivial: on five HME-QA videos the trigger produced 2.4 false positives per minute. For always-worn devices, even a manageable false-positive rate can affect perceived reliability, storage load, and privacy exposure.

The paper therefore identifies multimodal trigger design as an open direction, including IMU fusion, personalization, and self-supervised cross-modal training (Paruchuri et al., 3 Aug 2025). In the broader memory-QA literature, additional open problems concern memory ingestion, retrieval, and answer generation. ATM-Bench shows that long-horizon multi-evidence reasoning and memory updates over time remain unresolved (Mei et al., 2 Mar 2026), while AssoMem shows that similarity-dense memory repositories require more than nearest-neighbor retrieval (Zhang et al., 12 Oct 2025). A plausible implication is that future HME-QA systems will need an end-to-end architecture spanning selective sensing, structured storage, temporally aware retrieval, and evidence-grounded generation rather than a capture policy alone.