OpenLifelogQA: Multi-Modal Lifelog Querying

• OpenLifelogQA is a dataset and research program that enables answering diverse personal queries using 18 months of real lifelog data.
• It employs a multi-modal approach, integrating images, text, and sensor inputs to support temporal, geographic, and aggregate reasoning.
• The system leverages retrieval-to-reasoning pipelines and modality-specific encoders to synthesize complex, context-rich answers over episodic logs.

OpenLifelogQA denotes both a research ambition and a named dataset. As a research ambition, it is the attempt to answer arbitrary, user-driven questions over a person’s lifelog—an evolving timeline of experiences—so that personal assistants can provide advice in context. As a dataset, it is an open-ended multi-modal lifelog question-answering resource built on 18 months of real point-of-view lifelog data, intended to move beyond small-sized or synthetic QA datasets (Tan et al., 2023, Tran et al., 5 Aug 2025).

1. Conceptual foundations

In this line of work, a lifelog is a private, user-controlled database of episodes that capture experiences drawn from digital services such as photos, maps, shopping, streaming, and health applications, and, in the future, AR glasses. Episodes combine free text with structure, including time, location, participants, and measurements; they may also be nested, as in a multi-day trip containing daily sub-episodes. The resulting QA problem is not reducible to conventional document QA, because answers often depend on compositional selection over sets of episodes, temporal and geographical predicates, and aggregation across heterogeneous records (Tan et al., 2023).

A broader lifelogging agenda situates this problem within personal knowledge management. The position paper “Ten Questions in Lifelog Mining and Information Recall” describes lifelogs as personal big data stored in digital formats that keep personal life events, and frames a Personal Knowledge Base as “a set of personal knowledge represented in a structured format.” It further proposes temporal fact representations such as $\langle s, p, o, t \rangle$ and an extension $\langle s, p, o, t, e \rangle$ when emotion is modeled (Yen et al., 2020). This suggests that OpenLifelogQA occupies a boundary region between lifelog retrieval, temporal knowledge representation, and QA over personal episodic memory.

The practical motivation is explicit across the literature. Representative queries include “When did I go to Tokyo?”, “What did I eat on my second night in Paris?”, “How many times did I go to the dentist last year?”, and “Did my deep sleep improve in the week when I increased aerobic exercise?” These formulations require retrieval over longitudinal personal data, temporal alignment, structured reasoning, and answer synthesis rather than simple content lookup (Tan et al., 2023, Tian et al., 20 Jan 2026).

2. Benchmark ecology and dataset resources

The benchmark landscape is stratified by realism, modality, and evaluation protocol. “TimelineQA” provides synthetic lifelogs of imaginary people with diverse personas and a public benchmark that spans atomic, temporal, geographic, and multi-hop aggregate questions. “OpenLifelogQA,” by Tran, Nguyen, Jones, and Gurrin, uses 18 months of real point-of-view lifelog data and emphasizes open-ended answers grounded in images, time, location, and event descriptions. “LifeAgentBench” narrows the scope to digital health, but introduces SQL-verifiable long-horizon and cross-dimensional reasoning over diet, sleep, activity, and emotion. “LifeDialBench” addresses continuous conversational memory under a causality-preserving online protocol (Tan et al., 2023, Tran et al., 5 Aug 2025, Tian et al., 20 Jan 2026, Zheng et al., 13 Apr 2026).

Resource	Scale	Distinctive focus
TimelineQA	3,000 lifelogs; 128,023,476 entries; 600,000 atomic test questions	Synthetic episode-centric benchmark for atomic, temporal, geographic, and aggregate reasoning
OpenLifelogQA	14,187 Q&A pairs; 27,705 event descriptions; 514 days	Open-ended multi-modal QA on real 18-month point-of-view lifelog data
LifeAgentBench	22,573 questions; 100 participants; month-long AI4FoodDB	Long-horizon, cross-dimensional, multi-user lifestyle health reasoning with executable SQL ground truth
LifeDialBench	EgoMem 1,774 QA pairs; LifeMem 1,717 QA pairs	Continuous lifelog memory over ambient conversations with online evaluation

TimelineQA’s scale is unusually large for lifelog QA: 3,000 lifelogs totaling 128,023,476 entries, split across sparse, medium, and dense logs. Each entry averages 8.4 tokens across 25 event categories, and the benchmark exposes atomic QA, temporal reasoning, geographic reasoning, and multi-hop aggregates with ground-truth evidence sets for the multi-hop split (Tan et al., 2023). OpenLifelogQA, by contrast, is grounded in 722,606 point-of-view images from the LSC’24 corpus, with 14,187 Q&A pairs and 27,705 event descriptions spanning 514 days. Its questions are categorized as Atomic (58.8%), Aggregation (32.58%), and Temporal (8.63%), and each QA is linked to one or more event IDs for retrieval and verification (Tran et al., 5 Aug 2025).

LifeAgentBench contributes a health-centric but methodologically important slice of the same problem space. It is built from AI4FoodDB, a month-long multimodal lifestyle dataset with 100 participants, and contains 22,573 questions split into single-user and multi-user cases. Its operator-based generation pipeline maps each natural-language question $Q$ to an executable program $\pi(Q)$ and then to SQL, yielding deterministic ground truth for long-horizon aggregation, threshold checks, and trend analysis (Tian et al., 20 Jan 2026). LifeDialBench addresses a different modality regime—ambient conversation captured by wearable devices—and explicitly argues that offline evaluation produces temporal leakage. Its two subsets, EgoMem and LifeMem, support both MCQ and open-ended QA over continuous, multi-party lifelog streams (Zheng et al., 13 Apr 2026).

3. Data model, task structure, and formal semantics

The most explicit formalization comes from TimelineQA. A lifelog timeline is modeled as a finite set or sequence $L = \{e_i\}_{i=1}^N$, where each episode is $$e_i = (t_i^{start}, t_i^{end}, \ell_i^{start}, \ell_i^{end}, x_i, a_i)$$. Here $t_i^{start}, t_i^{end} \in T$, $\ell_i^{start}, \ell_i^{end} \in G$, $x_i$ is the free-text description, and $a_i$ are structured attributes such as participants, activity type, and metrics. Super-episodes contain sub-episodes through a containment relation $\langle s, p, o, t, e \rangle$0 (Tan et al., 2023).

Atomic QA is formulated as episode selection under a predicate $\langle s, p, o, t, e \rangle$1. The system finds an episode $\langle s, p, o, t, e \rangle$2 such that $\langle s, p, o, t, e \rangle$3 holds, then returns an attribute value or a span from $\langle s, p, o, t, e \rangle$4. Multi-hop aggregation introduces a selected set $\langle s, p, o, t, e \rangle$5 and computes $\langle s, p, o, t, e \rangle$6 where $\langle s, p, o, t, e \rangle$7. Temporal predicates include interval membership $\langle s, p, o, t, e \rangle$8, sequencing $\langle s, p, o, t, e \rangle$9, and gap duration $Q$0; geographic predicates include region membership and route-constrained selection during a trip super-episode (Tan et al., 2023).

Question taxonomies across the literature are largely compatible. TimelineQA separates atomic lookup/extraction, temporal reasoning, geographical reasoning, and multi-hop selection with aggregates. LifeAgentBench defines five task families: Fact Query, Aggregated Statistics, Numeric Comparison, Conditional Query, and Trend Analysis. OpenLifelogQA uses Atomic, Temporal, and Aggregation. Despite terminological variation, the shared substrate is a programmatic decomposition into selection, alignment, comparison, and aggregation over timestamped personal records (Tan et al., 2023, Tian et al., 20 Jan 2026, Tran et al., 5 Aug 2025).

Evaluation protocols differ by answer form. TimelineQA evaluates atomic QA with Exact Match and token-level F1, and multi-hop TableQA with denotation accuracy. LifeAgentBench uses Accuracy, SQL Validity, Execution Accuracy, and $Q$1 to separate SQL generation from final reasoning. OpenLifelogQA evaluates open-ended multimodal answers with BERTScore, ROUGE-L, and an LLM Score from 1 to 5. These metric choices reflect a genuine methodological divide: some benchmarks emphasize executable denotations and structured outputs, whereas others emphasize semantically faithful natural-language answers (Tan et al., 2023, Tian et al., 20 Jan 2026, Tran et al., 5 Aug 2025).

4. Retrieval, reasoning, and multimodal system design

A recurring architectural pattern is retrieval followed by typed reasoning. TimelineQA explicitly recommends a pipeline with dense episode retrieval, fine-tuned span-extractive QA for “what/where/when/who/yes-no” questions, normalization of times and locations for temporal and geographic predicates, table-style aggregation over selected episode sets, and hybridization between free-text extraction and structured schemas. Topic-aware tables such as annual_medical_care(date, place, medical_care_type, person) instantiate the structured stage, while dense retrievers and extractive readers handle the unstructured stage (Tan et al., 2023).

LifeAgentBench generalizes this design into a tool-augmented agent. LifeAgent implements an iterative thought-action-observation loop $Q$2 whose state stores the query, parsed intent, partial plan, evidence cache with SQL traces, and intermediate statistics. The agent decomposes a question into a retrieval agenda $Q$3, calls domain-specific retrieval and cohort aggregation tools, and offloads arithmetic to deterministic operators such as windowed means, threshold counts, cohort averages, and trend fitting. Because the answers are backed by SQL traces, the resulting pipeline is auditable end to end (Tian et al., 20 Jan 2026).

Real-image lifelog retrieval has produced a parallel retrieval-to-reasoning stack. “LifeIR at the NTCIR-18 Lifelog-6 Task” uses a five-stage pipeline: blurred-image filtering by edge weight summation, CLIP-based retrieval, LLM-based query rewriting, event-based candidate expansion, and Qwen2-VL reranking. Event embeddings are computed by averaging image features within temporally and visually coherent segments, and multi-round expansion refines the query representation with top retrieved image features before MLLM posterior filtering. Although developed for LSAT image retrieval, this architecture maps directly onto the evidence retrieval layer of OpenLifelogQA (Chen et al., 27 May 2025).

OpenLifelogQA also intersects with modality-specific models. LLaSA integrates LIMU-BERT with Vicuna-1.5-7B via an MLP projector and LLaVA-style multimodal conditioning, enabling QA over accelerometer and gyroscope streams. Its associated datasets, SensorCaps and OpenSQA, operationalize sensor-context instruction following for questions about activities, phases of motion, terrain, fatigue, movement quality, and sensor reliability (2406.14498). At the opposite end of the computational spectrum, the smartwatch-oriented interface of “A Natural Language Query Interface for Searching Personal Information on Smartwatches” parses queries into $Q$4, emphasizing low-latency on-device intent extraction for quantified-self data (Rawassizadeh et al., 2016). Indoor retrieval work further extends lifelog QA toward spatio-temporal grounding with room-level localization, event segmentation, and cross-modal evidence for questions such as “Where did I last see my keys?” and “When did I take my medication indoors?” (Datta, 2019).

5. Empirical findings and performance characteristics

The strongest consistent result in TimelineQA is that extractive QA substantially outperforms generative RAG for atomic queries when answers are spans in episode text. With fine-tuned retrieval, ExtractiveQA reaches EM 82.6 and F1 93.8, whereas RAG-Token reaches EM 40.3 and F1 57.5. Even with oracle retrieval, extractive remains stronger: EM 83.3 and F1 94.8 versus EM 73.7 and F1 84.4. For multi-hop aggregates, the best reported setting is TAPEX-large FT + Oracle at 59.0% denotation accuracy, with performance dropping from 85.1% for evidence sets of 0–10 records to 4.3% for evidence sets larger than 1,000 records (Tan et al., 2023).

LifeAgentBench shows that long-horizon aggregation remains a major bottleneck for general LLMs. Under Context Prompting, GPT-4o achieves 57.02% Accuracy, while the best open-source model, Qwen-2.5-7B, reaches 40.45%. Under Database-augmented Prompting, Gemini 2.5 Lite attains 39.04% Accuracy and GPT-4o 34.71%. Yet when correct evidence is provided, final reasoning is often reliable: the average Execution Accuracy across models is 25.94%, seven models have $Q$5, and GPT-4o reaches 95.65%. The proposed LifeAgent baseline raises average accuracy on the hardest subsets from 7.74% in CP and 9.43% in DP to 40.16%, with multi-item answers improving from 0.36% and 3.04% to 32.31% (Tian et al., 20 Jan 2026).

OpenLifelogQA’s baseline experiments emphasize the difficulty of open-ended aggregation even when the correct context is already provided. LLaVA-NeXT-Interleave 7B obtains BERTScore 0.897, ROUGE-L 0.2587, and LLM Score 3.9665 on the test split. The 0.5B variant scores slightly higher on BERTScore and ROUGE-L—0.9063 and 0.2613—but lower on LLM Score at 3.3873, because lexical or semantic overlap can remain high even when numeric answers are wrong. The 7B per-type breakdown shows Atomic LLM Score 4.2574, Temporal 4.2115, and Aggregation 3.3362, making aggregation the most difficult category (Tran et al., 5 Aug 2025).

Continuous conversational memory exhibits a related pattern. LifeDialBench reports that simple raw-text retrieval can outperform more elaborate memory compression systems. On LifeMem with qwen-plus in the online setting, RAG reaches 75.16 overall on MCQ and 46.18 on open-ended QA, A-Mem reaches 74.51 and 49.54, MemOS reaches 70.45 and 39.88, and Mem0 reaches 63.46 and 20.20. The paper attributes the degradation to lossy compression and shows that retrieval breadth also matters: with Qwen3-8B on LifeMem, increasing top-$Q$6 from 10 to 40 improves both A-Mem and MemOS (Zheng et al., 13 Apr 2026).

Sensor-aware QA exhibits stronger performance when the modality is narrower and supervision is tailored to the signal. LLaSA reports F1 scores of 0.84 on HHAR, 0.83 on MotionSense, 0.81 on Shoaib, 0.72 on UCI, and 0.65 on the unseen SHL dataset in close-ended activity recognition from IMU context, while open-ended evaluation on PAMAP2 indicates richer activity-specific interpretations than a text-only Vicuna-13b baseline (2406.14498). A plausible implication is that OpenLifelogQA will benefit from modality-specialized encoders when extending beyond image-plus-metadata settings.

6. Limitations, controversies, and future directions

Several recurrent limitations constrain current OpenLifelogQA systems. Synthetic lifelogs can provide scale and exact supervision, but they lack the linguistic diversity, multi-modal noise, and social variability of real-world records. Real lifelog datasets, by contrast, often remain narrow in subject coverage: OpenLifelogQA is based on a single anonymous lifelogger, and LifeAgentBench covers about one month per user. LifeDialBench avoids raw conversational audio in EgoMem because direct ASR was impractical under background noise, overlapping speakers, sparse meaningful dialogue density, and variable mic distance (Tan et al., 2023, Tran et al., 5 Aug 2025, Tian et al., 20 Jan 2026, Zheng et al., 13 Apr 2026).

A second limitation concerns the boundary between retrieval and reasoning. Multiple benchmarks demonstrate that oracle or high-quality evidence selection is often the decisive factor. TimelineQA’s table-based readers perform best when the ground-truth set of episodes is available; LifeAgentBench shows high $Q$7 once correct evidence is supplied; OpenLifelogQA’s published baseline assumes that the relevant event descriptions and representative images are already given as context. A common misconception is that strong answer quality under gold-context evaluation implies a solved end-to-end system; the benchmark results do not support that interpretation (Tan et al., 2023, Tian et al., 20 Jan 2026, Tran et al., 5 Aug 2025).

A third controversy concerns memory abstraction. LifeDialBench explicitly reports a counterintuitive result: sophisticated memory systems fail to outperform a simple RAG-based baseline. The documented explanation is that over-designed structures and lossy compression damage high-fidelity context preservation, especially for detail retrieval and temporal grounding. This does not imply that structure is unhelpful in general; rather, the evidence suggests that structure must not discard the raw evidence needed for retrieval and answer synthesis (Zheng et al., 13 Apr 2026).

Privacy and governance remain central. OpenLifelogQA removes faces and readable text and follows GDPR-oriented precautions; LifeDialBench emphasizes consent mechanisms, indicators, opt-outs, and on-device processing; earlier lifelogging work identifies data ownership, public disclosure, and cloud-based storage as major concerns. Future directions repeatedly named across the literature include richer multi-modal lifelogs with images and videos, stronger temporal and geographic knowledge integration, privacy-preserving retrieval and on-device computation, longer-horizon evaluation, richer event diversity, and counterfactual lifelogs for bias analysis (Tran et al., 5 Aug 2025, Zheng et al., 13 Apr 2026, Yen et al., 2020).

In aggregate, OpenLifelogQA has emerged as a technically specific research program: represent personal experience as structured-yet-open episodic memory, retrieve the right evidence under temporal and contextual constraints, and produce faithful natural-language answers over text, images, sensors, and conversation. Existing results establish that extractive methods remain strong for atomic spans, table-pretrained or tool-augmented systems are preferable for aggregation, modality-specialized encoders can improve signal interpretation, and retrieval quality remains the principal bottleneck for realistic deployment (Tan et al., 2023, Tian et al., 20 Jan 2026, Tran et al., 5 Aug 2025).