- The paper introduces HippoCamp, a benchmark built from 42.4 GB of multimodal personal files, 581 evidence-grounded questions, and 46,100 annotated agent trajectories across three realistic user profiles.
- The evaluation finds that ChatGPT Agent Mode achieves the strongest results—48.3% profiling accuracy and 62.8% factual-retention accuracy—while profiling is substantially harder than factual retrieval, especially in media-rich and socially complex environments.
- The paper identifies post-retrieval processing as the main bottleneck and recommends explicit entity modeling, minimal evidence sets, filesystem-aware search, and final verification to reduce grounding, attribution, and hallucination errors.
HippoCamp is a benchmark for evaluating multimodal agents on realistic, device-scale personal file systems, addressing a gap left by existing agentic evaluations that target web interaction, tool use, or document understanding in generic, fully observable settings. The benchmark instantiates three archetypal personal computing environments from real user data—42.4 GB across more than 2K files spanning text, documents, images, video, and audio—and pairs them with 581 evidence-grounded QA pairs and 46.1K densely annotated structured trajectories. Evaluation of state-of-the-art MLLMs and agentic methods reveals that even the strongest commercial system reaches only 48.3% accuracy on user profiling, with failure analysis localizing the dominant bottleneck to post-retrieval stages: multimodal perception, evidence grounding, entity binding, and verification.
Motivation and positioning
Existing benchmarks evaluate agents against public data or task-bounded retrieval spaces. Retrieval-centric suites such as HotpotQA, MultiModalQA, M3DocRAG, and BrowseComp assume web or document corpora; agentic environments such as WebShop, GAIA, and VisualWebArena probe goal-driven action but not long-lived personal context; lifelog benchmarks such as LoCoMo are text-only and small (300 questions), while EgoLifeQA and Ego-R1-Bench restrict personalized evaluation to egocentric video/audio streams. HippoCamp is distinguished by combining all five modalities, real file-system hierarchy, temporal metadata, and user-level inference targets within a single benchmark. Its central premise is that personalized digital assistance requires integrating search over heterogeneous files, modality-specific perception, and longitudinal reasoning—capabilities that current pipelines treat separately.
Benchmark construction
The corpus derives from interviews with 100+ participants screened under reproducible criteria: at least 500 user files covering four of five modalities and ten extensions, at least three months of longitudinal activity, and evidence completeness sufficient for auditable cross-file corroboration of key personal facts. Selected contributors are aggregated into three archetypal profiles—Bei Weiwei (student/content creator), Adam Turner (legal executive), and Victoria Anne Clarke (senior financial analyst)—under distribution-preserving partitioning followed by automated coherence checks on timestamps, entities, and project threads, minimal-edit repair, and human validation. Privacy processing removes system-generated artifacts, pseudonymizes identifiers, sanitizes embedded metadata, and requires participant sign-off; limited curated public-domain material (FinanceBench- and LegalBench-RAG-derived) supplements underrepresented professional document forms after rewriting to match the fictional identity system.
Annotation follows a hybrid pipeline: domain-aware annotators author user-driven questions grounded in their own files, while proprietary LLMs generate candidates conditioned only on restricted contextual metadata (paths, timestamps, directory structure). All candidates pass intent- and pattern-level de-duplication, coverage balancing, and human consolidation into structured trajectory records containing the question, normalized answer, minimal supporting file set, localized evidence objects with atomic locators (page indices, table cells, timestamp ranges), stepwise rationale traces linked by evidence IDs, and capability labels decomposing required behavior into search, perception, and reasoning. A minimalist gold-trajectory principle records the smallest sufficient support set rather than exhaustive solution paths. Modality-normalized atomic units (AUs) provide a diagnostic interface for distinguishing file-level hits from fine-grained grounding.
Tasks
Two task families are defined. Factual retention requires retrieving and reasoning over verifiable file-grounded facts—for example, extracting visa photo constraints from a policy PDF and verifying compliance against candidate images, or aligning earnings-call transcripts across quarters to attribute revenue-growth drivers. Profiling sits atop the annotation hierarchy and requires synthesizing weak, distributed signals across time into coherent user-level inferences, decomposed into five subtasks: preferences, behavioral patterns, scheduling information, retrospective reflections, and workflows. Profiling answers are rarely anchored to a single decisive statement; they demand event-to-trait abstraction, referent disambiguation among co-occurring entities, and constraint-aware planning consistent with historical decision policies.
Complexity is characterized along three axes plus a scalar difficulty score computed via log–quantile normalization of eight factors with interaction coupling terms. Evidence breadth is heavy-tailed (354 single-file queries overall, but non-trivial mass beyond 20 evidence files per profile); reasoning depth peaks at 5–8 steps with a tail beyond 14. The difficulty distribution has mean 57.5 with 26.7% of questions scoring ≥70; profiling is systematically harder (mean 89.1, 93.3% ≥70) than factual retention (mean 53.8). Question-level judge scores decline monotonically with binned difficulty, supporting the score's validity as a hardness measure.
Experimental results
Evaluation covers RAG baselines (Standard RAG, Self-RAG), search agents (ReAct with Gemini-2.5-flash and Qwen3-30B-A3B, Search-R1), and autonomous agents (Dockerized terminal agents with multiple backends, plus ChatGPT Agent Mode in its hosted configuration), under profile-isolated access with LLM-as-a-judge answer scoring (GPT-4o reference) and file-level retrieval metrics against annotated minimal evidence sets.
| Method |
Profiling F1 / Acc |
Factual Retention F1 / Acc |
| Standard RAG |
18.4 / 26.7 |
30.0 / 30.2 |
| Self-RAG |
15.2 / 10.0 |
31.9 / 27.5 |
| ReAct (Qwen3-30B-A3B) |
11.8 / 13.5 |
43.1 / 28.5 |
| ReAct (Gemini-2.5-flash) |
18.5 / 20.0 |
26.5 / 38.7 |
| Search-R1 |
10.8 / 5.0 |
41.0 / 25.3 |
| Terminal Agent (GPT-5.2) |
11.1 / 30.0 |
24.6 / 48.2 |
| ChatGPT Agent Mode |
21.0 / 48.3 |
35.3 / 62.8 |
Several findings stand out. First, retrieval quality and answer quality decouple in both directions: ReAct (Qwen3) achieves 43.1% factual-retention F1 but only 28.5% accuracy (retrieval without synthesis), while Terminal Agent (GPT-5.2) shows the inverse pattern (24.6% F1 vs. 48.2% Acc), suggesting reliance on parametric knowledge rather than grounded evidence—a caveat the authors flag explicitly. Second, capability decomposition shows perception is the most universal bottleneck: profiling perception accuracy ranges from 13.2% (Self-RAG) to 28.5% (ChatGPT Agent Mode), roughly half the best system's search accuracy (56.5%). Third, profile difficulty tracks structural clarity rather than domain content: ChatGPT Agent Mode reaches 90.3% factual accuracy on Adam's document-centric legal environment but only 31.2% on Bei's socially entangled, media-rich college environment, where entity ambiguity is amplified. Fourth, an efficiency–capability trade-off is pronounced: ChatGPT Agent Mode averages roughly 786 seconds per query versus about 5.5 seconds for Standard RAG, and its runs frequently require re-execution due to incomplete outputs, so reported figures should be read as point estimates under execution instability—the paper concedes this variance openly, along with the omission of Claude Sonnet 4.5 from vacuum settings due to unreliable long-document handling.
Failure analysis
The paper identifies a recurring five-stage failure pipeline: retrieval mismatch, grounding avoidance, hard evidence hallucination, entity misattribution, and verification deficit. Standard RAG retrieves topically similar but contextually irrelevant files (e.g., financial disclosures matching "health" keywords); Self-RAG's self-reflection cannot recover from off-target initial retrieval, collapsing to 0.0% profiling accuracy on Victoria. Search-R1 exhibits grounding avoidance—34.2% profiling F1 but 5.0% accuracy, including 0.0% on two profiles—defaulting to generic parametric responses despite available evidence. Terminal agents fabricate file paths and metadata, producing internally consistent but invented evidence chains. Even ChatGPT Agent Mode commits entity misattribution errors, binding genuine health records to the user's pet rather than the account holder. No evaluated method includes an explicit final-stage verification loop re-binding answers to a minimal coherent evidence set. The one consistent success pattern is iterative discovery: ChatGPT Agent Mode's advantage stems not from stronger one-shot retrieval but from revisiting hypotheses and correcting errors across pipeline stages.
Design implications
From these failures the paper derives concrete architectural principles: treating file-system hierarchies, temporal regularities, and attachment relations as inductive biases for search rather than flattening corpora into vector indexes; forming a minimal sufficient evidence set before answering instead of summarizing broad candidate pools; maintaining explicit profile-local entity models to resolve first-person references among co-occurring entities; and adding verification as an explicit terminal stage that checks answer traceability to grounded evidence. These recommendations follow directly from the observed decoupling between retrieval metrics and answer correctness, though the paper does not empirically validate any proposed architecture—it leaves open whether such designs close the measured gap.
Limitations and open questions
Several limitations bear on interpretation. The benchmark comprises only three profiles aggregated from screened contributors, each an archetype rather than a literal user; generalization to other profile distributions is untested. The difficulty score is heuristic, with weights and interaction terms chosen by design judgment rather than fitted to human performance data. LLM-as-a-judge evaluation introduces residual judge-side variance, mitigated but not eliminated by stratified human audit. Hosted commercial agent modes cannot be standardized against Docker-based agents at the tooling level, so cross-regime comparisons assess matched information access rather than identical affordances. Non-determinism in hosted APIs means results are point estimates. Open questions include whether verification loops and entity modeling can be implemented without prohibitive latency, and how AU-level supervision might support training under privacy constraints.
Conclusion
HippoCamp provides a device-scale, densely annotated testbed demonstrating that current agents fail at personalized multimodal file reasoning primarily after retrieval—in perception, grounding, entity binding, and verification—with the best commercial system reaching only 48.3% profiling accuracy. By pairing QA tasks with stepwise trajectories and capability labels, it enables failure localization that single-metric benchmarks obscure, and establishes a foundation for diagnosing and developing next-generation personal file-system agents.
