- The paper introduces PersonalHomeBench, a benchmark that evaluates agentic AI in personalized smart home environments through realistic, multimodal data and persona-driven tasks.
- It demonstrates that models achieving >85% accuracy in sole reasoning experience significant performance drops in tool-mediated contexts, especially in plan generation and counterfactual tasks.
- The study highlights critical challenges in tool coordination, long-horizon reasoning, and preference alignment, urging further research for robust, user-centered smart home assistants.
PersonalHomeBench: A Benchmark for Evaluating Agentic AI in Personalized Smart Home Environments
Motivation and Context
Advancements in foundational LLMs and agentic AI systems have increased interest in deploying autonomous assistants in real-world domains, notably smart homes. Despite progress on task-driven benchmarks, there remains a critical gap in evaluating agentic systems under the demands of personalization, long-horizon contextualization, and multimodal inputs in evolving, realistic home environments. Existing smart home or agentic benchmarks are limited in coverage, only partially address personalization, and typically ignore genuine heterogeneity of environments, long-term memory, proactive behavior, and partial observability.
PersonalHomeBench directly addresses these limitations by introducing a benchmark framework for rigorous, multi-faceted evaluation of personalized agentic assistants in complex smart home scenarios. The framework systematically integrates realistic synthetic and real-home data, context-rich and evolving environments, persona-driven user modeling, and a comprehensive set of agentic tasks beyond simple device control or question answering.
Benchmark Design and Task Taxonomy
PersonalHomeBench’s generation pipeline reflects realistic household evolution, beginning with a minimal household scaffold augmented by Nemotron Personas to instantiate diverse and demographically-grounded multi-person households. Each household is progressively annotated with appliance configurations, behavioral memories, contextual events, and, for the multimodal setting, video and audio groundings. All data are validated for consistency and compliance with a detailed schema.
Tasks are divided into reactive and proactive regimes:
- Reactive tasks: Information Grounding (IG), Counterfactual Reasoning (CF), and Multiple Choice (MC). IG focuses on factual extraction under variable context, CF requires robustness to hypothetical perturbations, and MC assesses discriminative reasoning with plausible yet nuanced distractors.
- Proactive tasks: Feature Recommendation (FR) and Plan Generation (PG). FR requires agents to rank the contextual relevance of appliance features, leveraging real-world API inventories; PG mandates the synthesis of executable, multi-step, preference/alignment-aware plans under non-trivial environmental and household states.
Both text-only and multimodal instances are supported, enabling fine-grained ablation studies across input modality, observability, and personalization.
PersonalHomeTools is provided as a domain-specific API suite, mediating structured access to static metadata, event/memory retrieval, environmental perception, and appliance control (covering 40+ device types). Core functional modules—Transcriber, Memory Retriever, Video Understanding Agent—allow for tightly integrated perception, context tracking, and tool-used reasoning. Fine control over tool use is crucial: tasks are not fully specified by static context, and effective tool invocation and argumentation are essential for correct completion, especially under partial observability.
Experimental Protocol
The evaluation spans eleven models covering several families (GPT, Gemini, Qwen3, Nemotron) and model sizes (from small 4B open models suitable for edge deployment to proprietary 100B-scale models). Each is tested both in “Sole Reasoning” (entire context upfront, no tool-use required) and “With Tools” (contextual information only accessible via API calls). Gemini 2.5 Pro is employed both as a data generator and as an LLM-based Role Playing Judge (RPJ) for plan evaluation, including subjective satisfaction and relative plan preference.
Metrics include accuracy (reactive tasks), MAP@1 and RRS (FR), program validity and win-rate by RPJ (PG), and ablation per difficulty and reasoning mode. Counterfactual tasks are carefully partitioned by whether the answer does or does not change. RPJ supports both absolute (scalar, persona-resolved) and relative (joint plan ranking) scoring.
Key Empirical Findings
Across all evaluated models, performance deteriorates sharply moving from sole reasoning to tool-mediated contexts. While many models achieve >85% reactive accuracy when given the full context, tool-mediated accuracy can drop by tens of percentage points—even in large proprietary models. For proactive tasks, valid structured plan generation essentially vanishes for most models when tool-use is required, regardless of underlying model scale.
Performance is especially brittle in:
- Counterfactual reasoning (CF), where most models fail to maintain answer consistency in the presence of hypothetical perturbations, with pronounced degradation in the answer-unchanged regime.
- Plan Generation validity, where most model outputs fail to produce syntactically correct, executable structured plans when interacting with tool APIs, despite logic-level sensibility.
Notably, explicit “thinking” or reasoning-promoted versions do not close this gap. Increasing model size and reasoning depth leads to only marginal improvement, and in certain agentic settings, may worsen tool-use error rates.
The Role of Personalization and Modality
Ablation of personalization demonstrates that reasoning over personalized attributes introduces substantial complexity, with up to a 22-point accuracy drop in sole-reasoning. However, PersonalHomeTools significantly mitigates this deficit by supporting targeted retrieval of persona-grounded details; in tool-mediated settings, performance is nearly unaffected by personalization, which validates the need for explicit, API-accessible user context in realistic agentic environments. Multimodal (video-grounded) context improves reactive task accuracy under sole reasoning, but does not consistently benefit plan validity or proactive satisfaction.
Reflection and Subjective Assessment
The use of the RPJ as a self-reflection loop enables iterative plan improvement, as shown by increased absolute subjective scores and plan validity for the Gemini 2.5 Pro agent. Human validation of RPJ scoring shows strong alignment (over 65% agreement and no systematic disagreement), justifying its use for scalable evaluation of subjective satisfaction and plan alignment.
Implications and Outstanding Challenges
These results expose three primary unsolved problems for agentic AI in personalized, dynamic home environments:
- Tool-use brittleness: Models fail to reliably compose correct tool call sequences, properly condition calls on context, and manage API argumentation for real-world device control, especially in presence of complex grounding and partial observability.
- Long-horizon and counterfactual reasoning limitations: Multi-hop retrieval, robust temporal state tracking, and the maintenance of causal coherence in hypothetical settings are fundamentally limiting, particularly under constrained tool-mediated access.
- Preference and routine alignment: Even when plans are logically plausible, generating coherent, preference-aligned, and validly executable multi-step routines remains a significant challenge.
These weaknesses are not significantly ameliorated by current model scaling, reasoning augmentation, or the introduction of multimodal context. Effective agentic reasoning in complex personalized settings requires advances in tool-use competence, adaptive long-term memory and context modeling, robust planning architectures, and tighter integration of explicit user profiling.
PersonalHomeBench establishes a unified, extensible platform for diagnosing the multifaceted failures of current agentic AI in personalized smart environments. Its utility is both practical—by exposing specific tool usage and reasoning bottlenecks relevant for real-world deployment—and theoretical, by providing a controlled measurement environment to assess architectural advances, new memory models, preference learning algorithms, and advanced reasoning/tracking protocols. The benchmark’s support for compact, edge-suitable models and detailed modality control also enables research into secure, privacy-preserving local deployment, mitigating some adoption risks.
Subsequent work may extend PersonalHomeBench with richer embodied interaction, hybrid physical/digital task settings, more advanced video-grounded tasks, expanded multimodal plan judging, and human-in-the-loop or RLHF-trained evaluation/plan generation. The benchmark highlights the need for future research into holistic agent architectures that effectively unify structured tool-use, dynamic context management, and deep preference modeling for robust personalized autonomy.
Conclusion
PersonalHomeBench and PersonalHomeTools set a new standard for evaluating agentic AI systems in the personalized smart home domain. The benchmark demonstrates that high scores in idealized sole-reasoning environments are not indicative of agentic readiness in realistic, tool-mediated, and context-evolving settings. The empirical results reveal persistent, foundational limitations in tool coordination, memory-perception integration, counterfactual robustness, and preference alignment, all of which must be addressed by future research if robust, user-centered smart home assistants are to become practical and trustworthy.
Reference: "PersonalHomeBench: Evaluating Agents in Personalized Smart Homes" (2604.16813)
