Evaluating the Utility of Personal Health Records in Personalized Health AI

Abstract: Patient-managed Personal Health Records (PHRs) promises to empower patients to better understand their health; but information in the record is complex, potentially hindering insights. In this study, we assess the potential of LLMs (LLMs, Gemini 3.0 Flash) to provide helpful answers to user health queries, when provided clinical data from PHRs as context. A total of 2,257 user queries were drawn from 3 different distributions to represent patient questions: shorter web search queries, longer questions derived from templates of chatbot conversations, and questions patients asked to their healthcare team (patient calls). Queries were matched with de-identified PHRs (from a pool of 1,945). Gemini responses were generated (1) without PHR context; (2) with a basic summary of demographics, conditions, and medications; (3) with full, extensive clinical notes. For evaluation, we leveraged an existing rating framework (SHARP), and developed a new framework for specific error modes when interpreting PHRs. Evaluation was performed using autoraters for the full set, and with clinician ratings for a subset (n=95), with both sets of raters knowing the full PHR context. We see significant improvements in the helpfulness of answers to all question types with PHR data (p < 0.001, paired t-test). We also observe potential gains in safety, accuracy, relevance and personalization of answers. Our PHR evaluation framework further identifies gaps in LLM understanding of particular aspects of complex PHRs, such as temporal disorientation, and rare but meaningful confabulations. These results suggest potential for PHR data to help people with a wide range of user needs; and provide a framework for monitoring for gaps in LLM answers based on PHR context. This study motivates further work to assess and realize potential benefits to users from understanding their health records.

• The paper demonstrates that integrating PHR data significantly improves LLM response helpfulness and personalization, as measured by the SHARP framework.
• The study employs rigorous empirical evaluation with 2,255 queries and 1,945 de-identified PHRs using both clinician and autorater assessments.
• The paper identifies challenges such as temporal misalignment and data grounding errors, and suggests iterative self-critique to reduce these failure modes.

Evaluating the Utility of Personal Health Records in Personalized Health AI: Expert Analysis

Study Overview and Motivation

The proliferation of LLMs for consumer-facing health applications has raised critical questions about their capacity to provide personalized, actionable, and clinically safe responses—especially when augmented with contextual data such as Personal Health Records (PHRs). "Evaluating the Utility of Personal Health Records in Personalized Health AI" (2605.18937) presents an extensive empirical evaluation of LLMs (Gemini 3.0 Flash) operating over consumer health queries, with direct, structured access to PHR data at varying levels of granularity. The study addresses fundamental challenges in realizing truly personalized AI health advice, exploring how PHR context impacts performance across multiple axes, including helpfulness, safety, and fidelity to clinical nuance.

The methodological architecture involves 2,255 user queries, sampled from real-world sources (web search, chatbot dialogs, patient-provider calls), systematically paired to 1,945 de-identified PHRs. The evaluation leverages both human clinician raters and automated "autoraters," employing the SHARP (Safety, Helpfulness, Accuracy, Relevance, Personalization) framework along with a bespoke suite of PHR contextualization axes. This design allows for rigorous disentanglement of the marginal utility and residual risks of layered PHR context—from "no PHR" (fully generic), to structured "basic PHR," to full longitudinal clinical notes.

Figure 1: Study schema: query sampling, PHR pairing, response generation with varied PHR context, and dual-channel evaluation (human and automated) across aggregate and PHR-specialized axes.

Quantitative Findings: Effect of PHR Context on LLM Response Utility

The presence of PHR context drives a statistically significant boost in answer helpfulness, across both auto-rating and clinical human annotation. On a normalized scale (−1 to 1), clinician-rated overall helpfulness improved from a mean of 0.13 (no PHR) to 0.53 (basic PHR) and 0.49 (full PHR). Autorater results indicate an even more pronounced gain, from 0.18 (no PHR) to 0.85 (basic PHR) and 0.95 (full PHR) (p < 0.001 for all pairwise comparisons). The effect holds over all query sources—web, chatbot, direct patient calls. While improvements from basic to full PHR are subtler in clinician scoring, autorating detects significant incremental benefit.

This increase in utility primarily manifests on the axes of helpfulness, actionability, and motivation. The incorporation of structured or unstructured PHR context enables the model to reference patient-specific chronology, comorbidities, medications, and laboratory data, resulting in more clinically salient reasoning, individualized risk stratification, and precisely tailored guidance.

Figure 2: Improvement in SHARP framework measures (Autorater and Clinician scores) across exposure to no PHR, basic PHR, and full PHR context.

Further, the SHARP framework metrics reveal that specificity and motivation scores also benefit from PHR availability. Actionability increases from a clinician-rated 0.61 (no PHR) to 0.85 (basic PHR) and 0.84 (full PHR). Motivation sees a similar gain. While gains on safety and accuracy axes are less statistically dramatic (owing to high baseline performance), autorater ratings suggest additional but more modest improvements.

Qualitative and Error Analysis: Nuanced Gains and Remaining Failure Modes

In-depth rater feedback indicates that LLMs operating without PHR context mostly provide generic, albeit safe, advice, failing to flag red-flag symptoms, contraindications, or drug-disease interactions highly relevant to the actual user. When supplied with even basic PHR data, responses more accurately reference historical conditions and current medications, increasing safety netting and personalization; however, certain failure modes—inadequate differential diagnosis, focus on irrelevant history, and episodic tunnel vision—persist.

Analyses using the PHR-specific rubric (16 axes across Fidelity, Robustness, Utility, Safety, and Equity) expose several residual error modes even for full PHR context. The most critical and persistent are:

1. Chronological and temporal blindness: Inability to appropriately track the patient timeline leads to misattribution of outdated medications/labs, logic failures in pregnancy/disease progress, and erroneous recommendations based on resolved events. Time-related errors constitute up to 6% of full PHR answers.
2. Groundedness failures: Hallucination and confabulation, including fabricated labs and conditions, or improper mapping of real data points. This occurs both in responses to insufficient (basic) and in complex (full) PHRs, with overall rates of 1-3%.
3. Failure to identify internal PHR data issues: Absence of robustness regarding missing/incomplete data, and lack of detection of internal contradictions or biologically implausible values.
4. Context omission and lack of data synthesis: Defaulting to generic/boilerplate templates and failing to integrate the relevant clinical context, such as ignoring sociodemographic barriers or critical comorbidities.

The study demonstrates that a simple "self-critique" loop—autorater flagging of problematic outputs followed by agent revision—can reduce major failure classes (time-blindness, grounding errors) by over 90%, suggesting that iterative post-processing or further fine-tuning can mitigate many (though not necessarily all) high-risk errors.

Figure 3: Distribution of PHR-specific rubric losses (by axis), showing axis-specific error targets and reduction after automated self-critique cycles.

Implications and Limitations

The empirical findings strongly support the claim that integration of PHR context into LLM-driven health conversations yields meaningfully better and more personalized consumer health advice, not only for complex, context-rich queries but also extending to generic search interactions. These improvements are most salient for nuanced clinical decision-making, such as risk stratification, medication interaction, and actionable lifestyle/disease management guidance. This directly addresses a noted gap in current AI health applications, where lack of personalized context is a key driver of suboptimal (or potentially hazardous) recommendations.

However, the study also exposes persistent risks inherent to free-text PHR ingestion—temporal misalignment, confabulation, and low robustness to missing/contradictory input. Automated or human-in-the-loop strategies for error remediation will be essential if LLMs are to serve as trustworthy, regulatory-compliant decision support.

There are notable constraints regarding generalizability: the PHR dataset is US-centric with clinical and demographic biases; not all query types or modalities of context provision were explored; the fine-grained impact of different PHR structuring on model performance remains an open question.

Theoretical and Practical Trajectories

By formalizing and applying a structured, multifactorial, and failure-mode-aware evaluation framework, the paper establishes a research baseline for both model optimization and benchmarking in longitudinal, context-rich healthcare QA. The design—pairing synthetic (basic) and real-world (full) PHR conditions with diverse real queries—enables systematic mapping of LLM strengths and vulnerabilities in a practical deployment scenario.

Future investigations will need to address longitudinal health outcomes, user trust, and equity impacts; evaluate the efficacy of advanced retrieval and summarization architectures for large-scale PHRs; and extend beyond English and US health systems. There is also a clear need for standardization of PHR-aware benchmarking, robust adversarial stress-testing (for confabulation, bias, and fairness), and principled risk-management strategies, especially as consumer integration of PHRs accelerates and regulatory standards tighten.

Conclusion

This work provides robust quantitative and qualitative evidence for both the value and the hazards of integrating personal health records into LLM-driven health Q&A, with PHR context facilitating substantial increases in helpfulness, personalization, and actionability. At the same time, it exposes characteristic and sometimes subtle error modes in temporal reasoning, data grounding, and context sensitivity—risks which, while amenable to post-generation correction, require ongoing methodological vigilance. The analytical paradigm introduced yields a practical template for iterative evaluation and model improvement in the development of truly personalized, safe, and reliable health AI systems.