MCP-Persona: Benchmarking LLM Agents on Real-World Personal Applications via Environment Simulation

Abstract: The Model Context Protocol (MCP) has emerged as a transformative standard for connecting LLMs with external data sources and tools, and has been rapidly adopted across personal applications and development platforms. However, existing benchmarks predominantly focus on generic information-seeking tools and fail to capture the practical challenges posed by personal social applications, where tools interact with individual accounts or local databases. To bridge this critical gap, we introduce MCP-Persona, the first benchmark specifically designed for evaluating agent performance on real-world, personalized MCP tools. MCP-Persona encompasses a diverse set of widely-used applications, ranging from social media platforms like Reddit and Xiaohongshu (Rednote) to enterprise collaboration suites such as Lark (Feishu) and Slack. Our extensive experiments on various state-of-the-art (SOTA) agents demonstrate their significant struggles with personalized tool use, thereby highlighting the benchmark's crucial role in identifying and addressing these limitations. MCP-Persona is publicly available at https://github.com/wwh0411/MCP-Persona}{https://github.com/wwh0411/MCP-Persona.

• The paper introduces a simulation-based benchmark that replicates realistic, personalized environments to evaluate LLM agents' tool orchestration.
• It leverages a three-component simulator—Tool-Traverse, context trees, and Persona-Gen—to emulate authentic user state and complex task flows.
• Experimental results show sub-50% accuracy even for SOTA models, underscoring the need for improved context grounding and multi-step reasoning.

MCP-Persona: Environment-Simulated Benchmarking of LLM Agents on Real-World Personalized Applications

Motivation and Positioning

MCP-Persona addresses the core deficiency in current tool-augmented LLM agent benchmarks: the lack of dedicated evaluation in environments reflecting real-world, personalized application usage. Existing protocols such as Model Context Protocol (MCP) have enabled LLMs to leverage external tools, but previous benchmarks focus primarily on generic, stateless, and information-seeking APIs, circumventing the complexity of enterprise collaboration, social media, and other applications inherently coupled with private user state and personalized workflow. This critical gap impedes research on robust, privacy-compliant agents capable of high-fidelity tool orchestration in actual user scenarios.

Architecture Overview and Simulation Paradigm

The MCP-Persona framework consists of an environment simulator constructed from three components: tool simulation, context modeling, and authentic, human-relevant task generation. The overall architecture centers on the interaction among MCP tools, context trees representing user and application state, and tasks expressing realistic user goals.

Figure 1: System overview of MCP-Persona, establishing an environment simulator based on three core components: tools, contexts, and tasks connected via controlled interfaces.

For tool simulation, MCP-Persona introduces Tool-Traverse, a technique that captures operational semantics of real MCP servers by systematically traversing authentic function calls. This method utilizes seed calls for both successful and adversarial (failure-inducing) scenarios to reconstruct the complete behavioral manifold, including precise error-handling, as encountered in production environments. These operational traces are translated into executable Python modules with LLM-based code synthesis, ensuring that simulation aligns with the heterogeneous, stateful, and error-prone reality of real applications.

Personalized environment state is manifested as a Context-Tree: a hierarchical structure representing all application- and user-specific entities and their relations, constructed from both tool metadata and empirical observation of tool call behaviors. Leaf and intermediate nodes are populated with realistic values, leveraging authentic data where feasible (e.g., content from public Xiaohongshu posts, sanitized for privacy), probabilistically synthesized content, and structured relations reflective of actual application semantics.

Task synthesis in MCP-Persona leverages Persona-Gen, a pipeline coupling chain-of-tool-invocation sampling with context injection and naturalistic instruction fuzzification. Abstract, dependency-coherent tool chains are augmented with realistic user context and then "fuzzed"—that is, essential information is intentionally omitted in surface instructions, imitating the ambiguity and implicitness characteristic of real user directives. Human verification ensures the validity and solvability of all resulting tasks.

Figure 2: Illustration of tool-context interaction; left: execution trajectory of tool calls in Lark-MCP; right: evolution of the tree-structured context as stateful tools modify the environment.

Dataset Characteristics and Task Analysis

MCP-Persona encompasses 173 human-verified tasks distributed over 24 MCP servers (12 personalized), including domains such as workplace collaboration (Lark, Slack), social media (Instagram, Reddit), and personal content management (Notion, Obsidian). Tasks are further diversified into single-server and cross-server (hybrid) compositions, requiring the agent to plan and coordinate tool calls across multiple accounts and disparate data silos.

Task distributions reflect realistic workflow heterogeneity: while some require only simple CRUD operations or entity resolution, others necessitate multi-step, cross-context, and stateful coordination with implicit variable resolution and logical dependency chains. Average token statistics per task further underscore the increased complexity and context volume compared to prior benchmarks.

Figure 3: Distribution of task types and domain coverage, showcasing the diversity and realism of query patterns.

Figure 4: Distribution of average token count per task, indicating heavy context and long-horizon reasoning requirements typical in MCP-Persona.

Experimental Methodology and Results

Evaluation of LLM agents follows a rigorous, dual-protocol paradigm: checkpoint-based metrics analyze intermediate parameter and response validity at each reasoning step, while execution-based metrics assess end-to-end alignment of state transitions and final task completion.

The results are unambiguous: no evaluated agent surpassed 50% accuracy on either checkpoint or execution-based measures. Even contemporary SOTA proprietary models (Claude-Sonnet-4.5, GPT-5) exhibit severe limitations, especially in tasks involving unseen tools, nontrivial dependency chains, and implicit context inference. For example, GPT-5 achieved only 41.45% execution accuracy overall. Open-weight models performed substantially worse.

Performance breakdown by application type reveals further trends:

• Email-oriented tasks exhibit highest accuracy, reflecting their simplicity and limited dependency structure.
• Social media and enterprise collaboration scenarios yielded poor results, mapping to both cross-user/entity reasoning and rich, dynamic context modeling.
• Content-management tools proved most challenging due to long-context navigation and requirement for robust entity manipulation.

Additionally, the hodgepodge cross-server subset consistently yielded the lowest accuracy, highlighting the acute challenge of orchestrating tool calls in heterogeneous, multi-application context.

A key finding is the general failure of agents to perform environment exploration and to recover implicit entity references, often skipping required tool invocations or misidentifying context-relevant entities. This underexploration mirrors failure modes known in open-domain multi-hop reasoning, confirming that naïve LLM-based agents remain inadequate for real-world personal workflow automation.

Tool Simulation Fidelity and Ablations

Tool-Traverse, the code-based simulation paradigm, was validated against vanilla baselines using real interaction traces with Lark-MCP. The approach reached 94% behavioral alignment F1 and tripled the structure-aware lexical similarity metrics (TF-IDF, METEOR) compared to documentation-based simulators, establishing that high-fidelity simulation is critical to realistic agent evaluation.

Skill incorporation experiments confirmed that well-aligned, manually refined skill documentation yields moderate but consistent performance gains over public, less tailored skill guides or no skills at all. Tool-candidate ablation further demonstrated that constraining the available tool set to ground-truth relevant tools improves agent stability, partly mitigating distraction from irrelevant option space.

Moreover, injecting distractor context—realistic, but unnecessary, additional information—significantly degrades agent performance, highlighting current models' insufficient robustness to the "context confusion" posed by information overload in authentic environments.

Theoretical and Practical Implications

MCP-Persona places a strong constraint on the ambition of current LLM-based agent research: despite massive advances in generative capabilities, the gap between prototype tool-use in synthetic, stateless environments and robust, reproducible tool orchestration in realistic personalized scenarios remains substantial. Key weaknesses include insufficient exploration, fragile entity resolution, and poor handling of implicit stateful context. The demonstrated simulation and evaluation platform now enables iterative improvement on these axes, enabling measurement-driven progress toward robust agents in privacy-sensitive and context-intensive domains.

Practically, the simulation-first, privacy-preserving methodology of MCP-Persona will facilitate reproducible, large-scale experimentation without the prohibitive cost, privacy risk, and deployment friction of real-account-based benchmarking—a primary obstacle impeding academic and industrial progress.

Theoretically, the findings align with and extend recent work highlighting both the critical difficulty and necessity of multi-step, personalized tool orchestration for the next phase of LLM-based agent capabilities. Lessons from MCP-Persona point to the urgent need for better tool grounding, hierarchical memory, entity linking, and specialty skill integration.

Future Directions

MCP-Persona should serve as the primary testbed for evaluation and improvement of next-generation, personalization-aware agents. Immediate avenues include:

• Model architecture innovations for context-aware tool grounding and implicit information recovery;
• Integration of explicit memory and reasoning modules to address multi-step, long-context benchmarking challenges;
• Robustness training via adversarial distractors and environment randomization;
• Extension to other privacy-sensitive, high-stakes domains (e.g., finance, healthcare) contingent on advances in privacy-preserving simulation.

Conclusion

MCP-Persona establishes a comprehensive, reproducible, and privacy-compliant standard for benchmarking tool-augmented LLM agents on tasks representative of actual personalized application use. The benchmark surfaces critical limitations in current SOTA agents, particularly regarding implicit context handling, exploration sufficiency, and stateful, multi-application reasoning. Advances in tool-oriented agent modeling, memory, and grounding are essential for closing this gap. MCP-Persona will be central to empirical progress in this space and sets the agenda for future research at the intersection of applied agent modeling and realistic user-centric evaluation scenarios (2606.02470).