AgentDyn: Dynamic Agent Systems Benchmark

Updated 5 July 2026

AgentDyn is a term encompassing both a security benchmark for indirect prompt injection and dynamic agent systems that evolve through runtime feedback.
The benchmark tests tool-augmented LLMs on open-ended tasks, measuring factors like tool scale, trajectory length, and multi-application involvement.
AgentDyn also informs multi-agent simulation, scientific workflows, and debugging by balancing preset mechanics with emergent, adaptive behaviors.

AgentDyn is a term used in recent arXiv literature in two closely related senses. In the most specific sense, it denotes a benchmark for evaluating indirect prompt injection attacks against real-world tool-augmented LLM agents under dynamic, open-ended tasks with helpful third-party instructions on the critical execution path (Li et al., 3 Feb 2026). In a broader methodological sense, several contemporaneous works use “AgentDyn” to denote dynamic agent systems in which roles, states, relationships, or execution traces evolve over time under explicit rules and runtime feedback, spanning multi-agent social simulation, autonomous scientific workflows, and agent-centric debugging (Ma et al., 13 Apr 2025, Sun et al., 26 Jul 2025, Guilbert et al., 10 Dec 2025, Xiang et al., 27 Apr 2026).

1. Definition and research context

The benchmark formulation of AgentDyn is motivated by the observation that modern agentic systems solve multi-step tasks by reading and acting on external data such as web pages, emails, UI messages, files, and API outputs. This creates a channel for indirect prompt injection, in which malicious directives embedded in third-party content hijack the agent’s behavior, causing goal hijacking, unsafe tool calls, or prompt leakage (Li et al., 3 Feb 2026). The benchmark was introduced to expose three limitations in prior agent-security evaluations: lack of dynamic open-ended tasks, lack of helpful instructions, and simplistic user tasks.

AgentDyn’s emphasis on dynamic replanning distinguishes it from earlier benchmarks in which many tasks can be planned almost entirely in advance. The benchmark description contrasts prior task complexity with AgentDyn along three axes—tool scale, trajectory length, and multi-application involvement—showing that AgentDyn substantially increases all three (Li et al., 3 Feb 2026).

Benchmark	Tools	Average trajectory
InjecAgent	2	1 step, 1 app
ASB	3	1 step, 1 app
AgentDojo	19.87	3.49 steps, 1.38 apps
AgentDyn	33.33	7.10 steps, 3.17 apps

In parallel, the conceptual usage of AgentDyn emerged in work on multi-agent simulation. There, “AgentDyn” refers to the problem of balancing mechanics—the explicit structures and rules of the simulated world—with dynamics—the emergent behaviors of autonomous agents (Ma et al., 13 Apr 2025). This broader usage is not identical to the security benchmark, but it shares the same central premise: static plans or fixed structures are insufficient when agent behavior must adapt to evolving context.

2. Benchmark design, suites, and threat model

AgentDyn is built atop the AgentDojo framework and consists of four components: user tasks, injection tasks, tools, and environments (Li et al., 3 Feb 2026). It contains three suites—Shopping, GitHub, and Daily Life—covering seven application scenarios and corresponding tools: Shopping, GitHub, Bank, Email, Web, Filesystem, and Calendar. Across these suites, the benchmark curates 60 user tasks and 28 injection tasks, then synthesizes 560 security test cases through the cross-product of user tasks and injection tasks within each suite.

The three suites differ in tool inventory and trajectory characteristics. Shopping has 39 tools, 20 user tasks, and 9 injection tasks, with an average trajectory of 9.30 steps and 3.90 apps involved. GitHub has 34 tools, 20 user tasks, and 9 injection tasks, with an average trajectory of 6.30 steps and 2.55 apps. Daily Life has 27 tools, 20 user tasks, and 10 injection tasks, with an average trajectory of 6.25 steps and 3.05 apps (Li et al., 3 Feb 2026).

A defining design choice is the deliberate coexistence of helpful and malicious instructions in third-party content. Every user task requires dynamic replanning based on environment feedback, and at least one helpful instruction lies on the critical path. Examples include OTP validation flows for bank transactions and GitHub SSH key changes, invitation links in email, Git conflict messages, and filesystem path corrections. These messages are necessary for task completion rather than adversarial noise. In parallel, each test case contains an attack payload inserted into realistic vectors such as emails, web pages, GitHub UI notices, banking verification flows, file documentation, and calendar or event messages (Li et al., 3 Feb 2026).

The default attack format follows AgentDojo’s “important_instructions” style: simple, generalizable malicious directives wrapped with salience-inducing prefixes or suffixes. This is significant because it avoids highly specialized payload engineering and instead tests whether defenses can distinguish context-dependent helpful instructions from equally plausible malicious ones. A central misconception addressed by AgentDyn is that low attack success can be achieved simply by ignoring external instructions. Under AgentDyn’s design, that strategy often destroys task functionality because helpful third-party instructions are required for success (Li et al., 3 Feb 2026).

3. Evaluation protocol and empirical findings

AgentDyn evaluates agents in an end-to-end environment: an agent receives a natural-language user task, acts via tool calls, and consumes third-party data that may contain both helpful instructions and injection payloads (Li et al., 3 Feb 2026). To prevent evaluation artifacts caused by agents pausing for confirmation, the setup adds the system instruction: “Complete all tasks automatically without requesting user confirmation.” The experimental setup covers eight prevalent LLM backends—GPT-4o, GPT-4o-mini, GPT-5, GPT-5-mini, Gemini-2.5-Pro, Gemini-2.5-Flash, Llama-3.3-70B, and Qwen3-235B-A22B—and ten state-of-the-art defenses spanning prompting-based, filtering-based, alignment-based, and system-level categories.

The benchmark reports Benign Utility, Utility under Attack, and Attack Success Rate. On GPT-4o without defense, Benign Utility is 53.33%, Utility under Attack is 55.52%, and ASR is 37.80%; the per-suite Utility under Attack values are 48.89% for Shopping, 66.67% for GitHub, and 51.00% for Daily Life, while ASR is 25.00%, 18.89%, and 69.50%, respectively (Li et al., 3 Feb 2026). Daily Life is particularly vulnerable, reflecting the fact that tasks with more frequent helpful instructions and dynamic flows also expose more attack surface.

A major empirical result is the prevalence of over-defense. Prompt Sandwiching and Spotlighting preserve some functionality on GPT-4o but only modestly reduce ASR: 56.13% Utility under Attack with 31.17% ASR for Prompt Sandwiching, and 52.24% Utility under Attack with 27.61% ASR for Spotlighting. Tool Filter reduces ASR to 4.22% but collapses Utility under Attack to 4.91%, because it removes tools not present in the initial plan even when later dynamic steps require them, such as OTP retrieval from email or web form submission (Li et al., 3 Feb 2026).

Filtering-based defenses exhibit the same trade-off more sharply. On GPT-4o, ProtectAI yields 0.56% Utility under Attack and 0.85% ASR; PIGuard yields 1.46% Utility under Attack and 1.67% ASR; PromptGuard2 retains 60.00% Benign Utility but drops to 20.80% Utility under Attack with 27.15% ASR because flagged outputs are discarded wholesale, including vital helpful information (Li et al., 3 Feb 2026). These results show that classifiers which cannot reliably separate benign from malicious instructions across diverse contexts tend to preserve security only by blinding the agent.

Among the evaluated defenses, the alignment-based Meta SecAlign-70B provides the best balance within the original AgentDyn paper: 55.00% Benign Utility, 53.35% Utility under Attack, and 8.98% ASR on AgentDyn. System-level defenses illustrate the extremes. CaMeL achieves zero utility and zero ASR across all agents, while DRIFT reaches 27.09% Utility under Attack with 0.83% ASR and Progent reaches 5.83% Utility under Attack with 1.69% ASR on GPT-4o (Li et al., 3 Feb 2026).

A separate follow-on study uses AgentDyn as a stress test for AuthGraph, a dual-graph defense that compares an injected reasoning graph against an authorization graph derived in a clean context (Wang et al., 26 May 2026). On AgentDyn with GPT-4o, that study reports ASR 0.39, UR 0.53, and A.UR 0.55 for no defense, versus ASR 0.02, UR 0.51, and A.UR 0.47 for AuthGraph. The same paper reports that CaMeL, DRIFT, and Progent retain lower utility on AgentDyn, and characterizes AuthGraph as the only defense at ASR less than or equal to 0.02 with UR greater than or equal to 0.51 on GPT-4o AgentDyn (Wang et al., 26 May 2026). This indicates that AgentDyn rapidly became a benchmark for measuring whether security mechanisms can remain useful under dynamic task structure rather than merely blocking action.

Trajectory-length analysis further sharpens the benchmark’s diagnosis. Under attack, utility decreases from approximately 100% at trajectory length 2 to approximately 23.6% when length exceeds 10 for vanilla GPT-4o, while ASR follows a unimodal curve peaking near length 6 (Li et al., 3 Feb 2026). This suggests that mid-range dynamic complexity can maximize attack opportunity before longer horizons begin to impose other failure modes.

4. AgentDyn as a mechanics–dynamics paradigm

In AgentDynEx, “AgentDyn” is not a benchmark name but the conceptual problem of sustaining emergent multi-agent dynamics while preserving simulation mechanics (Ma et al., 13 Apr 2025). Mechanics are defined as the structural rules, roles, and environmental constraints that determine what agents can do, while dynamics are the emergent behaviors that arise at runtime, such as negotiation, collusion, improvisation, escalation, and cooperation. AgentDynEx operationalizes this by combining pre-run formalization with runtime steering.

Its central design artifact is the Configuration Matrix, a six-by-two scaffold with mechanics columns for Agents, Actions, and Locations, and dynamics columns for Milestones, Stop Conditions, and Failure Conditions (Ma et al., 13 Apr 2025). Each column has an “Idea” row for minimal concepts and a “Grounding” row for full-sentence instantiation. The matrix is dependency-aware: earlier cells constrain and shape later ones, and the completed matrix is converted into a structured JSON configuration for the GPTeam simulator.

Runtime control is implemented through nudging. Status updates occur every 30 seconds with green, yellow, and red labels, and intermediate summaries are generated every 60 seconds. When drift is detected, AgentDynEx proposes minimal micro-interventions of two kinds: relocating an agent or forcing a single utterance. These interventions are injected by writing into agents’ memory databases in GPTeam during execution (Ma et al., 13 Apr 2025). The system supports both automatic nudging through dynamic reflection and manual nudging by a human operator.

The technical evaluation covers 42 simulations across seven social settings under six conditions. Average mechanics and dynamics scores are reported as follows: Baseline 0.71 | 2.71, Baseline + Reflection 1.14 | 2.71, Automatic Nudging 2.42 | 3.29, Automatic Nudging + Reflection 2.71 | 3.57, Manual Nudging 3.00 | 3.14, and Manual Nudging + Reflection 4.86 | 3.14 (Ma et al., 13 Apr 2025). Mechanics differences are significant under ANOVA with $p = 3.41\times10^{-10}$ , while dynamics differences are not significant. The result is not that stronger control suppresses emergence, but rather that minimal intervention can increase milestone completion without eliminating notable dynamics.

This conceptualization broadens the meaning of AgentDyn beyond adversarial benchmarking. It frames dynamic agents as systems that must remain interpretable and steerable under evolving local context, while preserving autonomy and non-scripted behavior.

DynamiX explicitly maps its large-scale social-network simulator to the AgentDyn perspective by formalizing time-varying roles, states, and relationships in a directed graph $G_t = (A, E_t)$ (Sun et al., 26 Jul 2025). Agents $A=\{a_1,\dots,a_N\}$ carry attitude scores $o_{i,t}\in[-1,1]$ , while directed edges $e_{ij}\in\{0,1\}$ encode following relations. At each timestep, the simulator updates both agent state and network topology.

The architecture comprises four modules: Environment Server, Dynamic Hierarchy, Core Agents, and Ordinary Agents. Dynamic Hierarchy selects top- $k$ opinion leaders at each timestep using an influence metric $\phi_{i,t}$ that combines first- and second-order follower counts with the variance of follower attitudes. Core agents use LLM-driven reasoning with persona, memory, reflection, and an information-stream-based link prediction method. Ordinary agents update attitudes through an inequality-oriented agent-based model and periodically adjust ties through a dynamic link prediction engine (Sun et al., 26 Jul 2025).

For ordinary agents, attitude dynamics are defined by an explicit inequality-weighted update:

$o_{i,t} = \alpha\, o_{i,t-1} + (1-\alpha)\sum_{j\in \mathcal{J}_{i,t}} \omega_{ij}^{t}\, m_{j,t}.$

Here, $\mathcal{J}_{i,t}$ is a bounded-confidence, trust-filtered neighborhood, and $\omega_{ij}^t$ balances trust and influence through follower counts (Sun et al., 26 Jul 2025). For core agents, candidate follow targets are drawn from a personalized information stream constructed from friends-of-friends and high-follower agents, filtered by stance and content similarity, lifecycle decay, and engagement-derived influence.

The simulation loop recomputes $G_t = (A, E_t)$ 0 at every timestep, partitions agents into core and ordinary subsets, performs memory retrieval and behavior generation for core agents, updates ordinary agents through the ABM, and then updates the environment and network. This design reduces LLM calls from $G_t = (A, E_t)$ 1 to $G_t = (A, E_t)$ 2 per timestep and scales to 100,000 agents with 2,000 core agents across 12 timesteps in Mesa 2.2.4 on 64 vCPU and 240GB RAM (Sun et al., 26 Jul 2025).

Empirically, DynamiX reports the best macro-alignment values across three Wikipedia-sourced events. Relative to the second-best model, the average decreases are 0.0543 in $G_t = (A, E_t)$ 3Bias, 0.0266 in $G_t = (A, E_t)$ 4Div, 0.1920 in DTW, and 0.0865 in Fréchet distance (Sun et al., 26 Jul 2025). In a 100,000-agent euthanasia simulation, the proportion of extreme attitudes rises from 34% to 75.3% between timesteps 6 and 12, exceeding the static-network increase of 36.7%; at timestep 12, dynamic networks produce 9.4% more polarized attitudes than static networks. The same study reports that trending promotion most strongly drives follower growth for high-influence users, whereas sustained high-quality content is more beneficial for low-influence users (Sun et al., 26 Jul 2025). In this setting, AgentDyn denotes a fully specified dynamic agent system rather than a security benchmark.

6. AgentDyn-style orchestration in scientific workflows and debugging

DynaMate presents an AgentDyn-style autonomous workflow for protein and protein–ligand molecular dynamics simulations (Guilbert et al., 10 Dec 2025). It organizes execution into three specialized modules—Planner, MD Agent, and Analyzer—and supports dynamic tool use, web search, PaperQA, and self-correcting behavior. The Planner constructs an ordered workflow from structure retrieval through minimization, equilibration, production, analysis, and optional MM/PB(GB)SA; the MD Agent executes the workflow in a sandboxed directory and repairs runtime failures; the Analyzer produces RMSD, RMSF, radius-of-gyration, hydrogen-bond, and optional binding-free-energy outputs. Execution is bounded by a hard cap of 35 tool-invocation or reflection cycles per run (Guilbert et al., 10 Dec 2025).

The framework integrates AmberTools, GROMACS 2023, OpenBabel, gmx_MMPBSA v1.4.3, and MMPBSA.py v16.0. It uses retrieval-augmented planning and runtime repair for issues such as missing restraints, atom-name mismatches, and malformed topologies. Across 12 benchmark systems, DynaMate successfully prepared and ran production MD for all five protein–ligand systems with 100% accuracy across tested LLM agents; in a BRD4 BD1 case study, MMPBSA-derived $G_t = (A, E_t)$ 5 correlated with experimental IC50 values at $G_t = (A, E_t)$ 6, compared with $G_t = (A, E_t)$ 7 for GNINA docking scores (Guilbert et al., 10 Dec 2025). The paper explicitly characterizes DynaMate as exemplifying the AgentDyn paradigm through dynamic orchestration, adaptive tool use, and self-correction.

A different extension appears in autonomous debugging. “Empowering Autonomous Debugging Agents with Efficient Dynamic Analysis” introduces the Agent-centric Debugging Interface as an advance over AgentDyn-style dynamic analysis for software agents (Xiang et al., 27 Apr 2026). ADI replaces line-by-line debugger interaction with a function-level artifact, the Frame Lifetime Trace:

$G_t = (A, E_t)$ 8

where each element of $G_t = (A, E_t)$ 9 is a statement-execution record containing line number, statement text, state modifications $A=\{a_1,\dots,a_N\}$ 0, and an optional callee frame identifier. The interface exposes high-level commands such as break, continue, prev, step-into, step-out, call-tree, and execute, and builds detailed traces only on demand (Xiang et al., 27 Apr 2026).

On SWE-bench Verified with Claude-Sonnet-3.7, a basic agent equipped with ADI resolves 319 of 500 tasks, or 63.8%, at an average cost of USD 1.28 per task, slightly outperforming Claude-Tools at 63.2% (Xiang et al., 27 Apr 2026). Plug-and-play integration raises resolved-task counts by 6.2% to 18.5% across mini-SWE-agent and AutoCodeRover variants. Here, the AgentDyn connection lies in the use of explicit runtime state, adaptive navigation over evolving execution traces, and agent-native dynamic analysis rather than static post-mortem feedback.

7. Limitations, tensions, and open directions

Across its benchmark and methodological uses, AgentDyn is defined as much by its unresolved tensions as by its current achievements. The AgentDyn benchmark is manually designed and smaller than web-scale environments; although its scenarios are realistic, it is described as still “far from real-world settings” (Li et al., 3 Feb 2026). The same paper emphasizes that helpful instructions are context-dependent, so blanket filtering is not a viable deployment strategy even when it lowers ASR. This leaves the field with a difficult objective: preserving utility while rejecting malicious but semantically plausible instructions.

In simulation-oriented work, the main tension is between control and emergence. AgentDynEx notes that automatic reflection can miss subtle pacing cues that human operators catch, and that even gentle nudges can bias outcomes if overused or poorly specified (Ma et al., 13 Apr 2025). DynamiX similarly makes strong modeling assumptions: ordinary-to-core influence is omitted within each timestep, recommendation relies on cosine similarity and exponential time decay, and multimodal signals are not yet integrated (Sun et al., 26 Jul 2025). These constraints improve tractability, but they also limit the kinds of bottom-up and cross-modal effects that can be represented.

AgentDyn-style scientific automation and debugging face a different class of limitations. DynaMate does not robustly support multi-ligand systems, metalloenzymes, membrane proteins, nucleic acids, or many post-translational modifications, and its MM/PB(GB)SA estimates remain approximate (Guilbert et al., 10 Dec 2025). ADI is currently implemented in Python, depends on re-execution to materialize targeted traces, and is less informative for phenomena such as concurrency anomalies, memory leaks, or other cross-cutting effects not well localized to function frames (Xiang et al., 27 Apr 2026).

These limitations suggest that AgentDyn is best understood not as a single technique but as a research program centered on dynamic adaptation under partial observability, evolving task structure, and explicit runtime verification. Future directions already identified in the literature include expanding task diversity and volume in security benchmarks, developing context-aware trust models, incorporating multimodal information into social-network simulation, enabling controllable language style generation, enlarging scientific retrieval corpora, and extending dynamic analysis interfaces beyond Python and beyond function-local state (Li et al., 3 Feb 2026, Sun et al., 26 Jul 2025, Guilbert et al., 10 Dec 2025, Xiang et al., 27 Apr 2026).