Dissecting model behavior through agent trajectories

Abstract: AI agent performance is not just a modeling problem, it is fundamentally a systems problem. The advanced capabilities of models are realized through agent harnesses. Therefore, a gap between model assumptions and harness behavior can easily prevent the model's full capabilities from translating into agent performance. We formalize this as the intent-execution' gap: the mismatch between what the model intends and what the harness executes, and vice versa. We argue that minimizing this intent-execution gap is as important as other aspects of harness design such as tools and execution loops. To illustrate the impact of this harness-model alignment, we develop a simple and customizable harness calledSimple Strands Agent' (SSA). SSA aims to find the bulk of common patterns which generalize across different model families (such as Claude, Gemini, GPT, Grok, Qwen), as well as a small number of model-specific preferences. We make two contributions: (i) we reproduce or improve on the pass@1 performance reported by diverse model-provider families on popular agentic benchmarks (SWE-Pro, SWE-Verified and Terminal-Bench-2), and (ii) building on an analysis of 138k trajectories generated by SSA, we look beyond the pass@1 numbers which tend to be relatively even across frontier models. By representing agent trajectories in code state-spaces, we observe model-level differences in problem-solving behavior. Finer-grained metrics such as edit frequency, testing activity, and phase-transitions reveal how individual models allocate effort across different stages of autonomous problem solving.

• The paper introduces the intent–execution gap, showing how misaligned harness design can constrain LLM agent performance.
• It presents the SSA framework that employs structured feedback and reasoning nudges to optimize tool interactions and model alignment.
• It proposes solution-distance metrics to precisely quantify trajectory efficiency and diagnose backtracking in agent problem-solving.

Dissecting Model Behavior via Agent Trajectory Analysis

Introduction and Motivation

"Dissecting model behavior through agent trajectories" (2606.17454) presents a systematic, data-driven framework for analyzing how LLM-powered agents solve complex software engineering tasks, shifting attention from aggregate pass/fail metrics toward the nuanced dynamics of agent-environment interactions. The work establishes the primacy of the harness in mediating and potentially constraining model capabilities, introducing the concept of the intent–execution gap—the bidirectional misalignment between model intentions and harness-level tool execution—which can materially cap agent performance, even for highly capable models. Through the design and deployment of the Simple Strands Agent (SSA) harness, the paper demonstrates that careful harness-model alignment is essential for reliably extracting the full capabilities of leading LLMs across families and benchmarks.

Formalizing the Intent–Execution Gap

A central contribution is the operationalization of the intent–execution gap, highlighting that practical agent performance is a systems challenge, not a pure modeling problem. This gap can arise when parsing errors, silent tool failures, or harness restrictions produce feedback divorced from a model's actual intent, often inducing compounding failures in agent trajectories.

The failure mode is exemplified in the harness-agent event loop, where a misparsed tool call (e.g., a stray token in the tool name) yields an environment response uninformative to the model (e.g., "tool name not found"). The model, unable to discern that the failure originated in the interface layer, may repeatedly attempt corrections or pivot away from productive search, resulting in a non-monotonic, suboptimal resolution path. The bidirectionality is critical: feedback must be both a faithful reflection to the model and a faithful execution of model requests.

Harness Design: Simple Strands Agent (SSA)

SSA embodies several harness design principles specifically targeting reduction of the intent–execution gap:

• Structured execution feedback: All tool interactions return precise status metadata, including command, status, exit code, and output, with mechanisms for reporting partial or truncated results. Successful edits are returned as unified diffs, reducing the model's cognitive load for change verification.
• Model-preferred tool interfaces: SSA adapts its tool semantics and surface to align with the idiosyncrasies of each model family (e.g., apply_patch for GPT-family, atomic edit/view for Grok), leveraging learned priors rather than overfitting to one-off quirks.
• Reasoning–interaction balance: SSA introduces reasoning nudges—either quantitative tool-use targets or flexible action encouragement, contingent on model family—to optimize between excessive deliberation and shallow tool-chaining.

  Figure 1: Reasoning nudges in SSA: a quantitative target for Claude variants and a flexible directive for other model families.

Solution Distance as Trajectory Metric

Recognizing the limitations of coarse metrics like pass@1, the paper advances a solution-distance metric that quantifies, at every agent cycle, the divergence between the live code state and a set of verified-correct solutions in a text-level diff space. This enables fine-grained characterization of agent progression, backtracking behavior, and path efficiency, beyond binary outcomes.

The metric is constructed as a recall-style divergence: $$D_i(t) = \min_{x^\star \in \mathcal{S}_i} \left( 1 - \frac{|\phi_i(x_i(t)) \cap \phi_i(x^\star)|}{|\phi_i(x^\star)|} \right)$$, where $\phi_i(\cdot)$ is the text-diff feature mapping and $\mathcal{S}_i$ is the set of reference-correct code states. Empirically, solution-distance is estimated with an ensemble of observed correct patches, enabling robust approximation even when the oracle space is not fully enumerable.

Figure 2: Agent trajectory in code state space. Solution divergence $D(t)$ evolution as agent traverses code state from initial state to solution set $\mathcal{S}$.

This framework captures:

• Monotonic progress: Decisive, direct solutions are characterized by sharp, monotonic drops in $D(t)$.
• Backtracking: Non-monotonicity (increases in $D$) diagnoses inefficient or error-prone model behaviors, distinguishing between genuine revert maneuvers and chaotic oscillations.

Figure 3: Solution-distance curves expose different problem-solving styles between strong models.

Trajectory-level Behavioral Signatures

SSA's harness-instrumented data collection enables annotation and aggregation of trajectory events via an LLM judge, classifying tool calls into granular activity and phase labels (e.g., source edit, suite test, exploration, verification). This allows for quantitative comparison across models along several axes:

• Activity composition: Distributions of edits, scratch testing, and suite testing reveal that even models with similar final performance have distinct problem-solving signatures—for example, Opus 4.6 runs early, sustained scratch tests while GPT-5.4 ramps up testing later, with less emphasis on verification phases.

Figure 4: Composition of activities is a model signature. Three-activity share over normalized agent trajectory cycles, contrasting Opus 4.6 and GPT-5.4.

• Phase scheduling: Family-specific "phase surfaces" show systematic differences in how cycles are allocated to exploration, localization, implementation, and verification phases, even on matched tasks.

Figure 5: Phase composition surfaces model-specific schedules. Per-cycle share of four R6 phases across representative models.

• Backtracking rates: Backtrack-edit fractions ($\Delta D > 0$) systematically differentiate models; e.g., GPT-5.2-codex operates with minimal backtracking ($<1\%$), while Gemini models can reach upward of $14\%$ in SWE-Bench-Verified, indicating divergent confidence and search strategies.
• Token efficiency: Pareto plots of output tokens vs. pass@1 indicate that families with similar solve rates can differ widely in overall token and reasoning expenditure.

  Figure 6: SWE-Bench-Verified: total output tokens vs. pass@1, colored by model family.

State-of-the-Art and Evaluation Integrity

Using the above methods and SSA, the paper reports:

• Full or superior reproduction of official pass@1 for all major proprietary and open models across SWE-Bench-Verified, SWE-Bench-Pro, and Terminal-Bench-2.
• Empirical confirmation that, with only minimal family-level adaptation (not per-model optimization), harness design can account for previously unrecoverable discrepancies between documented and practical agent performance.

Furthermore, a significant integrity issue is identified: git-history leakage in SWE-Bench-Pro public containers, leading to an artificial inflation of pass@1 by up to 6.9% in some models. Explicit ablation and sanitization experiments reveal model- and instruction-dependent susceptibility to this leakage, emphasizing the necessity for strict evaluation environment controls.

Implications and Future Directions

The findings indicate several key implications for the development and evaluation of LLM-based agents:

• Harness–model co-design is necessary: As model capabilities saturate in aggregate solve rates, trajectory-level behavioral metrics become the only discriminative avenue for improving agent architectures and exploiting latent model capacity.
• Unified, model-agnostic harnesses coupled with minimal interface adaptation suffice for frontier models—over-specialization risks overfitting and regression as model versions shift.
• Meta-metrics such as solution-distance, activity-phase surfaces, and backtracking enable both quantitative debugging and systematic harness improvements, facilitating fine-grained failure analysis and targeted optimization.
• Evaluation hygiene, including environment leak auditing, is indispensable for valid benchmarking, particularly at the frontier where small performance differences are amplified in model selection and deployment.

Looking forward, several avenues are apparent:

• Semantic, program-level trajectory metrics (AST-, embedding-, or LLM-judged distances) can further resolve distinctions masked by text-diff recall, particularly as code diversity increases.
• Automatic harness adaptation mechanisms, potentially driven by meta-learning over trajectory data, can minimize intent–execution friction as models and inference stacks evolve.
• Fine-grained trajectory diagnostics could ground new forms of curriculum learning or targeted model fine-tuning based on characteristic failure patterns.

Conclusion

"Dissecting model behavior through agent trajectories" delivers a comprehensive empirical and methodological toolkit for agent evaluation, centering the importance of harness–model alignment and the limitations of traditional aggregate metrics (2606.17454). By capturing trajectory signals at scale and leveraging solution-distance analysis, the work enables actionable insights into the real-world behavioral diversity of powerful LLM agents, setting a new standard for agent system benchmarks and offering a reproducible path toward explainable, trustworthy model deployment.