Determine whether late-trajectory evidence is most relevant for LLM judging

Determine whether the final portion of an LLM agent’s execution trajectory generally contains the most relevant evidence for judging Agent Completion Unfaithfulness, relative to the full trajectory, which may introduce distracting tool errors, intermediate drafts, and irrelevant reasoning steps.

Background

The paper compares LLM-as-a-judge baselines using only the final answer, the final answer plus the last 30% of the trajectory, and the full execution trajectory. The last-30% setting performs close to, and sometimes slightly better than, full-trajectory judging. The authors attribute this observation to a conjecture that late execution states contain the evidence most relevant to detecting deviations from the user’s requirements, while earlier trajectory content may add noise. The conjecture remains an empirical and conceptual question about how execution evidence should be selected for reliable post-hoc judging of agent faithfulness.