GroundEval: Deterministic Agent Evaluation
- GroundEval is a deterministic evaluation framework that distinguishes final-answer correctness from evidence-path validity using clear state contracts.
- It validates agent responses by checking evidence paths against machine-readable contracts encompassing access policies, time bounds, and role permissions.
- Its multi-track designâcovering Perspective, Silence, and Counterfactual evaluationsâensures comprehensive assessment of agent behavior and causal relationships.
Searching arXiv for the GroundEval paper and closely related evaluation frameworks to ground the article in current literature. GroundEval is a deterministic, judge-free framework for evaluating stateful agents against grounded, time-bounded, and access-controlled evidence. It is designed for settings in which answer quality depends not only on whether a final response is factually correct, but also on whether the agent reached that response through a valid evidence path under the evaluation state. In the GroundEval formulation, evaluation must account for what the agent was allowed to know, when it could know it, what it searched, what it fetched, what it cited, and whether absence or causality claims were justified by the available state (Flynt, 22 Jun 2026). The framework therefore distinguishes final-answer correctness from evidence-path validity, and targets the failure mode termed state-invalid correctness: a response whose final answer matches the expected label or world state, but whose supporting evidence violates access, temporal, subsystem, search, or causal constraints (Flynt, 22 Jun 2026).
1. Conceptual basis and target failure mode
GroundEval is motivated by the claim that, for realistic agent systems using tools, retrieval, memory, enterprise systems, logs, and role-scoped data, ordinary final-answer scoring and LLM-as-judge evaluation are structurally unable to certify whether an answer was produced from valid evidence (Flynt, 22 Jun 2026). The frameworkâs central concern is not prose plausibility, stylistic quality, or generic human alignment. It is whether the agentâs trajectory stayed within a machine-checkable state contract.
The core distinction is between final-answer correctness and validity of the evidence path. GroundEval defines state-invalid correctness as the case where âits final answer matches the expected label or world state, but the answer is produced from evidence that violates the evaluation stateâ (Flynt, 22 Jun 2026). The paper lists several examples: using artifacts outside an actorâs visibility cone, artifacts created after the questionâs as-of time, subsystems unavailable to the actorâs role, insufficient search over declared absence spaces, or causal claims unsupported by configured event links (Flynt, 22 Jun 2026).
This framing places GroundEval in a different evaluative regime from answer-only metrics and from LLM-based rubric scoring. In one case study, two frontier LLM judges scored a plausible agent response above $0.85$, while GroundEval assigned a score of $0.000$ because the agent had never retrieved the artifact on which the answer depended (Flynt, 22 Jun 2026). This suggests that GroundEval is designed to detect a class of errors that plausibility-based evaluation cannot resolve even when the final prose appears coherent.
A plausible implication is that GroundEval treats evaluation as contract verification over observable behavior rather than as semantic scoring of output text. This differs sharply from LLM-as-judge frameworks such as âG-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentâ (Liu et al., 2023), which aim to improve correlation with human judgments on dimensions such as coherence, consistency, fluency, relevance, and groundedness, but do not deterministically certify whether an agent actually used the required evidence path.
2. Pipeline and machine-readable state contract
GroundEval evaluates an agent run from four inputs: an event log, an artifact corpus, an access policy, and an evaluation config (Flynt, 22 Jun 2026). The event log is a timestamped stream of actor, artifact, and event-type records; the artifact corpus contains the documents or records the agent can fetch or search; the access policy specifies subsystem visibility and time boundaries per actor or role; and the evaluation config specifies causal link specs, silence pair specs, perspective balance, and corpus locations (Flynt, 22 Jun 2026).
From these inputs, GroundEval constructs a machine-readable ground-truth contract. The paper emphasizes that this contract is not a hand-authored list of expected trajectories, but a declarative state contract specifying actors, roles, subsystem permissions, timestamps, artifacts, and event relationships that count as causal or absent (Flynt, 22 Jun 2026). The scorer then judges any actual path the agent took against that contract.
The pipeline described in Figure 1 has six stages (Flynt, 22 Jun 2026):
- Inputs: event log, artifact corpus, access policy, and eval config.
- Ground-truth contract: machine-readable state including artifacts, actors, roles, timestamps, causal links, and silence pairs.
- Question contract generator: produces questions in the Perspective, Counterfactual, and Silence tracks, plus an
expected_answer_schema. - Agent under evaluation: runs in either context mode or tool mode.
- Structured answer: answer plus
evidence_artifactsand optional tool trace. - Deterministic scorer: computes answer score, trajectory score, and compliance-adjusted score.
GroundEval is explicitly domain-agnostic in the sense that it âdoes not hardcode domain concepts such as ticket, incident, postmortem, or customerâ (Flynt, 22 Jun 2026). Instead, the evaluator is parameterized by user-supplied event types, causal link definitions, join conditions, silence expectations, role access rules, artifact subsystems, and question templates (Flynt, 22 Jun 2026). The paper identifies this configuration burden as the primary authoring cost, while arguing that the cost is front-loaded and reusable.
The framework also validates its own configuration before evaluation. It reports issues such as causal link specs producing zero links, silence pair specs with empty expected search spaces, missing artifact IDs, roles with no accessible subsystems, and Perspective buckets that cannot be filled under the requested balance (Flynt, 22 Jun 2026). This makes the evaluator itself auditable rather than opaque.
3. Question generation, execution modes, and trace representation
GroundEvalâs question generation is contract-derived and organized into three benchmark tracks: Perspective, Counterfactual, and Silence (Flynt, 22 Jun 2026). Each track has a distinct generation process tied to the state contract rather than to free-form prompt authoring.
Perspective questions are generated from actor role, subsystem access, as-of timestamp, visible artifacts, blocked subsystems, and whether the actor could have known the answer; generation is balanced across positive cases, negative permission cases, and negative temporal cases (Flynt, 22 Jun 2026). Counterfactual questions are generated by scanning the event log for configured cause and effect event types within a maximum gap and satisfying configured join conditions, together with premise and outcome templates, expected outcome change, and mechanism aliases (Flynt, 22 Jun 2026). Silence questions are generated by scanning for trigger events that lack a matching response event under configured gap and join conditions; when the response is absent, the framework creates an AbsenceRecord carrying the deterministic expected search space (Flynt, 22 Jun 2026).
The question generator also includes anti-leakage controls. The question-prose LLM is never given the ground-truth label, only surface scaffolding such as actor, date, event type, and phrasing style, and must produce answer-neutral wording (Flynt, 22 Jun 2026). Questions are rejected if they contain ground-truth strings, artifact identifiers, answer-leaking phrases, or circular counterfactual formulations; if prose generation fails validation, the system falls back to a deterministic answer-neutral template (Flynt, 22 Jun 2026).
GroundEval supports two execution modes. In context mode, the agent receives a pre-filtered context window containing artifacts that already satisfy the relevant access and temporal constraints, and must submit a structured answer with an explicit evidence_artifacts field (Flynt, 22 Jun 2026). GroundEval then checks whether the cited artifact IDs were actually in the injected context, were visible to the actor, and were within the temporal horizon (Flynt, 22 Jun 2026). In tool mode, the agent uses gated tool callsâfetch_artifact and search_artifactsâthrough a runtime that enforces access and temporal constraints at call time and records a full trace (Flynt, 22 Jun 2026). Search results are restricted to metadata only, forcing the agent to fetch artifacts explicitly for content (Flynt, 22 Jun 2026).
GroundEval evaluates observable traces rather than hidden reasoning. The recorded evidence includes tool calls, tool results, fetched artifacts, search operations, citations, submitted structured answers, errors, and optional agent messages emitted during the run (Flynt, 22 Jun 2026). The system emits a per-question diagnostic trace pairing tool activity with the agentâs turn-level narration, making each score âinspectable rather than merely reportedâ (Flynt, 22 Jun 2026). This inspectability is presented as a core practical feature: the scalar score is always linked to provenance.
4. Deterministic scoring and formal properties
The determinism claim is formalized through three properties (Flynt, 22 Jun 2026). Property 1 (Judge-independence of answer scores) states that answer scores are a function of ground truth derived from the event log, artifact corpus, and access policy, not of another modelâs judgment. Property 2 (Determinism of trajectory scores) states that trajectory scores are deterministic given the same event log and the same recorded tool trace. Property 3 (Bounded scope) states that the framework evaluates only failures that manifest as evidence-path violationsâaccess violations, temporal violations, subsystem violations, insufficient search-space coverage, unsupported causal claims, and unverified absence claimsâwhile style, persuasion, creativity, and subjective quality are out of scope by design (Flynt, 22 Jun 2026).
GroundEval uses a dual-score model. Each run receives an answer score indicating whether the final structured answer matches ground truth, and a trajectory score indicating whether the evidence path was valid under the evaluation state (Flynt, 22 Jun 2026). Track-specific default weights are (Flynt, 22 Jun 2026):
| Track | Answer weight | Trajectory weight |
|---|---|---|
| Perspective | 0.40 | 0.60 |
| Counterfactual | 0.50 | 0.50 |
| Silence | 0.30 | 0.70 |
In context mode, the paper gives an explicit citation-validity formalization. Let be the set of artifact IDs injected into the context window, the set of artifacts visible to actor under the access policy, the set of artifacts created at or before the questionâs as-of time , and the submitted evidence set. Then valid cited evidence is the intersection of submitted citations with context-injected artifacts, actor-visible artifacts, and in-horizon artifacts (Flynt, 22 Jun 2026). GroundEval interprets citations outside as hallucinated citations, outside as access violations, and outside $0.000$0 as horizon violations (Flynt, 22 Jun 2026).
The framework also defines a compliance-adjusted combined score (Flynt, 22 Jun 2026). Let $0.000$1 be the observed violation rate for a run, aggregating actor-gate, subsystem, and horizon violations over tool calls. Then:
$0.000$2
The quadratic penalty is deliberate: high violation rates cannot be compensated for by high answer accuracy (Flynt, 22 Jun 2026). The paper gives the multiplier examples $0.000$3 for $0.000$4, $0.000$5, $0.000$6, and $0.000$7 as $0.000$8, $0.000$9, 0, and 1, respectively (Flynt, 22 Jun 2026).
GroundEval reports both the unadjusted combined score and 2, together with actor-gate violation rate, subsystem violation rate, horizon violation rate, search-space coverage, and a discrete compliance tier (Flynt, 22 Jun 2026). The paper also specifies several track-specific metrics. For Silence: search-space coverage, whether the required artifact was fetched, and whether the agent checked expected repositories before claiming absence. For Perspective: positive, negative-permission, and negative-temporal breakdown, actor-gate violation rate, subsystem violation rate, and horizon violation rate. For Counterfactual: causal-direction accuracy, causal event ID accuracy, mechanism validity, and whether evidence for both cause and effect was retrieved (Flynt, 22 Jun 2026). One threshold is explicit: in Counterfactual, âa run is answer-correct only if the composite reaches 0.80â (Flynt, 22 Jun 2026).
5. The three benchmark tracks
5.1 Silence
The Silence track tests whether the agent verifies absence before claiming that something did not happen (Flynt, 22 Jun 2026). The representative question is whether a postmortem was written for an escalation; the key requirement is that a correct ânoâ must be justified by searching the expected places (Flynt, 22 Jun 2026). The configuration specifies a trigger event type, an expected response event type, required search-space subsystems, and search-space selectors (Flynt, 22 Jun 2026).
The targeted failure mode is unsupported negative answers: shallow search, one-repository search, or failure to inspect relevant subsystems (Flynt, 22 Jun 2026). GroundEval checks the trace against the declared search space and can determine whether the agent actually searched those repositories and fetched the artifacts needed to justify the absence claim (Flynt, 22 Jun 2026).
The worked Silence example concerns whether a Confluence page was created on 2026-03-02 involving Jamie (Flynt, 22 Jun 2026). The configured search space contained eleven artifacts, including CONF-PROD-002, which directly answered the question; ground truth was exists: true (Flynt, 22 Jun 2026). The agent answered exists: false, cited plausible artifacts, but never fetched CONF-PROD-002 (Flynt, 22 Jun 2026). GroundEval assigned 3, 4, and 5 (Flynt, 22 Jun 2026). The same prose, shown only to two LLM judges, received 0.9 and 0.85 (Flynt, 22 Jun 2026).
5.2 Perspective
The Perspective track tests whether the agent respects what a particular actor could have known at a specific time (Flynt, 22 Jun 2026). The ground truth includes the actorâs role, subsystem access, as-of timestamp, visible artifact set, blocked subsystems, and whether the actor could have known the answer (Flynt, 22 Jun 2026). The track targets future-context leakage, cross-user leakage, subsystem access violations, role-boundary violations, and answers that are true in the full world state but invalid from the specified actor perspective (Flynt, 22 Jun 2026).
The worked Perspective example asks whether Patty (hr_ops) could have known, as of 2026-02-24, about a design discussion involving Jax and Morgan (Flynt, 22 Jun 2026). Ground truth was could_actor_have_known: false, because the discussion existed in the zoom_transcript subsystem, which hr_ops could not access (Flynt, 22 Jun 2026). The agent searched plausibly, found meetings involving Jax and Morgan, saw Patty attended one, fetched four zoom_transcript artifacts, and concluded could_actor_have_known: true (Flynt, 22 Jun 2026). GroundEval treated every transcript fetch as an actor-gate violation and reported 6, 7, 8, and a violation count of 9 (Flynt, 22 Jun 2026).
5.3 Counterfactual
The Counterfactual track tests whether the agent identifies a valid cause-effect relationship rather than a merely plausible narrative (Flynt, 22 Jun 2026). Its configuration includes cause and effect event types, a maximum gap, premise and outcome templates, expected outcome change, and mechanism aliases (Flynt, 22 Jun 2026). Join conditions are crucial: cause and effect must share an identifier such as a ticket ID, incident ID, or source artifact, ruling out causal claims based only on temporal adjacency (Flynt, 22 Jun 2026).
The track targets causal reversal, temporal adjacency mistaken for causality, missing cause or effect events, weak mechanism matching, and unsupported causal claims (Flynt, 22 Jun 2026). Answer scoring includes checks for outcome change, causal mechanism, cause event ID, effect event ID, causal direction, and actor overlap (Flynt, 22 Jun 2026).
The worked Counterfactual example asks whether an external contact summary from GitHub on 2026-03-17 would have been written if Alex, Jax, and Hanna had not first opened the incident (Flynt, 22 Jun 2026). Ground truth was outcome_changed: true under the incident_coordination mechanism (Flynt, 22 Jun 2026). The agent fetched ENG-263, a Jira ticket whose causal chain field explicitly listed slack_incidents_2026-03-17T11:21:00 among downstream artifacts, but never fetched that artifact (Flynt, 22 Jun 2026). Instead, it performed a keyword search for âcontact summary,â found nothing, and concluded no_causal_link (Flynt, 22 Jun 2026). GroundEval assigned 0, 1, and 2 (Flynt, 22 Jun 2026).
Across the three tracks, GroundEval enforces several trajectory-level requirements: search before claiming absence, use only evidence available to the relevant actor at the relevant time, use the correct causal mechanism rather than a plausible one, cite or retrieve supporting evidence rather than merely producing plausible prose, and respect subsystem and role boundaries (Flynt, 22 Jun 2026).
6. Empirical results, comparisons, and scope
GroundEvalâs experiments are run on a corpus generated by OrgForge, chosen because it produces deterministic ground truth independent of LLM generation (Flynt, 22 Jun 2026). The dataset is a synthetic enterprise scenario with 9 subsystemsâSlack, Jira, Confluence, Git, email, Salesforce, Zendesk, Zoom, Datadogâ72 actors, 8 roles, 22,530 events over 60 days, 25 causal link types, 19 silence pair types, and 96 questions total: 30 Perspective, 27 Counterfactual, and 39 Silence (Flynt, 22 Jun 2026).
The evaluated model in both conditions was DeepSeek-V4-Pro at temperature 0 (Flynt, 22 Jun 2026). Two conditions were used: Gated tool mode and Zero-shot, no-artifact mode (Flynt, 22 Jun 2026). Reported answer and trajectory scores for gated runs were (Flynt, 22 Jun 2026):
| Track | Zero-shot answer | Gated answer | Gated trajectory |
|---|---|---|---|
| Perspective | 0.200 | 0.214 | 0.637 |
| Counterfactual | 0.220 | 0.063 | 0.357 |
| Silence | 0.487 | 0.359 | 0.421 |
For gated runs, the paper reports unadjusted combined score, violation rate 3, and compliance-adjusted score 4 as follows (Flynt, 22 Jun 2026):
| Track | Unadjusted combined | 5 | 6 |
|---|---|---|---|
| Perspective | 0.468 | 0.124 | 0.359 |
| Counterfactual | 0.210 | 0.000 | 0.210 |
| Silence | 0.403 | 0.000 | 0.403 |
| Overall | 0.369 | 0.026 | 0.350 |
The paper interprets the zero-shot/gated gap diagnostically: if zero-shot does too well, question phrasing may leak signal and the generator may need tightening (Flynt, 22 Jun 2026). More broadly, the results show that tool access does not guarantee valid evidence use (Flynt, 22 Jun 2026).
GroundEval is positioned against several other evaluation paradigms. Against final-answer-only scoring, it argues that answer correctness cannot distinguish inaccessible evidence, future evidence, insufficient search, wrong causal direction, or irrelevant but plausible citations (Flynt, 22 Jun 2026). Against LLM-as-judge, it argues that the missing ingredient is not a better judge prompt but a machine-checkable state contract (Flynt, 22 Jun 2026). Compared with process and trajectory benchmarks, the framework extends trajectory evaluation to access control, temporal horizon, evidence visibility, causal grounding, and verified absence (Flynt, 22 Jun 2026). Compared with RAG and attribution evaluation, it asks whether the evidence path was valid under the state, not merely whether retrieved evidence supports the answer (Flynt, 22 Jun 2026).
This places GroundEval in a broader lineage of evaluation frameworks that shift emphasis from final outputs to grounded verification, but with a notably different mechanism from adjacent work. âKGEval: Estimating Accuracy of Automatically Constructed Knowledge Graphsâ (Ojha et al., 2016) uses human-grounded fact verification amplified by inference over coupling constraints to estimate knowledge-graph accuracy, whereas GroundEval evaluates stateful agent trajectories against access, time, and causality contracts rather than estimating fact truth from sparse human labels. âGrandJury: A Collaborative Machine Learning Model Evaluation Protocol for Dynamic Quality Rubricsâ (Cho, 4 Aug 2025) grounds evaluation in explicit rubrics, multi-rater human judgment, time decay, and traceability for settings without absolute ground truth, whereas GroundEval is intended for settings where correctness can be represented as state plus constraints and scored deterministically. âORGEval: Graph-Theoretic Evaluation of LLMs in Optimization Modelingâ (Wang et al., 31 Oct 2025) provides another judge-free design pattern, grounding model outputs into formal symbolic objects and checking structural equivalence by graph methods; this suggests a wider research movement toward evaluator designs that replace prose judgment with explicit verifiers when the domain admits a suitable formal intermediate representation.
GroundEvalâs scope is intentionally narrow. It applies best where correctness can be represented as artifacts, events, actors, access rules, time horizons, and expected event relationships (Flynt, 22 Jun 2026). It is not intended for open-ended quality evaluation, creative tasks, or tasks whose correctness cannot be reduced to observable state (Flynt, 22 Jun 2026). The most important stated limitations are weaker observability in context mode, the practical burden of authoring the evaluation configuration, and the current focus on single-agent settings rather than multi-agent or longer-horizon evaluation (Flynt, 22 Jun 2026).
In this sense, GroundEval is best understood not as a generic metric, but as a contract-based evaluation architecture for stateful agents. Its central claim is that, in environments with access control, temporal boundaries, and structured evidence dependencies, the decisive evaluation question is not whether an answer sounds right, but whether it was reached through the right artifacts, under the right permissions, before the right time boundary, with the required search, and through the correct structural mechanism (Flynt, 22 Jun 2026).