---
title: 'GroundEval: Deterministic Agent Evaluation'
url: https://www.emergentmind.com/topics/groundeval
type: topic
---

# GroundEval: Deterministic Agent Evaluation

Searching arXiv for the GroundEval paper and closely related evaluation frameworks to ground the article in current literature.
GroundEval is a deterministic, judge-free framework for evaluating stateful agents against grounded, time-bounded, and access-controlled evidence. It is designed for settings in which answer quality depends not only on whether a final response is factually correct, but also on whether the agent reached that response through a valid evidence path under the evaluation state. In the GroundEval formulation, evaluation must account for what the agent was allowed to know, when it could know it, what it searched, what it fetched, what it cited, and whether absence or causality claims were justified by the available state [2606.22737]. The framework therefore distinguishes final-answer correctness from evidence-path validity, and targets the failure mode termed *state-invalid correctness*: a response whose final answer matches the expected label or world state, but whose supporting evidence violates access, temporal, subsystem, search, or causal constraints [2606.22737].

## 1. Conceptual basis and target failure mode

GroundEval is motivated by the claim that, for realistic agent systems using tools, retrieval, memory, enterprise systems, logs, and role-scoped data, ordinary final-answer scoring and LLM-as-judge evaluation are structurally unable to certify whether an answer was produced from valid evidence [2606.22737]. The framework’s central concern is not prose plausibility, stylistic quality, or generic human alignment. It is whether the agent’s trajectory stayed within a machine-checkable state contract.

The core distinction is between *final-answer correctness* and *validity of the evidence path*. GroundEval defines *state-invalid correctness* as the case where “its final answer matches the expected label or world state, but the answer is produced from evidence that violates the evaluation state” [2606.22737]. The paper lists several examples: using artifacts outside an actor’s visibility cone, artifacts created after the question’s as-of time, subsystems unavailable to the actor’s role, insufficient search over declared absence spaces, or causal claims unsupported by configured event links [2606.22737].

This framing places GroundEval in a different evaluative regime from answer-only metrics and from LLM-based rubric scoring. In one case study, two frontier LLM judges scored a plausible agent response above \(0.85\), while GroundEval assigned a score of \(0.000\) because the agent had never retrieved the artifact on which the answer depended [2606.22737]. This suggests that GroundEval is designed to detect a class of errors that plausibility-based evaluation cannot resolve even when the final prose appears coherent.

A plausible implication is that GroundEval treats evaluation as contract verification over observable behavior rather than as semantic scoring of output text. This differs sharply from LLM-as-judge frameworks such as “G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment” [2303.16634], which aim to improve correlation with human judgments on dimensions such as coherence, consistency, fluency, relevance, and groundedness, but do not deterministically certify whether an agent actually used the required evidence path.

## 2. Pipeline and machine-readable state contract

GroundEval evaluates an agent run from four inputs: an event log, an artifact corpus, an access policy, and an evaluation config [2606.22737]. The event log is a timestamped stream of actor, artifact, and event-type records; the artifact corpus contains the documents or records the agent can fetch or search; the access policy specifies subsystem visibility and time boundaries per actor or role; and the evaluation config specifies causal link specs, silence pair specs, perspective balance, and corpus locations [2606.22737].

From these inputs, GroundEval constructs a machine-readable ground-truth contract. The paper emphasizes that this contract is not a hand-authored list of expected trajectories, but a declarative state contract specifying actors, roles, subsystem permissions, timestamps, artifacts, and event relationships that count as causal or absent [2606.22737]. The scorer then judges any actual path the agent took against that contract.

The pipeline described in Figure 1 has six stages [2606.22737]:

1. **Inputs**: event log, artifact corpus, access policy, and eval config.  
2. **Ground-truth contract**: machine-readable state including artifacts, actors, roles, timestamps, causal links, and silence pairs.  
3. **Question contract generator**: produces questions in the Perspective, Counterfactual, and Silence tracks, plus an `expected_answer_schema`.  
4. **Agent under evaluation**: runs in either context mode or tool mode.  
5. **Structured answer**: answer plus `evidence_artifacts` and optional tool trace.  
6. **Deterministic scorer**: computes answer score, trajectory score, and compliance-adjusted score.  

GroundEval is explicitly domain-agnostic in the sense that it “does not hardcode domain concepts such as ticket, incident, postmortem, or customer” [2606.22737]. Instead, the evaluator is parameterized by user-supplied event types, causal link definitions, join conditions, silence expectations, role access rules, artifact subsystems, and question templates [2606.22737]. The paper identifies this configuration burden as the primary authoring cost, while arguing that the cost is front-loaded and reusable.

The framework also validates its own configuration before evaluation. It reports issues such as causal link specs producing zero links, silence pair specs with empty expected search spaces, missing artifact IDs, roles with no accessible subsystems, and Perspective buckets that cannot be filled under the requested balance [2606.22737]. This makes the evaluator itself auditable rather than opaque.

## 3. Question generation, execution modes, and trace representation

GroundEval’s question generation is contract-derived and organized into three benchmark tracks: Perspective, Counterfactual, and Silence [2606.22737]. Each track has a distinct generation process tied to the state contract rather than to free-form prompt authoring.

Perspective questions are generated from actor role, subsystem access, as-of timestamp, visible artifacts, blocked subsystems, and whether the actor could have known the answer; generation is balanced across positive cases, negative permission cases, and negative temporal cases [2606.22737]. Counterfactual questions are generated by scanning the event log for configured cause and effect event types within a maximum gap and satisfying configured join conditions, together with premise and outcome templates, expected outcome change, and mechanism aliases [2606.22737]. Silence questions are generated by scanning for trigger events that lack a matching response event under configured gap and join conditions; when the response is absent, the framework creates an `AbsenceRecord` carrying the deterministic expected search space [2606.22737].

The question generator also includes anti-leakage controls. The question-prose LLM is never given the ground-truth label, only surface scaffolding such as actor, date, event type, and phrasing style, and must produce answer-neutral wording [2606.22737]. Questions are rejected if they contain ground-truth strings, artifact identifiers, answer-leaking phrases, or circular counterfactual formulations; if prose generation fails validation, the system falls back to a deterministic answer-neutral template [2606.22737].

GroundEval supports two execution modes. In **context mode**, the agent receives a pre-filtered context window containing artifacts that already satisfy the relevant access and temporal constraints, and must submit a structured answer with an explicit `evidence_artifacts` field [2606.22737]. GroundEval then checks whether the cited artifact IDs were actually in the injected context, were visible to the actor, and were within the temporal horizon [2606.22737]. In **tool mode**, the agent uses gated tool calls—`fetch_artifact` and `search_artifacts`—through a runtime that enforces access and temporal constraints at call time and records a full trace [2606.22737]. Search results are restricted to metadata only, forcing the agent to fetch artifacts explicitly for content [2606.22737].

GroundEval evaluates observable traces rather than hidden reasoning. The recorded evidence includes tool calls, tool results, fetched artifacts, search operations, citations, submitted structured answers, errors, and optional agent messages emitted during the run [2606.22737]. The system emits a per-question diagnostic trace pairing tool activity with the agent’s turn-level narration, making each score “inspectable rather than merely reported” [2606.22737]. This inspectability is presented as a core practical feature: the scalar score is always linked to provenance.

## 4. Deterministic scoring and formal properties

The determinism claim is formalized through three properties [2606.22737]. **Property 1 (Judge-independence of answer scores)** states that answer scores are a function of ground truth derived from the event log, artifact corpus, and access policy, not of another model’s judgment. **Property 2 (Determinism of trajectory scores)** states that trajectory scores are deterministic given the same event log and the same recorded tool trace. **Property 3 (Bounded scope)** states that the framework evaluates only failures that manifest as evidence-path violations—access violations, temporal violations, subsystem violations, insufficient search-space coverage, unsupported causal claims, and unverified absence claims—while style, persuasion, creativity, and subjective quality are out of scope by design [2606.22737].

GroundEval uses a dual-score model. Each run receives an **answer score** indicating whether the final structured answer matches ground truth, and a **trajectory score** indicating whether the evidence path was valid under the evaluation state [2606.22737]. Track-specific default weights are [2606.22737]:

| Track | Answer weight | Trajectory weight |
|---|---:|---:|
| Perspective | 0.40 | 0.60 |
| Counterfactual | 0.50 | 0.50 |
| Silence | 0.30 | 0.70 |

In context mode, the paper gives an explicit citation-validity formalization. Let \(C\) be the set of artifact IDs injected into the context window, \(V_a\) the set of artifacts visible to actor \(a\) under the access policy, \(T_t\) the set of artifacts created at or before the question’s as-of time \(t\), and \(E\) the submitted evidence set. Then valid cited evidence is the intersection of submitted citations with context-injected artifacts, actor-visible artifacts, and in-horizon artifacts [2606.22737]. GroundEval interprets citations outside \(C\) as hallucinated citations, outside \(V_a\) as access violations, and outside \(T_t\) as horizon violations [2606.22737].

The framework also defines a compliance-adjusted combined score [2606.22737]. Let \(v \in [0,1]\) be the observed violation rate for a run, aggregating actor-gate, subsystem, and horizon violations over tool calls. Then:

\[
S_{\text{adj}} = \left[w_a \cdot S_{\text{answer}} + w_t \cdot S_{\text{traj}}\right] \cdot (1-v)^2
\]

The quadratic penalty is deliberate: high violation rates cannot be compensated for by high answer accuracy [2606.22737]. The paper gives the multiplier examples \((1-v)^2\) for \(v=0\%\), \(25\%\), \(50\%\), and \(75\%\) as \(1.00\), \(0.56\), \(0.25\), and \(0.06\), respectively [2606.22737].

GroundEval reports both the unadjusted combined score and \(S_{\mathrm{adj}}\), together with actor-gate violation rate, subsystem violation rate, horizon violation rate, search-space coverage, and a discrete compliance tier [2606.22737]. The paper also specifies several track-specific metrics. For Silence: search-space coverage, whether the required artifact was fetched, and whether the agent checked expected repositories before claiming absence. For Perspective: positive, negative-permission, and negative-temporal breakdown, actor-gate violation rate, subsystem violation rate, and horizon violation rate. For Counterfactual: causal-direction accuracy, causal event ID accuracy, mechanism validity, and whether evidence for both cause and effect was retrieved [2606.22737]. One threshold is explicit: in Counterfactual, “a run is answer-correct only if the composite reaches 0.80” [2606.22737].

## 5. The three benchmark tracks

### 5.1 Silence

The Silence track tests whether the agent verifies absence before claiming that something did not happen [2606.22737]. The representative question is whether a postmortem was written for an escalation; the key requirement is that a correct “no” must be justified by searching the expected places [2606.22737]. The configuration specifies a trigger event type, an expected response event type, required search-space subsystems, and search-space selectors [2606.22737].

The targeted failure mode is unsupported negative answers: shallow search, one-repository search, or failure to inspect relevant subsystems [2606.22737]. GroundEval checks the trace against the declared search space and can determine whether the agent actually searched those repositories and fetched the artifacts needed to justify the absence claim [2606.22737].

The worked Silence example concerns whether a Confluence page was created on 2026-03-02 involving Jamie [2606.22737]. The configured search space contained eleven artifacts, including **CONF-PROD-002**, which directly answered the question; ground truth was `exists: true` [2606.22737]. The agent answered `exists: false`, cited plausible artifacts, but never fetched **CONF-PROD-002** [2606.22737]. GroundEval assigned \(S_{\mathrm{ans}} = 0.000\), \(S_{\mathrm{traj}} = 0.273\), and \(S_{\mathrm{adj}} = 0.191\) [2606.22737]. The same prose, shown only to two LLM judges, received **0.9** and **0.85** [2606.22737].

### 5.2 Perspective

The Perspective track tests whether the agent respects what a particular actor could have known at a specific time [2606.22737]. The ground truth includes the actor’s role, subsystem access, as-of timestamp, visible artifact set, blocked subsystems, and whether the actor could have known the answer [2606.22737]. The track targets future-context leakage, cross-user leakage, subsystem access violations, role-boundary violations, and answers that are true in the full world state but invalid from the specified actor perspective [2606.22737].

The worked Perspective example asks whether Patty (`hr_ops`) could have known, as of 2026-02-24, about a design discussion involving Jax and Morgan [2606.22737]. Ground truth was `could_actor_have_known: false`, because the discussion existed in the `zoom_transcript` subsystem, which `hr_ops` could not access [2606.22737]. The agent searched plausibly, found meetings involving Jax and Morgan, saw Patty attended one, fetched four `zoom_transcript` artifacts, and concluded `could_actor_have_known: true` [2606.22737]. GroundEval treated every transcript fetch as an actor-gate violation and reported \(S_{\mathrm{ans}} = 0.000\), \(S_{\mathrm{traj}} = 0.384\), \(S_{\mathrm{adj}} = 0.230\), and a violation count of \(7\) [2606.22737].

### 5.3 Counterfactual

The Counterfactual track tests whether the agent identifies a valid cause-effect relationship rather than a merely plausible narrative [2606.22737]. Its configuration includes cause and effect event types, a maximum gap, premise and outcome templates, expected outcome change, and mechanism aliases [2606.22737]. Join conditions are crucial: cause and effect must share an identifier such as a ticket ID, incident ID, or source artifact, ruling out causal claims based only on temporal adjacency [2606.22737].

The track targets causal reversal, temporal adjacency mistaken for causality, missing cause or effect events, weak mechanism matching, and unsupported causal claims [2606.22737]. Answer scoring includes checks for outcome change, causal mechanism, cause event ID, effect event ID, causal direction, and actor overlap [2606.22737].

The worked Counterfactual example asks whether an external contact summary from GitHub on 2026-03-17 would have been written if Alex, Jax, and Hanna had not first opened the incident [2606.22737]. Ground truth was `outcome_changed: true` under the `incident_coordination` mechanism [2606.22737]. The agent fetched **ENG-263**, a Jira ticket whose causal chain field explicitly listed `slack_incidents_2026-03-17T11:21:00` among downstream artifacts, but never fetched that artifact [2606.22737]. Instead, it performed a keyword search for “contact summary,” found nothing, and concluded `no_causal_link` [2606.22737]. GroundEval assigned \(S_{\mathrm{ans}} = 0.062\), \(S_{\mathrm{traj}} = 0.637\), and \(S_{\mathrm{adj}} = 0.350\) [2606.22737].

Across the three tracks, GroundEval enforces several trajectory-level requirements: search before claiming absence, use only evidence available to the relevant actor at the relevant time, use the correct causal mechanism rather than a plausible one, cite or retrieve supporting evidence rather than merely producing plausible prose, and respect subsystem and role boundaries [2606.22737].

## 6. Empirical results, comparisons, and scope

GroundEval’s experiments are run on a corpus generated by **OrgForge**, chosen because it produces deterministic ground truth independent of LLM generation [2606.22737]. The dataset is a synthetic enterprise scenario with **9 subsystems**—Slack, Jira, Confluence, Git, email, Salesforce, Zendesk, Zoom, Datadog—**72 actors**, **8 roles**, **22,530 events** over 60 days, **25 causal link types**, **19 silence pair types**, and **96 questions** total: **30 Perspective**, **27 Counterfactual**, and **39 Silence** [2606.22737].

The evaluated model in both conditions was **DeepSeek-V4-Pro** at temperature 0 [2606.22737]. Two conditions were used: **Gated tool mode** and **Zero-shot, no-artifact mode** [2606.22737]. Reported answer and trajectory scores for gated runs were [2606.22737]:

| Track | Zero-shot answer | Gated answer | Gated trajectory |
|---|---:|---:|---:|
| Perspective | 0.200 | 0.214 | 0.637 |
| Counterfactual | 0.220 | 0.063 | 0.357 |
| Silence | 0.487 | 0.359 | 0.421 |

For gated runs, the paper reports unadjusted combined score, violation rate \(v\), and compliance-adjusted score \(S_{\mathrm{adj}}\) as follows [2606.22737]:

| Track | Unadjusted combined | \(v\) | \(S_{\mathrm{adj}}\) |
|---|---:|---:|---:|
| Perspective | 0.468 | 0.124 | 0.359 |
| Counterfactual | 0.210 | 0.000 | 0.210 |
| Silence | 0.403 | 0.000 | 0.403 |
| Overall | 0.369 | 0.026 | 0.350 |

The paper interprets the zero-shot/gated gap diagnostically: if zero-shot does too well, question phrasing may leak signal and the generator may need tightening [2606.22737]. More broadly, the results show that tool access does not guarantee valid evidence use [2606.22737].

GroundEval is positioned against several other evaluation paradigms. Against final-answer-only scoring, it argues that answer correctness cannot distinguish inaccessible evidence, future evidence, insufficient search, wrong causal direction, or irrelevant but plausible citations [2606.22737]. Against LLM-as-judge, it argues that the missing ingredient is not a better judge prompt but a machine-checkable state contract [2606.22737]. Compared with process and trajectory benchmarks, the framework extends trajectory evaluation to access control, temporal horizon, evidence visibility, causal grounding, and verified absence [2606.22737]. Compared with RAG and attribution evaluation, it asks whether the evidence path was valid under the state, not merely whether retrieved evidence supports the answer [2606.22737].

This places GroundEval in a broader lineage of evaluation frameworks that shift emphasis from final outputs to grounded verification, but with a notably different mechanism from adjacent work. “KGEval: Estimating Accuracy of Automatically Constructed Knowledge Graphs” [1610.06912] uses human-grounded fact verification amplified by inference over coupling constraints to estimate knowledge-graph accuracy, whereas GroundEval evaluates stateful agent trajectories against access, time, and causality contracts rather than estimating fact truth from sparse human labels. “GrandJury: A Collaborative Machine Learning Model Evaluation Protocol for Dynamic Quality Rubrics” [2508.02926] grounds evaluation in explicit rubrics, multi-rater human judgment, time decay, and traceability for settings without absolute ground truth, whereas GroundEval is intended for settings where correctness can be represented as state plus constraints and scored deterministically. “ORGEval: Graph-Theoretic Evaluation of LLMs in Optimization Modeling” [2510.27610] provides another judge-free design pattern, grounding model outputs into formal symbolic objects and checking structural equivalence by graph methods; this suggests a wider research movement toward evaluator designs that replace prose judgment with explicit verifiers when the domain admits a suitable formal intermediate representation.

GroundEval’s scope is intentionally narrow. It applies best where correctness can be represented as artifacts, events, actors, access rules, time horizons, and expected event relationships [2606.22737]. It is not intended for open-ended quality evaluation, creative tasks, or tasks whose correctness cannot be reduced to observable state [2606.22737]. The most important stated limitations are weaker observability in context mode, the practical burden of authoring the evaluation configuration, and the current focus on single-agent settings rather than multi-agent or longer-horizon evaluation [2606.22737].

In this sense, GroundEval is best understood not as a generic metric, but as a contract-based evaluation architecture for stateful agents. Its central claim is that, in environments with access control, temporal boundaries, and structured evidence dependencies, the decisive evaluation question is not whether an answer sounds right, but whether it was reached through the right artifacts, under the right permissions, before the right time boundary, with the required search, and through the correct structural mechanism [2606.22737].

Source: https://www.emergentmind.com/topics/groundeval