User Journey Coverage Score Metrics

• User Journey Coverage Score is a metric that quantifies how completely a system follows a predefined sequence of tasks, emphasizing strict trace accuracy and parameter correctness.
• It is designed to differentiate true business-compliant execution from superficial task completion by enforcing exact tool-call sequences and state adherence.
• The metric serves as both a strict evaluative principle and a family of domain-specific measures, adaptable to scenarios like customer support, web interaction, and process mining.

In current literature, User Journey Coverage Score can be understood as a contract-based measure of how completely a system traverses and satisfies a user journey. The term is introduced explicitly in JourneyBench, where a journey is a specific path through a business SOP graph and scoring requires exact agreement between the expected and actual tool-call sequence before parameter correctness is credited (Balaji et al., 2 Jan 2026). Related work broadens the same concern to browser interaction, event-log modeling, scenario libraries, travel planning, recommender systems, and documentation, where coverage is tied to requirement-induced states, held-out trace replayability, operational-design-domain coverage, or user-request fulfillment rather than to superficial task completion alone (Meng et al., 2 Jun 2026, Kobialka et al., 4 Apr 2026, Gelder et al., 2024, Qu et al., 10 Oct 2025).

1. Definition and conceptual boundaries

JourneyBench gives the clearest explicit definition. It treats a user journey as a path through a business standard operating procedure represented as a Directed Acyclic Graph, with nodes as tasks and edges as valid transitions under business logic. The User Journey Coverage Score then measures whether an agent executed the expected tool-call sequence for that journey and whether the parameters used in those calls were correct. The motivation is to distinguish true business-compliant execution from superficial task completion: an agent may appear to solve a customer problem while skipping required checks, violating dependencies, or proceeding after missing information or tool failures (Balaji et al., 2 Jan 2026).

Related papers define adjacent but non-identical coverage objects. WebRISE evaluates MLLM-generated web artifacts against requirement-induced states and transitions rather than local checkpoints; UXBench gates report generation on whether a browser agent has exercised required pages, viewports, controls, and states; the ADS scenario-coverage literature distinguishes intended-domain coverage from observed-data coverage; and process-mining work on user journeys treats coverage as whether observed or held-out traces belong to the language of a learned behavioral model (Meng et al., 2 Jun 2026, Wang et al., 15 Jun 2026, Gelder et al., 2024, Kobialka et al., 4 Apr 2026). This suggests that a User Journey Coverage Score is best interpreted not as a universal scalar, but as a family of domain-specific measures that ask whether the journey implied by a contract, log, or scenario model has actually been covered.

A recurrent boundary condition across the literature is that coverage is not the same as completeness. The ADS scenario paper explicitly separates coverage from completeness, noting that one can have high coverage while still omitting relevant detail, and WebRISE likewise states that contract coverage is bounded by specified requirements, generated test items, and available DOM/visual assertions rather than all possible behaviors (Gelder et al., 2024, Meng et al., 2 Jun 2026).

2. Strict workflow-adherence formulation

JourneyBench formalizes the strictest current version of a User Journey Coverage Score. For a single conversation, it defines Tool Call Accuracy as

$$TCA_{conv} = \begin{cases} \frac{\sum_{i=1}^{L} C_i}{\sum_{i=1}^{L} E_i} & \text{if } T_{act} = T_{exp} \ 0 & \text{otherwise} \end{cases}$$

and then defines the User Journey Coverage Score as

$$\text{UJCS} = \frac{1}{N} \sum_{k=1}^{N} TCA_{conv_k}.$$

Here, $T_{act}$ is the actual tool-call sequence, $T_{exp}$ the expected sequence, $L=|T_{exp}|$, $P_{act}^{(i)}$ and $P_{exp}^{(i)}$ the actual and expected parameter sets for tool call $i$, $C_i = |P_{act}^{(i)} \cap P_{exp}^{(i)}|$, and $E_i = |P_{exp}^{(i)}|$. The decisive property is the trace-equality gate: any missing, extra, or misordered tool call yields $$\text{UJCS} = \frac{1}{N} \sum_{k=1}^{N} TCA_{conv_k}.$$0, and parameter overlap is scored only after exact trace alignment has been established (Balaji et al., 2 Jan 2026).

This construction makes UJCS a path-adherence metric before it is a parameter-accuracy metric. It is therefore especially suited to policy-sensitive settings such as customer support, where prerequisite checks, task ordering, and stopping conditions matter as much as end-state resolution. JourneyBench evaluates 703 conversations in three domains and reports that the Dynamic-Prompt Agent significantly boosts policy adherence; one headline result is that GPT-4o-mini with DPA achieves an average UJCS of 0.649, outperforming GPT-4o with SPA at 0.564 (Balaji et al., 2 Jan 2026).

The strictness of this design is both its main strength and its main limitation. It sharply penalizes unreachable or noncompliant branches, but it also assumes a single expected trace per scenario and does not grant partial credit for alternative compliant traces unless those alternatives are explicitly represented.

3. State-transition and evidence-gated generalizations

WebRISE reframes journey coverage for interactive web artifacts as conformance to a requirement-induced state-transition space. For each task $$\text{UJCS} = \frac{1}{N} \sum_{k=1}^{N} TCA_{conv_k}.$$1, it defines an Interaction Contract Graph

$$\text{UJCS} = \frac{1}{N} \sum_{k=1}^{N} TCA_{conv_k}.$$2

where $$\text{UJCS} = \frac{1}{N} \sum_{k=1}^{N} TCA_{conv_k}.$$3 are stable, replayable, requirement-relevant observable states, $$\text{UJCS} = \frac{1}{N} \sum_{k=1}^{N} TCA_{conv_k}.$$4 are user-intent-driven transitions, $$\text{UJCS} = \frac{1}{N} \sum_{k=1}^{N} TCA_{conv_k}.$$5 are DOM and visual predicates, and $$\text{UJCS} = \frac{1}{N} \sum_{k=1}^{N} TCA_{conv_k}.$$6 maps requirements to transitions and checks. It then defines state reachability $$\text{UJCS} = \frac{1}{N} \sum_{k=1}^{N} TCA_{conv_k}.$$7, transition validity $$\text{UJCS} = \frac{1}{N} \sum_{k=1}^{N} TCA_{conv_k}.$$8, and requirement coverage $$\text{UJCS} = \frac{1}{N} \sum_{k=1}^{N} TCA_{conv_k}.$$9, with explicit and implicit requirement splits $T_{act}$0 and $T_{act}$1. Requirement satisfaction is strict: a requirement is satisfied only if all mapped checks pass (Meng et al., 2 Jun 2026).

This formulation shifts journey coverage away from isolated action evidence toward end-to-end state consistency. WebRISE’s motivating failures are not failed clicks but broken downstream semantics such as stale cart totals, unsynchronized pagination, or lost hidden state. Its defect-injection study reports detection of 16/25 injected state-related defects, versus 8/25 under a broad checkpoint-style criterion and 1/25 under a strict one, yielding $T_{act}$2 to $T_{act}$3 better detection of state errors. The benchmark also shows that visual quality is not a proxy for interaction correctness: for Qwen3.6-35B-A3B on Markdown, $T_{act}$4 but $T_{act}$5 and $T_{act}$6; even the strongest model reaches only $T_{act}$7 and $T_{act}$8 in its best modality (Meng et al., 2 Jun 2026).

UXBench reaches a related conclusion from the opposite direction: critique quality should be gated by evidence collection. It defines a browser trajectory $T_{act}$9 and maintains coverage-tracking sets for pages seen, viewports seen, controls done, and states seen. Unmet coverage is computed as the union of required goals not yet satisfied, and termination is allowed only when the unmet set is empty and there is no blocking failure. Its appendix formalizes an evidence-confidence output $T_{exp}$0: high when all required coverage goals are met and no blocking failure remains, med when page and viewport goals are met, and low otherwise (Wang et al., 15 Jun 2026).

Taken together, these two papers define a broader technical lineage for User Journey Coverage Score: coverage is not merely whether an endpoint is reached, but whether required states are reachable, user-intended transitions execute correctly, and collected evidence is sufficient to justify downstream judgments.

4. Coverage of observed behavior and intended domain

The process-mining and automata-learning literature treats journey coverage as language inclusion. In the user-journey setting, a trace $T_{exp}$1 is a sequential event sequence, an event log $T_{exp}$2 is a multiset of traces, and a learned transition system $T_{exp}$3 covers a journey when the trace belongs to the model language $T_{exp}$4. The paper compares process mining and automata learning, notes that directly follows systems include all log traces by construction, and identifies recall against ground truth and held-out trace acceptance as the most direct proxies for coverage. Its hybrid method switches between process mining and automata learning depending on log size and variant count, because sparse logs and well-distributed logs require different generalization biases (Kobialka et al., 4 Apr 2026).

The ADS scenario-coverage paper defines an orthogonal but highly relevant distinction: Coverage Type I asks whether collected scenarios cover all relevant aspects of the Operational Design Domain, while Coverage Type II asks whether collected scenarios cover all relevant aspects present in the driving data. It operationalizes these with tag-based coverage for ODD breadth and time-based, actor-based, and actor-over-time-based coverage for empirical data coverage. Crucially, it also distinguishes coverage from completeness and emphasizes that 100% coverage is not always necessary or meaningful, because some tag-category combinations may be structurally implausible or extremely rare (Gelder et al., 2024).

For User Journey Coverage Score design, these papers support a two-level view. One level measures whether the intended journey space has been modeled at all; the other measures whether the modeled space actually explains observed behavior. A score built only from one side risks either overfitting the contract or ignoring large regions of observed user activity.

5. Domain-specific variants and operational proxies

Several domain papers do not define UJCS explicitly but provide operational proxies for it.

Domain	Journey object	Closest coverage proxy
Travel planning	User-requested itinerary	Hard-gated validity plus Information Completeness and User Request Fulfillment (Qu et al., 10 Oct 2025)
Search ranking	Multi-step booking funnel	Milestone path $T_{exp}$5 plus negative milestones (Tan et al., 2023)
Sequence modeling for Airbnb	Guest event history	7-year non-view history, 21-day view history, 80/200 caps, and a 7-day attribution window covering over 90% of searches before booking (Zha et al., 17 Jun 2026)
Mobility sensing	Origin-to-destination trip capture	88% fully detected, 9% clipped, 3% not detected (Camilleri et al., 2019)
Developer documentation	Information journey	Four stages—Exploration, Understanding, Practice, Application—with content, organization, and maintenance ranked highest (Gao et al., 2023)

TripScore is especially important because it makes a strong gating claim: format must pass first, then commonsense. If format fails, $T_{exp}$6; if format passes but commonsense fails, the final reward is 0; only then do soft and preference components contribute, with real-world User Request Fulfillment receiving weight $T_{exp}$7. This implies that, in planning domains, mentioning requested elements does not constitute journey coverage unless the itinerary is structurally valid and executable (Qu et al., 10 Oct 2025).

Airbnb’s ranking papers model journeys as milestone structures rather than strict paths. Journey Ranker decomposes the positive funnel into click, long click, payment page, reservation request, booking, and uncancelled booking, while separately modeling rejection, cancellation by host, and cancellation by guest; JourneyFormer operationalizes historical-journey coverage through selective retention of heterogeneous event histories under production constraints. These works suggest a softer interpretation of journey coverage: not exact path compliance, but support for productive progression through meaningful milestones (Tan et al., 2023, Zha et al., 17 Jun 2026).

The documentation study adds a stage-based human-information perspective. It reports that 98.11% of respondents agreed that Exploration, Understanding, Practice, and Application roughly describe their learning process, and that documentation priorities rank as Content 3.28, Organization 3.12, and Maintenance 2.93. This implies that, for documentation systems, journey coverage is naturally expressed as stage coverage weighted by the quality dimensions developers value most (Gao et al., 2023).

6. Limitations, misconceptions, and current debates

A first misconception is that journey coverage is equivalent to endpoint success. JourneyBench rejects that assumption by zeroing conversations with missing, extra, or misordered tool calls even when an apparent solution is reached; WebRISE makes the same point for web interaction by showing that pages can appear responsive while violating state-consistency obligations (Balaji et al., 2 Jan 2026, Meng et al., 2 Jun 2026).

A second misconception is that broad reach implies effective coverage. In overlapping marketing journeys, raw reach and isolated A/B lift can be misleading because users are co-exposed to multiple journeys. The hierarchical causal lift model therefore decomposes pure and global effects under overlap, and reports that pure lifts are substantially larger than observed lifts: about $T_{exp}$8 for Offer, $T_{exp}$9 for Search, and $L=|T_{exp}|$0 for Flow. This supports an overlap-aware interpretation of coverage in which unique reach, shared reach, and interaction-adjusted marginal contribution must be separated (Pellegrini, 27 Apr 2026).

A third misconception is that any reachability proxy is equivalent to full journey completion. CovAgent improves Android activity coverage far beyond the “30% curse” by inferring activation conditions and generating dynamic instrumentation; however, its own limitations note that launching a target activity does not necessarily mean that all internal features are meaningfully initialized, and dynamic instrumentation can inflate apparent journey coverage relative to natural user operation (Minn et al., 29 Jan 2026).

Finally, the literature consistently treats coverage as relative to a model boundary. WebRISE covers self-contained front-end HTML rather than full production systems; the ADS scenario paper warns that coverage is not completeness; JourneyBench assumes accurate SOP graphs and deterministic expected traces; TripScore hard-gates validity before preference satisfaction; and the documentation study shows that official documentation is only one resource within a broader information ecology (Meng et al., 2 Jun 2026, Gelder et al., 2024, Qu et al., 10 Oct 2025, Gao et al., 2023).

User Journey Coverage Score is therefore best viewed as an evaluative principle rather than a single fixed metric. In its strictest form, it is exact trace adherence plus parameter correctness. In broader formulations, it becomes state reachability, transition validity, requirement satisfaction, evidence sufficiency, milestone support, or request fulfillment. Across these variants, the common idea is stable: a system covers a user journey only when it supports the right sequence, under the right constraints, with the right evidence, rather than merely producing a plausible local outcome.