---
title: User Journey Coverage Score (UJCS)
url: https://www.emergentmind.com/topics/user-journey-coverage-score-ujcs
type: topic
---

# User Journey Coverage Score (UJCS)

The User Journey Coverage Score (UJCS) is a family of automated evaluation metrics designed to quantify the adherence of agents or generative systems to specified, multi-step processes (user journeys) over structured domains such as customer support, workflow automation, and use case recovery. UJCS accommodates both policy-aware agent evaluation and use case extraction by rigorously scoring system outputs against a reference, often graph-based, specification. By enforcing stepwise, parameter-sensitive, and coverage-aware correctness, UJCS enables granular assessments of functional compliance and process alignment. Notably, UJCS has been instantiated in two domains: policy execution for customer support LLM agents [2601.00596] and use case recovery from software artifacts [2512.13360], both featuring domain-appropriate mathematical formalizations unified by a focus on strict sequential and coverage fidelity.

## 1. Formal Definitions and Metric Variants

There exist two primary variants of UJCS:

1. **Policy Compliance for LLM Agents** [2601.00596]: Here, UJCS quantifies the fidelity with which an agent executes a standard operating procedure (SOP), represented by a task graph, over simulated conversations.

2. **Use Case Recovery Coverage** [2512.13360]: UJCS aggregates multiple aspects of alignment when comparing recovered use cases to ground truth, emphasizing actor, naming, path structure, and coverage.

### 1.1. Policy-Adherence UJCS

Given an SOP-encoded expected trace
$$
T_{\mathrm{exp}} = [(t_1, P_{\mathrm{exp}}^{(1)}), \ldots, (t_L, P_{\mathrm{exp}}^{(L)})]
$$
and an agent-produced actual trace
$$
T_{\mathrm{act}} = [(t'_1, P_{\mathrm{act}}^{(1)}), \ldots, (t'_{L'}, P_{\mathrm{act}}^{(L')})],
$$
UJCS is computed by:

- **Tool Call Accuracy per conversation:**
  $$
  TCA_{\mathrm{conv}} =
  \begin{cases}
    \dfrac{\sum_{i=1}^L |P_{\mathrm{act}}^{(i)} \cap P_{\mathrm{exp}}^{(i)}|}{\sum_{i=1}^L |P_{\mathrm{exp}}^{(i)}|} & \text{if}\ T_{\mathrm{act}} = T_{\mathrm{exp}} \\
    0 & \text{otherwise}
  \end{cases}
  $$
- **UJCS Aggregation:**
  $$
  \mathrm{UJCS} = \frac{1}{N} \sum_{k=1}^N TCA_{\mathrm{conv}_k}
  $$

### 1.2. Use Case Recovery UJCS

UJCS is defined as the equally-weighted mean of four sub-scores:

- Actor correctness $(\mathrm{Acc}_A)$
- Name accuracy $(\mathrm{Acc}_N)$
- Path fidelity $(\mathrm{Acc}_P)$
- Behavioral coverage $(\mathrm{BC})$

Let $U$ be the set of reference use cases and $M$ the number of missed use cases. Then:

- **Omission rate:** $\mathrm{OR} = M / U$
- **Behavioral coverage:** $\mathrm{BC} = 1 - \mathrm{OR}$
- **Sub-score aggregation:**
    $$
    \mathrm{UJCS} = \frac{\overline{\mathrm{Acc}_A} + \overline{\mathrm{Acc}_N} + \overline{\mathrm{Acc}_P} + \mathrm{BC}}{4}
    $$
  with mean values taken across matched use cases.

## 2. Scoring Dimensions and Mathematical Details

UJCS metrics operate by enforcing both stepwise and holistic system compliance across policy and structure dimensions.

### 2.1. Tool Trace Alignment and Parameter Accuracy

UJCS for policy execution is highly sensitive to trace order and parameter matching. The metric assigns zero if the agent skips, reorders, or misnames any tool call; otherwise, it computes the fraction of correctly supplied parameters, allowing fine-grained error attribution [2601.00596].

### 2.2. Semantic and Structural Alignment

In use case recovery:
- **Actor correctness ($\mathrm{Acc}_A$):** Weighted combination of SBERT-based semantic similarity and role taxonomy alignment.
- **Name accuracy ($\mathrm{Acc}_N$):** Cosine similarity over verb and noun phrases.
- **Path fidelity ($\mathrm{Acc}_P$):** Jaccard index over directory segments.
- **Behavioral coverage ($\mathrm{BC}$):** Proportion of ground-truth use cases matched [2512.13360].

Weighting for each sub-score, as well as detailed computation, is precisely defined in the original sources.

## 3. Interpretation and Diagnostic Guidance

UJCS scores span the unit interval $[0, 1]$, where higher values indicate stronger adherence or coverage.

| UJCS Range      | Interpretation                                          |
|-----------------|--------------------------------------------------------|
| 1.0             | Perfect compliance: no step skipped, reordered, or mis-parameterized |
| 0.8–1.0         | Strong adherence, minor slips                          |
| 0.5–0.8         | Moderate compliance, partial coverage                  |
| <0.5            | Poor adherence, substantial errors or omissions        |

A plausible implication is that UJCS thresholds (e.g., 0.9) can act as cutoffs for deployment readiness, as suggested in business-facing scenarios [2601.00596], while for recovery tasks, diminishing UJCS strongly correlates with both process and semantic drift [2512.13360].

## 4. Methodological Workflow

The computation of UJCS requires deterministic ground-truth modeling, structured output logging, and aggregation logic:

1. **Policy Evaluation:**
   - Encode the SOP as a DAG with tasks, inputs, and branching.
   - Generate reference traces by BFS enumeration.
   - Simulate agent-user interaction and log tool calls.
   - Compute per-conversation $TCA_{\mathrm{conv}}$.
   - Average across reflections to obtain UJCS [2601.00596].

2. **Use Case Recovery:**
   - For each ground-truth use case, match to the most similar candidate.
   - Compute $\mathrm{Acc}_A$, $\mathrm{Acc}_N$, $\mathrm{Acc}_P$.
   - Aggregate omission information into $\mathrm{BC}$.
   - Form UJCS via arithmetic mean of sub-scores [2512.13360].

Automation of trace extraction, branch condition simulation, and systematic error analysis are procedural recommendations.

## 5. Strengths, Limitations, and Cross-Domain Properties

UJCS enforces strict sequence and parameter validity, capturing multi-step logic and branching in complex SOPs [2601.00596], and multidimensional fidelity in use case recovery [2512.13360]. Its design achieves:

**Strengths:**
- High sensitivity to full-path adherence.
- Capability to dissect errors at parameter, action, or abstraction level.
- Applicability to deterministic workflow scenarios.

**Limitations:**
- Unforgiving to minor reordering or synonymous variants—trace misalignment results in a zero score.
- Ignores aspects unrelated to structural compliance, such as dialog quality or recovery from errors.
- Assumes deterministic conditions; substantial real-world nondeterminism may require adaptive extensions.
- In use case extraction, high domain specificity and codebase modularity can depress UJCS by increasing omission and misalignment rates [2512.13360].

This suggests UJCS is best used in concert with metrics targeting complementary desiderata such as conversational quality or user satisfaction.

## 6. Practical Application and Domain Insights

For policy-adherence evaluation, deployment requires: (i) reliable SOP graph construction; (ii) exhaustive generation of valid user journeys; (iii) deterministic simulation of agent interactions; and (iv) comprehensive, structured agent output logging [2601.00596]. Use case recovery demands multi-stage semantic, syntactic, and behavioral matching, with acknowledged sensitivity to abstraction variance and domain complexity [2512.13360].

Empirically, well-structured, shallow task domains consistently yield high UJCS (≈ 0.8–0.9); deep, multi-module, or domain-specific projects typically realize significantly lower scores (0.3–0.6) due to increased path and role ambiguity [2512.13360].

UJCS thresholds can serve as regression gates, deployment criteria, or drift monitors. It is recommended to validate SOP or reference-set accuracy whenever business or system changes occur.

## 7. Metric Comparisons and Future Directions

UJCS extends beyond conventional task-completion and tool-selection metrics by demanding stepwise and parameter fidelity rather than goal-only checks [2601.00596]. Tool Trace Alignment operates as a binary precursor, while UJCS admits partial parameter correctness, admitting finer error granularity. In current literature, no formal correlation between UJCS and external subjective metrics (e.g., conversational prowess) has been established.

Future refinement directions may include adaptation for nondeterministic environments, incorporation of partial trace equivalence, or hybridization with quality-focused and human-centric ratings, particularly for mixed-initiative or dialog-mediated workflows.

Source: https://www.emergentmind.com/topics/user-journey-coverage-score-ujcs