---
title: Weighted Pathway Success Rate (WPSR)
url: https://www.emergentmind.com/topics/weighted-pathway-success-rate-wpsr
type: topic
---

# Weighted Pathway Success Rate (WPSR)

The Weighted Pathway Success Rate (WPSR) is a quantitative evaluation metric introduced in the NaturalGAIA benchmark to rigorously assess the capabilities of Large Language Model (LLM)-driven Graphical User Interface (GUI) agents on long-horizon, multi-application tasks. WPSR accounts for complexity and real-world task diversity by weighting agent successes according to both the number of atomic steps and the cross-application breadth of the pathways traversed. This ensures that the metric reflects holistic, high-difficulty planning and execution, while maintaining robustness against inflation by trivial or short single-application benchmarks [2508.01330].

## 1. Definition and Formal Specification

WPSR is constructed atop the decomposition of GUI tasks into “Causal Pathways” (CPs). Each CP is a strictly ordered sequence of atomic tasks, each defined as a deterministic, verifiable operation within a single application. Pathways are indexed by \(j\) (the CP) and \(i\) (task instance within the CP).

For every task instance, a raw difficulty score \(D_{j,i}\) is assigned:
- \(D_{j,i}\) increases with both the number of atomic steps in the pathway and the count of distinct applications spanned.
- These raw scores are normalized to form weights \(w_{j,i}\):
  $$
  w_{j,i} = \frac{D_{j,i}}{\sum_{k,\ell} D_{k,\ell}}
  $$
  ensuring \(\sum_{j,i} w_{j,i} = 1\).

The outcome for each task is denoted as
$$
\mathbb{S}_{\text{task}(j,i)} = \begin{cases}
1, & \text{if the agent completes the entire pathway \(j\) on instance \(i\)}\\
0, & \text{otherwise}
\end{cases}
$$

The WPSR aggregates these values as:
$$
\boxed{
\text{WPSR} = \sum_{j,i} w_{j,i}\, \mathbb{S}_{\text{task}(j,i)}
}
$$
This directly penalizes pathway failures at any step and proportionally emphasizes harder tasks.

## 2. Decomposition of Causal Pathways and Its Impact

Causal Pathways delineate task structure as a chain of atomic, causally linked operations. Each node is atomic by design, both in minimality (single deterministic operation) and verifiability (clear input-output signature). Pathways with more steps and greater cross-application scope obtain higher difficulty scores, reflecting higher demands on agent capabilities:

| Pathway ID | Atomic Steps | Distinct Apps | Raw Difficulty \(D\) |
|------------|--------------|---------------|----------------------|
| A          | 2            | 1             | 2                    |
| B          | 2            | 1             | 2                    |
| C          | 4            | 2             | 4                    |

If an agent fails any step, the CP “collapses”; the process is fully transparent and allows unambiguous attribution of failure.

## 3. Calculation: Example and Weight Assignment

Assume tasks as above: A (\(D=2\)), B (\(D=2\)), C (\(D=4\)). If an agent succeeds on A and C but not B, the weights and WPSR are calculated:

- Total difficulty: \(2 + 2 + 4 = 8\)
- Weights: \(w_A = 0.25\), \(w_B = 0.25\), \(w_C = 0.5\)
- Task outcomes: Succeeds on A and C (\(1\)), fails B (\(0\))

Then,
$$
\text{WPSR} = 0.25 \times 1 + 0.25 \times 0 + 0.5 \times 1 = 0.75
$$

By contrast, the unweighted success rate is \(2/3 \approx 0.67\), underscoring WPSR’s amplification of success on the most complex pathway.

## 4. Key Properties and Analytical Benefits

WPSR offers several advantages over traditional, unweighted metrics:

- **Difficulty Awareness**: Harder, longer, and cross-application pathways are weighted more heavily, ensuring the metric is calibrated to the challenge posed by each task.
- **Holistic Long-Horizon Evaluation**: WPSR directly reflects integration of perception, planning, and execution across many apps, thus capturing end-to-end competencies beyond atomic skill.
- **Robustness to Benchmark Composition**: Insertion of trivial, short CPs does not significantly alter scores, as these are down-weighted proportionally. Agents cannot exploit benchmark weaknesses to achieve artificially inflated results.
- **Failure Attribution**: Collapsed pathways precisely localize weaknesses, enabling targeted analysis of agent limitations [2508.01330].

## 5. Empirical Usage and Interpretive Guidelines

In NaturalGAIA evaluations, WPSR is reported alongside unweighted Success Rate (SR), MATCR, and p-ATSR. Table 1 of [2508.01330] reveals substantive findings:

| Model                | WPSR   | SR        |
|----------------------|--------|-----------|
| Claude-sonnet-4      | 34.6%  | 42.9%     |
| Qwen2.5-VL-7B (pre-RFT) | 3.3%   | [not shown]|
| Qwen2.5-VL-7B (RFT)  | 10.8%  | [not shown]|

The notable disparity between WPSR and SR, especially for top-performing models, indicates the degree to which traditional success rates mask weaknesses in complex, multi-step workflows. This suggests that WPSR is more sensitive to “ceiling effects” in smaller LLMs and penalizes partial competence on holistic, long-horizon tasks.

## 6. Comparative and Methodological Context

WPSR is distinctive in its dual normalization by both pathway length and cross-application complexity. Unlike metrics that focus on single-step accuracy or aggregate only per-task success, WPSR enforces an authenticity in multi-step, multi-app adversarial benchmarking. Its adoption in the NaturalGAIA suite sets a reproducibility norm for LLM agent evaluation, while its automated verification pipeline leverages the atomic structure of CPs for systematic scaling and attribution.

A plausible implication is that WPSR or metrics with comparable design principles may become foundational in the assessment of future, more general-purpose autonomous agents operating in complex, structured interaction domains [2508.01330].

Source: https://www.emergentmind.com/topics/weighted-pathway-success-rate-wpsr