Weighted Pathway Success Rate (WPSR)

• Weighted Pathway Success Rate (WPSR) is a metric that quantifies agent success by weighting multi-step tasks based on the number of atomic operations and cross-application complexity.
• It applies normalized difficulty scores to sequential tasks, ensuring precise failure attribution and robust evaluation of long-horizon planning and execution.
• Empirical comparisons show WPSR’s sensitivity to task complexity, revealing performance gaps that traditional unweighted metrics may overlook in LLM evaluations.

The Weighted Pathway Success Rate (WPSR) is a quantitative evaluation metric introduced in the NaturalGAIA benchmark to rigorously assess the capabilities of LLM-driven Graphical User Interface (GUI) agents on long-horizon, multi-application tasks. WPSR accounts for complexity and real-world task diversity by weighting agent successes according to both the number of atomic steps and the cross-application breadth of the pathways traversed. This ensures that the metric reflects holistic, high-difficulty planning and execution, while maintaining robustness against inflation by trivial or short single-application benchmarks (Zheng et al., 2 Aug 2025).

1. Definition and Formal Specification

WPSR is constructed atop the decomposition of GUI tasks into “Causal Pathways” (CPs). Each CP is a strictly ordered sequence of atomic tasks, each defined as a deterministic, verifiable operation within a single application. Pathways are indexed by $j$ (the CP) and $i$ (task instance within the CP).

For every task instance, a raw difficulty score $D_{j,i}$ is assigned:

• $D_{j,i}$ increases with both the number of atomic steps in the pathway and the count of distinct applications spanned.
• These raw scores are normalized to form weights $w_{j,i}$:

$w_{j,i} = \frac{D_{j,i}}{\sum_{k,\ell} D_{k,\ell}}$

ensuring $\sum_{j,i} w_{j,i} = 1$.

The outcome for each task is denoted as

$$\mathbb{S}_{\text{task}(j,i)} = \begin{cases} 1, & \text{if the agent completes the entire pathway \(j\) on instance \(i\)}\ 0, & \text{otherwise} \end{cases}$$

The WPSR aggregates these values as:

$$\boxed{ \text{WPSR} = \sum_{j,i} w_{j,i}\, \mathbb{S}_{\text{task}(j,i)} }$$

This directly penalizes pathway failures at any step and proportionally emphasizes harder tasks.

2. Decomposition of Causal Pathways and Its Impact

Causal Pathways delineate task structure as a chain of atomic, causally linked operations. Each node is atomic by design, both in minimality (single deterministic operation) and verifiability (clear input-output signature). Pathways with more steps and greater cross-application scope obtain higher difficulty scores, reflecting higher demands on agent capabilities:

Pathway ID	Atomic Steps	Distinct Apps	Raw Difficulty $D$
A	2	1	2
B	2	1	2
C	4	2	4

If an agent fails any step, the CP “collapses”; the process is fully transparent and allows unambiguous attribution of failure.

3. Calculation: Example and Weight Assignment

Assume tasks as above: A ($i$0), B ($i$1), C ($i$2). If an agent succeeds on A and C but not B, the weights and WPSR are calculated:

• Total difficulty: $i$3
• Weights: $i$4, $i$5, $i$6
• Task outcomes: Succeeds on A and C ($i$7), fails B ($i$8)

Then,

$i$9

By contrast, the unweighted success rate is $D_{j,i}$0, underscoring WPSR’s amplification of success on the most complex pathway.

4. Key Properties and Analytical Benefits

WPSR offers several advantages over traditional, unweighted metrics:

• Difficulty Awareness: Harder, longer, and cross-application pathways are weighted more heavily, ensuring the metric is calibrated to the challenge posed by each task.
• Holistic Long-Horizon Evaluation: WPSR directly reflects integration of perception, planning, and execution across many apps, thus capturing end-to-end competencies beyond atomic skill.
• Robustness to Benchmark Composition: Insertion of trivial, short CPs does not significantly alter scores, as these are down-weighted proportionally. Agents cannot exploit benchmark weaknesses to achieve artificially inflated results.
• Failure Attribution: Collapsed pathways precisely localize weaknesses, enabling targeted analysis of agent limitations (Zheng et al., 2 Aug 2025).

5. Empirical Usage and Interpretive Guidelines

In NaturalGAIA evaluations, WPSR is reported alongside unweighted Success Rate (SR), MATCR, and p-ATSR. Table 1 of (Zheng et al., 2 Aug 2025) reveals substantive findings:

Model	WPSR	SR
Claude-sonnet-4	34.6%	42.9%
Qwen2.5-VL-7B (pre-RFT)	3.3%	[not shown]
Qwen2.5-VL-7B (RFT)	10.8%	[not shown]

The notable disparity between WPSR and SR, especially for top-performing models, indicates the degree to which traditional success rates mask weaknesses in complex, multi-step workflows. This suggests that WPSR is more sensitive to “ceiling effects” in smaller LLMs and penalizes partial competence on holistic, long-horizon tasks.

6. Comparative and Methodological Context

WPSR is distinctive in its dual normalization by both pathway length and cross-application complexity. Unlike metrics that focus on single-step accuracy or aggregate only per-task success, WPSR enforces an authenticity in multi-step, multi-app adversarial benchmarking. Its adoption in the NaturalGAIA suite sets a reproducibility norm for LLM agent evaluation, while its automated verification pipeline leverages the atomic structure of CPs for systematic scaling and attribution.

A plausible implication is that WPSR or metrics with comparable design principles may become foundational in the assessment of future, more general-purpose autonomous agents operating in complex, structured interaction domains (Zheng et al., 2 Aug 2025).