Weighted Efficiency Score (WES) Analysis

• Weighted Efficiency Score (WES) is a flexible metric defined differently across fields, integrating domain-specific weights with a base efficiency measure such as task latency or energy per user.
• It applies tailored weighting mechanisms to prioritize outcomes like step efficiency in agents, energy efficiency in MIMO systems, forecast quality, or anomaly severity ranking.
• Empirical studies show that WES-based metrics significantly influence system design and optimization, highlighting trade-offs between success rates, resource usage, and user fairness.

Weighted Efficiency Score (WES) is not a single universally standardized metric across the recent arXiv literature. The term appears explicitly in benchmarking computer-use agents, where it denotes a success-weighted measure of step efficiency relative to human reference trajectories (Abhyankar et al., 19 Jun 2025). Closely related literatures use different names for structurally similar ideas: weighted-sum energy efficiency (WSEE) in user-centric uplink cell-free massive MIMO (Kim et al., 10 Feb 2025), weighted proper scoring rules for emphasizing selected outcome regions in probabilistic forecasting (Allen, 2023), and weighted sums of pairwise AUROCs for severity-aware anomaly ranking (Yi et al., 2024). In decision analysis and expert aggregation, by contrast, “efficiency” refers to perturbation stability or Pareto efficiency of weight vectors rather than to a single scalar weighted score (Kadenko et al., 2019, Bozóki et al., 2016).

1. Terminological scope and principal usages

Across these literatures, WES designates either an explicit metric or a natural shorthand for a weighted efficiency functional whose base object differs by domain: task trajectories, per-user energy efficiencies, probabilistic forecast scores, pairwise ranking performance, or weight-vector approximations. The common element is not a shared formula but the insertion of application-specific weights into an efficiency or quality criterion.

Literature	Exact named metric	Core weighted object
Computer-use agents	Weighted Efficiency Score (WES)	Success adjusted by human-normalized step efficiency and failure budget usage
Cell-free massive MIMO	Weighted-sum energy efficiency (WSEE)	Weighted sum of individual UE energy efficiencies
Probabilistic forecasting	Weighted scoring rules	Proper scoring rules modified by outcome weights $w$ or chaining functions $v$
Anomaly severity evaluation	WS-AUROC	Weighted sum of pairwise AUROCs across severity levels
Expert aggregation / PCM analysis	No formal WES	Stability- or Pareto-based efficiency of aggregated weights

A recurrent misconception is that “weighted efficiency score” names a single transportable metric. The cited work does not support that reading. Instead, each field binds “efficiency” to a different primitive: temporal efficiency in agent trajectories, energy efficiency in wireless systems, ranking efficiency in severity ordering, forecast quality in target outcome regions, or nondominance/stability in decision weights (Abhyankar et al., 19 Jun 2025, Kim et al., 10 Feb 2025, Yi et al., 2024, Allen, 2023, Kadenko et al., 2019, Bozóki et al., 2016).

2. Formal WES in computer-use agent benchmarking

The explicit formal use of WES appears in "OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents" (Abhyankar et al., 19 Jun 2025). There, WES is designed for computer-use agents (CUAs) whose practical bottleneck is end-to-end latency rather than raw benchmark success. The paper studies efficiency through latency, trajectory length, and the new Weighted Efficiency Score, and motivates step-based efficiency by reporting that planning and reflection account for 75\% to 94\% of total latency and that each successive step can take 3× longer than steps at the beginning of a task.

For a task $t$, the paper defines a success-side term and a failure-side term: $$\text{WES}^+ = \sum_t^n r_t \left(\frac{t_{exp}}{t_{actual}}\right), \qquad \text{WES}^- = \sum_t^n -(1-r_t)\left(\frac{t_{actual}}{S}\right).$$ Here $r_t \in \{0,1\}$ is the task success indicator, $t_{exp}$ is the expected number of human-performed steps from OSWorld-Human, $t_{actual}$ is the number of agent-generated steps, and $S$ is the maximum number of allowed steps. The success term rewards trajectories that are close to the human reference length; the failure term penalizes unsuccessful trajectories according to the fraction of the step budget they consume. The text states that $\text{WES}^+$ ranges from $0$ to $v$0 and $v$1 from $v$2 to $v$3. This suggests that the operational score is averaged over tasks, even though the displayed equations omit the normalization factor.

OSWorld-Human supplies the crucial reference quantity $v$4. It is a manually annotated version of OSWorld with 369 examples, the same number as the original benchmark, and provides two human references for each task: a single-action trajectory, in which each action is a step, and a grouped-action trajectory, in which multiple actions executable from the same visual observation are grouped. The grouped reference is stricter because it assumes fewer observe-plan-act cycles.

Empirically, the metric discounts success rates by inefficiency. The paper reports that even the highest-scoring agents on OSWorld take 1.4–2.7× more steps than necessary. It also notes that the best-performing agent on OSWorld attains 42.5\% success but only 17.4\% on the strictest grouped $v$5 metric. This is not merely a restatement of accuracy: $v$6 is explicitly a success-weighted efficiency metric, while $v$7 captures how much budget failed trajectories consume.

The metric has several clear methodological implications. It assumes that step count is a useful proxy for latency, because more steps trigger more large-model planning and reflection calls and because later steps are slower as prompt history grows. It also assumes that human trajectories provide a meaningful efficiency reference, but specifically a minimal humanly-perceived reference rather than a formal shortest-path optimum. The paper is explicit that $v$8 can favor systems that fail quickly, so it must be interpreted jointly with $v$9 rather than as a competence measure in isolation (Abhyankar et al., 19 Jun 2025).

3. Weighted-sum energy efficiency as a WES analogue in wireless systems

In user-centric uplink cell-free massive MIMO, the closest formal analogue to WES is weighted-sum energy efficiency (WSEE) rather than a metric literally named WES (Kim et al., 10 Feb 2025). The paper defines individual uplink energy efficiency for user equipment (UE) $t$0 as

$t$1

with uplink spectral efficiency

$t$2

and UE-side power consumption

$t$3

The aggregate weighted metric is

$t$4

The weights $t$5 are UE-specific design parameters. They encode priority, heterogeneous power budgets, minimum-rate requirements, PA efficiencies, circuit powers, and service differentiation. The paper is explicit that these weights are given, not optimized; the optimization variables are the transmit-power vector $t$6 and LSFD combining matrix $t$7, once weights and AP-UE associations are fixed.

The resulting WSEE maximization problem is a nonconcave multiple-ratio fractional programming problem with nested SINR fractions and inter-user interference coupling. The paper studies two algorithmic approaches: a Dinkelbach-like transform and a nested quadratic transform (QT) applied twice consecutively. Its main structural statement is Proposition 1, which proves that the transformed component $t$8 is concave in $t$9 for fixed $$\text{WES}^+ = \sum_t^n r_t \left(\frac{t_{exp}}{t_{actual}}\right), \qquad \text{WES}^- = \sum_t^n -(1-r_t)\left(\frac{t_{actual}}{S}\right).$$0, making the power-control subproblem globally solvable by CVX or interior-point methods. The nested-QT alternating-optimization algorithm is then shown to improve the objective monotonically and to converge to a local optimum.

The paper distinguishes WSEE from weighted-global energy efficiency (WGEE),

$$\text{WES}^+ = \sum_t^n r_t \left(\frac{t_{exp}}{t_{actual}}\right), \qquad \text{WES}^- = \sum_t^n -(1-r_t)\left(\frac{t_{actual}}{S}\right).$$1

on the ground that WSEE preserves individual-UE efficiency whereas WGEE can mask very poor performance of some users. Numerically, for $$\text{WES}^+ = \sum_t^n r_t \left(\frac{t_{exp}}{t_{actual}}\right), \qquad \text{WES}^- = \sum_t^n -(1-r_t)\left(\frac{t_{actual}}{S}\right).$$2, nested-QT improves WSEE by about 18\% over the full-power baseline and about 10\% over the Dinkelbach-like scheme at convergence. As $$\text{WES}^+ = \sum_t^n r_t \left(\frac{t_{exp}}{t_{actual}}\right), \qquad \text{WES}^- = \sum_t^n -(1-r_t)\left(\frac{t_{actual}}{S}\right).$$3 increases from $$\text{WES}^+ = \sum_t^n r_t \left(\frac{t_{exp}}{t_{actual}}\right), \qquad \text{WES}^- = \sum_t^n -(1-r_t)\left(\frac{t_{actual}}{S}\right).$$4 to $$\text{WES}^+ = \sum_t^n r_t \left(\frac{t_{exp}}{t_{actual}}\right), \qquad \text{WES}^- = \sum_t^n -(1-r_t)\left(\frac{t_{actual}}{S}\right).$$5, average EE and SE of high-priority UEs increase, while those of low-priority UEs decrease. Thus, in this domain, a weighted efficiency score is not primarily a fairness index; it is a priority-aware control objective (Kim et al., 10 Feb 2025).

4. Outcome-weighted evaluation in probabilistic forecasting

"Weighted scoringRules: Emphasising Particular Outcomes when Evaluating Probabilistic Forecasts" does not define a metric named WES, but it provides a general theoretical framework for constructing weighted forecast-evaluation scores that are highly relevant to any WES-like quantity (Allen, 2023). The starting point is a proper scoring rule

$$\text{WES}^+ = \sum_t^n r_t \left(\frac{t_{exp}}{t_{actual}}\right), \qquad \text{WES}^- = \sum_t^n -(1-r_t)\left(\frac{t_{actual}}{S}\right).$$6

with lower scores preferred, and propriety defined by

$$\text{WES}^+ = \sum_t^n r_t \left(\frac{t_{exp}}{t_{actual}}\right), \qquad \text{WES}^- = \sum_t^n -(1-r_t)\left(\frac{t_{actual}}{S}\right).$$7

for all $$\text{WES}^+ = \sum_t^n r_t \left(\frac{t_{exp}}{t_{actual}}\right), \qquad \text{WES}^- = \sum_t^n -(1-r_t)\left(\frac{t_{actual}}{S}\right).$$8. The paper then introduces two major weighted constructions.

The first is outcome-weighted scoring rules, which directly weight the score by a nonnegative function $$\text{WES}^+ = \sum_t^n r_t \left(\frac{t_{exp}}{t_{actual}}\right), \qquad \text{WES}^- = \sum_t^n -(1-r_t)\left(\frac{t_{actual}}{S}\right).$$9. For logarithmic scores, the paper gives the conditional likelihood score and censored likelihood score: $r_t \in \{0,1\}$0

$r_t \in \{0,1\}$1

If $r_t \in \{0,1\}$2, CoLS evaluates performance only when $r_t \in \{0,1\}$3, while CeLS also scores whether the forecast assigns appropriate probability mass to the event $r_t \in \{0,1\}$4.

The second is threshold-weighted scoring rules, which transform forecasts and observations via a chaining function $r_t \in \{0,1\}$5. For the threshold-weighted CRPS,

$r_t \in \{0,1\}$6

where $r_t \in \{0,1\}$7 satisfies

$r_t \in \{0,1\}$8

This formulation emphasizes errors after the outcome scale has been stretched or compressed by the weighting scheme. The paper extends the same logic to multivariate kernel scores, including weighted versions of CRPS, energy score, variogram score, and MMDS.

The weighting functions can be threshold-based, smooth, region-based, or user-specified. Examples given in the paper include

$r_t \in \{0,1\}$9

and multivariate region indicators

$t_{exp}$0

For threshold-weighted rules, common chaining functions include

$t_{exp}$1

with componentwise multivariate extensions.

The paper is careful about theory and implementation. Some weight or chaining choices can destroy propriety, though the functions considered in the paper remain proper, albeit not always strictly proper. Outcome-weighted scores additionally require $t_{exp}$2; if an indicator weight singles out a rare set to which the forecast assigns zero probability, the score may be undefined. In finite predictive samples, the paper recommends weighted CRPS over weighted log scores because the latter rely on kernel density estimation and are sensitive to bandwidth choice, especially in tails.

The empirical case studies show how weighting changes interpretation. In precipitation forecasting for Innsbruck, the ordinary CRPS gave nearly identical post-processing performance for logistic, Gaussian, and Student’s $t_{exp}$3 models, but threshold weighting toward high precipitation magnified their advantage over the raw ensemble. In Bayesian forecasts of US GDP growth, outcome-weighted CRPS and CoLS were zero in quarters with positive realized growth when $t_{exp}$4, whereas threshold-weighted CRPS and CeLS remained nonzero because they still reflected missed-event and false-alarm behavior. This suggests that a WES built from weighted proper scores must decide whether “efficiency” means conditional performance in a target region or joint performance on event occurrence and within-region sharpness (Allen, 2023).

5. Severity-weighted ranking in anomaly score distributions

In anomaly detection with discrete severity levels, the relevant WES-like construct is WS-AUROC, the weighted sum of pairwise AUROCs across severity levels (Yi et al., 2024). The paper defines

$t_{exp}$5

where $t_{exp}$6 denotes the normal condition, $t_{exp}$7 denote anomaly severity levels, $t_{exp}$8 is the anomaly-score distribution for level $t_{exp}$9, $t_{actual}$0 is the AUROC between $t_{actual}$1 and $t_{actual}$2, and the nonnegative weights $t_{actual}$3 sum to $t_{actual}$4.

The point of the construction is that standard AUROC evaluates normal-versus-anomaly discrimination but does not test whether more severe anomalies receive higher scores than less severe ones. WS-AUROC instead aggregates all pairwise ranking tasks: normal versus each severity level and severity versus severity. As a result, it is a threshold-independent measure of severity-aware ranking quality.

The paper studies three weighting schemes. Uniform weighting assigns

$t_{actual}$5

which makes WS-AUROC the arithmetic mean of all pairwise AUROCs. Index-difference weighting uses

$t_{actual}$6

so pairs farther apart in ordinal severity receive greater weight. Physics-based weighting uses

$t_{actual}$7

where $t_{actual}$8 is the physical quantity associated with severity level $t_{actual}$9, such as mass, displacement, or defect length. The paper reports that the physics-based method was the most sensitive.

The chief interpretive result is that AUROC and WS-AUROC can diverge sharply. A detector may obtain AUROC near $S$0 by separating anomalies from normal samples, yet still assign higher scores to mild anomalies than to severe anomalies. The paper highlights such a case for BPFI, where AUROC remains high but WS-AUROC falls because anomaly severity distributions are reversed or poorly ordered. In this literature, a WES-like metric therefore measures ranking efficiency over ordered severity levels, not energy use, not latency, and not forecast calibration (Yi et al., 2024).

6. Efficiency as robustness or Pareto optimality in decision analysis

A different branch of the literature uses “efficiency” in a decision-theoretic rather than weighted-sum sense. In "Comparing Efficiency of Expert Data Aggregation Methods," efficiency is defined by stability under perturbations of the input data rather than by closeness to a benchmark score (Kadenko et al., 2019). The paper studies combinatorial aggregation of pairwise-comparison data through spanning tree enumeration. In the ordinary method, spanning-tree-derived priority vectors are averaged equally; in the modified method, they are weighted by ratings $S$1 that depend on expert competence coefficients, scale informativeness, and penalties for inconsistency or incompatibility between reconstructed and observed comparison structures. The core efficiency indicator is

$S$2

the maximum relative deviation of estimated weights from true weights under controlled perturbations. On real expert data the comparison is not decisive, but in the simulation-based framework the weighted spanning tree enumeration method is reported to be more stable and therefore more efficient. Here a WES would be, at most, an inferred scalarization of perturbation robustness rather than a named metric.

In "Efficient weight vectors from pairwise comparison matrices," efficiency again has a different meaning: a weight vector is efficient if no other positive weight vector improves at least one pairwise approximation error without worsening any other, and weakly efficient if no other vector strictly improves all off-diagonal approximations simultaneously (Bozóki et al., 2016). The approximation problem is

$S$3

with pairwise component functions

$S$4

The paper proves that the principal eigenvector is always weakly efficient, but it can be inefficient; linear programs are then used both to test efficiency and to compute an efficient dominating vector. This is a Pareto notion of efficiency over pairwise errors, not a weighted averaging score. A plausible implication is that any scalar WES in this setting would be an auxiliary summary of dominance margin or distance to the efficient frontier, not the primitive concept itself.

7. Cross-domain distinctions, shared structure, and methodological cautions

The surveyed work shows that WES-like constructions differ most in their definition of the base efficiency primitive. In computer-use agents, efficiency is step economy relative to human trajectories (Abhyankar et al., 19 Jun 2025). In wireless systems, it is per-UE spectral-efficiency-per-power aggregated across users (Kim et al., 10 Feb 2025). In probabilistic forecasting, it is forecast quality under a proper scoring rule, but tilted toward selected outcomes or thresholds (Allen, 2023). In anomaly detection, it is pairwise severity-ordering quality aggregated over all severity pairs (Yi et al., 2024). In expert aggregation and pairwise comparison analysis, it is robustness to perturbation or Pareto nondominance of weight vectors (Kadenko et al., 2019, Bozóki et al., 2016).

The role of the weights also varies. They may encode human reference efficiency ratios and step-budget penalties, UE priority and heterogeneous constraints, outcome regions of operational interest, severity-gap importance, or quality ratings for spanning-tree-derived estimates. This means that “weighted” does not by itself identify the semantics of the score; the meaning comes from the quantity being weighted and the rationale for the weighting scheme.

Several methodological cautions recur. Weighted constructions can alter rankings, sometimes by design. Weighted proper scores can lose strict propriety or even become undefined when the forecast assigns zero mass to a target event (Allen, 2023). WSEE can intentionally push lower-priority users toward their minimum feasible performance in order to favor high-priority users (Kim et al., 10 Feb 2025). WS-AUROC depends on the chosen penalty philosophy and on the availability of credible severity labels or physical severity quantities (Yi et al., 2024). In OSWorld-Human, $S$5 can look favorable for systems that fail quickly, and the human reference trajectory is normative rather than globally optimal (Abhyankar et al., 19 Jun 2025).

This suggests that WES is best understood not as a universal formula but as a family of domain-specific weighted efficiency functionals. Across the cited literature, a technically defensible WES requires at least three explicit design choices: a base efficiency notion, a weighting mechanism, and a statement of what trade-off the weighting is intended to privilege. Without those ingredients, the term remains ambiguous even when it is mathematically well formed.