Performance Realisation Ratio

• Performance Realisation Ratio is defined as the realised improvement divided by the expected improvement, serving as a calibration metric.
• It measures the gap between forecasted gains and actual outcomes, highlighting issues like partial adoption and measurement mismatch.
• It is applied in areas like software engineering and clinical decision support to assess and improve agentic AI system performance.

Performance Realisation Ratio (PRR) is a dimensionless metric introduced by Lobentanzer et al. to quantify how closely the realised improvement from an agentic AI system matches the improvement expected before deployment. It is designed to make the expectation-realisation gap explicit on a common scale by comparing pre-deployment forecasts with controlled-trial or external-validation outcomes. In the reviewed evidence base, PRR is used across software engineering, clinical documentation, and clinical decision support, where substantial discrepancies are reported between expected gains and realised outcomes (Lobentanzer, 23 Feb 2026).

1. Formal definition

The formal definition is

$PRR = \frac{R}{E}$

where $E$ is the expected improvement and $R$ is the realised improvement. The same write-up defines Calibration Error on the same scale as

$CE = R - E.$

The quantities $E$ and $R$ must be expressed in the same units as will be measured. In some domains, such as percent speedup, both are signed: positive for a speedup and negative for a slowdown. This matters because PRR can therefore be negative when a system expected to improve performance instead degrades it.

Within this formulation, PRR is a compact summary of calibration. A value $PRR < 1$ indicates that realised benefits fell short of expectations; $PRR \approx 1$ indicates well-calibrated expectations; and $PRR > 1$ indicates under-confident expectations, meaning that realised improvement exceeded forecasts. The associated Calibration Error preserves the absolute gap rather than the ratio.

2. Computation and measurement discipline

The proposed computation procedure is explicitly stepwise. First, the forecasted improvement $E$ is identified. The source may be user self-reports, vendor marketing claims, or developer internal validation. Second, the realised improvement $E$0 is extracted from a controlled trial or external validation, as the observed mean difference or percentage change between treatment and control.

Third, both quantities are converted into consistent, unit-matched values. The canonical example given is that an expectation of “24 % speedup” is encoded as $E$1, whereas a realised “19 % slowdown” is encoded as $E$2. Only after this unit and direction matching does the ratio $E$3 become interpretable.

This measurement discipline is central rather than procedural detail. The same framework treats PRR as valid only when the outcome construct is aligned across expectation and evaluation. A plausible implication is that PRR is less a standalone KPI than a calibration statistic whose meaning depends on carefully harmonised measurement definitions.

3. Empirical estimates in reviewed domains

The reviewed domains show that PRR can reveal large expectation-realisation gaps even when pre-deployment expectations appear numerically precise. The paper’s headline example is software engineering, where experienced developers expected a 24% speedup from AI tools but were slowed by 19%, implying a 43 percentage-point calibration error. In clinical documentation, vendor claims of multi-minute time savings contrast with measured reductions of less than one minute per note, and one widely deployed tool showed no statistically significant effect. In clinical decision support, externally validated performance falls substantially below developer-reported metrics (Lobentanzer, 23 Feb 2026).

Domain	Expected and realised outcomes	PRR and CE
Software engineering (METR trial)	Expected: 24% speedup; realised: 19% slowdown	$E$4; $E$5
Clinical documentation (ambient scribes)	Expected: about 5 min saved; realised: about 0.50 min saving in the UCLA Nabla RCT and about 0.77 min saving in a DAX cohort	$E$6; $E$7 min
Clinical decision support (Epic Sepsis Model)	Expected AUROC: 0.80; realised AUROC: 0.63	$E$8; $E$9
Clinical decision support (Watson for Oncology, colon cancer concordance)	Expected concordance: 0.96; realised strict concordance: 0.489	$R$0; $R$1

These examples illustrate several regimes. A negative PRR arises when direction reverses, as in the software engineering case. Values between 0 and 1 indicate partial realisation of expected gains. None of the reported examples shows realised improvement exceeding forecast.

4. Mechanistic sources of low PRR

Lobentanzer et al. identify four mechanistic drivers that systematically depress PRR. Workflow integration friction and partial adoption reduce realised gains because low utilisation attenuates average treatment effects. The cited UCLA trial reported utilisation of 30–34%, and partial adoption converts per-use effects into much smaller intent-to-treat averages (Lobentanzer, 23 Feb 2026).

Verification and review burden further reduces net realised benefit. Time spent debugging or editing agentic outputs can exceed the time initially saved by generation. In this framing, realised improvement $R$2 is not gross generation speed but net benefit after human oversight cost.

Measurement construct mismatch arises when pre-deployment claims and post-deployment evaluations measure different things. The write-up contrasts broad claims such as “minutes saved per encounter” with narrower trial outcomes such as “time-in-note.” In clinical decision support, it similarly notes that developer AUROC evaluations may use sampling, thresholds, and end-points that inflate lab performance relative to field performance.

Heterogeneity in treatment effects means that average PRR can conceal opposing subgroup effects. Gains may concentrate among less experienced users or inefficient workflows, while experienced users may see $R$3. Aggregation across heterogeneous subgroups therefore lowers interpretability as well as average PRR.

Taken together, these drivers explain why substantial expectation shortfalls can persist even when a system appears technically capable under internal or laboratory evaluation.

5. Structured planning frameworks

To reduce the expectation-realisation gap, the paper proposes the Agentic Automation Canvas (AAC), which operationalises five design principles (Lobentanzer, 23 Feb 2026):

• Quantified benefit expectations
• Dual-perspective confidence: user forecasts versus developer technical forecasts
• Explicit human oversight cost accounting
• Metric alignment: the same units and granularity for $R$4 and $R$5
• Explicit modelling of heterogeneity: target populations and tasks

The same framework presents a simplified decision rule:

$R$6

with deployment justified only when $R$7. Here $R$8 is the expected adoption or utilisation rate, and $R$9 is the per-unit human review cost.

In the AAC, stakeholders fill out a structured table containing baseline performance, expected improvement $CE = R - E.$0 from both user and developer perspectives, anticipated adoption $CE = R - E.$1, verification cost $CE = R - E.$2, and a success criterion defined by $CE = R - E.$3. By formalising these quantities and versioning them, the framework enables later comparison of realised $CE = R - E.$4 against initial $CE = R - E.$5 and therefore retrospective computation of PRR.

6. Methodological limitations and future directions

The framework’s main caveat is that PRR is only as good as the forecast input $CE = R - E.$6. The paper explicitly notes dependence on accurate expectation formation and points to the planning fallacy as a source of bias. A second limitation is that partial adoption and compliance confound realised outcomes because real-world utilisation $CE = R - E.$7 may drift over time (Lobentanzer, 23 Feb 2026).

A third limitation is measurement mismatch. The write-up recommends that future trials pre-register outcome metrics that align exactly with procurement claims. A fourth limitation is aggregation across heterogeneous users and tasks: aggregate PRR can hide subgroup variation, so practitioners are advised to compute PRR by user cohort and task complexity. A fifth limitation concerns long-term effects. Downstream impacts on skill formation, including cognitive offloading, may further depress realised net benefit but are rarely captured in short-term realised outcomes.

The future suggestions are correspondingly specific: standardise pre-deployment expectation logs to reduce post-hoc rationalisation, include oversight costs in prospective ROI models, design trials that measure both gross generation time and net end-to-end time including review, and report PRR distributions rather than only means. This suggests an evolution from one-number benchmarking toward distributional and subgroup-aware calibration analysis.

7. Distinction from photovoltaic Performance Ratio

Performance Realisation Ratio should not be confused with the photovoltaic Performance Ratio, despite the similar wording. In grid-tied photovoltaic analysis, Performance Ratio is defined as the ratio of Final Yield to Reference Yield, $CE = R - E.$8, and functions as a resource-independent metric of a PV system’s real-world effectiveness (Taduran et al., 3 Oct 2025). By contrast, Performance Realisation Ratio is defined as realised improvement divided by expected improvement, $CE = R - E.$9, and is concerned with calibration between anticipated and observed benefit in agentic AI deployment (Lobentanzer, 23 Feb 2026).

The distinction is substantive as well as terminological. The photovoltaic metric normalises realised AC output by irradiance-derived reference yield; the agentic-AI metric normalises realised benefit by prior expectation. Both are dimensionless ratios, but they operate over different domains, different numerators and denominators, and different inferential questions. A plausible implication is that PRR belongs to deployment evaluation and expectation management, whereas photovoltaic Performance Ratio belongs to physical-system efficiency assessment.