HERD: Human Expectation Realization Degree

• Human Expectation Realization Degree (HERD) is a measure comparing pre-deployment expectations with post-deployment outcomes to reveal calibration errors.
• The framework employs matched metrics, such as speedup percentages and AUROC values, to quantify expectation gaps across domains like software engineering and clinical support.
• HERD incorporates factors like oversight costs and workflow integration friction to guide structured planning and align expected benefits with realized performance.

Searching arXiv for papers relevant to "Human Expectation Realization Degree" and expectation-realization frameworks. Human Expectation Realization Degree (HERD), an Editor's term, denotes the extent to which anticipated benefits are actually realized after deployment. In the current arXiv literature, the closest explicit formulation is the “expectation-realisation gap”: the difference between pre-deployment expectations—including user forecasts, vendor claims, and developer-reported performance—and post-deployment realized outcomes such as measured productivity, measured time savings, and externally validated model performance (Lobentanzer, 23 Feb 2026). The relevant literature does not standardize HERD as a named scalar metric or a universally adopted equation; rather, it operationalizes the construct through direct empirical comparison of expected and realized outcomes, often revealing substantial miscalibration.

1. Conceptual definition and terminological status

HERD refers to expectation–outcome alignment. In the most direct arXiv treatment, the underlying construct is framed as the expectation-realisation gap for agentic AI systems. The central question is whether projected gains are borne out under real deployment conditions, and if not, by how much they diverge (Lobentanzer, 23 Feb 2026).

The construct is asymmetric in practice. A high HERD-like state corresponds to expectations being well realized in deployment; a low HERD-like state corresponds to realized outcomes falling short of prior expectations; and an inverted or negative HERD-like outcome occurs when the realized effect reverses the expected direction, such as an anticipated speedup becoming an actual slowdown. This interpretation is directly supported by the paper’s mapping from expected benefit, realized benefit, and calibration error, although the paper itself does not adopt the HERD acronym or formalize a named “realization degree” equation (Lobentanzer, 23 Feb 2026).

A key implication is that HERD is not simply a property of model capability. The relevant evidence treats it as a joint property of forecasts, deployment context, workflow structure, validation protocol, and human oversight. This suggests that HERD is better understood as an empirical deployment construct than as a benchmark score.

2. Quantification logic

The operative framework consists of four quantities: expected benefit, realized benefit, calibration error, and net benefit after oversight and workflow costs. The comparison is performed in matched units whenever possible: percentage speedup versus slowdown, minutes saved per encounter, AUROC claimed versus AUROC externally validated, concordance claimed versus concordance observed, and perceived time reduction versus measured time reduction (Lobentanzer, 23 Feb 2026).

The clearest explicit formulation is the paper’s definition of calibration error on the time-change scale:

$$\text{calibration error} = \text{expected change} - \text{realized change}.$$

In the software engineering randomized controlled trial, developers expected a 24% speedup and instead experienced a 19% slowdown, yielding a 43 percentage-point calibration error:

$24\% - (-19\%) = 43\text{ percentage points}.$

The same result is represented through completion-time factors. The expected completion time factor was 0.76, while the realized completion time factor was 1.19, so tasks took

$1.19 / 0.76 \approx 1.56,$

or about 56% longer than developers expected (Lobentanzer, 23 Feb 2026).

The paper also advances a net-benefit logic in which apparent gains must be adjusted for human review and integration costs. It does not present a single adoption inequality as a formal theorem, but its decision logic is summarized by the implied relation

$$\text{Net expected benefit} = \text{expected gross gain} - \text{oversight / review / integration cost}.$$

This is central to HERD-like analysis because it distinguishes gross apparent improvement from realized operational benefit.

3. Empirical evidence across deployment domains

The expectation-realisation literature surveyed in the agentic-AI paper spans software engineering, clinical documentation, and clinical decision support, and it uses controlled trials, field experiments, pre/post implementation studies, and independent external validations (Lobentanzer, 23 Feb 2026).

Domain	Pre-deployment expectation or claim	Realized outcome or validation
Software engineering	24% speedup expected	19% slowdown; 43 percentage-point calibration error
Clinical documentation	“5 minutes saved per clinician per encounter on average”	measured reductions of less than one minute per note; one tool showed no statistically significant effect
Clinical decision support	Epic AUROC 0.76–0.83; Watson concordance as high as 96%	Epic AUROC 0.63; Watson 48.9% strict concordance and 65.8% acceptable concordance

In software engineering, the most prominent case is the METR randomised controlled trial of 16 experienced open-source developers across 246 real tasks. The expected gain was 24% faster completion, but the measured result was 19% slower completion. The same section also documents that the expectation-realisation gap is not uniformly negative: in the GitHub Copilot pre-release trial on a standardized JavaScript HTTP server task, treated participants completed the task 55.8% faster with a 95% CI: 21% to 89%, while participants self-estimated productivity gains of about 35%. In that instance, expectation was below reality rather than above it. The paper therefore treats the sign of the gap as context-dependent rather than fixed (Lobentanzer, 23 Feb 2026).

In clinical documentation, the contrast between vendor-style claims and measured effects is narrower but persistent. Microsoft publicly claimed “5 minutes saved per clinician per encounter on average” for DAX Copilot. In the UCLA randomized controlled trial involving 238 physicians, 14 specialties, and approximately 24,000 encounters per arm, Nabla produced a 9.5% reduction in time-in-note with 95% CI: -17.2 to -1.8 and P = 0.02, whereas DAX produced -1.7%, with 95% CI: -9.4 to +5.9 and P = 0.66, which was not statistically significant. The contextual factors were also material: the tools were used in only 30–34% of visits, and about 15% of physicians in treatment arms never used their assigned scribe (Lobentanzer, 23 Feb 2026).

The clinical documentation literature in the same review also reports modest realized time reductions. In a DAX cohort study of 99 providers across 12 specialties, documentation EHR time dropped from 5.3 to 4.54 minutes per patient, a saving of about 46 seconds, but after-hours EHR time worsened significantly. In an Abridge pre/post study of 332 physicians, mean time in notes fell from 5.11 to 4.16 minutes, a difference of 57 seconds with 95% CI: 29–85 seconds. Adoption rose from 15% to 50% over 8 weeks, yet total note creation by the scribe increased only from 5% to 15%. The paper interprets these patterns as evidence that realized benefit is often substantially smaller than headline expectations once actual workflow uptake is taken into account (Lobentanzer, 23 Feb 2026).

In clinical decision support, the gap appears through external validation. The Epic Sepsis Model, externally validated in 38,455 hospitalizations, achieved AUROC = 0.63 with 95% CI: 0.62–0.64, compared with Epic’s reported AUROC = 0.76–0.83. At the operational threshold, performance was 33% sensitivity, meaning it missed about two thirds of septic patients. For Watson for Oncology, IBM publicized concordance as high as 96% in lung cancer cases, whereas a later Korean study found 48.9% strict concordance, 65.8% acceptable concordance, and about 20% concordance for patients aged 70 and older (Lobentanzer, 23 Feb 2026).

4. Mechanisms that depress realization

The surveyed literature identifies four major drivers of the expectation-realisation gap: workflow integration friction, verification burden, measurement construct mismatches, and heterogeneity in treatment effects (Lobentanzer, 23 Feb 2026).

Workflow integration friction arises because AI systems are deployed into pre-existing organizational routines rather than used in isolation. The ambient-scribe studies supply direct examples: tool use in only 30–34% of visits and non-use by about 15% of clinicians assigned to treatment arms. Even if a system can save time when used, the realized effect depends on actual adoption. From a HERD perspective, this means the relevant realization quantity is deployment-conditional rather than merely capability-conditional.

Verification burden reflects the need for human checking, editing, debugging, and remediation. In software engineering, this includes time spent reviewing and fixing generated code. In clinical documentation, it includes correction of inaccurate notes and the possibility that after-hours EHR time worsens even when time-in-note decreases. The paper also cites security and quality offsets: 32.8% of Python snippets generated by Copilot were flagged with security issues, 24.5% of JavaScript snippets were flagged, and Copilot could replicate vulnerable code patterns at around 33%. These observations support the claim that gross productivity gains may fail to become net gains once oversight is included (Lobentanzer, 23 Feb 2026).

Measurement construct mismatches occur when expectations are expressed in one construct and evaluation in another. The paper gives several examples: “minutes saved per encounter” versus measured “time-in-note”; developer-reported AUROC versus externally validated AUROC; and lab-task performance versus field productivity. In HERD terms, a portion of the apparent gap can be an artifact of non-aligned measurement, which is why matched operational metrics are indispensable.

Heterogeneity in treatment effects means that benefits vary by user experience, baseline efficiency, task complexity, and local context. The review notes that less experienced developers benefit more from copilots, efficient physicians may gain little from AI scribes, experienced developers in familiar repositories were slowed, and customer-support gains were concentrated among less experienced workers. The paper therefore rejects the assumption of a stable, globally positive treatment effect.

5. Planning frameworks and evaluative practice

The practical response proposed in the agentic-AI literature is not to assume that a model’s benchmark or developer-reported performance will translate directly into realized deployment value. Instead, the paper argues for a structured planning process in which benefit expectations are made explicit and quantified before deployment, with later comparison against realized outcomes (Lobentanzer, 23 Feb 2026).

The implementation framework identified is the Agentic Automation Canvas (AAC). The paper states that planning should: make benefit expectations explicit and quantified; include both user and developer confidence; deduct human oversight costs; link outcome metrics directly to original expectations; model heterogeneity by user group and task type; and version and archive plans early for later comparison. The AAC formalizes user expectations as quantified benefit metrics, baseline values, confidence levels from both user and developer perspectives, and explicit accounting for human oversight.

Within a HERD-oriented interpretation, these planning principles have two functions. First, they create a stable expectation baseline against which realization can be assessed. Second, they reduce calibration error caused by vague or shifting claims. A plausible implication is that HERD is not merely a retrospective audit quantity; it can also serve as a prospective design constraint if expectations, metrics, and oversight costs are specified in advance.

The same literature also documents a subjective–objective divergence that reinforces this point. In a study of 252 physicians, 86.5% perceived that documentation time had decreased, but there was no overall association between perceived reductions and objectively measured time changes, with OR = 0.975 and P = 0.144. Each 10 percentage-point increase in scribe usage was associated with about 30 seconds lower documentation time per scheduled hour, with P < 0.001. This indicates that expectation realization cannot be inferred reliably from user sentiment alone (Lobentanzer, 23 Feb 2026).

6. Related expectation–realization formalisms and acronym disambiguation

Expectation–realization reasoning also appears in arXiv work outside deployment evaluation, notably in metaphor detection. The paper “An Expectation-Realization Model for Metaphor Detection” does not define HERD, but it structures the model around an expectation component and a realization component. The expectation branch estimates literal word expectations from masked context; the realization branch computes the actual contextual meaning of the observed target word; and the system learns expectation-realization patterns that characterize metaphorical uses of words (Uduehi et al., 2023).

The architecture is explicitly dual-branch. It produces local and global embeddings for expectation and realization, combines them through non-linear layers, and computes the metaphor probability as

$$\hat{y} = \sigma(w^\mathsf{T}[h_{local} ; h_{global}] + b).$$

Its training objective is

$L^i = L^{i}_{CE} - L^{i}_{Sim},$

where the similarity term encourages expectation embeddings to remain close to the original pre-trained representations. The paper interprets metaphor as a form of surprise arising from the violation of literal word expectations. This is conceptually adjacent to HERD, but the object of analysis is linguistic metaphoricity rather than human forecasts of system performance, and the output is a classifier score rather than an interpretable human-degree metric (Uduehi et al., 2023).

The acronym also requires disambiguation because “herd” is established in epidemiology with a wholly different meaning. In “The disease-induced herd immunity level for Covid-19 is substantially lower than the classical herd immunity level”, the relevant quantities are the classical herd immunity level $h_C = 1 - 1/R_0$ and the disease-induced herd immunity level $h_D$. For the paper’s illustrative value $R_0 = 2.5$, the classical threshold is 60%, whereas the disease-induced threshold in the age-and-activity-structured model is 43.0%. In that literature, “herd” refers to population-level epidemic thresholds under heterogeneous mixing, not to expectation–outcome alignment (Britton et al., 2020).

The juxtaposition is instructive. In agentic-AI evaluation, expectation–realization concerns calibration between anticipated and realized benefit. In metaphor detection, expectation–realization concerns divergence between predicted literal meaning and realized contextual meaning. In epidemiology, “herd” denotes immunity thresholds in structured populations. The shared vocabulary does not imply a shared formalism.

7. Interpretive boundaries and common misunderstandings

A common misunderstanding is to treat HERD as though it were already a standardized scalar statistic in the literature. The available arXiv evidence does not support that interpretation. The closest explicit formalism is the expectation-realisation gap, and the reviewed paper states that it does not formalize HERD with a named equation (Lobentanzer, 23 Feb 2026).

A second misunderstanding is to assume that expectation–realization error is always negative. The software engineering evidence shows both directions: severe overestimation in the METR trial and underestimation in the GitHub Copilot pre-release trial. HERD-like analysis therefore concerns calibration quality, not merely disappointment. Whether expectations overshoot or undershoot depends on task context, measurement choice, and user population.

A third misunderstanding is to equate realized benefit with isolated model performance. The literature repeatedly shows that realized outcomes depend on adoption, oversight, workflow integration, and external validation. This suggests that any serious HERD formulation must be deployment-level rather than model-level. The empirical record supports that conclusion through repeated discrepancies between vendor claims, user perceptions, controlled-trial outcomes, and externally validated performance (Lobentanzer, 23 Feb 2026).

Taken together, the arXiv literature supports a precise interpretation of Human Expectation Realization Degree as a comparative deployment construct: the degree to which quantified expectations are matched by realized outcomes under actual use conditions. Its current empirical grounding lies primarily in expectation–realisation gap analysis for agentic AI, with related but distinct expectation–realization architectures in NLP and an unrelated established use of “herd” in epidemiology.