Graceful Degradation Score (GDS) in LLMs

• Graceful Degradation Score (GDS) is a partial-credit metric that quantifies the fraction of critical subtasks completed via a weighted sum, offering finer resolution than binary metrics.
• It is computed by programmatically checking each subtask’s success and aggregating scores across duration buckets to profile model reliability over long tasks.
• Empirical studies demonstrate that higher GDS values indicate agents that perform substantial useful work even as complete-task success rates (pass@1) decline with task duration.

Searching arXiv for the cited GDS paper and closely related “graceful degradation” papers to ground the article in the current literature. [arXiv search] query: (Khanal et al., 31 Mar 2026) Graceful Degradation Score Beyond pass@1 graceful degradation recovery (Chu et al., 2024, Dymond, 2021, Roozbehani et al., 2019) Graceful Degradation Score (GDS) is a partial-credit evaluation metric for long-horizon LLM agents introduced in “Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents” (Khanal et al., 31 Mar 2026). It quantifies, at the episode level, what fraction of a task’s critical work an agent completes when a benchmark task is decomposed into weighted subtasks. In contrast to binary capability metrics such as pass@1, GDS preserves information about useful intermediate progress, and was proposed precisely because capability and reliability diverge systematically as task duration grows. Within the paper’s four-metric reliability framework—Reliability Decay Curve (RDC), Variance Amplification Factor (VAF), Graceful Degradation Score (GDS), and Meltdown Onset Point (MOP)—GDS is the component that measures the quality of partial completion rather than only all-or-nothing success (Khanal et al., 31 Mar 2026).

1. Definition and mathematical form

In the formalization of the benchmark task, a task is modeled as

$$\mathcal{T} = (\mathcal{W}, \mathcal{S}, \mathcal{A}, d, \nu, n^*, \mathcal{E}),$$

where $\mathcal{S} = \{s_i\}$ is the set of subtasks and each subtask $s_i$ has a criticality weight $w_i$ satisfying $\sum_i w_i = 1$. For an episode trajectory $\tau$ on task $\mathcal{T}$, the Graceful Degradation Score is defined as (Khanal et al., 31 Mar 2026)

$$\text{GDS}(\tau, \mathcal{T}) = \sum_i w_i \cdot \mathbf{1}[\text{subtask } s_i \text{ completed correctly}].$$

The metric is bounded: $0 \le \text{GDS}(\tau, \mathcal{T}) \le 1.$ A value of $\text{GDS}=1$ holds iff all subtasks are completed correctly, which corresponds to full task success. A value of $\mathcal{S} = \{s_i\}$0 means that no subtask of positive weight was completed correctly. “Completed correctly” is checked programmatically through the evaluator $\mathcal{S} = \{s_i\}$1 using subtask-specific checks such as tests passing, expected files produced, or fields extracted correctly (Khanal et al., 31 Mar 2026).

The paper also defines an aggregated per-model, per-duration-bucket mean: $\mathcal{S} = \{s_i\}$2 The vector

$\mathcal{S} = \{s_i\}$3

is termed a GDS-based reliability profile. This profile is subsequently summarized by a regression slope over duration buckets, denoted Reliability Decay Slope (RDS) (Khanal et al., 31 Mar 2026).

Operationally, the paper interprets GDS near 1 as indicating that the agent almost always completes all key subtasks even on long tasks; moderately high GDS, such as $\mathcal{S} = \{s_i\}$4–$\mathcal{S} = \{s_i\}$5, as indicating that strict full-task success is often missed while substantial critical work is still completed; and very low GDS, such as $\mathcal{S} = \{s_i\}$6, as indicating that the agent typically does not meaningfully advance the task (Khanal et al., 31 Mar 2026). This contrasts with pass@1, under which “8 of 10 subtasks done” and “1 of 10 subtasks done” are both recorded as failure.

2. Position within the reliability framework

The reliability framework in (Khanal et al., 31 Mar 2026) contains four metrics with distinct scopes. RDC describes how binary success decays with duration; VAF quantifies how outcome variance increases from short to long horizons; GDS measures quality of partial completion per episode; and MOP quantifies trajectory-level meltdown dynamics. The paper characterizes RDC and VAF as population-level reliability, GDS as episode-level quality of partial failures, and MOP as within-episode failure dynamics.

The conceptual RDC is defined in terms of repeated full success: $\mathcal{S} = \{s_i\}$7 where

$\mathcal{S} = \{s_i\}$8

However, the paper then introduces RDS as the slope of a linear regression of GDS on bucket index $\mathcal{S} = \{s_i\}$9: $s_i$0 This use of GDS as the scalar “reliability level” at bucket $s_i$1 becomes especially important when pass@1 or pass$s_i$2 approaches zero on long or very-long tasks, because GDS remains informative even when strict success is rare (Khanal et al., 31 Mar 2026).

The contrast with VAF and MOP is exact rather than rhetorical. VAF is defined as

$s_i$3

so it measures horizon-induced variance across tasks, not graded success within a task. MOP is defined through a sliding-window entropy over tool-call distributions,

$s_i$4

$s_i$5

with onset $s_i$6 when

$s_i$7

Accordingly, GDS answers “how much useful work was done?”, VAF answers “how sensitive is success/failure to task choice at long vs short horizons?”, and MOP answers “did the episode enter a high-entropy meltdown regime, and when?” (Khanal et al., 31 Mar 2026).

A common misconception is to read GDS as a softened version of pass@1. The framework instead treats it as a distinct reliability observable: pass@1 records whether the entire evaluator $s_i$8 returns $s_i$9, whereas GDS records the weighted fraction of critical substructure successfully completed.

3. Computation and evaluation protocol

The experimental protocol in (Khanal et al., 31 Mar 2026) evaluates 10 models across 23,392 episodes on a 396-task benchmark spanning four duration buckets and three domains. For each $w_i$0 pair, the study runs $w_i$1 independent episodes; each episode is a full agent run, either ReAct or ReAct+memory, up to a step limit or finish(). Success for pass@1 is defined by $w_i$2, meaning all subtasks are judged correct, whereas GDS is computed from subtask-level programmatic checks.

Per episode, GDS is computed in three steps. First, subtasks $w_i$3 and weights $w_i$4 are defined with $w_i$5. Second, after the episode ends, evaluation logic is run for each subtask. The paper gives domain-specific examples: in software engineering (SE), tests can be focused on specific functions or features; in document processing (DP), required output files or fields are checked for presence and correctness; in web research (WR), extracted facts or structures are checked against ground truth or tolerances. Third, the weighted indicator sum is evaluated: $w_i$6 Aggregation across tasks and repeats within a bucket $w_i$7 is then

$w_i$8

where $w_i$9 tasks per bucket (Khanal et al., 31 Mar 2026).

The duration buckets are defined by human time: short $\sum_i w_i = 1$0 min, medium $\sum_i w_i = 1$1–$\sum_i w_i = 1$2 min, long $\sum_i w_i = 1$3–$\sum_i w_i = 1$4 min, and very long $\sum_i w_i = 1$5 min. Each bucket contains 33 tasks per domain across three domains, yielding 99 tasks per bucket (Khanal et al., 31 Mar 2026). Confidence intervals for GDS are not explicitly tabulated in the paper, although the details note that they can be computed by bootstrap across tasks.

This evaluation protocol matters because GDS is not a generic reward-shaping proxy. It is benchmarked as a post hoc, evaluator-grounded partial-credit metric whose semantics depend on an explicit subtask decomposition with criticality weights. A plausible implication is that the quality of the decomposition directly affects the interpretability of the resulting score.

4. Empirical behavior across duration, domains, and models

The global pattern reported in Table 7 of (Khanal et al., 31 Mar 2026) is that mean GDS across all models declines from $\sum_i w_i = 1$6 on short tasks to $\sum_i w_i = 1$7 on medium, $\sum_i w_i = 1$8 on long, and $\sum_i w_i = 1$9 on very-long tasks. The paper explicitly notes that this decay is much gentler than pass@1, which falls from $\tau$0 to about $\tau$1. The significance is that agents often continue to complete a substantial fraction of weighted subtasks even when strict full-task success is already eroding.

The domain-level pattern is strongly stratified:

Domain	Short $\tau$2 Very long	Drop
SE	$\tau$3	$\tau$4
WR	$\tau$5	$\tau$6
DP	$\tau$7	$\tau$8

These figures support the paper’s claim that reliability decay is domain-stratified: software engineering exhibits severe degradation, web research shows moderate degradation, and document processing is nearly flat (Khanal et al., 31 Mar 2026). The SE-to-DP contrast is central to the motivation for GDS, because a binary success metric cannot represent the difference between a domain in which partial work collapses and a domain in which partial work remains highly preserved.

At the model level, frontier systems maintain very high GDS even on very-long tasks. The reported values are: DeepSeek V3 $\tau$9; Kimi K2.5 $\mathcal{T}$0; and MiniMax M2.5 $\mathcal{T}$1 from short through very-long. Mid-tier models retain substantial but weaker partial success, for example Qwen3 32B $\mathcal{T}$2, Mistral 24B $\mathcal{T}$3, and Llama 3.3 70B $\mathcal{T}$4. Weak models exhibit low long-horizon GDS, including Mistral Nemo $\mathcal{T}$5 and Llama 3.1 8B $\mathcal{T}$6 (Khanal et al., 31 Mar 2026).

The gap between GDS and pass@1 widens with horizon. A specific example in the paper is Llama 3.3 70B on long tasks, where $\mathcal{T}$7 and $\mathcal{T}$8, leaving a $\mathcal{T}$9 margin of useful partial work that pass@1 does not record (Khanal et al., 31 Mar 2026). This widening gap is one of the clearest empirical justifications for the metric.

5. Interpretation, design utility, and interactions with failure modes

The paper interprets high long-horizon GDS, such as $$\text{GDS}(\tau, \mathcal{T}) = \sum_i w_i \cdot \mathbf{1}[\text{subtask } s_i \text{ completed correctly}].$$0–$$\text{GDS}(\tau, \mathcal{T}) = \sum_i w_i \cdot \mathbf{1}[\text{subtask } s_i \text{ completed correctly}].$$1, as indicating that failures tend to be localized: missing an edge case or a small formatting issue rather than failing to execute the critical subtask structure. Moderate long-horizon GDS, such as $$\text{GDS}(\tau, \mathcal{T}) = \sum_i w_i \cdot \mathbf{1}[\text{subtask } s_i \text{ completed correctly}].$$2–$$\text{GDS}(\tau, \mathcal{T}) = \sum_i w_i \cdot \mathbf{1}[\text{subtask } s_i \text{ completed correctly}].$$3, indicates that many runs do considerable useful work but leave gaps; from a deployment perspective, such outputs may remain valuable if they can be lightly reviewed and patched by a human. Low long-horizon GDS, such as $$\text{GDS}(\tau, \mathcal{T}) = \sum_i w_i \cdot \mathbf{1}[\text{subtask } s_i \text{ completed correctly}].$$4, indicates that failures are not graceful and that runs are largely wasted from a user perspective (Khanal et al., 31 Mar 2026).

The distinction between graceful and sharp degradation is therefore defined jointly over pass@1 and GDS. Graceful degradation occurs when performance drops gradually with duration but the agent continues to complete a large fraction of subtasks, so GDS remains high even as pass@1 falls. Sharp or catastrophic degradation occurs when pass@1 collapses and GDS also collapses, indicating that the agent is not merely failing to finish but is also doing little useful intermediate work. The paper identifies weaker-model performance on SE tasks as exhibiting this pattern (Khanal et al., 31 Mar 2026).

Several design and deployment implications follow directly. If two models have similar pass@1 but one has significantly higher GDS at long horizon, the higher-GDS model is preferable because partial outputs are more salvageable. If GDS remains high up to one duration bucket and then drops, task decomposition is suggested so that each subtask falls into the high-GDS range. The paper also describes intervention policies: runs with GDS around $$\text{GDS}(\tau, \mathcal{T}) = \sum_i w_i \cdot \mathbf{1}[\text{subtask } s_i \text{ completed correctly}].$$5–$$\text{GDS}(\tau, \mathcal{T}) = \sum_i w_i \cdot \mathbf{1}[\text{subtask } s_i \text{ completed correctly}].$$6 may be retained and human-edited, whereas runs consistently below $$\text{GDS}(\tau, \mathcal{T}) = \sum_i w_i \cdot \mathbf{1}[\text{subtask } s_i \text{ completed correctly}].$$7 may be auto-discarded or escalated to a stronger model (Khanal et al., 31 Mar 2026).

GDS also interacts nontrivially with other reliability phenomena. The paper’s “MOP paradox” is that frontier models have both the highest very-long GDS and the highest meltdown rates, up to $$\text{GDS}(\tau, \mathcal{T}) = \sum_i w_i \cdot \mathbf{1}[\text{subtask } s_i \text{ completed correctly}].$$8 of episodes at very long horizons. DeepSeek V3 and MiniMax M2.5 are explicitly cited as having very-long GDS of $$\text{GDS}(\tau, \mathcal{T}) = \sum_i w_i \cdot \mathbf{1}[\text{subtask } s_i \text{ completed correctly}].$$9 and $0 \le \text{GDS}(\tau, \mathcal{T}) \le 1.$0, while also exhibiting the highest meltdown rates. The interpretation given is that these models attempt ambitious multi-step strategies: when those strategies spiral, tool-call entropy spikes and MOP triggers; when they do not spiral, the models complete a large fraction of subtasks and often succeed (Khanal et al., 31 Mar 2026). High GDS and high MOP rate therefore do not contradict one another.

The relationship to VAF is similarly nontrivial. The paper reports that VAF is high for frontier models and low for weak models, and argues that high VAF is a capability signature rather than an instability signal. High-VAF models also have high GDS at long horizons, whereas low-VAF models often have low GDS and are uniformly poor. This is an important corrective to the naive reading that variance amplification itself signals fragility (Khanal et al., 31 Mar 2026).

A further empirical result concerns scaffolding. The memory scaffold (“Mem”) compared with ReAct never improves long+very-long GDS in Table 12; $0 \le \text{GDS}(\tau, \mathcal{T}) \le 1.$1 is never positive, memory hurts GDS for 6 of 10 models, and is essentially neutral for the rest. The paper attributes this to overhead in tool calls and context consumption, which reduces the amount of actual task work completed within step and context budgets (Khanal et al., 31 Mar 2026). In the terminology of the framework, naive episodic memory does not improve graceful degradation.

6. Limitations, external validity, and broader uses of “graceful degradation”

The paper’s own limitations for GDS are mostly structural. The metric depends on a meaningful subtask decomposition $0 \le \text{GDS}(\tau, \mathcal{T}) \le 1.$2 and weights $0 \le \text{GDS}(\tau, \mathcal{T}) \le 1.$3, and on the assumption that independently completing each subtask adds value. If subtasks are mis-weighted, overlap heavily, or fail to capture practical utility, GDS can misrepresent system quality. The benchmark evidence is restricted to SE, WR, and DP, and the paper explicitly notes that GDS has not yet been validated on embodied agents, multi-agent systems, or other complex domains. It also notes that human-time duration buckets may not align with agent complexity; the near-flat DP curve suggests that human “long” may be agent “short” in step complexity (Khanal et al., 31 Mar 2026).

Beyond long-horizon LLM agents, “graceful degradation” is an established concept in other fields, but the acronym GDS is not standardized there. In cyber-physical systems, “Integrating Graceful Degradation and Recovery through Requirement-driven Adaptation” treats degradation as temporarily weakening original system requirements and recovery as strengthening weakened requirements when the environment returns within expected bounds; the details explicitly state that the phrase “Graceful Degradation Score” is not used in that paper, although STL robustness, $0 \le \text{GDS}(\tau, \mathcal{T}) \le 1.$4, $0 \le \text{GDS}(\tau, \mathcal{T}) \le 1.$5, and cumulative robustness provide ingredients from which a scalar score can be inferred (Chu et al., 2024). In machine learning under OOD shift, “Graceful Degradation and Related Fields” defines graceful degradation as optimizing the drop in model performance as data moves further from the training distribution, but again does not introduce a specific scalar GDS; instead it surveys accuracy-vs-shift, ECE, Brier score, E99 error, entropy-based AUC, AUROC for OOD detection, hierarchical accuracies, and selective classification as the relevant observables (Dymond, 2021). In coding theory, “Low density majority codes and the problem of graceful degradation” studies smooth BER curves over the BEC and the structural incompatibility between near single-point optimality and graceful two-point performance for linear systematic codes, but does not define a scalar GDS either (Roozbehani et al., 2019).

This broader usage suggests that the distinctive contribution of (Khanal et al., 31 Mar 2026) is not the phrase “graceful degradation” itself, but the introduction of a concrete, evaluator-grounded partial-credit metric named Graceful Degradation Score for long-horizon LLM agents. A plausible implication is that the term may become field-specific unless later work standardizes analogous scalar formulations across CPS, OOD robustness, and coding-theoretic settings.

Within the reliability-science framing of (Khanal et al., 31 Mar 2026), GDS is best understood as the metric that restores resolution precisely where binary success metrics become least informative: long tasks, partial completion, and failure modes that remain operationally useful.