---
title: Graceful Degradation Score (GDS) in LLMs
url: https://www.emergentmind.com/topics/graceful-degradation-score-gds
type: topic
---

# Graceful Degradation Score (GDS) in LLMs

Searching arXiv for the cited GDS paper and closely related “graceful degradation” papers to ground the article in the current literature.
[arXiv search] query: 2603.29231 Graceful Degradation Score Beyond pass@1 graceful degradation recovery 2401.09678 2106.11119 1911.12263
Graceful Degradation Score (GDS) is a partial-credit evaluation metric for long-horizon LLM agents introduced in “Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents” [2603.29231]. It quantifies, at the episode level, what fraction of a task’s critical work an agent completes when a benchmark task is decomposed into weighted subtasks. In contrast to binary capability metrics such as pass@1, GDS preserves information about useful intermediate progress, and was proposed precisely because capability and reliability diverge systematically as task duration grows. Within the paper’s four-metric reliability framework—Reliability Decay Curve (RDC), Variance Amplification Factor (VAF), Graceful Degradation Score (GDS), and Meltdown Onset Point (MOP)—GDS is the component that measures the quality of partial completion rather than only all-or-nothing success [2603.29231].

## 1. Definition and mathematical form

In the formalization of the benchmark task, a task is modeled as
\[
\mathcal{T} = (\mathcal{W}, \mathcal{S}, \mathcal{A}, d, \nu, n^*, \mathcal{E}),
\]
where \(\mathcal{S} = \{s_i\}\) is the set of subtasks and each subtask \(s_i\) has a criticality weight \(w_i\) satisfying \(\sum_i w_i = 1\). For an episode trajectory \(\tau\) on task \(\mathcal{T}\), the Graceful Degradation Score is defined as [2603.29231]
\[
\text{GDS}(\tau, \mathcal{T}) = \sum_i w_i \cdot \mathbf{1}[\text{subtask } s_i \text{ completed correctly}].
\]

The metric is bounded:
\[
0 \le \text{GDS}(\tau, \mathcal{T}) \le 1.
\]
A value of \(\text{GDS}=1\) holds iff all subtasks are completed correctly, which corresponds to full task success. A value of \(\text{GDS}=0\) means that no subtask of positive weight was completed correctly. “Completed correctly” is checked programmatically through the evaluator \(\mathcal{E}\) using subtask-specific checks such as tests passing, expected files produced, or fields extracted correctly [2603.29231].

The paper also defines an aggregated per-model, per-duration-bucket mean:
\[
\text{GDS}(\mathcal{M}, d) = \mathbb{E}_{\mathcal{T}:\,d(\mathcal{T}) = d}\,\mathbb{E}_{\tau \sim \mathcal{M}}[\text{GDS}(\tau,\mathcal{T})].
\]
The vector
\[
(\text{GDS}(\mathcal{M}, \text{short}),\dots,\text{GDS}(\mathcal{M}, \text{very long}))
\]
is termed a GDS-based reliability profile. This profile is subsequently summarized by a regression slope over duration buckets, denoted Reliability Decay Slope (RDS) [2603.29231].

Operationally, the paper interprets GDS near 1 as indicating that the agent almost always completes all key subtasks even on long tasks; moderately high GDS, such as \(0.5\)–\(0.8\), as indicating that strict full-task success is often missed while substantial critical work is still completed; and very low GDS, such as \(<0.2\), as indicating that the agent typically does not meaningfully advance the task [2603.29231]. This contrasts with pass@1, under which “8 of 10 subtasks done” and “1 of 10 subtasks done” are both recorded as failure.

## 2. Position within the reliability framework

The reliability framework in [2603.29231] contains four metrics with distinct scopes. RDC describes how binary success decays with duration; VAF quantifies how outcome variance increases from short to long horizons; GDS measures quality of partial completion per episode; and MOP quantifies trajectory-level meltdown dynamics. The paper characterizes RDC and VAF as population-level reliability, GDS as episode-level quality of partial failures, and MOP as within-episode failure dynamics.

The conceptual RDC is defined in terms of repeated full success:
\[
\text{RDC}(\mathcal{M}, k): d \mapsto \text{pass}^k(\mathcal{M}, d),
\]
where
\[
\text{pass}^k(\mathcal{M}, d) = \text{average over tasks in bucket } d \text{ of } \Pr(\text{all k episodes succeed}).
\]
However, the paper then introduces RDS as the slope of a linear regression of GDS on bucket index \(b \in \{0,1,2,3\}\):
\[
\text{RDS}(\mathcal{M}) =
\frac{\sum_b (b - \bar{b})(\text{GDS}(\mathcal{M}, b) - \overline{\text{GDS}})}{\sum_b (b - \bar{b})^2}.
\]
This use of GDS as the scalar “reliability level” at bucket \(d\) becomes especially important when pass@1 or pass\(^k\) approaches zero on long or very-long tasks, because GDS remains informative even when strict success is rare [2603.29231].

The contrast with VAF and MOP is exact rather than rhetorical. VAF is defined as
\[
\text{VAF}(\mathcal{M}) = \frac{\sigma^2[\text{pass@1} \mid d=\text{long}]}{\sigma^2[\text{pass@1} \mid d=\text{short}]},
\]
so it measures horizon-induced variance across tasks, not graded success within a task. MOP is defined through a sliding-window entropy over tool-call distributions,
\[
p_t(\text{tool}_i) = \frac{|\{j \in [t-w,t] : a_j = \text{tool}_i\}|}{w},
\]
\[
H(t) = -\sum_i p_t(\text{tool}_i) \log p_t(\text{tool}_i),
\]
with onset \(t^*\) when
\[
H(t^*) > \theta_H \quad\text{and}\quad H(t^*) - H(t^*-w) > \delta.
\]
Accordingly, GDS answers “how much useful work was done?”, VAF answers “how sensitive is success/failure to task choice at long vs short horizons?”, and MOP answers “did the episode enter a high-entropy meltdown regime, and when?” [2603.29231].

A common misconception is to read GDS as a softened version of pass@1. The framework instead treats it as a distinct reliability observable: pass@1 records whether the entire evaluator \(\mathcal{E}(\mathcal{W})\) returns \(1.0\), whereas GDS records the weighted fraction of critical substructure successfully completed.

## 3. Computation and evaluation protocol

The experimental protocol in [2603.29231] evaluates 10 models across 23,392 episodes on a 396-task benchmark spanning four duration buckets and three domains. For each \((\text{model}, \text{task})\) pair, the study runs \(k=3\) independent episodes; each episode is a full agent run, either ReAct or ReAct+memory, up to a step limit or `finish()`. Success for pass@1 is defined by \(\mathcal{E}(\mathcal{W}) = 1.0\), meaning all subtasks are judged correct, whereas GDS is computed from subtask-level programmatic checks.

Per episode, GDS is computed in three steps. First, subtasks \(s_1,\dots,s_m\) and weights \(w_1,\dots,w_m\) are defined with \(\sum_i w_i = 1\). Second, after the episode ends, evaluation logic is run for each subtask. The paper gives domain-specific examples: in software engineering (SE), tests can be focused on specific functions or features; in document processing (DP), required output files or fields are checked for presence and correctness; in web research (WR), extracted facts or structures are checked against ground truth or tolerances. Third, the weighted indicator sum is evaluated:
\[
\text{GDS}(\tau,\mathcal{T}) = \sum_i w_i \cdot \mathbf{1}[s_i \text{ passed}].
\]
Aggregation across tasks and repeats within a bucket \(d\) is then
\[
\text{GDS}(\mathcal{M}, d) = \frac{1}{N_d k}\sum_{j \in d} \sum_{\ell=1}^k \text{GDS}(\tau_{j,\ell}, \mathcal{T}_j),
\]
where \(N_d = 33\) tasks per bucket [2603.29231].

The duration buckets are defined by human time: short \(\le 5\) min, medium \(5\)–\(30\) min, long \(30\)–\(120\) min, and very long \(\ge 120\) min. Each bucket contains 33 tasks per domain across three domains, yielding 99 tasks per bucket [2603.29231]. Confidence intervals for GDS are not explicitly tabulated in the paper, although the details note that they can be computed by bootstrap across tasks.

This evaluation protocol matters because GDS is not a generic reward-shaping proxy. It is benchmarked as a post hoc, evaluator-grounded partial-credit metric whose semantics depend on an explicit subtask decomposition with criticality weights. A plausible implication is that the quality of the decomposition directly affects the interpretability of the resulting score.

## 4. Empirical behavior across duration, domains, and models

The global pattern reported in Table 7 of [2603.29231] is that mean GDS across all models declines from \(0.81\) on short tasks to \(0.66\) on medium, \(0.61\) on long, and \(0.59\) on very-long tasks. The paper explicitly notes that this decay is much gentler than pass@1, which falls from \(0.76\) to about \(0.52\). The significance is that agents often continue to complete a substantial fraction of weighted subtasks even when strict full-task success is already eroding.

The domain-level pattern is strongly stratified:

| Domain | Short \(\rightarrow\) Very long | Drop |
|---|---:|---:|
| SE | \(0.90 \rightarrow 0.44\) | \(-0.46\) |
| WR | \(0.80 \rightarrow 0.63\) | \(-0.17\) |
| DP | \(0.74 \rightarrow 0.71\) | \(-0.03\) |

These figures support the paper’s claim that reliability decay is domain-stratified: software engineering exhibits severe degradation, web research shows moderate degradation, and document processing is nearly flat [2603.29231]. The SE-to-DP contrast is central to the motivation for GDS, because a binary success metric cannot represent the difference between a domain in which partial work collapses and a domain in which partial work remains highly preserved.

At the model level, frontier systems maintain very high GDS even on very-long tasks. The reported values are: DeepSeek V3 \(0.96, 0.95, 0.93, 0.87\); Kimi K2.5 \(0.97, 0.96, 0.86, 0.84\); and MiniMax M2.5 \(0.97, 0.95, 0.88, 0.89\) from short through very-long. Mid-tier models retain substantial but weaker partial success, for example Qwen3 32B \(0.91, 0.67, 0.59, 0.60\), Mistral 24B \(0.81, 0.67, 0.73, 0.70\), and Llama 3.3 70B \(0.84, 0.58, 0.55, 0.62\). Weak models exhibit low long-horizon GDS, including Mistral Nemo \(0.57, 0.21, 0.28, 0.23\) and Llama 3.1 8B \(0.32, 0.17, 0.11, 0.07\) [2603.29231].

The gap between GDS and pass@1 widens with horizon. A specific example in the paper is Llama 3.3 70B on long tasks, where \(\text{GDS}=0.55\) and \(\text{pass@1}=0.39\), leaving a \(0.16\) margin of useful partial work that pass@1 does not record [2603.29231]. This widening gap is one of the clearest empirical justifications for the metric.

## 5. Interpretation, design utility, and interactions with failure modes

The paper interprets high long-horizon GDS, such as \(0.85\)–\(0.95\), as indicating that failures tend to be localized: missing an edge case or a small formatting issue rather than failing to execute the critical subtask structure. Moderate long-horizon GDS, such as \(0.5\)–\(0.7\), indicates that many runs do considerable useful work but leave gaps; from a deployment perspective, such outputs may remain valuable if they can be lightly reviewed and patched by a human. Low long-horizon GDS, such as \(<0.3\), indicates that failures are not graceful and that runs are largely wasted from a user perspective [2603.29231].

The distinction between graceful and sharp degradation is therefore defined jointly over pass@1 and GDS. Graceful degradation occurs when performance drops gradually with duration but the agent continues to complete a large fraction of subtasks, so GDS remains high even as pass@1 falls. Sharp or catastrophic degradation occurs when pass@1 collapses and GDS also collapses, indicating that the agent is not merely failing to finish but is also doing little useful intermediate work. The paper identifies weaker-model performance on SE tasks as exhibiting this pattern [2603.29231].

Several design and deployment implications follow directly. If two models have similar pass@1 but one has significantly higher GDS at long horizon, the higher-GDS model is preferable because partial outputs are more salvageable. If GDS remains high up to one duration bucket and then drops, task decomposition is suggested so that each subtask falls into the high-GDS range. The paper also describes intervention policies: runs with GDS around \(0.6\)–\(0.7\) may be retained and human-edited, whereas runs consistently below \(0.2\) may be auto-discarded or escalated to a stronger model [2603.29231].

GDS also interacts nontrivially with other reliability phenomena. The paper’s “MOP paradox” is that frontier models have both the highest very-long GDS and the highest meltdown rates, up to \(19\%\) of episodes at very long horizons. DeepSeek V3 and MiniMax M2.5 are explicitly cited as having very-long GDS of \(0.87\) and \(0.89\), while also exhibiting the highest meltdown rates. The interpretation given is that these models attempt ambitious multi-step strategies: when those strategies spiral, tool-call entropy spikes and MOP triggers; when they do not spiral, the models complete a large fraction of subtasks and often succeed [2603.29231]. High GDS and high MOP rate therefore do not contradict one another.

The relationship to VAF is similarly nontrivial. The paper reports that VAF is high for frontier models and low for weak models, and argues that high VAF is a capability signature rather than an instability signal. High-VAF models also have high GDS at long horizons, whereas low-VAF models often have low GDS and are uniformly poor. This is an important corrective to the naive reading that variance amplification itself signals fragility [2603.29231].

A further empirical result concerns scaffolding. The memory scaffold (“Mem”) compared with ReAct never improves long+very-long GDS in Table 12; \(\Delta \text{GDS} = \text{Mem} - \text{ReAct}\) is never positive, memory hurts GDS for 6 of 10 models, and is essentially neutral for the rest. The paper attributes this to overhead in tool calls and context consumption, which reduces the amount of actual task work completed within step and context budgets [2603.29231]. In the terminology of the framework, naive episodic memory does not improve graceful degradation.

## 6. Limitations, external validity, and broader uses of “graceful degradation”

The paper’s own limitations for GDS are mostly structural. The metric depends on a meaningful subtask decomposition \(\{s_i\}\) and weights \(w_i\), and on the assumption that independently completing each subtask adds value. If subtasks are mis-weighted, overlap heavily, or fail to capture practical utility, GDS can misrepresent system quality. The benchmark evidence is restricted to SE, WR, and DP, and the paper explicitly notes that GDS has not yet been validated on embodied agents, multi-agent systems, or other complex domains. It also notes that human-time duration buckets may not align with agent complexity; the near-flat DP curve suggests that human “long” may be agent “short” in step complexity [2603.29231].

Beyond long-horizon LLM agents, “graceful degradation” is an established concept in other fields, but the acronym GDS is not standardized there. In cyber-physical systems, “Integrating Graceful Degradation and Recovery through Requirement-driven Adaptation” treats degradation as temporarily weakening original system requirements and recovery as strengthening weakened requirements when the environment returns within expected bounds; the details explicitly state that the phrase “Graceful Degradation Score” is not used in that paper, although STL robustness, \(\Delta_{weak}\), \(\Delta_{strong}\), and cumulative robustness provide ingredients from which a scalar score can be inferred [2401.09678]. In machine learning under OOD shift, “Graceful Degradation and Related Fields” defines graceful degradation as optimizing the drop in model performance as data moves further from the training distribution, but again does not introduce a specific scalar GDS; instead it surveys accuracy-vs-shift, ECE, Brier score, E99 error, entropy-based AUC, AUROC for OOD detection, hierarchical accuracies, and selective classification as the relevant observables [2106.11119]. In coding theory, “Low density majority codes and the problem of graceful degradation” studies smooth BER curves over the BEC and the structural incompatibility between near single-point optimality and graceful two-point performance for linear systematic codes, but does not define a scalar GDS either [1911.12263].

This broader usage suggests that the distinctive contribution of [2603.29231] is not the phrase “graceful degradation” itself, but the introduction of a concrete, evaluator-grounded partial-credit metric named Graceful Degradation Score for long-horizon LLM agents. A plausible implication is that the term may become field-specific unless later work standardizes analogous scalar formulations across CPS, OOD robustness, and coding-theoretic settings.

Within the reliability-science framing of [2603.29231], GDS is best understood as the metric that restores resolution precisely where binary success metrics become least informative: long tasks, partial completion, and failure modes that remain operationally useful.

Source: https://www.emergentmind.com/topics/graceful-degradation-score-gds