---
title: 'Rerun Crisis in AI: Cost and Stability'
url: https://www.emergentmind.com/topics/rerun-crisis
type: topic
---

# Rerun Crisis in AI: Cost and Stability

Searching arXiv for the cited papers on “Rerun Crisis” and related rerun-stability work.
The **Rerun Crisis** denotes a class of scalability and reproducibility failures in AI systems that arise when nominally repeated or semantically equivalent operations do not behave as stable, low-cost reruns. In the most explicit formulation, the term refers to the **redundant recomputation of deterministic workflows in continuous-loop agents, leading to linear growth in inference cost with respect to execution frequency** [2604.09718]. In a closely related but distinct measurement context, rerun analysis is used to establish a **rerun-stability baseline** for retrieval-augmented recommendation systems, against which natural-language paraphrases of the same buyer intent are shown to be substantially less stable than same-prompt reruns [2605.27440]. Taken together, these usages identify a broader systems problem: repeated execution, whether operational or evaluative, can be dominated by architectural redundancy or by prompt-phrasing artifacts rather than by underlying task semantics.

## 1. Definition and conceptual scope

The paper **"Agentic Compilation: Mitigating the LLM Rerun Crisis for Minimized-Inference-Cost Web Automation"** defines the Rerun Crisis as follows: *“The Rerun Crisis refers to the redundant recomputation of deterministic workflows in continuous-loop agents, leading to linear growth in inference cost with respect to execution frequency.”* This definition is presented for LLM-driven web agents that operate through continuous inference loops, repeatedly querying a model to evaluate browser state and select actions [2604.09718].

In that setting, the crisis is tied to **fixed, repeatable workflows** such as data extraction or form filling. The underlying logic does not change per run, yet standard agent architectures invoke the LLM at every step of every run. The result is **linear cost scaling** in both the number of sequential actions and the number of executions, making large-scale deployment economically difficult [2604.09718].

A second line of work uses reruns not to characterize execution cost, but to characterize **measurement stability**. In **"Paraphrase Brittleness in Production Retrieval-Augmented Commercial Recommendation: Reproducibility Below the Rerun-Stability Baseline"**, **rerun stability** is defined as the similarity of brand recommendation sets generated when the **identical prompt is issued multiple times** under otherwise constant conditions. That baseline is then contrasted with **paraphrase brittleness**, which measures the similarity of recommendation sets for different paraphrases of the same underlying buyer intent [2605.27440].

This suggests that “Rerun Crisis” has acquired a broader technical meaning in contemporary AI discourse: repeated execution or repeated evaluation may fail not because the task is intrinsically hard, but because the surrounding architecture or measurement unit is poorly aligned with the deterministic or intent-level structure of the problem.

## 2. Architectural form: continuous-loop web agents

In web automation, the Rerun Crisis is presented as a **structural, not temporary** bottleneck. The paper states that the problem is not merely one of current API pricing; it follows directly from how existing frameworks such as **ReAct** and **Chain-of-Thought** are architected. For deterministic workflows, recomputing the action plan at each step and at each iteration is described as wasteful because the LLM repeats identical or nearly identical inferences [2604.09718].

The paper formalizes the cost of a continuous agent for a workflow of \(N\) actions repeated \(M\) times, where each DOM snapshot at step \(i\) contains \(S_i\) tokens and the per-token inference cost is \(C_t\), as

\[
\text{Cost}_{cont} = M \times \sum_{i=1}^{N} [S_i \times C_t].
\]

The stated scaling is \(O(M \times N)\). Even with caching or diffing, the paper says the cost remains proportional to \(M \times N\) [2604.09718].

The paper gives a concrete example: a **5-step workflow** over **500 iterations** can cost approximately **150.00 USD** in inference costs, and even with **aggressive caching** this remains near **15.00 USD**. It further notes that community benchmarks show **1.00–3.20 USD per typical workflow on leading models** [2604.09718].

The same paper contrasts this with a **Compile-and-Execute** design that decouples LLM reasoning from browser execution. A **one-shot LLM invocation** processes a token-efficient semantic representation from a **DOM Sanitization Module (DSM)** and emits a deterministic **JSON workflow blueprint**. A lightweight runtime then drives the browser **without further model queries**. The corresponding cost is formalized as

\[
\text{Cost}_{oneshot} = 1 \times (S_{compile} \times C_t) + C_{exec},
\]

with **amortized \(O(1)\)** inference scaling rather than \(O(M \times N)\) [2604.09718].

The DSM is described as compressing the web page’s HTML to a semantic, compact “skeleton,” with **up to 85% reduction in token size**, pruning non-content nodes such as `<script>`, `<style>`, and hidden elements, stripping volatile CSS classes, and preserving stable identifiers such as `data-*` attributes and ARIA roles [2604.09718]. The LLM is prompted once with the sanitized DOM, URL, and natural-language user intent, and the execution engine interprets the JSON blueprint deterministically, stopping on error or completion [2604.09718].

## 3. Quantitative signatures in automation

The empirical signature of the web-automation Rerun Crisis is the gap between continuous inference and one-shot compilation. The paper reports **per-compilation costs between 0.002 USD and 0.092 USD across five frontier models**, in contrast to the **150 USD** continuous-loop example for 500 iterations and the **15 USD** cached case [2604.09718].

It also reports the following per-full-compilation results:

| Model | Cost (USD) | Success |
|---|---:|---|
| Claude Opus 4.6 | 0.0916 | ✓ |
| Claude Sonnet 4.5 | 0.0599 | ✓ |
| GPT-5.2-Codex | 0.0377 | ✓ |
| Qwen3.5 397B | 0.0172 | ✓ |
| Qwen3 Coder Next | 0.0020 | ✓ |

The paper characterizes this as a **1,500x** lower cost relative to an unoptimized continuous agent for realistic workflow sizes [2604.09718].

Reliability is evaluated across **data extraction**, **form filling**, and **tech stack detection**. The reported **zero-shot compilation success rates** are **80–94%**, and when a workflow compiles, the reported **execution accuracy** is **95–98%**. The paper attributes many failures to localized issues such as a **bad selector** or **schema violation**, which are described as easily patched via **Human-in-the-Loop (HITL)** review because of the modular JSON blueprint [2604.09718].

The task-level results are reported as follows:

| Task modality | Successful blueprints | Execution accuracy |
|---|---:|---:|
| High-Volume Extraction | 46/50 (92%) | 98.0% |
| Form Filling | 8/10 (80%) | 95.0% |
| Tech Stack Detection | 47/50 (94%) | 96.0% |

The paper states that minimal HITL patching can elevate execution reliability to **near-100%** before execution [2604.09718].

The formal guarantee is limited to **deterministic, repeatable tasks**. The paper explicitly says the architecture is **not suitable** for **Canvas/WebGL apps**, **pixel-only UIs**, or **extremely dynamic UIs without stable semantic markers**, and that tasks requiring **real-time policy changes** or adaptation to unexpected UI events may still require a continuous agent approach [2604.09718]. It also notes that recompilation cost scales with the **rate of change** in the environment, denoted \(O(R)\), rather than with the number of executions \(O(M \times N)\) [2604.09718].

## 4. Measurement form: rerun stability and paraphrase brittleness

A second technical setting uses reruns as a baseline for evaluating recommendation reproducibility. In retrieval-augmented commercial recommendation, the key comparison is between **same-prompt reruns** and **different paraphrases of the same underlying buyer intent** [2605.27440].

The paper defines **rerun stability** as the similarity of brand recommendation sets when the **identical prompt** is issued multiple times under constant model and tool conditions. **Paraphrase brittleness** is then defined as the similarity of recommendation sets for different paraphrases of the same intent, such as **“best CRM software”** versus **“top CRM tools”** or **“CRM for SaaS startups”** [2605.27440].

Similarity is measured using **Jaccard similarity**:

\[
J(A, B) = \frac{|A \cap B|}{|A \cup B|}.
\]

The study reports approximately **6,000 same-prompt rerun controls** and approximately **6,000 paraphrase runs**, with fixed system prompt, temperature, and tool description, all runs completed on the same day to minimize model drift, and brand extraction performed by two LLM judges in consensus mode [2605.27440].

The central quantitative finding is that the **same-prompt rerun baseline** is substantially higher than paraphrase stability. Reported **recommendation-slot Jaccard** for within-cell same-prompt reruns is **0.50–0.61**. By contrast, **cosmetic rewordings** yield **0.288** with clustered **95% CI [0.215, 0.361]**, and **constraint-adding rewordings** yield **0.135** with **95% CI [0.098, 0.175]** [2605.27440].

The paper’s summary table is:

| Condition | Recommendation-slot Jaccard |
|---|---:|
| Within-cell rerun (same prompt) | 0.50–0.61 |
| Cross-provider rerun (same prompt) | 0.33 |
| Within-provider effort pairing (same prompt, low/high) | 0.59–0.64 |
| Cosmetic paraphrase | 0.288 ([0.215, 0.361]) |
| Constraint-adding paraphrase | 0.135 ([0.098, 0.175]) |

The paper further gives the transformation from Jaccard to per-list overlap:

\[
\text{Overlap} = \frac{2J}{1+J},
\]

which yields approximately **44.7%** overlap for \(J = 0.288\) and approximately **23.8%** overlap for \(J = 0.135\). The corresponding turnover is

\[
\text{Turnover} = 1 - \frac{2J}{1+J},
\]

which the paper reports as roughly **55%** for cosmetic paraphrase and roughly **76%** for constraint-adding paraphrase [2605.27440].

The paper also states that increasing single-sample reasoning effort does **not** close the gap: Jaccard shifts by **\(-0.015\) to \(+0.005\)** for reruns and is bounded by **\(\pm 0.05\)** for paraphrase conditions [2605.27440].

## 5. Mechanisms, interpretation, and controversy

The web-automation interpretation is architectural. The paper argues that the crisis arises because the system repeatedly recomputes the same plan for deterministic workflows. Its proposed remedy is to **compile once** and **execute many times**, using a deterministic intermediate representation and a non-probabilistic execution engine [2604.09718].

The recommendation-system interpretation is measurement-theoretic. The paper’s key empirical claim is that **the prompt string, not the underlying buyer intent, is the dominant input to which brands surface**. It argues that small wording changes trigger large shifts in retrieval, and that models then elaborate on whichever brands the retrieval step surfaces [2605.27440].

These two mechanisms are distinct. In the first, the problem is **redundant recomputation** of a stable workflow. In the second, the problem is **structural instability of the measurement unit**, because a fixed prompt is not a faithful proxy for an intent distribution. A plausible implication is that the term “Rerun Crisis” now spans both **execution inefficiency** and **evaluation unreliability**, provided the common feature is a mismatch between repeated system use and the semantic object one actually wishes to preserve.

The recommendation paper frames this mismatch as a direct challenge to **AEO/GEO** practice. It argues that tracking a brand’s **“AI visibility”** by counting mentions over a fixed set of prompts yields a metric whose dominant source of variance is **which paraphrase the tracker happens to issue**, not the model’s behavior toward the brand. It concludes that **“Prompt-by-prompt mention tracking is structurally unstable as a unit of measurement.”** [2605.27440]

The automation paper, by contrast, frames the issue as a scalability barrier for enterprise web workflows. It states that the continuous-loop architecture blocks economically viable scaling even as raw hardware or model pricing improves, because the inefficiency is built into the interaction pattern itself [2604.09718].

## 6. Mitigation strategies and limits

For web automation, the primary mitigation is the **Compile-and-Execute architecture**. Its three main stages are the **DOM Sanitization Module**, **One-Shot LLM Compilation**, and a **Deterministic Execution Engine**. The JSON intermediate representation is central because it is modular, machine-readable, and amenable to **localized, code-free corrections** by human reviewers [2604.09718].

The paper states that the architecture is **model-agnostic**, that the **\(O(1)\)** cost guarantee holds for any model on deterministic fixed workflows, and that operational reliability should increase as LLMs improve. However, it also states that the approach is constrained to **repetitive, structure-amenable workflows** and is not a universal substitute for continuous agents [2604.09718].

For recommendation evaluation, the paper discusses **sampling many paraphrases per intent** as a mitigation that reduces artifact **in principle**, but argues that the natural buyer-phrasing space is much larger than any commercial tracker’s prompt budget. It therefore concludes that meaningful improvement likely requires **“a different unit rather than a larger prompt set.”** It further suggests that **full paraphrase-space integration** or **downstream metrics** such as buyer conversions may better reflect intent-level behavior because they average over buyer phrasing rather than fixing a single prompt string [2605.27440].

The recommendation paper also identifies scope limits: it is based on **single-day measurement**, has **English/US/UK/EU bias**, uses **consensus-based brand extraction**, and keeps the reference catalog and retrieval system fixed [2605.27440]. The automation paper similarly notes empirical scope limits, emphasizing that its benchmarks come from **proprietary enterprise use cases**, not public agent benchmarks, because the goal is to isolate the **economic scaling claim** [2604.09718].

## 7. Significance and relation to broader rerun methodology

The term “rerun” also appears in other technical literatures as a method for quantifying counterfactual improvement. In transmission resilience analysis, historical outage data can be **rerun** after modifying outages or restoration times in order to recompute resilience metrics such as **Event Size**, **Nadir**, **MVA-days out**, and **\(D_{95\%}\)** [2501.06042]. That usage is methodologically different from the Rerun Crisis, but it is relevant because it treats rerunning as a disciplined way to separate underlying event structure from hypothetical interventions.

This suggests a useful contrast. In transmission-resilience reruns, repeated replay is a tool for **quantifying benefits** under controlled modifications [2501.06042]. In the Rerun Crisis literature, repeated replay reveals that the system itself may be the source of uncontrolled variance or unnecessary cost [2604.09718; 2605.27440].

Accordingly, the contemporary significance of the Rerun Crisis lies in its identification of a failure mode that is easy to overlook in benchmark-centric evaluation. A system may perform adequately on single executions while remaining economically non-scalable under repetition, or it may appear measurable under fixed prompts while its outputs are actually dominated by paraphrase sensitivity rather than intent-level consistency. The emerging literature treats these as foundational design issues rather than incidental implementation defects [2604.09718; 2605.27440].

Source: https://www.emergentmind.com/topics/rerun-crisis