Rerun Crisis in AI: Cost and Stability

• Rerun Crisis is a phenomenon where repeated execution of fixed AI workflows causes linear scaling of inference costs due to redundant recomputation.
• A compile-and-execute approach decouples expensive continuous inference, reducing costs by up to 1,500x in deterministic web automation tasks.
• In recommendation systems, rerun analysis reveals that identical-prompt executions yield higher stability than paraphrased prompts, questioning reliable measurement.

Searching arXiv for the cited papers on “Rerun Crisis” and related rerun-stability work. The Rerun Crisis denotes a class of scalability and reproducibility failures in AI systems that arise when nominally repeated or semantically equivalent operations do not behave as stable, low-cost reruns. In the most explicit formulation, the term refers to the redundant recomputation of deterministic workflows in continuous-loop agents, leading to linear growth in inference cost with respect to execution frequency (Chundru, 8 Apr 2026). In a closely related but distinct measurement context, rerun analysis is used to establish a rerun-stability baseline for retrieval-augmented recommendation systems, against which natural-language paraphrases of the same buyer intent are shown to be substantially less stable than same-prompt reruns (Jack et al., 22 May 2026). Taken together, these usages identify a broader systems problem: repeated execution, whether operational or evaluative, can be dominated by architectural redundancy or by prompt-phrasing artifacts rather than by underlying task semantics.

1. Definition and conceptual scope

The paper "Agentic Compilation: Mitigating the LLM Rerun Crisis for Minimized-Inference-Cost Web Automation" defines the Rerun Crisis as follows: “The Rerun Crisis refers to the redundant recomputation of deterministic workflows in continuous-loop agents, leading to linear growth in inference cost with respect to execution frequency.” This definition is presented for LLM-driven web agents that operate through continuous inference loops, repeatedly querying a model to evaluate browser state and select actions (Chundru, 8 Apr 2026).

In that setting, the crisis is tied to fixed, repeatable workflows such as data extraction or form filling. The underlying logic does not change per run, yet standard agent architectures invoke the LLM at every step of every run. The result is linear cost scaling in both the number of sequential actions and the number of executions, making large-scale deployment economically difficult (Chundru, 8 Apr 2026).

A second line of work uses reruns not to characterize execution cost, but to characterize measurement stability. In "Paraphrase Brittleness in Production Retrieval-Augmented Commercial Recommendation: Reproducibility Below the Rerun-Stability Baseline", rerun stability is defined as the similarity of brand recommendation sets generated when the identical prompt is issued multiple times under otherwise constant conditions. That baseline is then contrasted with paraphrase brittleness, which measures the similarity of recommendation sets for different paraphrases of the same underlying buyer intent (Jack et al., 22 May 2026).

This suggests that “Rerun Crisis” has acquired a broader technical meaning in contemporary AI discourse: repeated execution or repeated evaluation may fail not because the task is intrinsically hard, but because the surrounding architecture or measurement unit is poorly aligned with the deterministic or intent-level structure of the problem.

2. Architectural form: continuous-loop web agents

In web automation, the Rerun Crisis is presented as a structural, not temporary bottleneck. The paper states that the problem is not merely one of current API pricing; it follows directly from how existing frameworks such as ReAct and Chain-of-Thought are architected. For deterministic workflows, recomputing the action plan at each step and at each iteration is described as wasteful because the LLM repeats identical or nearly identical inferences (Chundru, 8 Apr 2026).

The paper formalizes the cost of a continuous agent for a workflow of $N$ actions repeated $M$ times, where each DOM snapshot at step $i$ contains $S_i$ tokens and the per-token inference cost is $C_t$, as

$$\text{Cost}_{cont} = M \times \sum_{i=1}^{N} [S_i \times C_t].$$

The stated scaling is $O(M \times N)$. Even with caching or diffing, the paper says the cost remains proportional to $M \times N$ (Chundru, 8 Apr 2026).

The paper gives a concrete example: a 5-step workflow over 500 iterations can cost approximately 150.00 USD in inference costs, and even with aggressive caching this remains near 15.00 USD. It further notes that community benchmarks show 1.00–3.20 USD per typical workflow on leading models (Chundru, 8 Apr 2026).

The same paper contrasts this with a Compile-and-Execute design that decouples LLM reasoning from browser execution. A one-shot LLM invocation processes a token-efficient semantic representation from a DOM Sanitization Module (DSM) and emits a deterministic JSON workflow blueprint. A lightweight runtime then drives the browser without further model queries. The corresponding cost is formalized as

$$\text{Cost}_{oneshot} = 1 \times (S_{compile} \times C_t) + C_{exec},$$

with amortized $O(1)$ inference scaling rather than $M$0 (Chundru, 8 Apr 2026).

The DSM is described as compressing the web page’s HTML to a semantic, compact “skeleton,” with up to 85% reduction in token size, pruning non-content nodes such as <script>, <style>, and hidden elements, stripping volatile CSS classes, and preserving stable identifiers such as data-* attributes and ARIA roles (Chundru, 8 Apr 2026). The LLM is prompted once with the sanitized DOM, URL, and natural-language user intent, and the execution engine interprets the JSON blueprint deterministically, stopping on error or completion (Chundru, 8 Apr 2026).

3. Quantitative signatures in automation

The empirical signature of the web-automation Rerun Crisis is the gap between continuous inference and one-shot compilation. The paper reports per-compilation costs between 0.002 USD and 0.092 USD across five frontier models, in contrast to the 150 USD continuous-loop example for 500 iterations and the 15 USD cached case (Chundru, 8 Apr 2026).

It also reports the following per-full-compilation results:

Model	Cost (USD)	Success
Claude Opus 4.6	0.0916	✓
Claude Sonnet 4.5	0.0599	✓
GPT-5.2-Codex	0.0377	✓
Qwen3.5 397B	0.0172	✓
Qwen3 Coder Next	0.0020	✓

The paper characterizes this as a 1,500x lower cost relative to an unoptimized continuous agent for realistic workflow sizes (Chundru, 8 Apr 2026).

Reliability is evaluated across data extraction, form filling, and tech stack detection. The reported zero-shot compilation success rates are 80–94%, and when a workflow compiles, the reported execution accuracy is 95–98%. The paper attributes many failures to localized issues such as a bad selector or schema violation, which are described as easily patched via Human-in-the-Loop (HITL) review because of the modular JSON blueprint (Chundru, 8 Apr 2026).

The task-level results are reported as follows:

Task modality	Successful blueprints	Execution accuracy
High-Volume Extraction	46/50 (92%)	98.0%
Form Filling	8/10 (80%)	95.0%
Tech Stack Detection	47/50 (94%)	96.0%

The paper states that minimal HITL patching can elevate execution reliability to near-100% before execution (Chundru, 8 Apr 2026).

The formal guarantee is limited to deterministic, repeatable tasks. The paper explicitly says the architecture is not suitable for Canvas/WebGL apps, pixel-only UIs, or extremely dynamic UIs without stable semantic markers, and that tasks requiring real-time policy changes or adaptation to unexpected UI events may still require a continuous agent approach (Chundru, 8 Apr 2026). It also notes that recompilation cost scales with the rate of change in the environment, denoted $M$1, rather than with the number of executions $M$2 (Chundru, 8 Apr 2026).

4. Measurement form: rerun stability and paraphrase brittleness

A second technical setting uses reruns as a baseline for evaluating recommendation reproducibility. In retrieval-augmented commercial recommendation, the key comparison is between same-prompt reruns and different paraphrases of the same underlying buyer intent (Jack et al., 22 May 2026).

The paper defines rerun stability as the similarity of brand recommendation sets when the identical prompt is issued multiple times under constant model and tool conditions. Paraphrase brittleness is then defined as the similarity of recommendation sets for different paraphrases of the same intent, such as “best CRM software” versus “top CRM tools” or “CRM for SaaS startups” (Jack et al., 22 May 2026).

Similarity is measured using Jaccard similarity:

$M$3

The study reports approximately 6,000 same-prompt rerun controls and approximately 6,000 paraphrase runs, with fixed system prompt, temperature, and tool description, all runs completed on the same day to minimize model drift, and brand extraction performed by two LLM judges in consensus mode (Jack et al., 22 May 2026).

The central quantitative finding is that the same-prompt rerun baseline is substantially higher than paraphrase stability. Reported recommendation-slot Jaccard for within-cell same-prompt reruns is 0.50–0.61. By contrast, cosmetic rewordings yield 0.288 with clustered 95% CI [0.215, 0.361], and constraint-adding rewordings yield 0.135 with 95% CI [0.098, 0.175] (Jack et al., 22 May 2026).

The paper’s summary table is:

Condition	Recommendation-slot Jaccard
Within-cell rerun (same prompt)	0.50–0.61
Cross-provider rerun (same prompt)	0.33
Within-provider effort pairing (same prompt, low/high)	0.59–0.64
Cosmetic paraphrase	0.288 ([0.215, 0.361])
Constraint-adding paraphrase	0.135 ([0.098, 0.175])

The paper further gives the transformation from Jaccard to per-list overlap:

$M$4

which yields approximately 44.7% overlap for $M$5 and approximately 23.8% overlap for $M$6. The corresponding turnover is

$M$7

which the paper reports as roughly 55% for cosmetic paraphrase and roughly 76% for constraint-adding paraphrase (Jack et al., 22 May 2026).

The paper also states that increasing single-sample reasoning effort does not close the gap: Jaccard shifts by $M$8 to $M$9 for reruns and is bounded by $i$0 for paraphrase conditions (Jack et al., 22 May 2026).

5. Mechanisms, interpretation, and controversy

The web-automation interpretation is architectural. The paper argues that the crisis arises because the system repeatedly recomputes the same plan for deterministic workflows. Its proposed remedy is to compile once and execute many times, using a deterministic intermediate representation and a non-probabilistic execution engine (Chundru, 8 Apr 2026).

The recommendation-system interpretation is measurement-theoretic. The paper’s key empirical claim is that the prompt string, not the underlying buyer intent, is the dominant input to which brands surface. It argues that small wording changes trigger large shifts in retrieval, and that models then elaborate on whichever brands the retrieval step surfaces (Jack et al., 22 May 2026).

These two mechanisms are distinct. In the first, the problem is redundant recomputation of a stable workflow. In the second, the problem is structural instability of the measurement unit, because a fixed prompt is not a faithful proxy for an intent distribution. A plausible implication is that the term “Rerun Crisis” now spans both execution inefficiency and evaluation unreliability, provided the common feature is a mismatch between repeated system use and the semantic object one actually wishes to preserve.

The recommendation paper frames this mismatch as a direct challenge to AEO/GEO practice. It argues that tracking a brand’s “AI visibility” by counting mentions over a fixed set of prompts yields a metric whose dominant source of variance is which paraphrase the tracker happens to issue, not the model’s behavior toward the brand. It concludes that “Prompt-by-prompt mention tracking is structurally unstable as a unit of measurement.” (Jack et al., 22 May 2026)

The automation paper, by contrast, frames the issue as a scalability barrier for enterprise web workflows. It states that the continuous-loop architecture blocks economically viable scaling even as raw hardware or model pricing improves, because the inefficiency is built into the interaction pattern itself (Chundru, 8 Apr 2026).

6. Mitigation strategies and limits

For web automation, the primary mitigation is the Compile-and-Execute architecture. Its three main stages are the DOM Sanitization Module, One-Shot LLM Compilation, and a Deterministic Execution Engine. The JSON intermediate representation is central because it is modular, machine-readable, and amenable to localized, code-free corrections by human reviewers (Chundru, 8 Apr 2026).

The paper states that the architecture is model-agnostic, that the $i$1 cost guarantee holds for any model on deterministic fixed workflows, and that operational reliability should increase as LLMs improve. However, it also states that the approach is constrained to repetitive, structure-amenable workflows and is not a universal substitute for continuous agents (Chundru, 8 Apr 2026).

For recommendation evaluation, the paper discusses sampling many paraphrases per intent as a mitigation that reduces artifact in principle, but argues that the natural buyer-phrasing space is much larger than any commercial tracker’s prompt budget. It therefore concludes that meaningful improvement likely requires “a different unit rather than a larger prompt set.” It further suggests that full paraphrase-space integration or downstream metrics such as buyer conversions may better reflect intent-level behavior because they average over buyer phrasing rather than fixing a single prompt string (Jack et al., 22 May 2026).

The recommendation paper also identifies scope limits: it is based on single-day measurement, has English/US/UK/EU bias, uses consensus-based brand extraction, and keeps the reference catalog and retrieval system fixed (Jack et al., 22 May 2026). The automation paper similarly notes empirical scope limits, emphasizing that its benchmarks come from proprietary enterprise use cases, not public agent benchmarks, because the goal is to isolate the economic scaling claim (Chundru, 8 Apr 2026).

7. Significance and relation to broader rerun methodology

The term “rerun” also appears in other technical literatures as a method for quantifying counterfactual improvement. In transmission resilience analysis, historical outage data can be rerun after modifying outages or restoration times in order to recompute resilience metrics such as Event Size, Nadir, MVA-days out, and $i$2 (Ahmad et al., 10 Jan 2025). That usage is methodologically different from the Rerun Crisis, but it is relevant because it treats rerunning as a disciplined way to separate underlying event structure from hypothetical interventions.

This suggests a useful contrast. In transmission-resilience reruns, repeated replay is a tool for quantifying benefits under controlled modifications (Ahmad et al., 10 Jan 2025). In the Rerun Crisis literature, repeated replay reveals that the system itself may be the source of uncontrolled variance or unnecessary cost (Chundru, 8 Apr 2026, Jack et al., 22 May 2026).

Accordingly, the contemporary significance of the Rerun Crisis lies in its identification of a failure mode that is easy to overlook in benchmark-centric evaluation. A system may perform adequately on single executions while remaining economically non-scalable under repetition, or it may appear measurable under fixed prompts while its outputs are actually dominated by paraphrase sensitivity rather than intent-level consistency. The emerging literature treats these as foundational design issues rather than incidental implementation defects (Chundru, 8 Apr 2026, Jack et al., 22 May 2026).