DragonCrawl: LLM-Driven Mobile Chaos Testing

• DragonCrawl is an LLM-based platform for mobile chaos testing that dynamically navigates mobile apps using natural language interpretation and adaptive test execution.
• It orchestrates testing by integrating LLM-driven UI navigation with uHavoc fault injection to simulate degraded backend conditions and trigger real-time validations.
• Empirical results highlight 99% test reliability and savings of 39,000 testing hours, with successful detection of critical resilience issues in Uber’s apps.

Searching arXiv for the specified paper to ground the article and citation. DragonCrawl is an LLM-based mobile testing platform and the LLM-driven core of Uber’s mobile chaos-testing pipeline. In the system described in "Scaling Mobile Chaos Testing with AI-Driven Test Execution" (Marcano et al., 5 Feb 2026), DragonCrawl sits in front of instrumented iOS and Android test devices, dynamically crawls through screens by interpreting their content in natural language, and coordinates with uHavoc, a service-level fault injection system. The platform is designed to address the combinatorial explosion of flows, locations, and failure scenarios that arise when validating mobile applications under degraded backend conditions. Since Q1 2024, the combined system has executed over 180,000 automated chaos tests across 47 critical flows in Uber’s Rider, Driver, and Eats applications, representing approximately 39,000 hours of manual testing effort (Marcano et al., 5 Feb 2026).

1. Definition and system role

DragonCrawl is presented as the adaptive AI-driven execution layer in an automated mobile chaos-testing system. Its function is not merely to replay fixed scripts, but to navigate mobile applications under degraded backend conditions without requiring manually written test cases for each combination of user flow, city, and failure type. In production, it operates with instrumented mobile app instances that support header injection for uHavoc, network logging through an in-app bug reporter, and distributed tracing through Jaeger (Marcano et al., 5 Feb 2026).

Within the overall pipeline, DragonCrawl is downstream of a Test Scheduler and upstream of assertion, failure detection, and automated root cause analysis. The Test Scheduler chooses a user flow and a fault-injection scenario, then renders a DragonCrawl test specification containing a Flow Definition, Mid-state Assertions, End-state Assertions, and a Fault Configuration. DragonCrawl Test Runner then launches a fresh mobile app instance, visits screens, captures XML or textual hierarchies and screenshots, and issues prompts to GPT-4o at each step. A failure is registered if the end screen is never reached or a key element is missing; upon failure, logs, spans, and screen transition traces are collected for automated RCA and ticketing (Marcano et al., 5 Feb 2026).

This organization suggests that DragonCrawl should be understood as an execution and orchestration substrate for mobile resilience validation rather than as a standalone UI crawler.

2. Architecture and integration with uHavoc

The system architecture is defined by a header-based coupling between DragonCrawl and uHavoc. DragonCrawl wraps every outgoing mobile RPC in the test with the uHavoc headers specified in the test’s Fault Configuration block, including X-Fault-Tier, X-Fault-Type, and X-Fault-Scope. The instrumented mobile app then issues RPCs carrying these headers, and uHavoc intercepts and applies the specified failure at the service mesh. The paper states that this guarantees that only test traffic sees injected errors while production traffic remains safe (Marcano et al., 5 Feb 2026).

The pipeline can be summarized as follows:

Stage	Function	Key artifacts
Test Scheduler	Chooses flow and fault scenario; renders test specification	Goal, assertions, fault configuration
DragonCrawl Test Runner	Launches app, captures screens, prompts GPT-4o	XML/text hierarchies, screenshots
uHavoc Fault Injector	Intercepts tagged RPCs and injects faults	Aborts, timeouts, latency
Assertion & Failure Detection	Verifies UI conditions with VQA	Mid-state and end-state checks
Automated RCA & Ticketing	Ranks offending RPCs and files tickets	Network logs, Jaeger spans, screen trace

This architecture is explicitly intended to connect end-user mobile behavior with service-level fault injection. A plausible implication is that the system closes a gap between backend-centric chaos engineering and user-visible resilience validation, because the same test run simultaneously exercises the client UI, the network path, and backend service dependencies.

3. Test specification and execution model

DragonCrawl models test execution as a next-action retrieval problem. At step $t$, it has a textual description of the current screen $S_t$, a high-level goal $G$, and a history of previous actions $H = (a_1,\ldots,a_{t-1})$. From the XML hierarchy it enumerates a small candidate set $A_t$ of legal UI actions, such as tapping an element or filling a field, and scores each candidate according to the LLM’s conditional probability. The selected action is

$a_t = \arg\max_{a \in A_t} P(a \mid S_t, G, H).$

The prompt template used during execution is stated explicitly: “You are an automated tester. The screen shows: <text of S_t>. Goal: <G>. History: <H>. Which UI action do you take now? Return JSON {“action”:..., “target_id”:...}.” Assertions are evaluated with a VQA prompt of the form: “Given this screenshot, does element ‘X’ appear? Answer True or False.” (Marcano et al., 5 Feb 2026)

Offline evaluation uses a Precision@k retrieval metric:

$$\mathrm{Precision@k} = \frac{1}{N}\sum_{i=1}^{N} \frac{|\{\mathrm{relevant\_actions\_in\_top\_k_i}\}|}{k}.$$

The execution formalism is significant because it frames mobile interaction under faulted conditions as constrained action selection over a dynamic UI state rather than as deterministic script replay. This suggests why the system can remain effective when backend degradation changes latency, partial rendering, or navigation pathways.

4. Adaptive execution under degraded backend conditions

A central property of DragonCrawl is adaptive AI-driven execution. The execution loop adapts to degraded backend conditions through three mechanisms specified in the paper: fault-aware timeouts, cycle detection, and baseline comparison. Actions that time out under Tier 2 faults trigger alternate retry logic. If the same action repeats more than $m$ times, the system re-prompts with an instruction to avoid previous actions. Every chaos test is paired with a control run without faults, and only delta errors are triaged (Marcano et al., 5 Feb 2026).

The paper expresses the control flow in LaTeX-style pseudocode:

$$\begin{aligned} & \textbf{while not end\_state and } t < T_{\max}: \ & \quad S_t \leftarrow \text{capture\_screen()} \ & \quad A_t \leftarrow \text{enumerate\_actions}(S_t) \ & \quad a_t \leftarrow \arg\max_{a\in A_t} P(a\mid S_t, G, H) \ & \quad \text{if cycle\_detected}(a_t,H):\; \text{re\_prompt\_avoid\_history}() \ & \quad \text{execute}(a_t);\; H\leftarrow H\cup\{a_t\}. \end{aligned}$$

This design addresses a recurrent weakness of conventional mobile automation in faulted environments: backend failures can perturb timing, reorder visible affordances, or prevent expected screens from appearing. The explicit use of baseline comparison further constrains triage to behavior attributable to injected faults rather than to preexisting flakiness. In that sense, DragonCrawl is not simply adaptive at the UI level; it is adaptive in a fault-differential sense.

5. Empirical performance and retrieval metrics

The reported evaluation covers execution reliability, retrieval precision, and automated root-cause attribution. Since Q1 2024, the system has run 180,000+ automated chaos-test executions across 47 core flows, with estimated savings of 39,000 manual-testing hours. Test reliability under fault injection is reported as at least 99%, and automated RCA reaches precision@5 of 0.88 (Marcano et al., 5 Feb 2026).

The paper reports DragonCrawl Precision@k under T2–T5 faults as follows:

Metric	Baseline	T5	T4	T3	T2
Precision@1	0.9723	0.9716	0.9716	0.9512	0.9408
Precision@2	0.9623	0.9641	0.9641	0.9473	0.9408
Precision@3	0.9423	0.9398	0.9398	0.9318	0.9408

Root Cause Analysis Precision@k is reported as 0.50 at Precision@1, 0.71 at Precision@3, and 0.88 at Precision@5. The Root Cause score for an RPC call $c$ is defined as

$S_t$0

where $S_t$1 for 5xx, 4xx, and 2xx respectively; $S_t$2 is the baseline error frequency; $S_t$3; and $S_t$4 from the LLM’s semantic endpoint classification (Marcano et al., 5 Feb 2026).

These results indicate that degradation in action-selection precision under stronger fault tiers remains limited relative to baseline. A plausible implication is that the next-action retrieval framing is robust to moderate UI perturbation induced by backend failures.

6. Operational findings and resilience defects

Operationally, the system is described as running nightly at scale. Each night, all 47 DragonCrawl-supported flows run under Tier 2+ fault scenarios. The reported pass rate is 99.27%, with pre-deployment latencies of approximately $S_t$5–$S_t$6 s and $S_t$7–$S_t$8 s. Automating account provisioning and fault setup requires 2 RPC calls with $S_t$9 s, and the paper states that DragonCrawl’s 7 min supervised-test equivalent, compared with a 13 min manual test, yields full automation and cumulative savings of 39,000 hours to date (Marcano et al., 5 Feb 2026).

The most consequential reported outcome is issue discovery. The system identified 23 resilience risks; 70% were architectural dependency violations in which non-critical service failures degraded core user flows. Twelve issues were severe enough to prevent trip requests or food orders, and two caused application crashes that were detectable only through mobile chaos testing rather than backend testing alone (Marcano et al., 5 Feb 2026).

The paper also reports a monthly issue-discovery trend peaking in early 2024 and tailing off as teams remediated most risks, as well as a service-tier interaction heatmap highlighting hot spots at Tier 2 to Tier 1 failures. These observations suggest that DragonCrawl was used not only as a test executor but also as an empirical instrument for identifying architectural coupling patterns.

7. Interpretation, scope, and common misconceptions

A common misconception in resilience validation is that backend testing alone is sufficient to characterize end-user impact. The reported discovery of two crash defects detectable only through mobile chaos testing directly contradicts that assumption (Marcano et al., 5 Feb 2026). In the system described, failure manifestation depends on the interaction among mobile UI state, client RPC issuance, injected backend faults, and assertion logic; backend-only observation therefore omits part of the causal chain.

Another misconception is that large-scale chaos testing for mobile clients necessarily requires manually authored test cases for each scenario. DragonCrawl is explicitly introduced to eliminate the need to manually write test cases for each combination of user flow, city, and failure type. The mechanism enabling that substitution is adaptive AI-driven test execution grounded in screen interpretation, candidate-action enumeration, and fault-differential validation (Marcano et al., 5 Feb 2026).

The paper’s conclusion is that DragonCrawl’s LLM-driven adaptation to UI variations, combined with uHavoc’s safe service-level fault injection, breaks the combinatorial-explosion barrier in mobile chaos testing. From an encyclopedic standpoint, the salient significance of DragonCrawl lies in this synthesis: it joins LLM-mediated UI navigation, service-mesh fault injection, VQA-based assertion, and automated RCA into a continuous nightly validation loop for end-user resilience at production scale (Marcano et al., 5 Feb 2026).