uHavoc: Uber’s Mobile Chaos Testing Platform

• uHavoc is Uber’s service-level fault-injection platform designed to test mobile resilience in large-scale distributed systems.
• It features a three-part architecture—Fault Scheduler, Injection Filter, and Safety Guard—to inject controlled failures while preserving production safety.
• Integrated with DragonCrawl, uHavoc leverages adaptive AI-driven test execution to maintain high pass rates and enable rapid automated root-cause analysis.

Searching arXiv for the specified paper to ground the article and citation. uHavoc is Uber’s service-level fault-injection platform for resilience testing in mobile applications deployed over large-scale distributed systems. As described in "Scaling Mobile Chaos Testing with AI-Driven Test Execution" (Marcano et al., 5 Feb 2026), it is designed to work in concert with DragonCrawl, an LLM-based mobile testing platform, so that chaos experiments can span end-to-end mobile user flows under realistic backend failures. The system addresses a specific scaling problem: traditional chaos engineering approaches cannot cover the combinatorial explosion of flows, locations, and failure scenarios that arise in mobile environments, whereas adaptive AI-driven test execution can navigate degraded backend conditions without requiring manually written test cases for each combination.

1. System role in mobile chaos testing

uHavoc operates as an intermediary in Uber’s RPC service mesh. Its purpose is to inject service-level failures into test-tenancy traffic while preserving production safety, thereby enabling resilience validation at the level of complete mobile experiences rather than isolated backend interactions (Marcano et al., 5 Feb 2026). This placement is central: it allows failures to be introduced where mobile API requests traverse the service mesh, while still keeping the blast radius constrained to explicitly marked experiments.

The platform is presented as part of an integrated workflow with DragonCrawl. At test launch, DragonCrawl’s orchestrator provisions a test user account with the correct experiment flags and injects havoc headers into every mobile API request. Those headers then propagate through the client to the service mesh, where uHavoc applies the configured faults in real time. This architecture enables tests to exercise full user flows in Rider, Driver, and Eats under degraded backend conditions rather than merely simulating backend perturbations in isolation.

A plausible implication is that uHavoc is less a standalone chaos tool than a fault-injection substrate for AI-driven mobile execution. The paper’s emphasis on end-to-end flows, true/false VQA assertions, and adaptive navigation suggests that its main significance lies in making mobile chaos testing operationally scalable rather than in introducing a new abstract failure model.

2. Internal architecture and tenancy controls

uHavoc consists of three main subsystems (Marcano et al., 5 Feb 2026).

Subsystem	Function	Safety or isolation role
Fault Scheduler	Maintains per-test tenancy isolation and selects which services and failure types to target	Keeps experiments scoped per test
Injection Filter	Examines incoming RPCs for special havoc headers and applies faults according to configured rules	Acts only when matching header-based policies are present
Safety Guard	Enforces strict blacklists and blast-radius controls	Ensures only test-tenancy traffic is affected

All microservices on Uber’s Software Networking infrastructure are onboarded by default, and production requests carry no havoc headers. The system therefore preserves customer safety by design: traffic without the designated headers is not subject to injected failure. The Safety Guard further constrains impact by combining blacklists with blast-radius controls, which is particularly important because the platform sits directly in the RPC path.

Tier classifications from 0 to 5 reflect service criticality. These tiers can be used to scope injections to “non-critical” services only, which is operationally important in experiments intended to detect architectural dependency violations rather than induce indiscriminate failure. This tiering mechanism is also reused in policy matching and in downstream root-cause ranking.

3. Fault models and header-based injection semantics

uHavoc currently implements three service-level fault models: abort faults, timeout faults, and latency faults (Marcano et al., 5 Feb 2026). Abort faults inject error responses by returning configured 4xx or 5xx codes in place of a normal RPC reply. Timeout faults simulate complete network partitions by dropping calls after a specified deadline. Latency faults impose artificial delays, with the paper giving an example range of $100\mathrm{ms}$–$5\mathrm{second}$.

Each model is implemented through header-based injection. When the Injection Filter detects a havoc header, it consults an in-memory policy that matches caller and callee tiers and then either stalls the request, aborts it, or kills it. The paper gives the following example:

$5\mathrm{second}$3

This example makes clear that the injection decision is not merely service-specific; it is conditioned on service tiers and mode parameters. The design therefore supports controlled sweeps over service-criticality classes and failure types, which is important for systematic resilience validation across many flows.

The platform’s use of abort, timeout, and latency faults situates it squarely within service-level chaos engineering, but the paper’s results indicate that the consequences of such faults must be evaluated at the mobile UI level. In particular, the operational findings include failures that blocked trip requests or food orders and two native mobile crashes that were invisible to backend-only testing, which suggests that service-layer perturbations can manifest as client-visible failure modes not recoverable from server-side observability alone.

4. Integration with DragonCrawl and test execution model

Integrating uHavoc with DragonCrawl required extending DragonCrawl’s test specification to include a fault configuration block (Marcano et al., 5 Feb 2026). A typical enhanced test case contains a Flow Definition describing the end-to-end user steps, Mid-state and End-state Assertions written as true/false VQA prompts, and a Fault Configuration that names the failure types, the target service tiers, and any percentage-of-requests scope.

To guard against environmental noise and mobile UI drift, each chaos run is paired with a control run that carries no headers. This pairing allows test analysts to isolate genuine resilience failures from unrelated variation in the execution environment. That control strategy is especially important in mobile testing, where UI changes, asynchronous loading behavior, and device-level non-determinism can otherwise confound fault attribution.

DragonCrawl’s AI agent is capable of detecting injected slowdowns or missing UI elements and adapts its action sequence in real time by waiting for delayed screens or selecting alternate UI paths while still honoring the end-state goal. The paper states that this adaptive execution removes the need to manually author separate test scripts for each city, language, or failure scenario. In context, this is the core scaling mechanism of the overall system: uHavoc provides realistic perturbations, while DragonCrawl preserves test progress under those perturbations.

5. Scheduling, runtime adaptation, and evaluation metrics

Fault-injection scheduling occurs at two levels. Nightly pipelines iterate over all 47 core flows in Rider, Driver, and Eats, sweeping combinations of service-tier targets and failure models. Ad-hoc drills can also sample random subsets of services for region-failover simulations (Marcano et al., 5 Feb 2026). This dual mode distinguishes continuous regression-style validation from exploratory drills aimed at broader failure scenarios.

The system defines test reliability $R$ as the ratio of passing executions to total executions, where a “pass” means DragonCrawl reached the golden end-state screen within the timeout. Across 300 trip-flow runs per failure tier, uHavoc+DragonCrawl maintained $R \ge 0.99$ even when injecting latency or aborts into Tier-2 services. The controlled study ran 300 chaos experiments per fault tier on the trip-request flow, over 6,000 UI interactions each, and measured precision@k, latency percentiles $(P50, P95, P99)$, and pass rates.

Latency increased by up to 20 seconds at $P50$ when targeting Tier-2 failures but remained within DragonCrawl’s adaptive waiting thresholds. The paper reports that $P95$ and $P99$ showed larger spreads, reflecting UI pop-ups and fallback screens, while pass-rate under Tier-2 injection dipped only to 0.99. Nightly pass rates in production held at 99.27%.

These measurements are significant because they distinguish resilience testing from brittle scripted automation. A plausible implication is that the principal performance criterion is not minimal latency under fault but the ability to preserve end-state reachability and diagnostic fidelity despite injected degradation.

6. Automated root-cause analysis and operational findings

For automated root-cause analysis, uHavoc collects in-app network logs and examples of error screens, then applies a three-phase pipeline: error detection via regex and LLM classification, causal ranking using a composite score, and trace analysis to confirm the top candidates (Marcano et al., 5 Feb 2026). The score is defined as

$$\mathrm{Score}(c)=F_{\mathrm{status}(c)} \bigl(1 - \mathrm{normalFailureRate}(c)\bigr) F_{\mathrm{tier}(c)} F_{\mathrm{category}(c)}.$$

Here $F_{\mathrm{status}}$ assigns weights 1.0 for 5xx, 0.5 for 4xx, and 0.2 for 2xx; $5\mathrm{second}$0 gives 1.0 for Tier-0 down to 0.1 for Tier-5; and $5\mathrm{second}$1 uses LLM-derived semantic relevance scores. Evaluation treats RCA as a ranked recommendation problem and measures precision@k as

$5\mathrm{second}$2

In offline tests over real failures, Precision@5 reached 0.88, Precision@3 was 0.71, and Precision@1 was 0.50. The paper states that this reduced mean debug time from approximately 3 hours to under 5 minutes.

The production record since Q1 2024 is also quantitative. The integrated system executed over 180,000 automated chaos tests across 47 flows and saved an estimated 39,000 developer hours. It uncovered 23 resilience risks: 70% architectural dependency violations, 15% timeout misconfigurations, 10% missing fallbacks, and 5% circuit-breaker issues. Twelve of these issues blocked core trip or order flows, and two caused native mobile crashes invisible to backend-only tests. Teams now resolve discovered issues with a median fix time of 64 days, often dedicating sprints to resilience improvements.

One recurrent misconception in resilience engineering is that backend chaos testing alone is sufficient. The reported detection of two native mobile crashes that were not observable through backend-only testing directly contradicts that view. The evidence presented for uHavoc instead supports an end-to-end perspective in which service-level perturbations, mobile UI behavior, and automated RCA must be evaluated together.