Weak Unsoundness in Analysis Models

• Weak unsoundness is a structured relaxation of full soundness, where failures are permitted under constrained conditions such as precision orders, rare-event models, or restricted semantic envelopes.
• In static analysis, it is formalized using partial orders over analysis variants, highlighting how increased precision can reveal semantic gaps and inconsistencies.
• Applications span Java call-graph analysis, LLM-based GUI testing, and dynamic heap enrichment, emphasizing practical trade-offs between ideal guarantees and operational realism.

Weak unsoundness denotes a family of relaxed soundness notions used when full soundness is either unattainable, operationally too costly, or empirically contradicted by the behavior of real analysis and execution systems. In Java call-graph construction, it arises whenever a supposedly more precise analysis variant introduces edges instead of only pruning false positives, thereby revealing a violation of the expected precision order (Zhong et al., 1 Apr 2026). In natural-language GUI testing with LLM agents, it denotes the condition that conformant implementations pass under all normal contexts while false failures may still occur in a small-probability rare-event subset of contexts (Salva et al., 23 Sep 2025). In static-analysis practice more broadly, it is closely aligned with “practically unsound” analysis or “soundiness”: analyses that are sound under a restricted semantic model but unsound in the presence of reflection, native methods, invokedynamic, dynamic loading, and related dynamic constructs (Grech et al., 2019).

1. Conceptual position between soundness and arbitrary unsoundness

Weak unsoundness is not a single domain-independent formal property. Rather, it is a structured relaxation of soundness in which the permitted failures are constrained: by an expected precision order, by a rare-context model, or by an explicitly restricted semantic envelope. This distinguishes it from unrestricted unsoundness, which offers no systematic guarantee about missed behaviors or erroneous rejections.

For static call-graph construction, strong soundness can be stated as

$$\forall E \in Exec(P): Cg_{dyn}(E) \subseteq Cg_{stat}(P),$$

meaning that every dynamic call edge of every execution is contained in the statically computed call graph. The corresponding weakly unsound position, as described in the HeapDL work, is that the analysis over-approximates most control flow but admits known gaps whenever corner-case language features are unhandled (Grech et al., 2019). In the LLM-based GUI-testing setting, full soundness would require that no conformant implementation is ever falsely rejected, but the paper argues that this is unattainable in practice because natural-language instructions and LLM behavior are inherently unsound (Salva et al., 23 Sep 2025). In the call-graph consistency setting, weak unsoundness is diagnosed not by a dynamic oracle but by violations of declared precision or soundness partial orders, without requiring ground truth (Zhong et al., 1 Apr 2026).

Setting	Strong baseline	Weakly unsound relaxation
Static call-graph soundness	$$\forall E \in Exec(P): Cg_{dyn}(E) \subseteq Cg_{stat}(P)$$	Sound under a restricted semantic model; gaps for dynamic constructs
Call-graph precision refinement	More precision should only remove false positives	A “more precise” variant introduces new edges
NL GUI test execution	No conformant AUT is falsely rejected	False rejection occurs only in rare contexts

A plausible implication is that weak unsoundness is best understood as a domain-relative compromise between ideal semantic guarantees and operational realism, rather than as a universally fixed weakening of soundness.

2. Formalization in call-graph construction

The call-graph framework in “Detecting Call Graph Unsoundness without Ground Truth” introduces partial orders over analysis variants and uses them to formalize expected semantic relations among algorithms and configurations (Zhong et al., 1 Apr 2026). Let $P$ be a set of Java programs, and for any analysis variant $X$ and program $p \in P$, let $CG_X(p)$ be the produced call graph. Each edge is assumed semantically either a true-positive or a false-positive, although that classification is unknown in practice.

The precision partial order is defined by

$$X_1 \preceq_p X_2 \iff \forall p \in P.\; FP(CG_{X_2}(p)) \subseteq FP(CG_{X_1}(p)).$$

Thus, $X_2$ is at least as precise as $X_1$ if it never introduces new spurious edges beyond those already present in $X_1$. Dually, the soundness partial order is

$$\forall E \in Exec(P): Cg_{dyn}(E) \subseteq Cg_{stat}(P)$$0

so $$\forall E \in Exec(P): Cg_{dyn}(E) \subseteq Cg_{stat}(P)$$1 is at least as sound if it preserves all truly reachable edges of $$\forall E \in Exec(P): Cg_{dyn}(E) \subseteq Cg_{stat}(P)$$2.

The same work defines semantic consistency under an expected precision order $$\forall E \in Exec(P): Cg_{dyn}(E) \subseteq Cg_{stat}(P)$$3 by

$$\forall E \in Exec(P): Cg_{dyn}(E) \subseteq Cg_{stat}(P)$$4

Any edge newly introduced by the more precise variant must therefore have been spurious in the less precise one. Weak unsoundness is then characterized by the existence of a program and an edge such that

$$\forall E \in Exec(P): Cg_{dyn}(E) \subseteq Cg_{stat}(P)$$5

Because $$\forall E \in Exec(P): Cg_{dyn}(E) \subseteq Cg_{stat}(P)$$6 and $$\forall E \in Exec(P): Cg_{dyn}(E) \subseteq Cg_{stat}(P)$$7 are not observable, the methodology flags any nonempty

$$\forall E \in Exec(P): Cg_{dyn}(E) \subseteq Cg_{stat}(P)$$8

under an expected precision order as evidence of a violation.

This formalization is notable because it reframes weak unsoundness as a relational property between analysis variants rather than as an absolute comparison against a dynamic ground truth. The central theoretical insight is that monotone precision refinement cannot be assumed: increasing nominal precision may expose missing true behavior, amplify inconsistencies, or reveal framework-specific semantic assumptions that were previously hidden.

3. Ground-truth-free detection and empirical manifestations

The ground-truth-free detection method is based on metamorphic testing over partial orders (Zhong et al., 1 Apr 2026). Its inputs are a grammar of valid analysis variants, a set of declared precision orders, and a suite of benchmark programs. For each framework run, method names are normalized and non-semantic identifiers are stripped so that call graphs from different variants remain comparable. For every ordered pair $$\forall E \in Exec(P): Cg_{dyn}(E) \subseteq Cg_{stat}(P)$$9 and every benchmark program $P$0, the method runs both analyses, computes

$P$1

and

$P$2

and reports a precision violation whenever $P$3. Under a soundness-driven order, a nonempty $P$4 is analogously a soundness violation.

This yields three principal theoretical conclusions. First, algorithmic precision orders frequently break within frameworks due to modern language features such as lambdas, reflection, and native modeling. Second, configuration choices strongly interact with analysis algorithms, producing “synergistic failures” that exceed the effects of algorithm-only or configuration-only changes. Third, cross-framework comparisons reveal irreconcilable semantic gaps, indicating that different frameworks may operate over incompatible notions of call-graph ground truth.

The empirical study covers Soot, SootUp, WALA, and Doop. The algorithm portfolios include Soot with CHA, RTA, and VTA; SootUp with CHA and RTA; WALA with RTA, 0-CFA, 0-Container-CFA, and 0-1-CFA; and Doop with Context-Insensitive, 1-CallSiteSens, 1-TypeSens, and 1-ObjSens. Configuration variants include Baseline, Field-Sensitive, Object-Sensitive, and FS+OS. The reported metrics are Jaccard similarity,

$P$5

and violation counts, defined as the number of $P$6 or $P$7 edges.

Representative failure modes are tied to specific language and framework features. In Soot, CHA $P$8 RTA because invokedynamic is collapsed to an opaque sink, missing lambda target edges. In WALA, enabling OS from BS activates speculative Class.forName handlers and injects approximately 500 edges, with 39% similarity. In Doop, native stubs add thousands of edges relative to Soot and WALA, yielding near-zero Jaccard even when edges subsume. Within frameworks, Soot shows 100% configuration invariance for FS and OS relative to BS; SootUp shows CHA $P$9 RTA with approximately 6 violations and 95.8% similarity due to better invokedynamic modeling; WALA exhibits approximately 39% similarity for OS alone, approximately 94% for FS, and approximately 43% for OS+FS; and Doop shows approximately 110 injected edges from 1-TypeSens to 1-CallSiteSens with 99.3% similarity because type-based heap merges spuriously connect flows. Across frameworks, Soot versus WALA gives approximately 10–21% similarity with approximately 140–200 violations, SootUp versus WALA also gives approximately 10–21%, Soot versus SootUp gives approximately 32%, and Doop versus others is near zero because of analysis-scope mismatch.

These observations directly challenge the common assumption that static-analysis results are semantically comparable across tools once algorithm names are matched. A plausible implication is that evaluation protocols that treat “CHA,” “RTA,” or “object sensitivity” as tool-independent categories can conflate algorithmic names with incompatible framework semantics.

4. Weak unsoundness in natural-language GUI testing with LLM agents

In “On the Soundness and Consistency of LLM Agents for Executing Test Cases Written in Natural Language,” weak unsoundness is formalized over an Input/Output Labelled Transition System and an ioco conformance relation (Salva et al., 23 Sep 2025). Let

$X$0

be a deterministic IOLTS model of the specification, and let

$X$1

be a natural-language test case such that

$X$2

with each assertion $X$3 true on the GUI state $X$4. If $X$5 is the augmented IOLTS produced by the execution algorithm, then an implementation under test passes the test case iff

$X$6

The paper defines weak unsoundness with respect to $X$7 for ioco as

$X$8

and

$X$9

Equivalently, every conformant implementation passes under normal contexts, while there exists a conformant implementation and a rare context in which the test fails. Here $p \in P$0 is the universe of execution contexts and $p \in P$1 is a small-probability rare-event subset that includes phenomena such as LLM hallucination, timeouts, prompt noise, or transient GUI timing issues.

The motivation is that natural-language test cases are never truly sound in the classical sense. Ambiguity in the test instructions and nondeterministic LLM behavior imply that false failures cannot be ruled out absolutely. The paper therefore links weak unsoundness to industrial quality control through Six-Sigma thresholds. For a binary-result agent with success probability $p \in P$2, the standard deviation is

$p \in P$3

The 3-$p \in P$4 level corresponds to $p \in P$5 and $p \in P$6, and is described as a minimal acceptable quality level in many industries. Proposition 1 states that if each specialized agent’s standard deviation is below $p \in P$7 only in rare contexts, then the test case is weakly unsound.

The execution architecture uses specialized navigation, readiness, and assertion agents, together with guardrail actions inserted into the instrumented IOLTS. The paper also defines atomic-action consistency scores,

$p \in P$8

$p \in P$9

$CG_X(p)$0

and overall test consistency,

$CG_X(p)$1

The evaluation covers eight publicly available LLMs ranging from 3B to 70B parameters, and reports that Meta Llama 3.1 70B demonstrates acceptable capabilities in NL test case execution with execution consistency above the level 3-sigma.

This usage of weak unsoundness differs sharply from the call-graph setting. It does not mean “missing some behaviors because of unmodeled language features”; it means “accepting a bounded, rare-context false-failure rate while preserving normal-context correctness.” The shared structure is the deliberate restriction of where unsoundness is allowed to appear.

5. Relation to soundiness and dynamic countermeasures

The HeapDL work situates weak unsoundness within a long-standing problem in static analysis: real systems routinely exhibit behaviors that purely static models do not capture, particularly through native code, reflection, invokedynamic, lambdas, and dynamic loading (Grech et al., 2019). In that account, weakly unsound analysis is “sound under a restricted semantic model” but admits known gaps for these dynamic constructs. This position is also described as “practically unsound” or “soundiness.”

HeapDL counters such unsoundness by incorporating heap snapshots as additional inputs to static analysis. The runtime collects HPROF-style heap dumps with allocation tracking, recording condensed stack traces for allocations. At program exit or a manual trigger, it writes a full heap dump containing live heap objects, concrete types, field pointers, array contents, primitive values, class-loader metadata, and allocation stack traces. A modified JHat then emits Datalog facts including ObjectFieldValue, StaticFieldValue, ArrayContentsValue, CallGraphEdge, and Reachable. Concrete objects are matched to abstract allocation-site objects using line numbers, bytecode offsets, type, and symbol information where available; if no exact match is possible, HeapDL invents a fresh abstract object keyed by concrete type and allocation context.

The enrichment mechanism addresses precisely the dynamic features that make static analyses weakly unsound. ClassData objects preserve dynamically loaded bytecode and loader identity. EdgeCtx objects make invocation contexts explicit, including invokedynamic bootstrap calls and native effects, so the heap dump directly contains dynamic call edges even when the static analyzer cannot otherwise model them. ObjAndCtx objects recover object-sensitive heap contexts. On the DaCapo benchmarks, the reported median call-graph-edge recall rises from 76.9% for static analysis plus Tamiflex to 99.5% for static analysis plus Tamiflex plus HeapDL, with a median increase of 24% in statically inferred call-graph edges and 86% in heap-object relation size. The trade-off is significant runtime overhead: full allocation tracking on Oracle JVM incurs approximately 39$CG_X(p)$2 slowdown, and the heap-enrichment agent adds another median factor of approximately 1.8$CG_X(p)$3.

These results do not establish absolute soundness for all executions; the paper explicitly notes continued dependence on exercised code paths, incomplete capture of some native side effects, and possible over-approximation from abstraction. Nevertheless, the work shows that weak unsoundness need not be treated as a fixed property of a static analyzer. Dynamic evidence can substantially narrow the unsound region while remaining “virtually” non-intrusive to the analysis logic.

6. Interpretation, misconceptions, and practical significance

A common misconception is that weak unsoundness is merely a softer name for arbitrary error. The surveyed uses do not support that interpretation. In the call-graph partial-order framework, weak unsoundness is diagnosed through specific violations of declared precision or soundness relations, and therefore has a precise relational meaning (Zhong et al., 1 Apr 2026). In the LLM-agent setting, it is explicitly tied to rare-event contexts and industrial quality thresholds rather than unconstrained nondeterminism (Salva et al., 23 Sep 2025). In the static-analysis setting of HeapDL, it denotes analyses that are reliable for ordinary bytecodes but known to be blind to particular dynamic features (Grech et al., 2019).

A second misconception is that greater nominal precision should monotonically improve semantic quality. The large-scale study across Soot, SootUp, WALA, and Doop rejects this assumption: increasing nominal precision can add edges, destabilize configurations, and amplify cross-framework inconsistency. The paper therefore recommends explicit semantic contracts over configurations, joint algorithm-configuration reasoning, and the use of partial-order checks in continuous integration. It also identifies root causes in concrete proportions: 45% of violations from static initializer misalignment, 30% from lambda/invokedynamic, 15% from reflection, and the remainder from lifecycle and native boundaries (Zhong et al., 1 Apr 2026).

A third misconception is that cross-tool disagreement necessarily indicates an implementation defect. The call-graph study instead shows that cross-framework differences may reflect incompatible notions of analysis scope, entry-point modeling, native stubs, or lifecycle semantics. This suggests that “same algorithm” labels are insufficient for fair comparison unless algorithmic and configuration semantics are calibrated across frameworks.

Across the three research threads, weak unsoundness emerges as a technically disciplined acknowledgement that semantic guarantees are often conditional. Sometimes the condition is an expected partial order between analysis variants; sometimes it is confinement of false failures to $CG_X(p)$4; sometimes it is a restricted semantic model that excludes hard dynamic features unless supplemented by runtime evidence. The unifying idea is not the abandonment of soundness, but the explicit localization, measurement, and mitigation of its failures.