- The paper introduces partial-order metamorphic testing that detects call-graph semantic violations by comparing algorithms, configurations, and framework outputs without external ground truth.
- Experiments reveal major failures involving WALA’s speculative reflection, lambda and invokedynamic handling, static initialization, and algorithm–configuration interactions, including 516 WALA configuration violations and 146 synergistic violations.
- The findings show that cross-framework call graphs often have low semantic similarity—about 10–21% for Soot and WALA—so evaluations must normalize outputs and account jointly for analysis settings and framework-specific modeling choices.
Motivation and problem statement
Java static analysis frameworks such as Soot, SootUp, WALA, and Doop underpin vulnerability detection, malware analysis, and program optimization, so silent errors in their outputs propagate directly into downstream security decisions. The paper distinguishes semantic violations — analyses that complete successfully but produce outputs violating expected semantic properties — from self-revealing failures like crashes or timeouts. A single missing call-graph edge can sever an entire data-flow path, allowing vulnerabilities of the Log4Shell class to evade taint analysis entirely.
Detecting semantic violations is difficult for three reasons the authors identify explicitly: authoritative ground truth is unavailable for real programs; algorithms and configuration options are heterogeneous across frameworks; and frameworks operate over different intermediate representations (Jimple for Soot/SootUp, SSA for WALA, Datalog for Doop), producing call graphs that differ syntactically even when semantically equivalent. A motivating example illustrates the stakes: Soot's CHA treats invokedynamic as an opaque placeholder node, dropping lambda-resolved edges that WALA's RTA recovers via LambdaMetafactory modeling — rendering lambda-wrapped callbacks unreachable and invisible to taint tracking.
Methodology: partial orders as semantic contracts
The core insight is that analysis algorithms and configurations are partially ordered by precision and soundness. Formally, A1⪰rA2 iff FP(A1(p))⊆FP(A2(p)) for all programs p, and dually for soundness over true positives. Violations of these orders constitute semantic inconsistencies detectable without any external oracle, since they depend only on relationships among the framework's own outputs.
Two methodological extensions make this practical. First, implicit soundness from precision: because increasing precision should not change which behaviors are truly possible, a precision order induces soundness relations in both directions between the compared variants. Second, algorithm–configuration pair ordering: if both algorithm and configuration follow their respective precision orders, the composed pair is expected to refine the weaker pair without altering true positives. When an edge appears in the nominally more precise variant but not the less precise one, at least one partial order must be violated — the diagnosis cannot be pinpointed to one contract, but the violation itself is unambiguous. The authors are careful to frame these orders as pragmatic contracts rather than proven specifications; violations may stem from implementation defects, undocumented design decisions, or incomplete language modeling.
Testing proceeds along two dimensions: intra-framework (configuration testing against a context-insensitive baseline, algorithm testing, and joint algorithm–configuration testing) and cross-framework (partially ordered pairs plus differential comparison of nominally equivalent algorithms). A normalization pipeline strips memory addresses, resolves internal node identifiers, filters metadata nodes, and unifies signatures — necessary because raw output differences otherwise drive similarity to zero (Soot vs. SootUp similarity rose from 0% to 33% after normalization).
Intra-framework results
Configuration violations (RQ1). Only Soot and WALA expose configurable precision options beyond baseline. Soot's object- and field-sensitivity options produce call graphs identical to baseline across all algorithms — conservative but non-refining behavior. WALA behaves very differently: enabling object sensitivity reshapes the call graph dramatically, yielding average similarity of only 39.1% relative to baseline and 516 violations. The root cause is reflection handling: object sensitivity activates WALA's speculative reflective-target recovery, which improves recall but introduces edges that violate expected precision refinement. Field sensitivity alone maintains over 94.3% similarity with only one violation (container abstraction), and notably, combining OS+FS produces fewer violations than OS alone — field sensitivity partially constrains the speculative edges introduced by object sensitivity.
Algorithmic violations under identical configurations (RQ2). SootUp exhibits a soundness inversion: its RTA resolves valid lambda edges that CHA misses, because CHA ignores invokedynamic targets while RTA actively resolves bootstrap methods. In Soot, CHA→RTA/VTA transitions show low similarity (~68–70%) with ~100 violations each; the RTA→VTA transition drops from 228 violations to 6 when OS+FS is enabled. In WALA, RTA exhibits structural regressions relative to more precise algorithms — omitting static initialization, re-thrown exception control flow, and MethodHandle intrinsics — while transitions among context-sensitive analyses show near-perfect consistency (99.5–100% similarity, zero violations). In Doop, the 1-call-site-sensitive → 1-type-sensitive transition triggers 110 violations due to heap abstraction merging container instances, and 1-object-sensitive fails to complete within solver limits.
Synergistic algorithm–configuration failures
The most consequential intra-framework finding concerns interactions. Under differing configurations, WALA's inter-algorithm similarity collapses from ~87–95% (identical configurations) to ~33% under BS→OS, with violations rising to 146 — exceeding the sum of violations observed under algorithm-only or configuration-only changes. The authors characterize this as a synergistic failure: object sensitivity introduces speculative reflection edges, and flow-sensitive propagation then carries data through those edges into methods unreachable under baseline, amplifying error beyond either factor alone. This result directly justifies treating algorithm–configuration pairs, rather than isolated factors, as the unit of evaluation.
A second notable pattern confirms that increased precision can be a prerequisite for soundness rather than merely pruning: field-sensitive RTA tracks MethodHandle objects through fields and resolves calls that CHA leaves unresolved, so the "more precise" configuration enables discovery of genuinely reachable behavior.
Cross-framework divergence
Cross-framework comparisons reveal irreconcilable semantic gaps. SootUp→Soot comparisons yield ~32% similarity regardless of Soot's configuration, dominated by SootUp truncating call graphs at invokedynamic sites (dummy sink nodes), with up to 416 violations for RTA→VTA. Soot↔WALA comparisons are worse: similarity of 10–21% across all algorithm pairs and configurations, invariant to precision increases on either side, indicating a fundamental design-philosophy gap rather than incidental noise — chiefly asymmetric reflection modeling (WALA heuristically resolves Class.forName targets; Soot treats reflection as opaque) and divergent entry-point selection (Soot includes library code as potential entries; WALA prunes it).
Doop comparisons yield near-zero Jaccard similarity, but the authors correctly attribute this to scope mismatch rather than disjoint semantics: Doop models implicit native and runtime flows, producing roughly 15,000-edge graphs versus ~150 for bytecode-constrained frameworks, so even perfect subsumption would score near zero. This is an honest caveat about their own metric rather than a claim about Doop's correctness.
Structural synthesis
Aggregating root causes, violations are highly skewed: static initialization accounts for approximately 45% (Soot eagerly includes <clinit> methods; WALA often omits them), dynamic invocation and lambdas roughly 30% (with invokedynamic inputs triggering over 1,400 cross-framework edge mismatches), reflection about 15%, and implicit lifecycle methods the remainder.
Three structural conclusions follow. First, the divergent construction problem: frameworks do not analyze the same logical program given identical bytecode — entry points, lifecycle methods, native boundaries, and finalizer modeling differ systematically (~188 edges per program), creating a gap no algorithmic refinement bridges. Second, a practical ceiling on precision: Doop's 1-object-sensitive analysis repeatedly exhausts solver resources, and intermediate relations reach 133.4 MB, meaning upper nodes of the refinement lattice are empirically unreachable and their orderings unvalidatable. Third, the closed-world assumption fails under speculative reflection recovery, where increased precision adds edges instead of pruning them.
Limitations
The authors state several limitations plainly. Internal validity depends on the normalization pipeline; residual divergence after normalization is asserted to be semantic, but the pipeline itself could mask or introduce artifacts. External validity rests on the JCG benchmark suite of synthetic feature-isolating tests — Soot's observed 100% configuration invariance may degrade on production applications with complex reflection usage. Construct validity is affected by unstandardized call-graph construction conventions and a 600-second timeout on Doop that defines the tractability boundary for high-precision comparisons. Additionally, the ground-truth-free detection localizes violations to "at least one violated contract" without attributing blame to a specific order, and the paper does not establish formal correspondences between Doop's algorithms and those of imperative frameworks, leaving those comparisons empirical.
Conclusion
This work reframes static analysis validation around testable semantic contracts derived from precision and soundness partial orders, eliminating dependence on ground truth. Its empirical study demonstrates that theoretical orderings break down routinely in practice — through lambda and reflection handling, algorithm–configuration synergy, and framework-specific program construction — and that cross-framework similarity is too low to support the common assumption of comparable results. The practical implication is that evaluations of static analyzers must jointly account for algorithms, configurations, and framework semantics, and that partial-order metamorphic testing offers a scalable foundation for doing so. Open questions include whether the observed stability patterns hold on production-scale Java applications and how to establish principled cross-framework correspondences for declarative analyzers like Doop.