TestWeaver: LLM-Driven Testing

• TestWeaver is an advanced framework that uses LLMs to guide both requirements elicitation and regression test generation for improved testing scope.
• It integrates an LLM-generated knowledge base with graph-based recommendations and dynamic slicing techniques to optimize test case development.
• Empirical evaluations demonstrate increased conceptual, line, and branch coverage while mitigating tester bias compared to traditional methods.

Searching arXiv for the named systems to ground the article in the cited papers. arXiv search: query for "TestWeaver" and related titles. TestWeaver designates two distinct but related uses in recent LLM-for-testing research. In one usage, the term refers to Weaver, an interactive system for requirements elicitation for model testing that helps testers decide what to test by combining LLM-generated knowledge bases, graph-based recommendation, and interactive concept exploration (Yang et al., 2023). In a later and separate usage, TestWeaver names a coverage-guided, LLM-based regression test generation framework that augments an LLM with lightweight program analysis and execution feedback to overcome the “coverage plateau” in automated test generation for Python code (Le et al., 2 Aug 2025). The shared motif is not a common implementation but a common emphasis on structuring and guiding test-related reasoning beyond naive prompting.

1. Terminological scope and naming

The literature uses the label TestWeaver in two ways. In the 2023 paper “Beyond Testers’ Biases: Guiding Model Testing with Knowledge Bases using LLMs,” the main system is called Weaver, and the term “TestWeaver” refers to the same tool in its role as a testing assistant (Yang et al., 2023). In the 2025 paper “TestWeaver: Execution-aware, Feedback-driven Regression Testing Generation with LLMs,” the name denotes a different framework centered on regression-test synthesis for code coverage improvement (Le et al., 2 Aug 2025). A later work, FutureWeaver, is not TestWeaver proper, but it extends the same “weaving” nomenclature to budget-aware planning of test-time compute in multi-agent systems (Jung et al., 12 Dec 2025).

Name in literature	Core objective	Representative paper
Weaver / TestWeaver	Requirements-driven model testing	(Yang et al., 2023)
TestWeaver	Execution-aware regression test generation	(Le et al., 2 Aug 2025)
FutureWeaver	Budget-aware multi-agent test-time compute planning	(Jung et al., 12 Dec 2025)

This naming overlap matters because the two TestWeaver usages operate at different levels of the testing stack. Weaver addresses the pre-test-generation question of which concepts, scenarios, and edge cases are important for evaluating a model in a specific application, whereas the later TestWeaver addresses how to generate regression tests that reach uncovered target lines in existing codebases.

2. TestWeaver as requirements-driven model testing

In the requirements-driven sense, TestWeaver is an interactive system for requirements elicitation for model testing. Its starting point is the observation that most model-testing work has focused on creating test cases, while the prior step of identifying what should be tested is “largely implicit,” driven by “ad-hoc intuition,” and poorly supported by tools (Yang et al., 2023). The paper explicitly connects this gap to software engineering practice, where requirements engineering and the V-model treat tests as artifacts derived from explicit requirements and design decisions.

The central motivation is tester bias. The paper identifies narrow conceptual focus, dependence on individual knowledge, and local exploitation vs. global exploration as recurring failure modes in manual model testing. Testers may “hill-climb” around a few observed failure modes and therefore miss important concepts that require domain expertise, are not obvious from initial error analysis, or arise from interactions among multiple factors. The stated consequence is reduced model reliability and fairness because large parts of the conceptual space remain untested (Yang et al., 2023).

Weaver operationalizes a different testing workflow. Given an application such as stance detection or code summarization, it uses an LLM to build a task-specific knowledge base, recommends a small, diverse, and relevant subset of concepts, lets testers explore and refine those concepts interactively, and then connects the selected concepts to a downstream test-case creation method, by default AdaTest. The system therefore sits before or around direct test generation: it structures the space of testing requirements rather than replacing case-level test synthesis (Yang et al., 2023).

The notion of a requirement is deliberately lightweight. Requirements are not formal logic specifications; they are conceptual categories such as “pedestrian detection at night,” “hateful memes with subtle symbols,” or “feminism in religious contexts” that can later be turned into concrete test suites. This top-down orientation distinguishes Weaver from bottom-up systems that begin with examples or already observed errors.

3. Knowledge-base construction, recommendation, and interaction

Weaver has three main components: an LLM-generated knowledge base, a graph-based recommendation algorithm, and an interactive interface (Yang et al., 2023). The process starts from a user-provided seed concept, which can be a task name, domain phrase, or tailored scenario description, such as “climate change” or “specific challenges that novice programmers might have in comprehending data visualization code.”

For knowledge-base construction, the system uses OpenAI’s text-davinci-003, and the paper also reports that LLaMA-2-13B-chat works reasonably well. The LLM is treated as a knowledge extractor. Prompts are derived in zero-shot form from ConceptNet relations, including TypeOf, PartOf, UsedFor, Causes, MotivatedBy, ObstructedBy, LocatedAt, CapableOf, CreatedBy, MadeOf, DoneBy, DoneTo, and RelatedTo, with 25 relations in total. The resulting knowledge base is a directed, labeled graph whose nodes are text concepts and whose edges encode relation types and, when relevant, path context for deeper expansion. Weaver initially pre-generates two layers from the seed and expands further on demand.

Because a full knowledge base may be too large to inspect directly, Weaver formalizes concept suggestion as a relevance-diversity optimization problem. For a fully connected graph $G$ over candidate concepts, it selects a $k$-node subgraph $G'$ by maximizing

$$\arg\max_{G' \subset G, |G'| = k} w_E(G') + \alpha \cdot w_V(G'),$$

where $w_E(G')$ captures diversity through edge weights and $w_V(G')$ captures relevance through node weights. Diversity is computed from SentenceBERT embeddings using cosine distance; relevance is estimated by constructing sentences of the form “{concept} often occurs in the context of {queried_concept}” and scoring them with GPT-2 perplexity, where lower perplexity indicates higher relevance. Because exact optimization is expensive, the system uses a greedy peeling algorithm that repeatedly removes the node with the smallest combined weight until only $k$ nodes remain (Yang et al., 2023).

The interface implements this recommendation machinery as an exploration environment. A tester enters a seed concept, receives an initial recommendation set, expands nodes to trigger further LLM expansion, asks for more recommendations anchored at a selected concept, and can manually add, edit, or delete concepts. When a concept is judged important, it is selected into an emerging requirements list. The interface thereby supports both externalized exploration and the “distillation of personal knowledge,” rather than imposing a fixed ontology. Selected concepts can then be passed to AdaTest for candidate input generation, model execution, labeling, and refinement, although the paper emphasizes that Weaver is agnostic to the specific test creation method (Yang et al., 2023).

4. Evaluation, applications, and limitations of Weaver

The evaluation of Weaver addresses both the intrinsic quality of the generated knowledge bases and the practical value of knowledge-base-guided requirements elicitation. In intrinsic evaluation, the paper considers hateful meme detection, pedestrian detection, and stance detection for feminism and climate change. Average recall of ground-truth concepts in the generated knowledge bases is reported as $90.6\%$, with per-task values of 86.9–93.1%; average sampled-edge precision is 80.5%; and inter-rater agreement for the manual concept-matching step is Cohen’s $\kappa = 69.4\%$ (Yang et al., 2023). The reported gaps concern very fine-grained concepts and interaction concepts such as “fossil fuel uses in developing countries.”

The user study involves $N = 20$ graduate students with varying ML/NLP experience in a within-subject design with Latin square counterbalancing. Participants test two stance-detection tasks with zero-shot GPT-3.5: stance toward feminism and stance toward combating climate change. Relative to a control interface without the knowledge base or recommendations, participants using Weaver identified 57.6% more concepts and covered 47.7% more concept clusters, with statistically significant condition effects under repeated-measures ANOVA. The paper further reports that participants found about 11 failing test cases per session, approximately 0.44 per minute, regardless of condition, while collectively discovering more than 200 failing test cases for zero-shot ChatGPT on stance detection (Yang et al., 2023).

The qualitative findings emphasize broader and less locally exploitative exploration. Survey responses report that 75% of participants agreed or strongly agreed that the knowledge base “helps me find more diverse model bugs,” 80% agreed that it “helps me test the model more holistically,” and 95% said they wanted to use Weaver for future testing. The paper also notes a shift in the global distribution of explored concepts, suggesting that Weaver mitigates some human biases while inheriting some LLM/knowledge-base biases (Yang et al., 2023).

Two case studies extend the scope beyond stance detection. In transcript summarization to instructions, Weaver helped refine vague concerns such as “useful summaries shouldn’t be literal” into testable concepts including “hidden thought process behind transcript” and “step ordering and grouping,” and revealed that generated instructions were always chronological even when different ordering would be preferable. In code understanding for novice programmers in data visualization, a more specific seed concept led Weaver to suggest more relevant concepts such as different parameters for customization and when to use different data visualization APIs, and the practitioner used those requirements against an existing dataset rather than AdaTest-generated inputs (Yang et al., 2023).

The limitations are explicitly tied to the knowledge source and study design. Weaver depends on LLM knowledge, reflects LLM biases and errors, was evaluated with graduate students rather than a broad practitioner sample, and treats model correctness as partly subjective. The paper also highlights issues of granularity, since suggested concepts may be too broad or too narrow, and notes that initial construction and on-demand expansion take seconds, implying interaction-cost and scalability constraints (Yang et al., 2023).

5. TestWeaver as execution-aware regression test generation

In the later usage, TestWeaver is a coverage-guided, LLM-based regression test generation framework for Python code that targets the “coverage plateau” observed in prior LLM-based approaches (Le et al., 2 Aug 2025). The setting is regression testing for existing codebases, starting from a possibly empty test suite and synthesizing additional tests to improve line and branch coverage. The target granularity is the individual uncovered target line.

The paper’s diagnosis is twofold. First, LLMs struggle with execution reasoning because they are trained mainly on static code rather than runtime traces or coverage information. Second, existing systems rely on passive feedback, repeatedly instructing the model to cover remaining lines without explaining why previous attempts failed. This creates conflicting objectives, provides no new execution insight, and bloats the prompt with irrelevant program context (Le et al., 2 Aug 2025).

TestWeaver responds with a three-phase architecture. Seed initialization generates about 10 tests per repository to cover “easy” lines. Phase 2 performs slice-guided test generation: for each uncovered line $k$0, the system computes a two-phase backward slice, prompts the LLM with the slice and the target line, executes the generated test, optionally repairs syntax using CoverUp’s program-analysis-based repair module, and retains successful tests. Phase 3 performs execution-aware test re-generation for lines still uncovered: it retrieves a closest test case using coverage traces and a control-dependence graph, executes that test on the slice, records variable states as execution in-line annotations, and prompts the LLM again with the annotated slice, the closest test, and explicit reasoning instructions (Le et al., 2 Aug 2025).

The slicing procedure is central. Phase 1 discards statements that cannot be executed before reaching $k$1 along any feasible control flow. Phase 2 then computes backward dependence closure over data dependencies and control dependencies. Conceptually, with a program dependence graph $k$2 and target node $k$3, the slice $k$4 is the smallest set containing $k$5 such that if $k$6 and $k$7, then $k$8. The paper reports a motivating example in which GPT-4o succeeded on a complex target line 9/100 times with the full function but 100/100 times with the slice, and later ablations show Phase-2 coverage dropping from 62% to 58% for lines and from 52% to 45% for branches when slicing is removed (Le et al., 2 Aug 2025).

The “closest test” mechanism operationalizes near-miss reuse through control dependence. For a target line $k$9, let $G'$0 denote the ordered list of control-dependent conditions governing $G'$1, and for a test case $G'$2, let $G'$3 denote the executed lines. Closeness is defined as

$G'$4

with smaller values indicating that $G'$5 reached a controlling condition near $G'$6. The selected test is then executed on the slice to generate execution in-line annotations of the form # (k) var1 = v1; var2 = v2; ..., where $G'$7 indexes execution order and the comment records in-scope variable values after that line. This gives the LLM localized dynamic state rather than only static code (Le et al., 2 Aug 2025).

Prompt design reflects the two phases. Phase 2 supplies only the code_slice and the target_line, with strict instructions to generate a single Pytest test. Phase 3 adds closest_test, code_slice_with_exec_inlines, and explicit separation between <thinking> and <answer> so that the LLM analyzes why the near-miss test fails and then produces a revised test. The implementation uses DeepSeek-V3-0324 as the base model, targets Python, and evaluates on the CodaMosa suite of 35 real-world Python projects, approximately 100k LOC, and 425 modules (Le et al., 2 Aug 2025).

6. Empirical performance, limitations, and conceptual extensions

On the full benchmark, TestWeaver reports 68% line coverage, 54% branch coverage, and 62% combined line+branch coverage within a 3-hour budget per repository, compared with 61%, 47%, and 55% for CoverUp, and 46%, 25%, and 40% for CodaMosa (Le et al., 2 Aug 2025). Stratified analyses by module size show smaller degradation as modules grow: for 0–150 lines, line coverage is 73%; for 150–500, 67%; and for 500–1100, 65%. By cyclomatic complexity, the reported values are 73% for CC 1–50, 68% for CC 50–100, 63% for CC 100–200, and 66% for CC 200–300.

The phase-wise analysis attributes gains to both static and dynamic guidance. Coverage rises from about 54% after seed initialization to about 62% after slice-guided generation and then to about 68% after execution-aware re-generation. Ablation on Phase 3 shows that removing execution in-lines reduces coverage from 68% to 63% for lines and from 62% to 56% for branches, while removing closest-test selection reduces it to 61% and 50% respectively. On typesystem.fields.Array, a closest test case yields 97% line coverage, compared with 88% for a random single test and 84% for five random tests, indicating that control-flow-aware retrieval is more useful than naive example accumulation (Le et al., 2 Aug 2025).

The framework’s stated target is not only higher final coverage but mitigation of the coverage plateau. In the typesystem.fields case study, CoverUp plateaus at 63% around prompt 98 and then accrues 69 re-prompts with no additional coverage, with a longest plateau of about 30 prompts. TestWeaver surpasses 63% at prompt 112, reaches 75% by prompt 120, records only 15 prompts with no coverage improvement, and has a maximum plateau length of 5 (Le et al., 2 Aug 2025).

Efficiency is mixed rather than uniformly favorable. Run to saturation, TestWeaver averages 2.13 hours/repo and \$G'$84.24</strong> for CoverUp, and <strong>5.59 hours</strong> and <strong>\$3.54 for CodaMosa. The paper interprets this as evidence that efficient test generation depends on smart, execution-aware prompting, not only on minimizing tokens (Le et al., 2 Aug 2025).

The limitations of the regression-testing TestWeaver are program-analysis and context related. Backward slicing can omit useful context, especially from dependent classes or type information; some generated tests remain unexecutable despite repair; extremely hard-to-reach lines may have no closest test found; and fixed retry budgets leave some lines uncovered. Threats to validity include Python-only evaluation, dependence on third-party control-dependency, slicing, and coverage tools, and the use of coverage rather than mutation score or real bug detection as the primary metric (Le et al., 2 Aug 2025).

A broader implication, marked here as interpretation, is that the “weaving” label has expanded from requirements-guided test scoping and execution-aware regression-test generation to explicit test-time orchestration. FutureWeaver treats the action space as $G'$9, imposes a hard budget constraint

$$\arg\max_{G' \subset G, |G'| = k} w_E(G') + \alpha \cdot w_V(G'),$$0

and selects actions by combining a short-term score $$\arg\max_{G' \subset G, |G'| = k} w_E(G') + \alpha \cdot w_V(G'),$$1 and a long-term budget-feasibility score $$\arg\max_{G' \subset G, |G'| = k} w_E(G') + \alpha \cdot w_V(G'),$$2 through $$\arg\max_{G' \subset G, |G'| = k} w_E(G') + \alpha \cdot w_V(G'),$$3 (Jung et al., 12 Dec 2025). This suggests a family resemblance rather than direct continuity: each system uses structured guidance to make test-related or inference-time exploration less myopic, but the objects being “woven” differ—concepts in Weaver, execution context in TestWeaver, and compute allocation in FutureWeaver.