TestAgent: Adaptive Testing Agents

• TestAgent is a research motif for decentralized, adaptive agents that autonomously plan, execute, and refine tests across various domains.
• The systems employ dynamic state management, interaction protocols, and external memory structures to enhance continuous integration and evaluation strategies.
• Applications span regression testing, large language model evaluation, adaptive human assessment, and unit test generation with measurable improvements in performance.

Searching arXiv for papers on “TestAgent” and closely related usage to ground the article in current literature. TestAgent is a recurrent research term used for several agentic systems centered on testing, evaluation, or assessment. In software engineering, the term was introduced to denote a test case re-imagined as an autonomous, adaptive software agent that can decide when to execute, how to update its purpose, and when to interact with other tests (Enoiu et al., 2018). Subsequent work used the same name for substantially different artifacts: an agent-based framework for domain-adaptive evaluation of LLMs via dynamic benchmark construction and exploratory interaction (Wang et al., 2024); an adaptive and intelligent expert for human assessment based on LLMs and adaptive testing (Yu et al., 3 Jun 2025); an Agentic Retrieval-Augmented Generation system for quality-engineering artifact creation (Hariharan et al., 12 Oct 2025); and a multi-agent framework for automatic unit test generation that imitates human testing workflows (Zhang et al., 10 Jul 2026). A plausible implication is that “TestAgent” has evolved from a specific concept in regression testing into a broader label for agentic systems that plan, execute, evaluate, or refine tests across several domains.

1. Origins in software test automation

The earliest formulation defines test agents, or TestAgents, as test cases endowed with more reasoning, adaptive behavior and learning capabilities by using the concepts of intelligent software agents (Enoiu et al., 2018). In this view, a software agent is a software system that operates in an execution environment which it can perceive and respond to, takes initiatives and selects own goals, and interacts with others when deemed fit (Enoiu et al., 2018). Applied to testing, the resulting artifact is a self-contained and self-aware test case capable of interactions with other test agents (Enoiu et al., 2018).

The motivation arises from growth of software size, lack of resources to perform regression testing, and failure to detect bugs faster, all within continuous integration and test automation settings (Enoiu et al., 2018). Existing regression test selection and prioritization mechanisms are described as centralized and complex, while test cases themselves are static and rather simplistic (Enoiu et al., 2018). The proposed shift is therefore toward decentralization: test agents should know for themselves when they should be executing, how they should update their purpose, and when they should interact with each other (Enoiu et al., 2018).

Traditional test cases are fixed executable scripts with inputs, expected results, and an embedded oracle, but with no decision-making, no interaction with other test cases, and no autonomy or adaptivity (Enoiu et al., 2018). Test agents preserve these basic elements while adding a notion of purpose or goal, perception capabilities, decision logic, an interaction model, and the ability to adapt their autonomy (Enoiu et al., 2018). This suggests that the original TestAgent concept is best understood not as a single framework but as a proposed redefinition of the test case abstraction itself.

2. Agent model, autonomy, and decentralized regression testing

The original TestAgent proposal models each agent with five main states: Idle, Execute, Interact, Regenerate, and Out of Order (Enoiu et al., 2018). In Idle, the agent is not committed to anything; in Execute, it runs its test task and may decide whether it needs assistance; in Interact, it handles incoming requests; in Regenerate, it undergoes redesign involving a test engineer; and in Out of Order, it can no longer serve its initial purpose (Enoiu et al., 2018). Example triggers for the last state include substantial code change that invalidates the test or irreconcilable mismatch with current software behavior (Enoiu et al., 2018).

Autonomy is analyzed through self-sufficiency and self-directness, with adaptive autonomy framed using dependence theory: an agent may be able to perform a task by itself at one time but not at another due to circumstantial changes such as low resource-consumption levels (Enoiu et al., 2018). In the testing setting, a test agent may initially achieve a coverage goal alone, but after a code change or resource limitation it may delegate subtasks, seek help, or transition to Out of Order or Regenerate (Enoiu et al., 2018).

Interaction is organized through willingness to give help and willingness to ask for help, with four interaction levels adapted to the test context: non-committal interactions, one-to-one dialogue, one-to-one delegation, and one-to-many dialogue or delegation (Enoiu et al., 2018). Broadcast information may include execution time, fault detection information, and test coverage; one-to-one dialogue may request fault history or recent coverage results; delegation may request execution of a subtask such as covering certain parts of the code; and one-to-many interaction may be used to trade off multiple test agents’ objectives with respect to test criteria and cost, such as maximizing overall coverage and minimizing total execution time (Enoiu et al., 2018).

Within continuous integration, this architecture replaces a centralized regression test system that performs selection, prioritization, and scheduling with a distributed approach in which agents themselves decide whether they should execute, how to update their purpose, and when and how to coordinate (Enoiu et al., 2018). The anticipated benefits are better responsiveness to local conditions and less dependence on a monolithic scheduler, but the paper also notes that global behavior becomes emergent and harder to predict or optimize (Enoiu et al., 2018).

3. TestAgent as dynamic evaluation of LLMs

A distinct use of the term appears in a framework for evaluating LLMs in vertical domains (Wang et al., 2024). Here, TestAgent is an agent-based evaluation framework that implements two constructs: Benchmark+, which generalizes a benchmark from static question-answer pairs to strategy-criterion pairs, and Assessment+, which turns evaluation into a dynamic, exploratory, multi-turn process (Wang et al., 2024). The framework is explicitly domain-adaptive, dynamic and interactive, and intended to provide richer than static scores (Wang et al., 2024).

Benchmark+ is defined as

$$\mathcal{B}^*(D;T;M;K):=\{(S^i(\cdot), C^i(\cdot))\}_{i=1}^{N},$$

where $D$ is a domain knowledge base, $T$ denotes user topics of interest, $M$ is the target model being evaluated, and $K$ is a kernel model used for generation and scoring (Wang et al., 2024). The strategy $S^i$ is a sequence over questions, beginning with an initial question generator and then a conditional distribution over follow-up questions based on interaction history; the criterion $C^i$ is generated by the kernel model and includes content-specific criteria $G$ and text-related criteria $E$ (Wang et al., 2024).

Assessment+ is driven by reinforcement learning. The action space is discrete, $\mathcal{A}=\{a_f,a_c\}$, corresponding to follow-up versus challenge (Wang et al., 2024). The next question is generated as

$D$0

and the reward function is

$D$1

encouraging score shifts and semantic change across turns (Wang et al., 2024). The evaluation score itself is defined as

$D$2

The framework was evaluated on medical, government, and reading comprehension domains, using DeepSeek-V2.5 as the kernel model and target models including GPT-4o and DeepSeek-V2.5 (Wang et al., 2024). On the medical domain, the reported scores over 50 conversations were 26.80 for GPT-4o and 26.68 for DeepSeek-V2.5; on the government domain, 23.39 and 23.42 respectively; and on reading comprehension, 48.93 and 48.70 (Wang et al., 2024). Human experts rated the automatically generated questions and criteria highly, with scores around 4.8–4.9 out of 5 for relevancy, scenarios, coherence, completeness, and logic (Wang et al., 2024). In this lineage, TestAgent is not a software test case but an evaluator that constructs and navigates dynamic benchmarks.

4. TestAgent for adaptive human assessment

Another reinterpretation places TestAgent within psychometrics and adaptive testing (Yu et al., 3 Jun 2025). Here the system is an LLM-powered adaptive testing agent that wraps a classic adaptive testing backbone based on cognitive diagnosis and item selection inside a multi-turn conversational interface (Yu et al., 3 Jun 2025). The stated goal is to move closer to an ideal adaptive testing process by supporting personalized question selection, capturing responses and anomalies, and providing precise outcomes through dynamic, conversational interactions (Yu et al., 3 Jun 2025).

The testing loop maintains a latent ability or trait vector $D$3. Given the previous sequence $D$4, the cognitive diagnosis model updates the ability values based on $D$5 using cross-entropy loss, and the question selection algorithm $D$6 chooses the next item according to

$D$7

(Yu et al., 3 Jun 2025). For graded responses, the paper uses a graded response model:

$D$8

with exact-category probability

$D$9

(Yu et al., 3 Jun 2025). The appendix also recalls the 1PL IRT model

$T$0

(Yu et al., 3 Jun 2025).

The LLM contributes question transformation, response interpretation, an Autonomous Feedback Mechanism, Anomaly Management, and diagnosis report generation (Yu et al., 3 Jun 2025). AFM checks domain relevance, response alignment, and logical coherence, and can generate a similar question if a response is unrelated, hard to judge, or inconsistent (Yu et al., 3 Jun 2025). AM addresses guessing anomaly, misleading anomaly, and overconfidence anomaly by combining the probability model with LLM reasoning over the dialogue history (Yu et al., 3 Jun 2025).

The reported datasets are MBTI with 1,000 test-takers, 60 questions, and 60,000 interaction records; MATH with 1,940 students, 1,485 questions, and 61,860 interactions; and SCL-90 with 500 test-takers, 90 questions, and 45,000 interactions (Yu et al., 3 Jun 2025). The paper reports that TestAgent achieves more accurate ability or trait estimation with about 20% fewer questions than strong adaptive baselines, and average improvements of 1.77% in AUC@5 and 0.91% in ACC@5 versus the best baselines (Yu et al., 3 Jun 2025). In a user study on MBTI with 50 volunteers, paired t-tests yielded $T$1 for perceived accuracy, $T$2 for fluency, $T$3 for speed, and $T$4 for interaction experience, all favoring TestAgent (Yu et al., 3 Jun 2025). This use of the term therefore denotes an assessment expert rather than a software-testing artifact.

5. Agentic quality engineering and structural testing of agents

In quality engineering for enterprise systems, TestAgent appears as an Agentic Retrieval-Augmented Generation system specialized for software testing and quality engineering artifact creation (Hariharan et al., 12 Oct 2025). The system combines autonomous AI agents with a hybrid vector-graph knowledge system to automate test plan, case, and QE metric generation (Hariharan et al., 12 Oct 2025). Its four primary components are a Multi-Agent Orchestration Layer, a Hybrid Vector-Graph Knowledge System, an Enhanced Contextualization Engine, and a Comprehensive Traceability Framework (Hariharan et al., 12 Oct 2025).

The planning and generation pattern is expressed as: $M$1 (Hariharan et al., 12 Oct 2025). Specialized agents include a Legacy Test Analysis & Business Intent Agent, Functional Change Mapping Agent, Integration Point Identification Agent, Modernized Test Case Agent, and Compliance Validation Agent (Hariharan et al., 12 Oct 2025).

The vector layer uses SingleStore or a similar distributed vector database, embedding dimensions 384, 768, and 1024, cosine, Euclidean, and dot-product distance metrics, a semantic similarity threshold of $T$5, and a Sentence Transformer encoder (Hariharan et al., 12 Oct 2025). The graph layer uses TigerGraph Cloud, supports 15+ predefined edge types and BFS, DFS, shortest path, and PageRank, and models entities such as requirements, test cases, business processes, SAP modules, interfaces, change requests, and defects (Hariharan et al., 12 Oct 2025). Retrieval combines semantic retrieval with graph traversal and context synthesis, followed by an 8-thread parallel synthesis and a 7-layer validation pipeline (Hariharan et al., 12 Oct 2025).

The reported progression in test artifact accuracy is from 65.2% for Basic RAG to 78.4% for Vector Search, 87.1% for Hybrid RAG, and 94.8% for the Agentic System (Hariharan et al., 12 Oct 2025). Table I reports Agentic RAG at 94.80% accuracy, 96.20% completeness, 95.70% consistency, 98.10% traceability, and 96.20% overall, compared with manual, template-based, basic RAG, GPT-4, and SAP TAO baselines (Hariharan et al., 12 Oct 2025). Additional results include 92.3% test case generation accuracy with 97% requirement traceability coverage, artifact creation time reduced from 240 hours to 36 hours per project phase, 35% cost savings, 85% improvement in test suite efficiency, 98.7% functional coverage over 25,000 test cases, and 92% reduction in production defects post-go-live (Hariharan et al., 12 Oct 2025). The ablation study reports accuracy losses of 12.3% without multi-agent orchestration, 15.7% without the hybrid vector-graph component, 18.2% without enhanced contextualization, and 8.9% without the traceability framework (Hariharan et al., 12 Oct 2025).

A different but related line concerns testing LLM-based agents themselves. Structural testing methods for such agents use OpenTelemetry traces to capture trajectories, mocking to enforce reproducible LLM behavior, and assertions to automate verification (Kohl et al., 25 Jan 2026). The framework defines one trace per user turn, spans for LLM calls, tool invocations, memory operations, and API calls to another agent, and a fixed attribute ai.conversation.id to link traces across a conversation (Kohl et al., 25 Jan 2026). A representative API is: $M$2 with assertions such as: $M$3 (Kohl et al., 25 Jan 2026). Although this paper does not name its framework TestAgent, it provides a structural testing substrate for agentic systems and illuminates how an implementation-focused TestAgent might be validated.

6. Repository-level unit test generation and arbitrary test execution

A recent software-engineering usage defines TestAgent as a multi-agent, LLM-driven system for automatic unit test generation that imitates how human developers plan, write, and review tests (Zhang et al., 10 Jul 2026). The architecture contains three specialized agents: a Requirement Planner, a Test Generator, and a Test Reviewer (Zhang et al., 10 Jul 2026). These agents operate over a test-specialized knowledge graph constructed by static analysis and enriched with testing artifacts such as test reports and failure analyses (Zhang et al., 10 Jul 2026).

The knowledge graph is assembled from source files by building ASTs, extracting entities, analyzing dependencies via IR and pointer analysis, and creating triples

$T$6

(Zhang et al., 10 Jul 2026). Similar test retrieval is based on dependency overlap. For a test $T$7, dependencies are

$T$8

and Jaccard similarity is

$T$9

(Zhang et al., 10 Jul 2026). Runtime tools include check_syntax, compile_test_cases, execute_test_cases, calculate_coverage, and calculate_mutation_score, with mutation score defined as

$M$0

(Zhang et al., 10 Jul 2026).

On six Java projects, the paper reports averages of 100.00% syntax correctness, 99.11% compilation success, 97.46% execution success, 92.34% line coverage, 90.24% branch coverage, and 83.69% mutation score (Zhang et al., 10 Jul 2026). Compared with EvoSuite, ChatUniTest, and HITS, TestAgent outperforms the LLM-based baselines across all reported metrics and exceeds EvoSuite by 40.10 percentage points in mutation score (Zhang et al., 10 Jul 2026). Removing the knowledge graph causes severe degradation to 92.02% compilation, 86.93% execution, 75.65% line coverage, 72.62% branch coverage, and 66.12% mutation score, making the graph the single most impactful ablated component (Zhang et al., 10 Jul 2026). The paper also reports 154 real bugs detected across six projects with 92.22% precision (Zhang et al., 10 Jul 2026).

A neighboring but differently named system is ExecutionAgent, an LLM-driven agent that prepares scripts for building an arbitrary project from source code and running its test cases (Bouzenia et al., 2024). It is not called TestAgent in the paper, but the details describe an autonomous test-execution agent that clones or accesses a repository, infers the build and test process, writes scripts such as Dockerfile and install_and_run.sh, executes commands through tools such as linux_terminal, read_file, and write_file, and iteratively refines the process based on feedback (Bouzenia et al., 2024). Evaluated on 50 open-source projects using 14 different programming languages, it successfully executes the test suites of 33/50 projects, matching ground-truth test results with a deviation of only 7.5%, improving over the best previously available technique by 6.6x, with an average execution time of 74 minutes and LLM cost of USD 0.16 per project (Bouzenia et al., 2024). This suggests a broader ecosystem of agentic testing systems around the TestAgent label, even when the naming differs.

7. Conceptual convergence and divergence

Across these works, the term TestAgent does not denote a single standardized architecture. Instead, it labels several classes of agentic systems whose common feature is the delegation of testing or assessment functions to adaptive, tool-using, context-sensitive agents. In the 2018 proposal, the agent is an individual test case with adaptive autonomy in regression testing (Enoiu et al., 2018). In dynamic LLM evaluation, it is an agent-based evaluator that constructs strategy-criterion benchmarks and conducts exploratory assessment (Wang et al., 2024). In adaptive human assessment, it is a conversational psychometric expert combining cognitive diagnosis with LLM-mediated interaction (Yu et al., 3 Jun 2025). In enterprise quality engineering, it is a multi-agent RAG system for traceable artifact generation (Hariharan et al., 12 Oct 2025). In unit test generation, it is a repository-level testing team composed of planner, generator, and reviewer agents (Zhang et al., 10 Jul 2026).

The convergent themes are clear. First, all variants replace rigid, static procedures with adaptive interaction. Second, most variants combine an explicit state representation or external memory with decision-making over what to ask, execute, or generate next. Third, many variants embed testing artifacts inside richer structures such as knowledge graphs, trace graphs, or dialogue histories. Fourth, several variants treat traceability as a first-class concern, whether through requirements-to-tests links in quality engineering (Hariharan et al., 12 Oct 2025), OpenTelemetry spans in structural testing (Kohl et al., 25 Jan 2026), or persistent focal-test relations in repository graphs (Zhang et al., 10 Jul 2026).

The divergences are equally important. The object under test differs fundamentally across papers: software code and regression suites (Enoiu et al., 2018), LLMs (Wang et al., 2024), human internal states (Yu et al., 3 Jun 2025), enterprise requirements and SAP migration artifacts (Hariharan et al., 12 Oct 2025), and repository-level units under test (Zhang et al., 10 Jul 2026). The operational substrate also varies, from decentralized software agents and continuous integration (Enoiu et al., 2018) to reinforcement learning over dialog actions (Wang et al., 2024), psychometric estimation over latent traits (Yu et al., 3 Jun 2025), hybrid vector-graph RAG with dynamic model routing (Hariharan et al., 12 Oct 2025), and static-analysis-driven knowledge graphs with mutation-guided review (Zhang et al., 10 Jul 2026).

A common misconception would be to treat TestAgent as the name of a single framework or benchmark. The literature instead shows a family of distinct systems sharing an agentic orientation toward testing, evaluation, or assessment. A plausible implication is that the term now functions as a research motif: a way to signal that testing is no longer a passive script or static benchmark, but an adaptive process carried out by agents with goals, memory, retrieval, reasoning, and interaction capabilities.