Agentic Multi-Model Testing Framework

• Agentic multi-model testing is a framework that decomposes evaluation into specialized roles such as planning, generation, and review.
• It leverages heterogeneous evaluators—including LLM-based judges and code-based analyzers—to ensure comprehensive, traceable assessments.
• Empirical studies demonstrate that closed-loop, multi-agent workflows deliver improved consistency, reduced runtime, and enhanced failure detection.

An agentic multi-model testing framework is a testing and evaluation architecture in which evaluation is itself treated as a staged, tool-using, and often agentic process rather than as a single end-to-end score. Across recent work, the term encompasses systems that decompose testing into specialized roles such as planning, generation, execution, critique, review, aggregation, and governance; combine heterogeneous evaluators such as LLM-based judges, code-based evaluators, or external tools; and preserve intermediate artifacts or traces for replay, audit, or diagnosis. In this literature, “multi-model” is realized in different ways: some systems are architected for heterogeneous evaluators but remain experimentally dominated by one backbone model, while others explicitly compare interchangeable backbones or orchestrate distinct specialist models at inference time (Lee et al., 17 Jan 2026).

1. Conceptual scope and problem setting

Recent frameworks converge on a shared critique of static or single-shot evaluation. In multi-agent and tool-using systems, failures are not confined to the final answer; they arise through routing, planning, tool calls, memory interaction, validation, external side effects, and long-horizon coordination. A single “LLM-as-a-Judge” or one-pass test generator therefore misses process-level reasoning, coordination failures, instability across repeated runs, partial evaluator coverage, and auditability requirements (Lee et al., 17 Jan 2026).

This broader view appears in several neighboring formulations. Neo is introduced as a “configurable, multi-agent framework for automated testing of LLM-based agents” with a closed-loop interaction cycle rather than replay of fixed prompts (Wang et al., 19 Jul 2025). The Agentic Testing Architecture (ATA) is described as a “closed-loop, self-correcting system” in which generation, execution, analysis, and repair continue until convergence (Naqvi et al., 5 Jan 2026). A trace-oriented assurance formulation shifts the unit of analysis from final outputs to Message-Action Traces (MAT), arguing that agentic AI failures include non-termination, role drift, unsupported-claim propagation, prompt/context injection, and unsafe external actions (Paduraru et al., 18 Mar 2026). In software engineering, the same logic is applied to static analyzers through seed generation, validation, mutation, and metamorphic comparison rather than a monolithic pipeline (Nnorom et al., 20 Jul 2025).

Within this spectrum, “multi-model” is not uniform. Some systems are explicitly model-agnostic by design aspiration or support multiple evaluator types without benchmarking many foundation models; others treat different backbones as interchangeable components or expose genuinely heterogeneous tool agents. A plausible implication is that the field currently defines multi-model testing more strongly at the architectural level than at the experimental level.

2. Architectural patterns and agent roles

A recurring design pattern is role decomposition. AEMA defines four explicit roles: Planning Agent, Prompt-Refinement Agent, Evaluation Agents, and Final Report Agent. The framework is “process-aware” because it reasons over execution traces rather than only final answers, and “verifiable” because each stage leaves a traceable artifact that can be replayed, inspected, or audited (Lee et al., 17 Jan 2026). Neo uses a Question Agent, Evaluation Agent, and Context Hub, with a probabilistic conversation controller governing turn-by-turn adaptation (Wang et al., 19 Jul 2025). ATA decomposes testing into Test Generation Agent (TGA), Execution and Analysis Agent (EAA), Review and Optimization Agent (ROA), plus an Orchestrator / Scheduler and shared stores such as an Artifact Store, Vector Database, and Metrics Store (Naqvi et al., 5 Jan 2026). StaAgent similarly separates Seed Generation Agent, Code Validation Agent, Mutation Generation Agent, and Analyzer Evaluation Agent (Nnorom et al., 20 Jul 2025).

These systems also converge on an externalized coordination substrate. Neo’s Context Hub stores domain-specific configurations, prompt templates, behavioral expectations, interaction history, and evolving state (Wang et al., 19 Jul 2025). ATA uses Git-based artifact storage, FAISS vector memory, and PostgreSQL metrics logging (Naqvi et al., 5 Jan 2026). A trace-based assurance stack formalizes instrumentation as MAT records with provenance links, contract verdicts, and replay artifacts (Paduraru et al., 18 Mar 2026). This suggests that a modern testing framework is less a single evaluator than a layered orchestration system with persistent state, typed artifacts, and explicit control flow.

The “multi-model” dimension is realized differently across architectures. AEMA is “conceptually yes, experimentally only partially” multi-model: it supports “LLM-based judges and reliable code-based evaluators,” retrieval over vector memory, and future dynamic use of “small LLMs versus larger ones,” but the reported deployment uses GPT-4o and ChromaDB (Lee et al., 17 Jan 2026). StaAgent is multi-agent but not runtime-heterogeneous: “the same LLM” powers all four agents in a given setting, while backbone choice is studied experimentally across CodeLlama, DeepSeek, Codestral, Qwen, and GPT-4o (Nnorom et al., 20 Jul 2025). Team-of-Thoughts is closer to a strongly heterogeneous design: an orchestrator-tool paradigm where the orchestrator dynamically activates tool agents backed by different model families and aggregates their outputs under explicit budget accounting (Wong et al., 18 Feb 2026).

3. Testing workflows, traces, and control loops

The central methodological move in this literature is to treat testing as a workflow. In AEMA, the lifecycle begins with a planner that converts high-level evaluation intent into an executable plan, filters evaluator functions through hybrid sparse+dense retrieval over docstrings, and refines the plan through a bounded generator–evaluator loop capped at five rounds and converging within three rounds in the reported setting (Lee et al., 17 Jan 2026). The Prompt-Refinement Agent then constructs schema-compliant JSON parameter bundles and retrieves or synthesizes few-shot exemplars, after which heterogeneous evaluation agents return normalized scores in $[0,1]$ plus qualitative feedback.

Neo’s workflow is conversational rather than trace-audit-centric, but structurally similar. It retrieves static and dynamic context from the Context Hub, generates a test input, queries the Target Agent, evaluates the response, writes the result back to shared state, and conditions the next turn on updated feedback (Wang et al., 19 Jul 2025). Its state vector is explicitly modeled as

$\mathbf{S = \langle F, I, T, FB \rangle}$

with flow type, intent type, tone index, and prior feedback. This allows adaptive multi-turn probing rather than fixed script replay.

ATA operationalizes a generate–execute–analyze–repair loop. The paper gives explicit pseudocode in which TGA generates tests, EAA executes them, ROA analyzes results and refines tests, metrics are computed, and the loop stops when coverage and failure-rate thresholds are satisfied (Naqvi et al., 5 Jan 2026). Convergence is operationalized by Coverage $\ge 95\%$ and Failure Rate $\le 2\%$, although the paper also discusses “zero test failures” as a conceptual ideal.

The most formal trace-first workflow appears in the assurance framework based on Message-Action Traces. One execution is defined abstractly as

$\tau \;=\; (s_0,a_0,o_1,s_1,\dots,s_T),$

and each instrumented step is represented as

$r_t = \langle t,\ i_t,\ \mathrm{role}(i_t),\ \hat{s}_t,\ a_t,\ o_{t+1},\ \mathrm{prov}_t,\ \mathcal{I}^{\mathrm{step}_t,\ \mathrm{verdict}_t \rangle .$

Here the framework adds step contracts, trace contracts, provenance links, localization of the first violating step, deterministic replay artifacts, stress testing under perturbation budgets, structured fault injection, and runtime governance at the language-to-action boundary (Paduraru et al., 18 Mar 2026). This suggests a broader interpretation of testing: not only scoring outputs, but instrumenting and constraining executions.

4. Evaluation targets, metrics, and benchmarking dimensions

The literature defines multiple objects of evaluation. AEMA explicitly targets planning reliability, score stability across repeated evaluations, and alignment with human judges under clean and degraded inputs (Lee et al., 17 Jan 2026). Its most formal metric is not a global trustworthiness score but a plan-quality score combining schema validity, agent selection accuracy, step-agent coherence, order preservation, and step efficiency via AHP-weighted aggregation: $$\mathrm{FinalScore} = w_F F + w_S S + w_{C_v} C_v + w_{O_r} O_r + w_{E_f} E_f.$$ The paper is explicit that this evaluates predicted plans against gold plans, not every evaluation dimension in the framework.

Neo focuses on realistic multi-turn coverage and adversarial probing. Its evaluation agent currently returns only binary success/failure, judging criteria such as intent alignment, coherence, and appropriateness, while richer dimensions such as factual grounding and policy compliance are left as future work (Wang et al., 19 Jul 2025). Yet the state-space analysis is more formal than the evaluation rubric: the number of possible interaction states is defined as a product over state-dimension cardinalities, and for a session of $n$ rounds the number of possible question trees is given as $S_n = n!$, with semantic labeling $L_n = (|I| \times |T|)^n$.

ATA uses a compact metric tuple

$\mathbf{S = \langle F, I, T, FB \rangle}$0

for coverage, failure rate, and runtime, optimizing

$\mathbf{S = \langle F, I, T, FB \rangle}$1

and stopping at thresholded convergence (Naqvi et al., 5 Jan 2026). StaAgent uses a metamorphic oracle rather than a scalar score: Type1 denotes inconsistent detection between a seed and semantically equivalent mutants, and Type2 denotes false negatives across variants (Nnorom et al., 20 Jul 2025).

The trace-based assurance framework offers the richest comparative metric suite. It defines estimators for task success, Success@k, non-termination rate, trace-contract violation rate, per-contract violation profiles, first-violation step distributions, unsupported claim rate, unsupported claim propagation, role-drift score, containment rate under fault injection, governance outcome distributions, blocked high-impact action rate, robustness curves under perturbation budgets, MTBF, and regression rate (Paduraru et al., 18 Mar 2026). This metric family is explicitly intended to support comparison across stochastic seeds, models, and orchestration configurations, a property that is especially relevant to multi-model testing.

A different but complementary benchmark logic appears in APTBench, which converts real agent tasks and successful trajectories into static multiple-choice and text-completion items for base models. It evaluates planning, action, and domain-specific atomic abilities across software engineering and deep research, and is presented as a lightweight proxy for “agentic potential” during pre-training (Qin et al., 28 Oct 2025). This suggests that an agentic multi-model testing framework can target both deployed agent systems and upstream latent capacity, depending on the abstraction level.

5. Empirical evidence and demonstrated gains

Across domains, the strongest empirical results come from closed-loop and decomposition-based frameworks relative to monolithic baselines. AEMA reports that in the finance invoice-validation setting, consensus on the correct evaluation plan was reached in 13 runs after one debate round, 13 runs after two rounds, and 4 runs after three rounds across 30 runs. In repeated evaluation, AEMA shows “narrow score variation” while a single LLM baseline exhibits “wider dispersion,” especially for Decision and Final. In human-alignment experiments, average absolute error across six steps is 0.018 for AEMA versus 0.077 for the single LLM baseline on good-quality invoices, and 0.037 versus 0.108 on blurry invoices (Lee et al., 17 Jan 2026).

Neo demonstrates throughput and adversarial realism rather than trace auditability. Against a production-grade Seller Financial Assistant chatbot, six human testers induced 7 breaks out of 120 malicious prompts (5.8% break rate), while Neo induced 4 breaks out of 120 (3.3% break rate), with all observed breaks occurring in the Mixed Attack category. Neo also generated 180 realism prompts in ~45 min versus ~16 hours for six humans, while preserving 100% topic accuracy and exposing weaknesses such as ~55% tone accuracy for mid-range tones and around 32% of sessions with unnatural follow-up transitions (Wang et al., 19 Jul 2025).

ATA reports substantial gains over one-pass GPT-4o-mini test generation. In Table 3, mean statement coverage improves from 72.8% to 94.9%, mean branch coverage from 61.5% to 91.7%, valid executable tests from 64.1% to 89.3%, total runtime per module falls from 3.5 hrs to 1.1 hrs, and QA validation effort drops from 11.8 hrs to 3.4 hrs. Test convergence stabilizes around 5.2 iterations average (Naqvi et al., 5 Jan 2026).

StaAgent’s evidence is framed around bug discovery yield and model sensitivity. Across five analyzers—SpotBugs, SonarQube, ErrorProne, Infer, and PMD—the framework found 64 problematic rules, of which 53 out of the 64 bugs cannot be detected by the SOTA baseline. The union contains 43 Type1 and 21 Type2 issues, and all findings were reported to developers, with 2 fixed and 3 confirmed at the time of writing (Nnorom et al., 20 Jul 2025).

Where the goal is heterogeneous model orchestration rather than evaluator auditing, Team-of-Thoughts shows the clearest performance gains from multi-model selection. On five reasoning and coding benchmarks, Team-of-Thoughts reaches 96.67% on AIME24 and 72.53% on LiveCodeBench v6, compared with 80.00% and 65.93% for the AgentVerse baseline. The paper also shows that the best orchestrator is task-dependent: for example, DeepSeek v3.2 attains 93.33% orchestration calibration accuracy on AIME2024, while GPT-5 Mini reaches 85.14% on MBPP+ (Wong et al., 18 Feb 2026).

6. Limitations, blind spots, and emerging directions

The literature is notably candid about limitations. AEMA acknowledges instability in continuous scores, substantial cost and latency because evaluation may be slower and more expensive than the business workflow itself, limited empirical scope in a single finance domain, and a GPT-4o-centric implementation despite broader architectural claims (Lee et al., 17 Jan 2026). Neo notes its coarse binary evaluator, limited realism fidelity, small human baseline, absence of true runtime multi-model experiments, prompt dependence, and concentration on a financial assistant chatbot (Wang et al., 19 Jul 2025). ATA reports no formal ablation study, weak formalization of “reinforcement,” modest benchmark scale, reproducibility concerns from LLM nondeterminism, and limited test-modality depth (Naqvi et al., 5 Jan 2026). StaAgent highlights test-executable validation bias, false-positive risk from weak seeds or tests, strong model sensitivity, prompt dependence, computational expense, and Java-specific scope (Nnorom et al., 20 Jul 2025).

A broader systemic blind spot appears in the empirical study of open-source testing practice. Across 39 agent frameworks and 439 agentic applications, Resource Artifacts and Coordination Artifacts consume over 70% of testing effort, while the Plan Body receives less than 5%, and the Trigger component appears in around 1% of tests. Novel agent-specific methods such as DeepEval are used in only about 1% of application tests, whereas traditional patterns such as parameterized testing, negative testing, and membership testing dominate (Hasan et al., 23 Sep 2025). This suggests that current practice is rational but skewed toward deterministic infrastructure, leaving prompts and FM-mediated reasoning under-tested.

The next directions in the literature push beyond single-output scoring toward richer comparative and governance-aware evaluation. The trace-based assurance framework proposes contracts, deterministic replay, budgeted counterexample search, structured fault injection, and runtime governance as integral parts of the testing surface rather than afterthoughts (Paduraru et al., 18 Mar 2026). CAT reframes evaluation around goal-task alignment through the Goal Achievement Index,

$\mathbf{S = \langle F, I, T, FB \rangle}$2

arguing that agentic systems can execute local tasks correctly while still failing their overarching goals (Dhrif, 26 Sep 2025). CARE, in multimodal medical reasoning, shows that explicit evidence generation, confidence filtering, and coordinator review improve accountability relative to end-to-end black-box reasoning (Du et al., 2 Mar 2026). AgentM$\mathbf{S = \langle F, I, T, FB \rangle}$3D demonstrates that naive Best-of-$\mathbf{S = \langle F, I, T, FB \rangle}$4 scaling is often insufficient, and that adaptive planning plus task-aligned critics are needed for reliable specialist orchestration (Jiang et al., 3 Mar 2026).

Taken together, these works indicate that an agentic multi-model testing framework is best understood not as a single benchmark or judge, but as a controlled evaluation stack with specialized roles, explicit artifacts, heterogeneous evaluators, replayable traces, governance hooks, and comparative metrics across models and configurations. A plausible implication is that the field’s main transition is from output scoring to execution assurance: trustworthy testing of agentic systems increasingly requires planning, decomposition, instrumentation, retrieval, critique, auditability, and governance to be designed into the evaluator itself.