MUTABOT: Domain-Specific Mutation Testing

• MUTABOT is a set of mutation testing frameworks that apply domain-specific, semantic operators to generate meaningful mutants and evaluate test suite effectiveness.
• In conversational systems, MUTABOT mutates artifacts like flows, intents, and contexts to reveal blind spots in chatbot test suites.
• In industrial robotics, MUTABOT targets high-level read/write commands, ensuring executable mutants and robust assessment through domain-tailored metrics.

MUTABOT denotes a set of domain-specific mutation-testing frameworks that adapt classical mutation testing to software artifacts for which generic code-level operators are poorly matched. In the literature provided, the name is used in two distinct research lines. One line concerns conversational systems, where MutaBot targets chatbot artifacts such as flows, intents, entities, parameters, actions, and contexts, first for Google Dialogflow and later for both Dialogflow and Rasa (Urrico et al., 2024, Clerissi et al., 1 Sep 2025). The other line concerns industrial robotic systems, where MUTABOT targets message-based ROS programs by mutating high-level read and write operations, including movement, rotation, gripper actions, and sensor reads, with execution mediated by Gazebo (Santos et al., 18 Nov 2025). Across these uses, the unifying principle is to generate meaningful mutants at the level of domain semantics while reducing invalid, broken, or equivalent mutants.

1. Scope and naming

The term “MUTABOT” is not confined to a single implementation or application domain. It has been used for mutation-testing frameworks in both conversational AI and industrial robotics.

Domain	Target artifacts	Representative papers
Conversational chatbots	Flows, intents, entities, parameters or slots, actions, contexts, rules, stories	(Urrico et al., 2024, Clerissi et al., 1 Sep 2025)
Industrial robotic systems	High-level send(...) and read(...) operations in message-based ROS programs	(Santos et al., 18 Nov 2025)

In the chatbot literature, MutaBot is described as a mutation-testing tool for conversational chatbots and, in its initial form, targets Google Dialogflow chatbots (Urrico et al., 2024). A later extension generalizes the approach to multiple platforms through a platform-agnostic meta-model and concrete connectors for Dialogflow and Rasa (Clerissi et al., 1 Sep 2025). In the robotics literature, MUTABOT is defined as a mutation-testing framework for industrial robotic systems that operates on high-level actuation and sensing operations rather than on arithmetic expressions, Boolean expressions, or character strings (Santos et al., 18 Nov 2025).

A plausible implication is that “MUTABOT” names a methodological family rather than a single canonical software artifact. The common denominator is the replacement of generic mutation operators with operators aligned to domain-specific semantics.

2. Mutation-testing model and metrics

MUTABOT follows the classical mutation-testing idea of seeding a system with small faults and re-running an existing test suite to determine whether the mutants are killed. In the chatbot setting, deployment or training failures create an additional category of broken mutants; in the robotics setting, physical infeasibility and simulator-level failures motivate explicit filtering of invalid mutants.

For the multi-platform chatbot formulation, if $G$ is the total number of mutants produced, $E$ the number of equivalent mutants, $B$ the number of broken mutants, and $K$ the number of mutants killed by the test suite, then the mutation score is

$MS = \frac{K}{G - E - B}\times 100\%.$

The same work also reports the simplified percentage metric

$\%K = \frac{K}{G - E - B}\times 100\%,$

with the stated note that the underlying principle is identical to classical mutation testing (Clerissi et al., 1 Sep 2025).

For industrial robotic systems, MUTABOT uses the standard mutation-testing metric over the set $M$ of non-equivalent, non-invalid mutants and test suite $T$:

$$MS = \frac{|\{\mu \in M \mid \text{some test in }T\text{ kills }\mu\}|}{|M|}\times 100\%.$$

The robotics work adopts threshold criteria from classic software-testing practice: $MS \ge 80\%$ indicates good test quality, and $E$0 indicates exceptional fault-detection capability (Santos et al., 18 Nov 2025).

The metric formulation is therefore shared across the different MUTABOT variants, but the denominator is domain-sensitive. Chatbot experiments must account for trainability and deployability, whereas robotic experiments must account for executable feasibility in a continuous, noisy, and constrained environment.

3. MutaBot for conversational chatbots

In its initial formulation, MutaBot is a mutation-testing framework specialized for conversational chatbots, motivated by the observation that chatbot logic is expressed through declarative artifacts rather than conventional control-flow and data-flow structures (Urrico et al., 2024). The framework is organized into a Controller, a Mutant Generator, a Test Executor, and a Result Analyzer. The Controller parses configuration parameters and orchestrates the pipeline. The Mutant Generator consumes the exported agent definition and produces mutated agents, each corresponding to one operator application. The Test Executor uses Botium to run predefined conversational test suites. The Result Analyzer aggregates outcomes into overall mutation scores, per-operator kill rates, and lists of surviving mutants.

The initial operator design spans three abstraction levels. At the conversational-flow level, representative operators include Delete Transition and Swap Transition Targets. If a conversational flow is modeled as a directed graph $E$1, Delete Transition removes a transition $E$2, yielding $E$3. Swap Transition Targets selects two transitions with the same source state and exchanges their target states. At the intent level, representative operators include Remove Entity and Shuffle Utterances. At the context level, representative operators include Clear Context and Decrement Lifespan. These operators are implemented as transformations over the JSON or YAML representation of the chatbot, producing redeployable mutants (Urrico et al., 2024).

The initial implementation targets Dialogflow ES. The workflow comprises agent export or import through the Dialogflow CLI, mutant generation through JSON-level transformations, and test execution through Botium convo files and botium-connector-dialogflow. The system supports concurrency and isolation by queuing mutants and storing each mutant’s results under a unique subdirectory (Urrico et al., 2024).

The empirical evaluation in this initial line used three real-world Dialogflow chatbots designated “Bot A,” “Bot B,” and “Bot C.” Two kinds of test suites were generated per bot: an Intent-Coverage Suite with one random utterance per intent, and a Flow-Coverage Suite with scripted multi-turn scenarios. The study reports an average of approximately 1,200 mutants per bot. Before test refinement, overall mutation scores ranged between 35% and 52%, with conversational-flow mutants identified as the hardest to kill, at an average kill rate of 28%. Intent-level mutations had a higher kill rate, reported as 60%, while context-level mutations varied; clearing all contexts achieved only 22% kill rate. After adding 15 new multi-turn scenarios and 25 targeted utterances checking context resets, the reported mutation score improved to 78–84% across the three bots (Urrico et al., 2024).

These results position MutaBot as an analysis mechanism for identifying blind spots in chatbot test suites, especially around multi-turn flows, fallback behavior, and context-dependent behavior.

4. Multi-platform extension for task-based chatbots

The later chatbot work extends Mutabot to multiple platforms and makes the platform abstraction explicit through a small platform-agnostic meta-model of a task-based chatbot consisting of intents, entities, parameters, actions, and flows (Clerissi et al., 1 Sep 2025). For each supported platform, the framework provides a parser, operator implementations, a deployment connector, and a test harness connector. The platform mappings are concrete: Dialogflow is handled through intents and contexts in JSON plus webhook code in JavaScript, while Rasa is handled through domain.yml, nlu.yml, stories.yml, rules.yml, and actions.py. Deployment uses the Dialogflow CLI or Rasa CLI/Docker, and test execution is mediated by Botium.

The multi-platform workflow is explicitly enumerated: ingest the original chatbot project; enumerate all operator-location pairs; clone the project and apply the mutation; attempt training or deployment; if deployment succeeds, execute the Botium suite; and then record the mutant as broken, equivalent if no test ever fails, or killed if at least one test fails (Clerissi et al., 1 Sep 2025).

This extension defines eleven platform-agnostic mutation operators in two groups. The structural operators are removeIntentFromNLU, removeEntity, removeRule, and removeStory. The flow-level operators are removeIntentFromStory, removeIntentFromRule, removeInteractionFromStory, removeInteractionFromRule, changeSessionExpTimeInt, changeSessionExpTimeFloat, and toggleCarryOverSlots. Each operator is applied exhaustively: if a target file contains $E$4 applicable locations, the framework generates $E$5 distinct mutants (Clerissi et al., 1 Sep 2025).

The experimental evaluation on Rasa used three publicly available chatbots from the BRASATO dataset: Rock Paper Scissors, PJs Chatbot, and Customer Service. Their reported properties were, respectively, 6 intents, 1 entity, 1 custom action, and 46 Botium tests; 7 intents, 8 entities, 4 actions, and 74 tests; and 20 intents, 2 entities, 18 actions, and 83 tests. After discarding broken mutants and manually identifying equivalent mutants, the reported outcomes were Rock Paper Scissors: $E$6, $E$7, $E$8, and $E$9; PJs Chatbot: $B$0, $B$1, $B$2, and $B$3; Customer Service: $B$4, $B$5, $B$6, and $B$7 (Clerissi et al., 1 Sep 2025).

The per-category analysis reports that structural deletions are easier for Botium to detect, with kill rates up to 87%, whereas flow-level mutations such as removeInteractionFromRule and session time adjustments are detected only 23–58% of the time. The stated explanation is that Botium focuses on intent recognition or timeout failures and has a weak oracle on response content; automatically generated scenarios are short, cover training-phrase exemplars, and rarely explore deep or constrained flows. The study also notes that entities or fallback intents not well represented in training data are never exercised by Botium, leaving related mutants alive (Clerissi et al., 1 Sep 2025).

5. MUTABOT for industrial robotic systems

In industrial robotics, MUTABOT adapts mutation testing to message-based ROS programs and treats high-level read and write commands as the primary mutation targets (Santos et al., 18 Nov 2025). The central claim is that classical mutation operators are poorly suited to robotic programs because robotic software involves message-based commands and interaction with the physical world. MUTABOT therefore focuses on send(...) and read(...) operations rather than arithmetic-operator or statement-deletion mutations, and it integrates with Gazebo so that mutants remain executably feasible in a continuous, noisy, and constrained environment.

The operator set is organized into High-Level Write (Actuation) Operations and High-Level Read (Sensing) Operations. For Move(\Delta x,\Delta y,\Delta z), the operators are inversion, bias, and drop; concretely, Move(\Delta) \to Move(-\Delta), Move(\Delta) \to Move(\Delta + \epsilon) for small $B$8, and command skipping. For Rotate(\theta), the operators are complement and small-angle error, including Rotate(\theta) \to Rotate(\theta + 180^\circ) and Rotate(\theta) \to Rotate(\theta + \delta \theta). For GripperClose() and GripperOpen(), the operators are swap, suppress, and duplicate. On the sensing side, ColorRead() may be mutated into a constant classification such as “red” regardless of the actual value; PositionRead() is mutated by Gaussian-noise injection, expressed as

$B$9

and Boolean sensors such as collisionDetected() are mutated by truth-value flipping (Santos et al., 18 Nov 2025).

Mutant generation is defined over the sequence of read and write calls in a robot script. The stated procedure iterates over each command $K$0 in the original script $K$1, applies each applicable operator in $K$2, and returns the set of mutated scripts. The paper illustrates the method on a five-line script containing send("pick"), send("lift/5"), a color-dependent turn command, send("drop/5"), and send("release"). Example mutants include send("lift/5") \to send("lift/-5"), read("color") \to constant "blue", and send("turn/90") \to send("turn/270"). Each mutated script is executed in simulation under the same test suite to determine whether it is killed (Santos et al., 18 Nov 2025).

The empirical study uses a pick-and-place task drawn from the Robotic Grasping and Manipulation Competition, specifically “Task Pool: Pick Up and Place Using Tongs.” The platform is a Kinova Gen3 robot with a Robotiq-2f-85 gripper, simulated in Gazebo via ROS Kortex API. The success criteria require the robot start pose to be within 0.02 cm in $K$3, the box start pose to be within 0.01 cm in $K$4, and the final box pose to be within 0.02 cm. The test suite places colored boxes on a conveyor, invokes the script, and verifies final box location and color-dependent placement (Santos et al., 18 Nov 2025).

The reported comparison between naive conventional operators and domain-specific operators is central. Conventional operators applied to string parameters would generate hundreds of mutants, most invalid or immediately killed by the parser; more than 70% are reported as invalid, and approximately 80% crash the simulator or generate immediate syntax errors. MUTABOT’s domain-specific operators yielded exactly 26 distinct, executable mutants, with less than 10% invalid or equivalent and more than 90% executing to completion in Gazebo. Across five repeated runs, the mutation scores were 92%, 85%, 92%, 88%, and 88%, yielding an average of approximately 89%, above the 80% threshold for good test quality. No formal $K$5-values were reported, but the narrow 85–92% range is described as demonstrating stability against the injected Gaussian noise (Santos et al., 18 Nov 2025).

6. Interpretation, limitations, and future directions

A recurring interpretive point across the MUTABOT literature is that domain-specific mutation operators reveal faults that generic mutation operators either miss or represent poorly. In chatbots, the relevant artifacts are not low-level statements but conversational flows, rules, stories, entities, and context-management structures. In robotics, the relevant artifacts are not isolated arithmetic expressions but actuator commands and sensor reads. This suggests that MUTABOT is best understood as a domain-sensitive mutation strategy rather than merely a renamed instance of conventional mutation testing.

The literature also identifies several limitations. In the robotics setting, results rely on Gazebo, and real hardware may exhibit wear, backlash, or sensor drift not modeled in simulation. The operators were tailored to pick-and-place, so welding, painting, or collaborative human–robot interaction may require different action vocabularies. The mutation score was measured only against scripted test oracles, without correlation to actual historical robot failures (Santos et al., 18 Nov 2025). In the chatbot setting, manual curation of equivalent mutants is labor-intensive; some flow-level mutations yield broken or inconsistent conversation definitions; no statistical significance tests have yet been performed in the multi-platform extension; and the operator set covers only eleven core faults, leaving additional conversational defects for later work (Clerissi et al., 1 Sep 2025).

Future work follows directly from these limitations. For industrial robotics, the stated directions are an automated toolchain for mutant generation, Gazebo execution, coverage and score computation, and test-case suggestion; extensions to multi-robot coordination, cluttered bin picking, and force-controlled assembly; and hardware-in-the-loop mutation through physical perturbations such as fixture perturbation, payload variation, sensor bias, and temporary occlusions (Santos et al., 18 Nov 2025). For chatbots, the stated directions are automatic detection and filtering of equivalent mutants via differential behavior analysis, scaling to large industrial chatbots with hundreds of intents, richer response-generation and composite-entity operators, and integration with other test generators and oracles, including LLM-based conversational testers (Clerissi et al., 1 Sep 2025).

A related methodological proposal comes from the fuzzing literature. “DARWIN: Survival of the Fittest Fuzzing Mutators” proposes that MUTABOT can maintain a small, concurrent Evolution Strategy over its mutator set, sample mutators from the currently best distribution, and update that distribution via single-bit flips or Gaussian steps whenever a mutation yields new coverage (Jauernig et al., 2022). In that formulation, the proposal is explicitly prospective rather than an implemented feature of the reported MUTABOT systems.

One common misconception is to treat MUTABOT as a single, domain-independent framework. The available literature instead shows two concrete instantiations with different artifact models, different notions of invalidity or brokenness, and different evaluation infrastructures. What they share is a methodological commitment: meaningful mutants should be generated where faults are actually expressed in the target domain.