Botium Testing Framework

• Botium Testing Framework is a modular tool for automating the testing of conversational agents via detailed script-driven interactions.
• It supports multiple platforms like Dialogflow, Rasa, Alexa, and Microsoft Bot Framework through a flexible, multi-channel architecture.
• Integration with Mutabot enables mutation testing to expose blind spots in dialogue flows, thereby improving test coverage and chatbot robustness.

Botium is a comprehensive framework dedicated to the automated testing of conversational agents, with emphasis on task-based chatbots deployed on platforms such as Dialogflow and Rasa. Leveraging a modular architecture, Botium enables rigorous validation through script-driven interactions, multi-channel abstraction, and algorithmic test case generation. Recent experiments demonstrate how mutation testing techniques, applied via the Mutabot framework and orchestrated alongside Botium, can expose systematic blind spots in coverage and robustness, advancing the state-of-the-art in evaluation for conversational modeling (Clerissi et al., 1 Sep 2025).

1. Architecture and Core Components

Botium is architected around a set of interconnected modules that support chatbot testing from script definition to multi-platform execution. The central element, Botium Core, is a Node.js library implementing the "BotDriver" abstraction, responsible for marshalling user-bot exchanges, parsing test scripts, and routing requests to channel drivers. Channel drivers interface with platform-specific APIs, such as Dialogflow’s REST API or Rasa’s HTTP interface, to facilitate real-time communication and extract intent predictions, entity values, and response texts.

Test cases are described using plain-text "Convo" files (.convo.txt), structured as turn-by-turn exchanges (e.g., #me Hello #bot INTENT greeting). Botium CLI and the web-based Botium Box offer management tools for suite execution, parallelism, and centralized reporting. The framework supports additional drivers for platforms such as Alexa and Microsoft Bot Framework, enabling extensibility across heterogeneous conversational infrastructures.

2. Algorithmic Test Generation Workflow

Botium’s test generator systematically constructs conversational test suites by analyzing training data, intent definitions, entity lists, and dialog flows (stories/rules). For each intent $I$ and every Story/Rule $S$ utilizing $I$, the generator:

• Produces minimal conversations by traversing the sequence of intents and actions defined in $S$.
• At each user turn, selects an example utterance associated with the current intent.
• Substitutes entity parameters with randomly sampled (configurable) values from allowed entity lists.

Negative tests are optionally generated by injecting fallback utterances (out-of-scope phrases) and synonym/paraphrase transformations. Each generated conversation is materialized as a separate Botium Convo file.

Configuration is available via command-line options (e.g., --utterances-per-intent N, --max-turns M, --include-fallback, --entity-value-file, --parallel), controlling sample sizes, conversation lengths, slot-filling parameters, fallback behavior, and execution parallelism. Generated test types include full conversations following story/rule flows, single-intent isolation tests, and slot-filling scenarios exploring parameter permutations.

3. Mutabot Mutation Testing Methodology

Mutation testing is integrated using Mutabot, which operationalizes established techniques from software testing to simulate conversational faults. A chatbot mutant $C'$ is defined as a replica of the original chatbot $C$ with a single conversational element perturbed via a mutation operator $\mu \in M$.

On Rasa (and analogously Dialogflow), eleven mutation operators are implemented, including:

• removeIntentFromNLU: deleting intent definitions
• removeEntity: deleting entity types
• removeRule/removeStory: deleting entire dialog flows
• removeIntentFromStory/removeIntentFromRule: removing intent turns from specific flows
• removeInteractionFromStory/removeInteractionFromRule: removing single user-bot exchanges
• changeSessionExpTimeInt/Float: perturbing session expiration timing
• toggleCarryOverSlots: toggling slot carry-over at session termination

Operators are applied exhaustively, producing $G$ total mutants per chatbot. Each mutant is evaluated as follows:

• If it fails to deploy or train, it is marked "Broken" ($B$).
• If it is semantically identical to the original, "Equivalent" ($E$).
• If killed by at least one Botium test, "Killed" ($K$).
• Surviving mutants ($S$) are those undetected by the suite: $S = G - B - E - K$.

The primary metric is the mutation score $MS$:

$MS = \frac{K}{G - B - E} \times 100\%$

4. Orchestration: Integrating Botium and Mutabot

Experimental orchestration utilizes a pipeline wherein Botium’s test generator establishes a baseline test suite for each chatbot and platform. Mutabot then generates conversational mutants corresponding to each operator and systematically deploys them. Each mutant is subjected to the Botium suite against its dedicated endpoint; statuses are recorded for each mutant based on deployment outcome, detection, or semantic equivalence. The following pseudo-code illustrates this approach:

for platform in {Dialogflow, Rasa}: for chatbot in SubjectChatbots: botiumSuite = Botium.generateSuite(chatbot.config) Botium.runSuite(botiumSuite) # warm-up run mutants = Mutabot.generateMutants(chatbot.source, platform) for m in mutants: if not m.deployable(): recordStatus(m, "Broken") continue if Mutabot.isEquivalent(m): recordStatus(m, "Equivalent") continue m.deploy() results = Botium.runSuite(botiumSuite, endpoint=m.endpoint) if results.anyTestFails() or results.anyTimeout(): recordStatus(m, "Killed") else: recordStatus(m, "Survived")

This pipeline yields comprehensive metrics per platform/chatbot combination, enabling comparative analysis of mutation detection efficacy.

5. Experimental Findings

Experiments on three Rasa (from the BRASATO dataset) and three Dialogflow chatbots demonstrate variation in mutation score and coverage. For Rasa, the following summary encapsulates outcomes:

Chatbot	G	B	E	K	MS (%)
Rock Paper Scissors	27	0	2	12	48
PJs Chatbot	54	13	6	27	77
Customer Service	102	12	18	31	43

Observed phenomena include:

• Element-removal mutants (e.g., deleted intents) are readily killed, as Botium immediately detects missing or unhandled intents.
• Session-time and carry-over slot mutations survive, revealing that Botium lacks oracles for timing and slot-propagation behaviors.
• Complex, conditional dialog flows are infrequently exercised, leaving numerous flow mutants undetected.
• Botium asserts only on intent matches, not on the actual bot response text—mutants that modify responses while preserving intent accuracy are missed.
• Slot and entity values absent from training utterances remain untested, indicating incomplete test coverage.

6. Evaluation and Recommendations

Systematic analysis reveals critical areas for improvement in Botium’s test generation and oracle design:

1. Richer Oracles: It is recommended to extend assertions beyond intent matching to include textual response patterns, slot values, invoked actions, and semantic similarity metrics.
2. Flow Coverage: Test generation should be augmented to cover all dialog branches, including user-driven conditional paths and if/else logic.
3. Timing and Session Tests: Incorporation of deliberate input delays and simulated session expiration is advised to expose mutations that alter session behavior.
4. Negative & Fallback Testing: Comprehensive coverage of unexpected utterances, out-of-vocabulary messages, and omission scenarios is necessary.
5. Data-Driven Slot Coverage: Systematic enumeration of slot/entity value lists—including edge cases and the full combinatorial space—will increase test effectiveness.
6. Equivalent-Mutant Detection: Automation in semantic differentiation or coverage-driven analysis is needed to accurately identify equivalent mutants ($E$), thus reducing manual verification effort.

A plausible implication is that by evolving Botium according to these recommendations, one may achieve higher mutation scores and enhanced confidence in the robustness and correctness of task-based chatbots (Clerissi et al., 1 Sep 2025).