WikiHowAgent: Multi-Agent Dialogue System

• WikiHowAgent is a multi-agent workflow that transforms static WikiHow tutorials into interactive, step-by-step teacher–learner conversations.
• It employs dedicated teacher and learner agents, an interaction manager, and an evaluator to manage dialogue flow and assess pedagogic quality.
• Empirical tests over 14,287 tutorials show high completion (95.2%) and strong clarity, while revealing mixed results in learner engagement and instructional depth.

WikiHowAgent is a multi-agent workflow for converting static WikiHow tutorials into interactive, step-by-step teacher–learner conversations and for assessing their pedagogic quality at scale. It combines a teacher agent, a learner agent, an interaction manager, and an evaluator, all organized around procedural learning dialogues grounded in real-world “how-to” content. In experiments over 14,287 tutorials and 114,296 simulated conversations, the workflow is used both as a generation system for instructional dialogue and as an evaluation framework that blends computational metrics, rubric-based scoring, and human judgment alignment (Pei et al., 7 Jul 2025).

1. Problem setting and research context

WikiHowAgent addresses a specific gap in AI4Education and instructional NLP: existing work on LLM-based virtual educators and learners often lacks scalability, does not fully exploit diverse, large-scale course content, and provides limited machinery for assessing pedagogic quality. Its central object is the procedural tutorial rather than the open-ended conversation. Static tutorial steps are treated as the substrate from which an interaction is generated, with the conversation constrained by tutorial order, clarification requests, and explicit completion conditions (Pei et al., 7 Jul 2025).

This procedural framing places WikiHowAgent within a broader line of research on LLMs over how-to knowledge. In related work on open-domain procedure customization, how-to procedures were represented as editable plans, and multi-agent architectures were used to customize a base plan subject to a user constraint through insert/replace edits plus executability verification. On the CustomPlans benchmark of 206 procedures, a sequential edit-based architecture outperformed an end-to-end prompted LLM by 10.5 percentage points in FullAcc, showing that decomposing procedural tasks into specialized agent roles can improve fidelity and executability (Lal et al., 2023). A plausible implication is that WikiHowAgent extends the same general decomposition principle from plan editing to pedagogic dialogue generation.

2. Core architecture

WikiHowAgent is modular. Both teacher and learner are wrappers around off-the-shelf LLMs, ranging from open-source 7–14 B-parameter models—DeepSeek, Qwen2, Gemma, OLMo2, OpenChat, Llama3, and Phi4—to GPT-4. The two agents share a common interface but are initialized with different prompt templates and internal reasoning modules. They also share a lightweight message buffer, a finite-state flag for needs_clarification, and counters tracking how many steps have been delivered. This design supports both homogeneous learning, where teacher and learner use the same backbone model, and heterogeneous learning, where different backbones are assigned to the two roles (Pei et al., 7 Jul 2025).

Component	Inputs/state	Function
Teacher agent	Tutorial summary, current step index and content, clarification flag	Answers questions or emits the next procedural instruction; appends FINISHED after the final step
Learner agent	Teacher’s last utterance	Asks a concise targeted question if unclear; otherwise acknowledges comprehension and says “let’s move on”
Interaction manager	Dialogue state, step counter, clarification status	Alternates teacher and learner nodes until steps are exhausted or 40 exchanges are reached
Evaluator	Entire conversation and original tutorial	Computes computational metrics and eight rubric scores

The teacher prompt is explicitly instructional. It begins: “You are an expert teacher guiding a learner step by step through a tutorial via multi-turn conversations,” and assigns ordered priorities: answer learner questions first, provide the current step when the learner asks to move on, append FINISHED after the last step, and highlight key phrases found in the tutorial. The learner prompt is explicitly inquisitive: “You are a curious student engaged in a step-by-step learning conversation,” and instructs the model to ask a brief specific question when unclear, otherwise acknowledge and request the next step, and stop when the teacher says FINISHED (Pei et al., 7 Jul 2025).

This architecture is notable for its symmetry. Both agents are structurally lightweight, while functional specialization is delegated to prompts, role-specific logic, and the interaction manager. The result is a workflow that can be re-instantiated across many backbone models without changing the orchestration layer.

3. Dialogue control and pedagogic assessment

The interaction manager is a simple state machine. It alternates between teacher and learner nodes, enforces turn-taking, and terminates when the tutorial’s steps are exhausted or when a preset maximum of 40 exchanges is reached. It monitors question tokens from the learner to raise the clarification flag, increments the current-step counter whenever the learner asks to “move on,” and ensures that every teacher message is followed by a learner reply and vice versa (Pei et al., 7 Jul 2025).

Once the conversation ends, evaluation proceeds in two stages. The computational metrics pipeline measures question rate, completion, diversity, BLEU, METEOR, ROUGE, and BERTScore. Question rate is defined as the proportion of learner turns that contain a question. Completion is the percentage of dialogues ending with FINISHED. Diversity is unique 2-gram diversity in the aggregate of teacher and learner utterances. BLEU, METEOR, and ROUGE measure n-gram-level similarity to the reference tutorial, while BERTScore measures context-aware semantic overlap through BERT embeddings (Pei et al., 7 Jul 2025).

The rubric-based stage formalizes eight pedagogic criteria: Clarity, Truthfulness, Engagement, Coherence, Depth, Relevance, Progress, and Naturalness. An LLM-based evaluator agent ingests the full conversation together with the original tutorial and returns a 1–5 score for each dimension. The rubrics are explicit. For example, a Clarity score of 5 requires “instructions [that] are highly clear, logically structured, and easy to understand,” whereas a score of 1 corresponds to “highly unclear, disorganized, or misleading” teaching (Pei et al., 7 Jul 2025).

The evaluation framework also embeds standard classification notation:

$$\text{Precision} = \frac{TP}{TP + FP}, \qquad \text{Recall} = \frac{TP}{TP + FN}, \qquad \text{F1} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}.$$

Although these formulas are standard, their inclusion situates WikiHowAgent’s evaluation regime within conventional quantitative assessment as well as education-oriented rubric scoring.

4. Dataset construction and knowledge organization

WikiHowAgent’s corpus is derived from 3,680 publicly licensed WikiHow articles spanning 17 high-level domains, including Automobiles, Cooking, and Technology, and 727 subtopics. Each article averages 1,516 tokens and four distinct methods, with each method averaging 5.2 steps. Treating each multi-step method as an independent tutorial and discarding one-step methods, the workflow retains 14,287 tutorials containing 49,344 individual instructional steps (Pei et al., 7 Jul 2025).

By executing the workflow in a homogeneous setting across eight backbone LLMs, the system generates $14{,}287 \times 8 = 114{,}296$ teacher–learner dialogues. No additional data augmentation is used; the only cleaning step is the removal of tutorials with fewer than two steps. All tutorials and dialogues are then organized into a hierarchical knowledge graph linking domains to topics to tutorial nodes (Pei et al., 7 Jul 2025).

This construction matters methodologically because it preserves the procedural granularity of WikiHow methods rather than collapsing an article into a single monolithic sequence. It also makes the dataset naturally compatible with both instructional sequencing and graph-based retrieval or analysis. The knowledge graph is not merely a storage convenience; it defines the hierarchical organization of the corpus.

5. Experimental findings

In the homogeneous setting, averaged across 114K+ dialogues and eight LLM backbones, WikiHowAgent reports 95.2% completion, 80.1% diversity, and 70.4% semantic alignment as measured by BERTScore. Learner question-asking varies substantially, with a mean of 38.6% and a standard deviation of 20.7%, indicating that some models, specifically OpenChat and Llama3, simulate active curiosity much more strongly than others. Rubric evaluation shows high average Relevance, with $\mu = 4.70/5$, and high Clarity, with $\mu = 4.61$, while Depth, Truthfulness, and Naturalness exhibit larger variability, with $\sigma \approx 0.5$–$0.7$. OpenChat and OLMo2 are identified as the most balanced performers (Pei et al., 7 Jul 2025).

In heterogeneous experiments, where OpenChat serves as teacher and evaluator while learner models vary, BLEU increases by 3.2% and ROUGE by 1.5%; completion rises to approximately 99%; and diversity increases slightly. At the same time, learner question rate falls by nearly 9%. The empirical pattern is therefore mixed: heterogeneous pairing improves surface alignment and completion while reducing simulated engagement (Pei et al., 7 Jul 2025).

Domain-wise analysis across the 17 areas shows that Clarity, Coherence, Relevance, and Progress remain robust across topics, whereas Engagement, Depth, Truthfulness, and Naturalness vary considerably with domain complexity. Human judgment alignment reveals the same asymmetry. Using 25 tutorials and 175 model-generated conversations rated independently by two master’s/PhD-level annotators, Clarity, Coherence, Relevance, and Progress show moderate to strong correlation between LLM-based scores and human judgments, with Pearson’s $r > 0.5$ and $p < 0.01$ in representative cases. By contrast, Depth, Engagement, Naturalness, and Truthfulness correlate weakly, indicating that the evaluator tends to track some pedagogic dimensions more reliably than others (Pei et al., 7 Jul 2025).

A case study illustrates the intended behavior of the workflow. The teacher incrementally expands on steps such as “state your main ideas” and “find support,” while the learner alternates between confirmations—“I understand, let’s move on”—and targeted clarification questions, ending with an outline of a persuasive speech body that mirrors the tutorial structure (Pei et al., 7 Jul 2025).

6. Interpretation, related procedural-agent work, and boundaries

WikiHowAgent demonstrates that large-scale instructional dialogue can be synthesized from procedural documents without abandoning step structure. Its teacher–learner conversations are not free-form chat in the ordinary sense; they are state-controlled traversals of tutorial steps, modulated by clarification requests and assessed through both text-overlap and pedagogy-specific rubrics (Pei et al., 7 Jul 2025).

Its nearest methodological relatives are multi-agent systems for procedural reasoning rather than general conversational agents. In open-domain procedure customization, a Modify-agent and a Verify-agent were arranged in several interaction patterns—UNIFIED, SEQUENTIAL, PARALLEL, and REVERSE-SEQUENTIAL—over a semi-symbolic edit language consisting of replace(i, "New text") and insert(i, "New step after step i"). The best configuration was SEQUENTIAL, which achieved 51.94% FullAcc versus 41.46% for end-to-end rewriting, and all edit-based agents improved ExecAcc relative to the end-to-end baseline (Lal et al., 2023). This suggests a broader pattern: procedural tasks benefit when LLM responsibilities are decomposed into targeted roles with explicit control structures.

At the same time, WikiHowAgent’s findings delimit what such workflows currently establish. High completion and strong Relevance or Clarity scores do not imply uniformly human-like pedagogy. The weak correlations for Depth, Engagement, Naturalness, and Truthfulness show that LLM-based evaluation can overrate conversational fluency relative to human perceptions of instructional richness (Pei et al., 7 Jul 2025). A plausible implication is that pedagogic adequacy remains only partially captured by current automated evaluators, especially on dimensions that depend on nuance rather than structural correctness or semantic proximity.

Within the research landscape, WikiHowAgent therefore occupies a specific position: a scalable, modular, multi-LLM workflow for procedural learning dialogue and pedagogic assessment, grounded in real-world tutorials, quantitatively characterized across multiple backbones, and methodologically adjacent to other multi-agent systems that treat procedural knowledge as a structured object rather than an unconstrained generation target (Pei et al., 7 Jul 2025).