Demo2Tutorial: Converting Demos to Tutorials

• Demo2Tutorial is an agentic framework that converts raw interaction logs into structured, multimodal tutorials with hierarchical abstraction and adaptive visual guidance.
• It employs a four-component pipeline (HE-Recorder, Action Parser, Step Planner, Tutorial Composer) to reconstruct user intent and distill procedural knowledge from authentic digital interactions.
• The framework enhances both human learning through clear, annotated guides and GUI-agent planning by delivering high content and visual scores on benchmark evaluations.

Demo2Tutorial is an agentic framework for converting raw human computer-use experience—specifically synchronized screen recordings and low-level interaction logs—into structured, multimodal software tutorials for both human learners and GUI agents. It was introduced as a pipeline comprising a Human Experience Recorder, a multimodal Action Parser, a bottom-up Step Planner with actor–critic refinement, and a Tutorial Composer that emits image–text interleaved instructions with key-frame selection and adaptive visual highlights. The framework is positioned around the claim that authentic, untrimmed digital interaction traces contain procedural knowledge that is difficult to recover from raw videos alone but can be distilled into reusable instructional representations (Bai et al., 2 Jun 2026).

1. Definition and problem setting

Demo2Tutorial addresses the transformation of passive demonstrations into active learning artifacts. In its formulation, human screen interaction traces are authentic, long-form, untrimmed records of what people see, decide, and do in desktop environments, but raw demonstration videos are passive, continuous, and often redundant; they lack pedagogical structure, explicit intent, and visual guidance. Tutorials, by contrast, are step-structured, intent-explicit, and visually annotated, making them suitable both for human instruction and as interpretable external knowledge for agents (Bai et al., 2 Jun 2026).

The scope is explicitly dual-use. For human learning, the goal is to generate pedagogically effective, multimodal, step-by-step tutorials that reduce learning time and improve clarity. For agent learning, the goal is to distill structured knowledge from demonstrations to improve GUI-agent planning and generalization without requiring imitation learning from raw trajectories. This makes Demo2Tutorial a knowledge-distillation system rather than a conventional end-to-end trajectory imitation pipeline (Bai et al., 2 Jun 2026).

A recurrent misconception is that a demonstration video is already a tutorial. Demo2Tutorial rejects that equivalence. Its core premise is that demonstrations must be reorganized, summarized, and visually grounded before they become reusable instructional artifacts. Another misconception is that the framework is primarily a data collection system; in fact, recording is only the first stage in a larger abstraction pipeline that reconstructs observation, action, and intent, then organizes them into a hierarchy of goals, chapters, and steps (Bai et al., 2 Jun 2026).

2. End-to-end architecture

The framework has four principal components: the Human Experience Recorder (HE-Recorder), the Multimodal Action Parser, the Step Planner, and the Tutorial Composer. HE-Recorder captures full-screen, full-resolution screen video at 30 FPS using FFmpeg (ddagrab) together with high-precision mouse and keyboard logs, extending KeyCastOW for logging rather than visualization. Synchronization is achieved through an on-screen timer and a user hotkey at the start, providing a shared temporal reference for frame-level alignment (Bai et al., 2 Jun 2026).

The Action Parser reconstructs perception, action, and inferred intent from synchronized streams. For each action, it uses before/after screenshots sampled around the action timestamp together with action-grounded visual prompts that draw a red bounding box at mouse coordinates. GPT-4o is used as a vision–LLM without additional training, relying instead on structured prompting and Chain-of-Thought instructions to produce five fields per action: observation before, observation after, differences between states, factual action description, and inferred user intent. Robustness is supported by the grounded prompts, timestamp calibration, and preprocessing such as merging consecutive keystrokes within a 1-second window into a single typing action and consolidating modifier keys into “Shortcut Key” actions when applicable (Bai et al., 2 Jun 2026).

The Step Planner performs bottom-up hierarchical abstraction. Consecutive actions that achieve a common sub-goal are merged into step-level units; related steps are clustered into chapter-level phases; and a task goal is synthesized from the chapters. For long sequences, chunking at natural breakpoints such as application switches or long temporal gaps is used to fit LLM context limits, after which outputs are merged with temporal and dependency consistency. The planner is coupled to a Critic agent in an iterative refinement loop: the actor produces a structured tutorial draft in JSON, the critic evaluates it along dimensions including coverage, granularity, orderliness, and learnability, and refinement continues until approval or a maximum iteration count is reached (Bai et al., 2 Jun 2026).

The Tutorial Composer converts the refined task graph into an image–text interleaved tutorial. For each step, it scores candidate frames within a temporal window using four criteria: OCR-derived text relevance via RapidOCR, image sharpness via Laplacian variance, motion stability, and Gaussian-weighted temporal proximity to the action. The selected key frame is then edited with adaptive highlights using SAM2 and OCR-derived regions, including click markers, drag trajectories, hotkey badges, magnification, cropping, and contrast adjustments. Outputs are emitted in structured markdown and JSON, allowing both human readability and reuse as multimodal training data (Bai et al., 2 Jun 2026).

3. Hierarchical representation and planning formalism

Demo2Tutorial formalizes the distilled tutorial as a hierarchical task graph. The notation introduced in the framework is

$G = (V, E),$

where

$$V = V_{\text{goal}} \cup V_{\text{chap}} \cup V_{\text{step}}$$

contains nodes for the overall goal, chapters, and steps, and the edge set includes containment edges and temporal precedence edges. Specifically, containment is expressed through $E_c \subset V_{\text{chap}} \times V_{\text{step}}$ and $E_g \subset V_{\text{goal}} \times V_{\text{chap}}$, while temporal precedence is encoded by $$E_t \subset (V_{\text{step}} \cup V_{\text{chap}}) \times (V_{\text{step}} \cup V_{\text{chap}})$$. Steps carry attributes such as textual instruction, intent, and bound visual evidence, while chapters aggregate child-step intents and temporal ranges (Bai et al., 2 Jun 2026).

This representation distinguishes Demo2Tutorial from flat action summarization. A step is not simply an action or a screen event; it is an abstraction over consecutive actions that together achieve a sub-goal such as “adjust font size to 24 pt.” A chapter is not merely a contiguous video segment; it is a logical phase such as “Basic Configuration” or “Content Editing.” The graph therefore functions simultaneously as a pedagogical structure and a compact task representation for downstream systems (Bai et al., 2 Jun 2026).

The actor–critic planner enforces pedagogical regularization over this graph. Its loop is initialized by grouping actions into steps, clustering steps into chapters, and synthesizing a goal. The critic then evaluates the draft and returns actionable feedback, causing the actor to refine granularity, chapter boundaries, ordering, and wording. This iterative process is not described as introducing new model training; it is prompt-driven refinement over structured intermediate representations (Bai et al., 2 Jun 2026).

A plausible implication is that the graph serves as a compression boundary between raw behavior and reusable knowledge. The framework reports stage-wise reduction from an average of 1,208 frames per demo to 13.07 trace steps after parsing, then to 3.93 draft steps after planning, and finally to 3.71 tutorial steps after composition, for an overall compression ratio of approximately 325.71× across 110 demos (Bai et al., 2 Jun 2026).

4. Tutorial composition, benchmark design, and outputs

The output of Demo2Tutorial is a multimodal tutorial in interleaved image–text form. Key-frame selection is central: candidate frames are scored against the step text and screen evidence rather than sampled uniformly, and selected frames are then visually edited so that the instructional locus is explicit. The system uses OCR to identify relevant text regions and segmentation to localize interface elements, after which action-specific overlays such as click markers, drag trajectories, and hotkey badges are applied (Bai et al., 2 Jun 2026).

The benchmark introduced alongside the framework, TutorialBench, is organized as triplets $$\langle \text{Task Goal}, \text{Raw Demonstration}, \text{Official Tutorial} \rangle$$. It contains 110 tutorials across seven software applications: Microsoft Word (17), PowerPoint (17), Excel (16), Adobe Acrobat (15), Premiere Pro (15), Photoshop (15), and After Effects (15). The reference tutorials are sourced from official vendor websites, and the recorded demonstrations are produced by human experts who study the task goal and official tutorial once but do not copy the tutorial during recording, preserving natural, untrimmed behavior. Complexity varies by application; After Effects has the highest average number of human-authored steps at 21.4, while Word has the lowest at 7.35 (Bai et al., 2 Jun 2026).

The system also produces artifacts intended for agent consumption. In addition to markdown tutorials, it emits structured JSON aligned to the task graph and key visual evidence. This enables integration into downstream GUI-agent frameworks such as Agent-S3, where the tutorial can be injected as structured external knowledge rather than latent knowledge embedded only in model parameters (Bai et al., 2 Jun 2026).

A representative case study concerns a PowerPoint task for creating a custom motion path and adjusting timing. The generated tutorial organizes the workflow into three chapters—preparing slide content, creating the motion path, and adjusting timing and easing—and binds each step to a screenshot with targeted highlights, such as a box around the “Animations” ribbon entry, a zoomed-in “Custom Path” menu, a drag trajectory overlay on the canvas, and OCR-confirmed timing fields. The reported interpretation is that such visual grounding allows users to jump directly to exact screenshots and parameter settings rather than scrubbing through raw video (Bai et al., 2 Jun 2026).

5. Evaluation methodology and empirical findings

Tutorial quality is evaluated with a VLM-as-Judge protocol using GPT-4o over five dimensions grouped into two domains. Content Score comprises Actionability, Completeness, and Conciseness; Visual Score comprises Annotation Quality and Image Relevance. Each dimension is scored in $[0,1]$ and reported as $\times 100$, with overall score defined as the unweighted mean:

$$S_{\text{overall}} = \frac{S_{\text{act}} + S_{\text{comp}} + S_{\text{conc}} + S_{\text{anno}} + S_{\text{img}}}{5}.$$

The paper also reports agreement analysis on a 63-tutorial subset: Spearman’s $\rho$ is 0.601 for inter-rater human agreement, 0.675 for Rater-1 vs. VLM, 0.671 for Rater-2 vs. VLM, and 0.755 for averaged humans vs. VLM (Bai et al., 2 Jun 2026).

The main TutorialBench results place Demo2Tutorial ahead of all reported baselines under this protocol. Demo2Tutorial achieves an overall score of 86.2, compared with 79.1 for Human (GT), 70.3 for Vanilla Multi-Agent, 64.3 for Vision-based, and 59.2 for Text-only. Its content sub-scores are 90.5 for Actionability, 92.3 for Completeness, and 70.8 for Conciseness; its visual sub-scores are 83.3 for Annotation Quality and 94.0 for Image Relevance, yielding a visual average of 88.7, compared with 70.5 for Human (GT). The reported interpretation is that adaptive visual grounding and key-frame selection are decisive contributors to this advantage (Bai et al., 2 Jun 2026).

The agent-learning evaluation uses OSWorld subsets for Chrome (17 tasks) and VLC (14 tasks) within Agent-S3, with GPT-o4-mini and GPT-5 as planning backbones. Under GPT-o4-mini, success rates improve from 47.1 to 58.8 on Chrome and from 53.4 to 56.1 on VLC when the full tutorial is added. Under GPT-5, success rates improve from 52.9 to 70.6 on Chrome and from 59.6 to 70.7 on VLC. Qualitatively, the paper states that text-only assistance gives marginal gains, image-only assistance helps variably, and the full multimodal tutorial yields the largest and most consistent improvements (Bai et al., 2 Jun 2026).

A human study with $$V = V_{\text{goal}} \cup V_{\text{chap}} \cup V_{\text{step}}$$0 evaluates a non-trivial PowerPoint motion path animation task with timing adjustments. Completion time is 131.6 seconds using tutorials versus 147.1 seconds using demo videos, corresponding to a 10.5% faster completion time, and 80% of participants prefer tutorials over demo videos. This result should not be conflated with a general superiority claim across all instructional settings; it is a task-specific study in the evaluation suite (Bai et al., 2 Jun 2026).

Component ablations further isolate system contributions. On a 21-task subset, the full system scores 90.2 overall, while removing Visual Highlight reduces the score to 66.2 and Annotation Quality to 3.8. Removing Task Hierarchy reduces overall score to 84.5 and lowers Conciseness from 84.8 to 64.8. Removing Iterative Refinement yields 87.7, and removing Key-Frame Selection yields 87.5. These results support the paper’s conclusion that adaptive visual highlights are indispensable, while hierarchy and iterative refinement further improve clarity and structure (Bai et al., 2 Jun 2026).

6. Relation to earlier tutorial-generation research

Demo2Tutorial belongs to a broader research line that treats tutorial construction as a computational synthesis problem. Earlier work in games framed tutorial generation as an AI problem encompassing natural-language descriptions of rules, instructive game levels, and agent-driven demonstrations. “Press Space to Fire”: Automatic Video Game Tutorial Generation formalized tutorial generation in the General Video Game AI framework and argued for outputs that teach controls, rules, strategies, and failure modes through text, levels, and demonstrations (Green et al., 2018).

AtDELFI later operationalized this game-oriented program by parsing VGDL rule descriptions, constructing a mechanic graph, extracting critical win and loss paths, generating explanatory text from templates, and attaching short agent-executed demonstration clips. Its results suggested that graph-based rule modeling works well for simple arcade-style games, but that more complex games require higher-level understanding than isolated mechanics (Green et al., 2018).

This lineage clarifies what Demo2Tutorial changes. The earlier systems are rule-centric and largely symbolic, deriving tutorials from explicit game descriptions and agent play. Demo2Tutorial instead begins with authentic, untrimmed human desktop interactions and reconstructs intent from multimodal evidence rather than from a formal rule language. This suggests a shift from explicit-rule tutorialization to experience distillation: instead of assuming access to machine-readable mechanics, the framework infers pedagogically useful structure from real usage traces (Bai et al., 2 Jun 2026).

The same “demo-to-tutorial” motif also appears in domain-specific reproducibility guides. The InstructFX2FX material, for example, presents “Demo2Tutorial: A Reproducible Guide to InstructFX2FX for Multi-turn Text-to-Preset Refinement,” where a demonstration-oriented description is converted into a stepwise guide for a stateful audio-effect refinement system (Yu et al., 20 Jun 2026). A plausible implication is that “Demo2Tutorial” functions both as the name of a specific software-tutorial framework and as a more general pattern for transforming demonstrations into reusable instructional artifacts across domains.

7. Limitations, robustness, and prospective directions

The framework’s limitations are tied to privacy, perception robustness, and domain variability. Screen recordings may contain sensitive documents, emails, or personally identifiable information, so deployment requires consent, on-device redaction such as blurring OCR-detected emails or IDs, and secure storage. Domain generalization is also incomplete: the reported system is centered on desktop applications, and extensions to mobile or web UIs, or to interfaces with heavy animation and transitions, may require more robust visual grounding and timing models (Bai et al., 2 Jun 2026).

Failure modes are documented. The planner may miss sub-procedures during abstraction, as in omitted deletion steps in Excel. The parser may confuse visually similar interface effects, such as misrecognizing Morph as Fade in PowerPoint. UI variability across software versions, themes, and localization affects OCR and segmentation performance, making the system sensitive to the quality of perception modules and prompt engineering (Bai et al., 2 Jun 2026).

Operationally, the framework is resource-intensive but does not require model fine-tuning. Average per-tutorial runtime is approximately 368 seconds with about 259K input tokens and 3K output tokens, at an API cost of approximately \$1.36. The implementation depends on prompting commercial or open VLMs such as GPT-4o or Qwen-VL-32B-Instruct, rather than on training a dedicated parser or planner model from scratch (Bai et al., 2 Jun 2026).

The reported future directions include mobile and web support, computational efficiency through pre-screening of salient segments, vision-only action inference for arbitrary web screencasts without HE-Recorder, and personalization of tutorial granularity. These directions are consistent with the framework’s central thesis: that structured tutorials distilled from human experience can function as reusable knowledge representations for both human learning and agent capabilities, provided the extraction process becomes more robust, efficient, and adaptive (Bai et al., 2 Jun 2026).