Demo2Tutorial: From Human Experience to Multimodal Software Tutorials

Abstract: Human experience in digital environments offers a vast, underexplored resource of authentic, untrimmed interactions that contain rich procedural knowledge. We introduce Demo2Tutorial, a framework that transforms this experience captured via screen recordings and interaction logs into structured, multimodal software tutorials for teaching both humans and agents. Demo2Tutorial first collects human experience via a dedicated recorder, then parses raw experience using a multimodal Action Parser to reconstruct perception, action, and intent. A Step Planner then abstracts these steps into hierarchical task graphs representing goals and steps. Finally, a Tutorial Composer transforms the parsed experience into structured, reusable image-text instructions. We evaluate the tutorial generation quality on a new benchmark derived from official software documentation. We further demonstrate that this distilled representation benefits (i) human learning, by automatically generating multimodal tutorials, and (ii) agent learning, by improving downstream GUI-agent planning and generalization. Experiments show Demo2Tutorial produces high-quality tutorials that surpass human-authored ones and significantly outperform baseline methods, while enabling both faster human task completion and improved GUI agent planning, demonstrating that structured tutorials distilled from human experience can serve as effective knowledge representations for advancing both human learning and agent capabilities. Code and data will be available at https://github.com/showlab/Demo2Tutorial.

• The paper introduces Demo2Tutorial, a pipeline for distilling raw human-computer interaction videos into step-by-step, multimodal tutorials.
• Key modules include HE-Recorder, Action Parser, Step Planner, and Tutorial Composer that ensure precise temporal alignment, semantic parsing, and effective visual cue selection.
• Results demonstrate superior tutorial quality with improved GUI agent performance and reduced human task time, achieving up to a 325× compression ratio.

Demo2Tutorial: Automated Distillation of Human Experience into Multimodal Software Tutorials

Motivation and Problem Statement

The Demo2Tutorial framework addresses the transformation of untrimmed, raw human computer-use demonstrations into structured, multimodal tutorials that are pedagogically actionable for both humans and GUI agents. While raw demonstration videos capture procedural task experience, they are inherently unstructured—lacking clear segmentation, instructional granularity, and explicit guidance (see (Figure 1)), making them inefficient for downstream consumption. In contrast, expertly crafted tutorials provide stepwise, visually annotated instructions but are laborious and costly to produce at scale.

Figure 1: Raw demonstration videos (left) lack actionable structure and guidance, whereas tutorials (right) offer stepwise, annotated instructions optimized for learnability.

Framework Overview and Technical Contributions

Demo2Tutorial introduces a pipeline integrating four tightly-coupled modules: HE-Recorder, Action Parser, Step Planner, and Tutorial Composer, each architected for robust experience distillation via temporally aligned, semantically rich, and visually grounded representations.

Figure 2: The Demo2Tutorial pipeline: synchronized recording, semantic parsing, hierarchical planning with actor-critic optimization, and adaptive visual-text composition.

HE-Recorder synchronizes high-fidelity screen capture with granular user action logs. Unlike standard screen recorders, it natively timestamps all mouse and keyboard events for precise alignment with visual changes, enabling downstream temporal and causal reasoning.

Action Parser employs multimodal LLMs (e.g., GPT-4o) for fine-grained grounding of user actions in context, incorporating bidirectional screenshot extraction and explicit action-region highlighting via visual prompt engineering. Output is a sequence of context-anchored semantic units: pre- and post-state, action, and intent.

Step Planner executes bottom-up aggregation of atomic interactions into semantically meaningful instructional steps, followed by hierarchical clustering into chapters and global goals. The planner is tightly coupled to a critic in an actor-critic loop, enforcing structural coherence, non-redundancy, and pedagogical quality through iterative curriculum-level feedback.

Tutorial Composer selects optimal key-frames based on a four-way score: OCR-based text-instruction alignment, sharpness, motion stability, and temporal proximity, augmented with adaptive visual highlights (segmentation, badges, focus effects). This stage ensures procedural clarity in image-text interleaving for both human readability and multimodal model fine-tuning.

Benchmark and Evaluation Protocol

To quantify tutorial quality, the authors introduce TutorialBench, spanning 110 tasks across seven productivity and creative applications, with reference triplets: $$(\text{Goal}, \text{Demonstration}, \text{Official Tutorial})$$ (see (Figure 3)). Human-authored tutorials serve as gold-standard references; demonstrations capture authentic, expert-driven, untrimmed workflows.

Figure 3: TutorialBench distribution and instructional complexity statistics across applications.

Scoring leverages a five-dimension metric evaluated by strong VLMs (GPT-4o): Actionability, Completeness, Conciseness (content), Annotation Quality, Image Relevance (visuals). This joint metric robustly reflects learning efficacy, actionable clarity, and visual grounding.

Results: Quantitative and Qualitative Analysis

Tutorial Generation Quality

Demo2Tutorial establishes a new state-of-the-art with an overall score of 86.2, exceeding both human-authored tutorials (79.1) and all baseline methods across content and visual axes. Notably, it produced substantially higher Visual Scores (88.7 vs. 70.5), indicating effective visual grounding and annotation (see Table in main text). The ablation analysis highlights the necessity of actor-critic iterative refinement and adaptive highlights—removal sharply reduces annotation quality (from 85.2 to 3.8).

Downstream Impact: GUI Agent Planning

In the context of GUI agent learning, the incorporation of tutorials into Agent-S3 (Gonzalez-Pumariega et al., 2 Oct 2025) planning yields consistent and significant gains: for GPT-5, Chrome domain success rates improved from 52.9% to 70.6%. Detailed ablations demonstrate that multimodal alignment—not text or visuals alone—is critical to agent performance (see (Figure 4)).

Figure 4: Success rates for GUI agents with varying levels of tutorial context; multimodal input offers the largest gains.

Human User Studies

Empirical studies with human participants show that access to structured tutorials reduces task completion time by 10.5% and is preferred by 80% of participants over raw demonstrations, establishing the dual pedagogical and computational efficacy (see (Figure 5)).

Figure 5: Human user study: generated tutorials yield faster completion and are strongly preferred.

Qualitative Assessment

Qualitative comparisons (see (Figure 6)) reveal that Demo2Tutorial uniquely provides step-aligned, annotated visuals with explicit pointers to interface regions, significantly enhancing interpretability compared to both vanilla multi-agent pipelines and official tutorials.

Figure 6: Demo2Tutorial output yields superior image-text alignment and actionable highlights versus baseline and official content.

Compression Characteristics

The pipeline achieves a mean compression ratio of $325.71\times$, transforming lengthy demonstrations (average $>1200$ frames) into concise, high-informational steps—major reduction occurring in the Action Parser and Step Planner stages (see (Figure 7)).

Figure 7: Compression analysis; compression is maximized at Action Parser stage via action-based frame filtering.

Representative Examples and Failure Modes

Generated tutorials for each major software domain exhibit fine-grained procedural clarity and visual alignment (see Figures 9–14). Failure cases reveal current pipeline challenges: missed instructional coverage (Planner condensation errors) and annotation misalignment due to VLM misrecognition (see Figures 15–16).

Implications and Future Directions

Demo2Tutorial demonstrates that agentic distillation pipelines can outperform manual curation both in instructional efficacy and visual quality, enabling scalable, high-fidelity knowledge transfer for software education and agent planning. These findings suggest that interpretable, multimodal instructional artifacts offer a tractable alternative to behavior cloning for GUI agents, enhancing explicit reasoning and sample efficiency.

Opportunities for extension include: fully vision-based action inference (eliminating recorder dependence), adaptive content pre-filtering to reduce computational cost, domain adaptation to mobile/web platforms, and user-personalized tutorial synthesis.

Conclusion

Demo2Tutorial (2606.03951) provides a reproducible, superior pipeline for converting unstructured human-computer interactions into actionable, visually grounded tutorials. Comprehensive experiments demonstrate strong gains for both human and AI learners, with the framework overtaking human-authored references on holistic quality metrics. This work substantiates the dual value of structured procedural knowledge and marks a salient direction for multimodal knowledge distillation in real-world interactive environments.