Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining

Abstract: Explicit skill libraries make computer-using agents easier to inspect, but it remains unclear whether such libraries can be mined from interaction data in a way that improves downstream policies. We study this question through a three-stage pipeline that segments GUI trajectories, clusters segments into candidate skills, and trains a skill-aware policy from the resulting annotations. The mined clusters are readable on the source benchmark: five of eight clusters have at least 0.95 purity against InteraSkill Workflows labels. However, readability does not imply transfer. GRPO improves IW skill-step accuracy only from 18.5\% to 20.5\%, leaves BrowseComp+ essentially unchanged, and underperforms trivial frequency priors on key source-domain metrics. We therefore present the method as a diagnostic study: trajectory mining can expose inspectable skill structure, but the current boundary detector, orderless segment representation, and offline reward model are insufficient for reliable cross-domain policy improvement.

• The paper introduces a three-phase pipeline to automatically generate SKILL.md from GUI interaction logs, emphasizing segmentation, clustering, and policy training.
• The methodology employs unsupervised and weakly-supervised techniques that achieve high recall in skill detection but suffer from low precision and limited cross-domain transfer.
• Empirical results reveal that while mined skills are interpretable, they offer negligible policy improvements compared to simple frequency-based baselines.

Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining

Introduction and Motivation

This work investigates the automatic derivation of explicit, inspectable skill libraries (SKILL.md) for Computer-Using Agents (CUAs) by mining skill structure from GUI interaction trajectories. The main challenge addressed is whether unsupervised or weakly-supervised segmentation and clustering of GUI action sequences yields actionable skills that benefit policy learning and transfer across domains. While human-designed skill libraries facilitate transparency and debugging, they are labor-intensive and inflexible. As such, the paper proposes a three-phase pipeline for automated SKILL.md generation and evaluates the pipeline with a focus on both the structure and transferability of the discovered skills.

Figure 1: Study design for automated SKILL.md generation. Evaluation spans pipeline diagnostics, source-domain accuracy, and transfer checks.

Methodology

The pipeline comprises three phases:

Trajectory Segmentation: Trajectories are segmented by detecting large Euclidean changes in fixed-length low-level action representations (click, type, scroll, etc.), using a percentile-based threshold ($\theta$). The segmentation is designed for computational simplicity; however, high recall comes at the cost of poor precision, with frequent over-segmentation at action discontinuities that occur within rather than between skill boundaries.

Figure 2: IW action-discontinuity scores. The selected threshold $\theta = 1.545$ optimizes F1 for source segmentation.

Skill Embedding and Clustering: Each segment is summarized by the empirical mean and diagonal variance of its feature vectors, producing bag-of-actions representations. Agglomerative clustering via squared Bures (diagonal 2-Wasserstein) distance groups these into candidate skills, with pseudo-labels used to train a supervised-contrastive MLP encoder, refining segment embeddings for better alignment with source ground-truth skills.

Figure 3: t-SNE projection of learned 16-dim skill embeddings on IW, illustrating well-separated, discriminative skill representations.

Skill-Aware Policy Training: Using the mined skill assignments, models—including a zero-shot Qwen3-8B, MLP, Transformer, and a GRPO-trained Qwen3-8B—are trained/evaluated on the skill prediction task over the source (IW) and held-out (WebArena, BrowseComp+) datasets. The primary downstream metric is skill-step sequence accuracy.

Empirical Results

Segmentation and Clustering Quality

Skill boundary detection achieves high recall (0.803) but low precision (0.419) on the IW dataset, with marked domain instability: direct threshold transfer to WebArena yields poor F1. Clustering with the Wasserstein metric yields interpretable groups aligned with ground-truth skills (NMI = 0.650 at $k=8$). Supervised-contrastive refinement significantly improves separability (NMI = 0.862).

Figure 4: Action-type distributions for each discovered IW cluster, confirming interpretability and cluster-skill mapping.

Figure 5: Supervised-contrastive encoder shows stable convergence, indicating no overfitting to pseudo-label clusters.

Skill Transfer and Policy Learning

While five out of eight clusters achieve at least 0.95 purity against ground-truth labels, skill structure does not translate into a transferable skill vocabulary. The main GRPO-trained Qwen3-8B policy achieves only marginal gains on the source—20.5% skill-step accuracy (vs. 18.5% baseline)—and negligible or negative transfer to WebArena (44.2% vs. 55.8%) and BrowseComp+ (43.3% vs. 43.5%). Zero-shot closed-source (GPT-5, Claude Sonnet~4.5) and open models achieve higher transfer accuracy, highlighting the ineffectiveness of the mining pipeline for cross-domain policy improvement.

(Table: Main Results from the Paper, summarized)

Model	IW	WebArena	BrowseComp+
Qwen3-8B (zero-shot)	18.5%	55.8%	43.5%
Qwen3-8B (GRPO)	20.5%	44.2%	43.3%
Llama-3.1-70B	30.0%	56.2%	51.9%
OLMo-3-7B	14.3%	54.5%	61.4%
GPT-5	24.5%	57.6%	59.5%

The MLP and Transformer models operating directly on skill embeddings outperform the GRPO policy and are matched or outperformed by a trivial most-frequent-skill baseline (Frequency), calling into question whether the learned structure is more than a byproduct of dataset priors.

Figure 6: Next-skill prediction accuracy. Transformers leveraging sequential dependencies outperform MLP and GRPO, but all struggle with error accumulation.

Automated vs. Hand-Crafted SKILL.md

Automated SKILL.md artifacts, generated from mining, occasionally outperform hand-written baselines in source-domain next-skill prediction, yet are consistently outperformed by trivial Frequency baselines across all evaluated data sizes. Thus, the current trajectory mining pipeline is not a reliable substitute for even simple statistical priors.

Figure 7: Data-efficiency curve. Mined SKILL.md is competitive with hand-written but not with the most-frequent-skill baseline.

Implications and Limitations

The authors provide a rigorous diagnostic of the state of skill mining for CUAs. Readable, high-purity skill clusters are achievable via action segmentation and clustering; yet, the current method lacks cross-domain generalization and policy improvement. The inability to outperform statistical priors suggests that simply mining frequently observed action patterns is insufficient; segment representations lacking action order, reward functions grounded solely in source-domain skill flows, and the absence of robust transfer diagnostics are all limiting factors. The study establishes that empirical evaluation of mined skills must benchmark against dataset priors and provide ablations to avoid confounding structural insights with class imbalance or prompt adaptation.

Figure 8: Cluster-skill label alignment further demonstrates source-domain discriminability but not actionable transfer effectiveness.

Future Directions

The empirical findings indicate several critical directions for future research:

• Order-Aware Segment Representations: Capturing temporal dependencies within segments (beyond bag-of-actions) is likely required for executable and transferable skills.
• Reward Model Generalization: Reward models must be adapted or jointly trained on both source and target domains for improved cross-domain performance.
• Ablative Diagnostics: Rigorous pipeline ablations are needed to isolate the bottlenecks—segmenter quality, clustering, and reward feedback.
• Integration with Human-Audit Loops: As skills extracted are not robust to domain drift, keeping them human-reviewable prior to deployment is essential for both safety and adaptability.

Conclusion

This study clarifies the limits of current methods for automated SKILL.md generation from GUI interaction logs: while readable and high-purity skills can be mined, they do not provide robust policy improvement or transfer when compared to naive baselines. Orderless segment representations, source-domain-only reward models, and insufficient controls result in no clear functional advantage over trivial priors. The primary contribution is the establishment of rigorous diagnostics and transparency standards for evaluating mined skill libraries, informing future CUA research seeking robust, interpretable, and transferable skill abstractions.