Beyond Domains: Reusing Web Skills via Transferable Interaction Patterns

Abstract: LLM web agents are usually deployed as tool callers: each turn, the model reads a fresh page observation and emits one structured tool action. When every action is a low-level primitive, horizons grow quickly and so do policy-facing LLM completions, dominating latency and cost on benchmarks such as Mind2Web and WebArena. Recent systems therefore wrap repeated interaction fragments as web skills: callable tools built from successful trajectories or induced programs, so one call can replace several primitives. However, prior skill libraries are still triggered mainly by instruction similarity or coarse site metadata, which yields low skill reuse on held-out sites and leaves much of the potential step and token reduction on the table. We present SkillMigrator, an agent that learns reusable web skills and transfers them across sites by matching layout structure rather than specific element references. Each induced skill is stored as a transferable interaction pattern (TIP): the skill paired with a structural sketch of the snapshot at induction time. At test time, SkillMigrator retrieves TIPs by layout similarity and grounds their references on the live page. The rest of the stack is standard: accessibility-snapshot observations with stable references, and fixed tool calling over primitives plus skill invocations. Compared with the state-of-the-art approaches, SkillMigrator reduces the average LLM-action count on successful trajectories by 8-10% across both WebArena and Mind2Web at matched success rate.

• The paper demonstrates that indexing web skills with transferable interaction patterns (TIPs) significantly reduces LLM-action counts while preserving task success rates.
• The methodology combines text similarity with layout-conditioned signals using a convex combination for precise cross-domain skill retrieval.
• The empirical results show an improved skill reuse rate and lower latency on benchmarks compared to existing frameworks like PolySkill.

Transferable Web Skill Reuse: A Detailed Analysis of SkillMigrator

Introduction and Motivation

The trajectory of LLM-based web agents is shaped by the need to translate natural language objectives into sequences of browser-level actions. While this paradigm ensures generality and broad task coverage, it suffers from high latency and cost due to the reliance on repeated LLM calls for each primitive action, especially in long-horizon or multi-step tasks. To amortize this cost, recent frameworks introduce web skills—procedural abstractions or callable macros constructed from successful traces. However, current popular strategies retrieve skills mainly using textual instruction similarity or metadata, sharply limiting skill reuse rates on unseen domains and novel websites.

The paper "Beyond Domains: Reusing Web Skills via Transferable Interaction Patterns" (2606.17645) introduces SkillMigrator, a web agent framework that advances the state-of-the-art by generalizing skill reuse across both unseen websites and domains. Key to this approach is the notion of transferable interaction patterns (TIPs), which are skill records indexed not just by task semantics, but also by layout-conditioned signals reflecting the structure of the inducing web snapshot. This hybrid retrieval mechanism supports robust cross-domain transfer while minimizing compounding error and LLM-action count.

Figure 1: Cross-domain skill reuse motivates SkillMigrator. Despite domain and surface variation, many web tasks reduce to isomorphic interaction patterns (e.g., fill-and-submit forms), which can be encoded as TIPs and reused globally.

Methodology: TIPs and Layout-Conditioned Retrieval

SkillMigrator operates in the conventional web-agent environment, leveraging Playwright-style accessibility snapshots for input and a fixed action space comprising both primitive actions and skill invocations. The innovation lies in:

• Skill Representation as TIPs: Each skill is stored as a tuple $(\iota_k, \sigma_k, \Phi_k, \tau_k)$, where $\iota_k$ is the high-level intent, $\sigma_k$ is an action template mined via structural cluster analysis, $\Phi_k$ is the slot schema with paraphrase pools, and $\tau_k$ is the accessibility-tree skeleton at induction. This abstracted format ensures that skills are robust to surface-level label, DOM, and vocabulary changes.
• Retrieval by Text and Layout Signals: At deployment, SkillMigrator computes a retrieval score via a convex combination of a sentence-embedding-based text similarity term and a normalized tree edit distance between the TIP’s source layout and the live page snapshot. The layout component enables cross-domain transfers where, for example, e-commerce and forum websites realize identical interaction schemas (Figure 1).
• Slot Grounding and Execution: SkillMigrator parses values for each slot using a Hungarian-matching algorithm over paraphrased keys, then binds slots to page elements, ensuring that only actionable, contextually appropriate primitives are emitted. Allowing the fallback to the reactive LLM policy mitigates spurious or partial matches.

A runtime control-flow schematic is provided for clarity (not shown), illustrating gating between skill invocation and fallback regimes dependent on the TIP match score.

Empirical Results

SkillMigrator is evaluated exhaustively on Mind2Web and WebArena under uniform agent settings, compared to ReAct, SkillWeaver, ASI, and PolySkill. Metrics include task success rate (SR), average LLM-action count on successful trajectories $(\bar{\mathcal{N}})$, and skill reuse rate.

• On WebArena, SkillMigrator achieves an 8.5% reduction in LLM-action count (from 5.9 to 5.4, compared to PolySkill) with no significant drop in SR (within $-3.6$ points).
• On Mind2Web, in the most challenging cross-domain (+Update) setting, SkillMigrator attains an LLM-action count of 6.2, outperforming PolySkill's 6.9, with comparable SR.
• The skill reuse rate in cross-domain scenarios climbs to 35% for SkillMigrator, up from PolySkill’s 31%, indicative of non-trivial generalization.

Figure 2: Skill reuse rate per test split for competing strategies, demonstrating SkillMigrator’s sustained reuse on unseen domains.

Figure 3: Average LLM-action count and success rate for WebArena, showcasing SkillMigrator's improved cost/efficiency Pareto frontier.

• Sensitivity analyses substantiate that the layout signal $\mathcal{L}$ is indispensable; ablations show a 6.5 percentage point SR drop when only text similarity is used. Tuning of the retrieval gate ($\beta$) trades off precision and recall with negligible loss in SR.

Figure 4: Performance as a function of the mixing weight $\alpha$ for text/layout retrieval signals, emphasizing the impact of layout conditioning.

Practical and Theoretical Implications

SkillMigrator's demonstration that layout-conditioned skill retrieval supports robust cross-domain transfer has several critical implications:

• Efficiency and Latency: By collapsing repeated primitive action sequences into single skill calls, agents become more tractable for real-world automation, especially outside the training distribution of domains/sites.
• Skill Library Generalization: TIP-based abstraction enables reuse even when task instructions and DOM structures diverge substantially in surface form but are isomorphic in role, yielding smaller, more effective libraries.
• Orthogonality: SkillMigrator can be composed atop prior skill induction pipelines (e.g., ASI, PolySkill), functioning as a generalized retrieval and grounding wrapper, not a competitor, for existing assets.

Limitations and Future Directions

SkillMigrator's gains are predicated on the structural correspondence between source and target pages. On visually similar but semantically distinct pages, or for workflows involving multimodal cues, layout signals are necessary but not sufficient. The next frontiers include:

• Extension to Visual/Multi-modal Cues: Incorporating visual layout descriptors or embedding vision-language foundation models for more robust grounding.
• Rich Action Templates: Generalizing TIPs to cover non-form, non-fill/click actions (e.g., drag-and-drop, modals).
• Automated Template Mining: Improving clustering/mining algorithms for $\iota_k$0, the action template basis, potentially via self-supervised trajectory aggregation.

Conclusion

SkillMigrator establishes that skill induction and invocation, when indexed by transferable interaction patterns encoding both intent and structural context, supports effective web automation across both unseen websites and domains. The explicit layout-conditioned retrieval mechanism is responsible for statistically significant improvements in LLM-call efficiency at no perceptible cost in task success, setting a new direction for cross-domain procedural knowledge reuse in web agent architectures.