Learn from Weaknesses: Automated Domain Specialization for Small Computer-Use Agents

Abstract: Computer-use agents (CUAs) have recently made substantial progress, but deploying a separate large expert for each software domain remains expensive. Small open computer-use agents are more practical specialization targets, but they remain substantially weaker and exhibit uneven domain-specific failures. A straightforward remedy is to synthesize large-scale training data for the target domain, yet we find that this naive approach yields only marginal improvements. Building on this observation, we introduce LearnWeak, an annotation-free specialization framework for small computer-use agents that uses a stronger reference agent to identify the student's weaknesses in the target domain, synthesize targeted tasks, and construct supervision automatically. LearnWeak further introduces an error-aware specialization objective that disentangles planning and execution errors, enabling more behaviorally precise updates than broad uniform supervision. On OSWorld, LearnWeak achieves average gains of 11.6 and 11.1 percentage points over EvoCUA-8B and OpenCUA-7B, respectively, across eight domains. We also validate that our student-aware dataset generation and training approaches outperform existing autonomous trajectory generation and training baselines. Our work highlights the importance of student awareness in both data synthesis and agent training, pointing toward a more principled and efficient path for specializing small computer-use agents in diverse domains.

• The paper presents an automated pipeline (LearnWeak) that identifies agent weaknesses and generates targeted tasks to improve domain-specific performance.
• The methodology employs a dual-stage process with weakness-aware data generation and error-aware preference optimization to refine agent behavior.
• Experimental results show significant gains across multiple domains, with improvements of up to 11.6 percentage points over baseline methods.

LearnWeak: Automated Domain Specialization for Small Computer-Use Agents

Motivation and Problem Statement

The proliferation of computer-use agents (CUAs) capable of interacting directly with graphical user interfaces (GUIs) has inaugurated a paradigm shift in end-to-end software automation. However, while proprietary frontier agents (e.g., Claude Sonnet, GPT-5.x, Kimi K2.x) achieve high performance, deploying large experts for each domain is prohibitively costly for enterprises and edge devices. Small open CUAs (e.g., EvoCUA-8B, OpenCUA-7B) are therefore preferable for real-world use-cases, but they suffer from both performance deficits and pronounced domain-specific failures—uneven generalization, catastrophic forgetting, and operational brittleness.

Prior attempts to bridge this gap via synthetic data expansion and naive finetuning have yielded marginal improvements. The paper "Learn from Weaknesses: Automated Domain Specialization for Small Computer-Use Agents" (2605.28775) tackles the specialization challenge by introducing a fully automated, annotation-free pipeline, LearnWeak, designed to efficiently identify student weaknesses, synthesize targeted tasks, and derive error-aware supervision.

Figure 1: Conceptual illustration of LearnWeak and performance gains after domain specialization, showing consistent improvements of the small student across target software domains.

LearnWeak Framework: Pipeline Design and Components

LearnWeak encapsulates domain specialization as a two-stage process: weakness-aware data generation (LearnWeak-GEN), followed by agent specialization via error-aware preference optimization (LearnWeak-DPO).

1. Weakness-aware Data Generation (LearnWeak-GEN)

LearnWeak-GEN begins with a seed set of domain-specific tasks executed by both teacher and student agents. It then:

• Compares teacher and student trajectories using an automatic verifier, isolating cases where the teacher succeeds and the student fails.
• Summarizes these failures into recurring weakness reports that capture systematic deficiencies (e.g., misapplied operations, localization errors).
• Exploits screenshot-driven query synthesis: diverse, informative screenshots are extracted from executed trajectories using embedding clustering and VLM-based reranking. New queries are generated by conditioning on both the weakness reports and screenshot context, balancing targeted coverage and open-ended exploration.
• This cyclical, iterative data-generation loop progressively refines the dataset toward the student’s unresolved failures, maximizing informativeness per sample.

  Figure 2: Overview of LearnWeak framework. LearnWeak-GEN iteratively constructs domain data by comparing teacher and student responses, summarizing student weaknesses, and generating new tasks conditioned on weakness reports and representative screenshots. LearnWeak-DPO then specializes the student with step-wise preference supervision and error-aware optimization.

2. Error-aware Preference Optimization (LearnWeak-DPO)

LearnWeak-DPO translates teacher-student discrepancies into step-level preference pairs, focusing exclusively on steps where the student’s tool execution differs from the teacher. It disentangles failures into:

• Planning-level errors: discrepancies in tool/function selection.
• Execution-level errors: parameterization mistakes or imprecise localizations.

A token-level masking strategy restricts loss computation to the behaviorally relevant spans—planning or execution—depending on the failure type. The policy update is applied modularly via domain-specific LoRA adaptations layered over the base student model, ensuring scalability and avoiding catastrophic interference.

Experimental Results: Empirical Evaluation

Comprehensive evaluation on OSWorld across eight domains (Gimp, Calc, Impress, Writer, OS, Thunderbird, VLC, VSCode) demonstrates strong numerical improvements:

• LearnWeak specialization elevates EvoCUA-8B from 50.69% to 62.24% mean success rate (+11.6), and OpenCUA-7B from 37.65% to 48.72% (+11.1).
• Gains are consistent across domains, with EvoCUA-8B surpassing its teacher (EvoCUA-32B) on Gimp, Thunderbird, and VSCode—a bold claim against standard imitation-based specialization.
• Data-generation baselines—human-validated, boosting, and naive zero-shot syntheses—are outperformed by LearnWeak by over 5.5 points on average under matched trajectory budgets.

Figure 3: EvoCUA-8B specialization.

Dataset Construction and Training Objective Analysis

LearnWeak-GEN outperforms alternative data-generation strategies due to its explicit student-awareness. Ablations confirm:

• Using weakness reports derived from other (non-target) student models degrades performance, validating the importance of model-specific error mining.
• The benefit of iterative generation depends on weak-report conditioning. Without it, additional rounds saturate or even decrease performance.
• Teacher strength only matters up to the point where successful trajectories serve as actionable supervision for student failures.

Token-level masking in LearnWeak-DPO consistently outperforms conventional SFT and DPO objectives. Standard full-response optimization yields modest gains, whereas error-aware masking delivers domain improvements of up to 9.6 points on average.

Qualitative Case Studies

In-depth before–after analyses on OSWorld benchmark tasks illustrate transformed agentic behaviors:

• In Calc, specialization corrects partial-range sorting failures, shifting selections from B1:D19 (baseline) to A1:D19 (specialized).
• For complex formula construction, the specialist abandons brittle VLOOKUP logic for robust, row-wise references.
• Impress case studies show improved recovery from interaction mode errors (e.g., exiting text-edit mode before completing object moves), as well as enhanced coverage in multistep tasks (e.g., recoloring all textboxes versus only obvious ones).

Figure 4: Case Study #1 (Domain: Libreoffice Calc)

Figure 5: Case Study #2 (Domain: Libreoffice Calc)

Figure 6: Case Study #3 (Domain: Libreoffice Impress)

Figure 7: Case Study #4 (Domain: Libreoffice Impress)

Practical and Theoretical Implications

Practically, LearnWeak introduces a scalable, annotation-free path for deploying domain-specialized small CUAs in privacy-sensitive and resource-constrained environments. The method’s modularity facilitates seamless expansion to multi-domain libraries via LoRA adapters, enabling context-sensitive specialization at inference.

Theoretically, the paper advances the notion that maximally informative supervision for specialization is not generic domain coverage or brute-force data scaling, but corrective feedback precisely targeting the agent’s persistent weaknesses. This aligns with broader trends in efficient preference-based alignment, error-driven curriculum learning, and capability gap-resistant training in RL and imitation learning.

Future Directions

• Systematic empirical studies of multi-adapter routing and domain library expansion.
• Deeper integration with token-level preference optimization schemes (e.g., TDPO, SparsePO, TGDPO).
• Extension to GUI agents in continually evolving environments, leveraging real-time self-improvement loops.
• Cross-modal adaptation to agents operating on mobile platforms and web-based applications.

Conclusion

The LearnWeak framework establishes domain-specialization protocols for small CUAs by automating the identification and repair of student-specific failures. The pipeline’s student-aware data synthesis and error-masked preference optimization yield robust improvements across multiple software domains, outperforming alternative autonomous generators and human-annotated baselines. The work underscores that targeted specialization, not indiscriminate data expansion, is essential for narrowing the performance gap between small open CUAs and proprietary frontier models, and provides a foundation for practical deployment and scalable expansion in mixed-domain agent libraries.