DPO Learning with LLMs-Judge Signal for Computer Use Agents (2506.03095v1)

Abstract: Computer use agents (CUA) are systems that automatically interact with graphical user interfaces (GUIs) to complete tasks. CUA have made significant progress with the advent of large vision-LLMs (VLMs). However, these agents typically rely on cloud-based inference with substantial compute demands, raising critical privacy and scalability concerns, especially when operating on personal devices. In this work, we take a step toward privacy-preserving and resource-efficient agents by developing a lightweight vision-LLM that runs entirely on local machines. To train this compact agent, we introduce an LLM-as-Judge framework that automatically evaluates and filters synthetic interaction trajectories, producing high-quality data for reinforcement learning without human annotation. Experiments on the OS-World benchmark demonstrate that our fine-tuned local model outperforms existing baselines, highlighting a promising path toward private, efficient, and generalizable GUI agents.

• The paper demonstrates a novel LLM-as-Judge framework that autonomously filters synthetic interaction trajectories, eliminating the need for human annotation.
• The methodology leverages local LLMs to ensure privacy and resource efficiency, significantly outperforming baselines on the OS-World benchmark.
• Empirical results reveal that optimal performance is achieved with shorter action sequences, underscoring the model’s scalability and robust task completion.

Overview of DPO Learning with LLMs-Judge Signal for Computer Use Agents

The paper "DPO Learning with LLMs-Judge Signal for Computer Use Agents" presents a significant contribution to the development of privacy-preserving and resource-efficient computer use agents (CUAs). These agents are designed to automate user interactions with graphical user interfaces (GUIs) through a lightweight vision-LLM that operates entirely on local machines. This stands in contrast to existing systems which primarily rely on cloud-based inference, posing issues related to privacy, latency, and resource demands.

The authors introduce an innovative LLM-as-Judge framework which utilizes LLMs to filter and evaluate synthetic interaction trajectories automatically. This process generates high-quality data for reinforcement learning without the need for human annotation, addressing the challenge of data acquisition for training these lightweight agents.

Key Insights and Results

The innovations proposed in this work focus on enhancing the usability and robustness of CUAs:

1. Privacy and Efficiency: By enabling the model to operate locally, privacy concerns are mitigated since no data needs to be transmitted over networks. This also makes the system suitable for deployment in bandwidth-limited or high-security environments.
2. LLM-as-Judge Framework: The framework leverages GPT-4o to score and rank generated responses, forming preference pairs for Direct Preference Optimization (DPO) training. This framework bypasses the costly process of manual data annotation and creates a scalable, efficient training protocol.
3. Empirical Performance: The fine-tuned local model was evaluated using the OS-World benchmark and was shown to outperform existing baselines significantly. Notably, it demonstrated superior task completion capabilities despite resource constraints.
4. Experimentation and Methodologies: The paper employed two configurations (15-step and 50-step) for testing agent actions, revealing that the DPO training models consistently outperformed baselines, particularly under the 15-step configuration. The experiments underscore that shorter, more efficient action sequences can lead to reliable task completion without sacrificing performance.

Implications and Future Directions

This research provides a robust foundation for future developments in the field of privacy-preserving GUI agents. The local-first approach indeed points toward more secure and efficient interactions with personal computing environments. Key implications for future work include:

• Refinement of Scoring Systems: Further enhancements to the LLM-as-Judge strategy could refine the quality of preference scoring, potentially leveraging multi-agent systems for enhanced decision-making.
• Broadened Application Scope: As the framework becomes more refined, it could be extended beyond traditional GUIs to include various immersive and next-generation interfaces, including AR and VR environments.
• Optimization for Extended Interactions: Although the model performs efficiently within short-action windows, optimizing for extended interaction scenarios remains a challenge. Addressing action sequencing and error recovery in long-horizon tasks could further extend usability.
• Future AI Collaboration: The integration and harmonization of LLMs with computer use agents pave the way for developing autonomous systems that require minimal human oversight while still maintaining high levels of accuracy and adaptability.

In conclusion, the approach presented in this paper marks significant progress toward self-sufficient and privacy-aware GUI agents. By circumventing some of the intrinsic burdens of data privacy and resource allocation, this work not only advances theoretical understanding but also lays pragmatic groundwork for deploying intelligent agents across varied computational ecosystems.