VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos (2510.19488v1)

Abstract: Training computer-use agents requires massive amounts of GUI interaction data, but manually annotating action trajectories at scale is prohibitively expensive. We present VideoAgentTrek, a scalable pipeline that automatically mines training data from publicly available screen-recorded videos at web scale, eliminating the need for manual annotation. Our approach addresses a key challenge: raw videos contain implicit demonstrations but lack explicit action labels. To solve this, we develop Video2Action, an inverse dynamics module (IDM) with two components: (1) a video grounding model that detects and localizes GUI actions with precise temporal boundaries and context, and (2) an action-content recognizer that extracts structured parameters like click coordinates and typed text with high fidelity. Applied to 39,000 YouTube tutorial videos, our pipeline generates 1.52 million interaction steps automatically. We leverage this data through continued pretraining followed by supervised fine-tuning. On OSWorld-Verified, our approach improves task success rates from 9.3% (SFT-only baseline) to 15.8%, a 70% relative improvement. On AgentNetBench, step accuracy increases from 64.1% to 69.3%. Our results demonstrate that passive internet videos can be transformed into high-quality supervision for computer-use agents, providing a scalable alternative to expensive manual annotation.

• The paper introduces a scalable pretraining method for computer-use agents by extracting structured GUI actions from unlabeled tutorial videos.
• It employs a two-stage Video2Action module that detects and parameterizes GUI events with millisecond precision, validated on 1.52M interaction steps.
• Experimental results show substantial performance gains on both offline and online benchmarks, emphasizing data scaling and long-horizon planning benefits.

VideoAgentTrek: Scalable Computer-Use Agent Pretraining from Unlabeled Videos

Introduction

VideoAgentTrek introduces a scalable methodology for pretraining computer-use agents by mining structured GUI interaction trajectories from large-scale, unlabeled screen-recorded tutorial videos. The approach addresses the prohibitive cost and limited diversity of manual annotation by leveraging the implicit supervision present in publicly available internet videos. The core innovation is the Video2Action inverse dynamics module, which automatically detects GUI actions and extracts their parameters, enabling the conversion of raw video into training-ready $$(\text{screenshot}, \text{action}, \text{parameters})$$ tuples. This pipeline is validated on 39,000 YouTube videos, yielding 1.52 million interaction steps and demonstrating substantial improvements in agent performance on established benchmarks.

Figure 1: Overview of VideoAgentTrek, illustrating the pipeline from video collection and filtering, through inverse dynamics-based action extraction, to agent training.

Video Collection and Preprocessing

The video collection strategy exploits channel coherence on YouTube, expanding from seed keywords to entire channels when quality thresholds are met. This maximizes recall while maintaining manageable precision, as subsequent filtering stages ensure data quality. The ScreenFilter module, built on YOLOv8x, efficiently identifies GUI interaction segments by detecting cursor presence, retaining only those video portions with sustained cursor activity. This process yields a corpus of 7,377 hours of verified GUI interactions from an initial pool of 10,000 hours.

Figure 2: Video candidate auto-discovery pipeline, showing iterative expansion from seed queries to high-quality channels and GUI-containing videos.

Video2Action: Inverse Dynamics Extraction

Video2Action is a two-stage inverse dynamics module designed to recover structured action supervision from unlabeled GUI videos:

1. Dense Event Detection: A VLM (Qwen2.5-VL-7B-Instruct) is fine-tuned to perform multi-class temporal event detection, localizing GUI actions (clicks, drags, scrolls, typing) with millisecond precision. Training utilizes OpenCUA logs for prompt-free supervision, mapping raw interaction events to typed spans with start and end timestamps.
2. Action Parameterization: For each detected segment, a recognizer predicts both the action type and its parameters (e.g., click coordinates, typed text). Training data is derived from temporally aligned OpenCUA logs, and the model is evaluated via blinded manual review on in-the-wild videos.
3. Inner Monologue Generation: To enhance planning and credit assignment, a brief rationale is generated for each action using GPT-5 Medium, conditioned on action details, keyframes, and ASR transcripts. This yields ReAct-style tuples for training.

   Figure 3: Overview of Video2Action, detailing the detection, parameterization, and rationale generation steps for each GUI action.

Agent Training Methodology

The agent training protocol consists of two stages:

• Stage 1 (Pretraining): The agent is pretrained for one epoch over 26B tokens from VideoAgentTrek trajectories, formatted as interleaved vision-text sequences. Images serve as conditioning context, with loss masked to textual outputs.
• Stage 2 (Supervised Fine-Tuning): The model is further trained for 8B tokens on curated, human-annotated trajectories, reformatted into a chat template for instruction-style SFT.

Training leverages Megatron-LM on large-scale GPU clusters (32–64× H100), with batch sizes of 256–512 and wall-clock times of 15–60 hours per stage.

Figure 4: Computer Use Agent Training Data samples for Stage-1 and Stage-2, illustrating the transition from video-mined to human-annotated supervision.

Figure 5: CUA Stage-1 Training: training loss curve and configuration details.

Figure 6: CUA Stage-2 Training: training loss curve and configuration details.

Experimental Results

Benchmark Performance

• AgentNetBench (Offline): VideoAgentTrek pretraining increases step success rate from 64.1% (SFT-only) to 69.3%, a 5.2-point absolute gain and 30.8-point improvement over the base model.
• OSWorld-Verified (Online): Task success rate improves from 9.3% (SFT-only) to 15.8% with VideoAgentTrek, a 70% relative improvement and more than triple the base model's performance.

  Figure 7: Experimental results on OSWorld-Verified and AgentNetBench, showing significant improvements from video pretraining and test-time scaling.

Data Scaling and Long-Horizon Planning

Performance scales monotonically with increased pretraining data. Models exposed to longer video trajectories demonstrate superior long-horizon planning, with task success rates on OSWorld-Verified increasing as the step budget grows (from 14.13% at 20 steps to 15.78% at 50 steps). SFT-only models show no such improvement, indicating that video pretraining instills the ability to leverage extended planning horizons and recover from intermediate errors.

Video2Action Module Evaluation

• Dense Event Detection: On a held-out test set (20,282 events), the detector achieves 0.88 precision and 0.70 recall (F1=0.78), with pointer-centric actions (click, scroll) most reliably localized.
• Action Parameterization: Manual review yields overall accuracy of 65.8%, with click and scroll actions exceeding 70% accuracy. Typing and drag/press actions are less reliable due to subtle visual cues and OCR noise, but remain sufficient for trajectory construction.

  Figure 8: Example training sample for the dense event detector, illustrating temporal event annotation.

  

  Figure 9: Dense event detector training loss and configuration.

  

  Figure 10: Example training sample for the action parameterization model, showing input-output alignment.

Implications and Future Directions

VideoAgentTrek demonstrates that passive internet videos can be transformed into high-quality, diverse supervision for computer-use agents, circumventing the cost and coverage limitations of manual annotation. The pipeline's scalability and generalization across platforms (Windows, macOS, web) suggest broad applicability for GUI automation, RPA, and interactive agent research. The open-source release of ScreenFilter and Video2Action enables reproducibility and further exploration.

Future work may focus on:

• Enhancing action parameterization accuracy, particularly for subtle or compound actions.
• Extending the pipeline to mobile and non-desktop environments.
• Integrating multimodal reasoning for richer intent and context extraction.
• Exploring self-supervised or reinforcement learning strategies atop mined trajectories.

Conclusion

VideoAgentTrek establishes a robust framework for mining agentic supervision from unlabeled screen recordings, achieving strong empirical gains in computer-use agent performance. The approach validates the utility of inverse dynamics for action extraction and demonstrates the benefits of large-scale, diverse pretraining data. This work provides a scalable, cost-effective alternative to manual annotation and lays the groundwork for future advances in general-purpose GUI agents.