OWA Toolkit: Unified Desktop Data Logging

• OWA Toolkit is a unified multimodal framework that synchronizes diverse desktop events at 60Hz using a standardized ocap recorder.
• It employs dual-layer encoding and H.265 compression to reduce data size dramatically, achieving up to 152× compression efficiency.
• Integrated within the D2E pipeline, the toolkit enables robust transfer learning, with success rates of 96.6% on LIBERO manipulation and 83.3% on CANVAS navigation.

The OWA Toolkit is a unified multimodal desktop data logging and compression framework that serves as the foundational data-acquisition layer in the D2E (Desktop to Embodied AI) pipeline. Its primary function is to collect, synchronize, and efficiently encode large-scale human-computer interaction episodes—including high-frequency screen captures, keyboard events, and mouse movements—into a standardized, compact, and schema-rich format suitable for large-scale internet data-driven transfer learning. The technical innovations of the toolkit address bandwidth, scalability, and downstream modeling requirements, establishing the substrate for cross-domain transfer from digital desktop interactions to robotics embodied AI tasks (Choi et al., 7 Oct 2025).

1. Unified Desktop Interaction Representation

The central feature of the OWA Toolkit is its universal schema for desktop activity capture. The ocap (Omnimodal CAPture) recorder collects episodic data in real time, synchronizing multiple modalities at 60 Hz (i.e., full HD/QHD video, audio, mouse, and keyboard streams) by interfacing directly with Windows APIs and using GStreamer for media processing.

All event modalities are encoded into a consolidated token stream using the specialized OWAMcap format. Built atop MCAP (an industry-standard container), OWAMcap introduces enhanced standardization with fine-grained message schemas. Each desktop event—be it a keypress, mouse movement, or video frame—is represented as a discrete token:

<EVENT_START>{TYPE}{TIMESTAMP}{DETAIL}</EVENT_END>

where {TYPE} is a modality class (e.g., SCREEN, KEYBOARD, MOUSE) and {DETAIL} is modality-specific. This uniform encoding allows diverse desktop actions to be ingested as a coherent, temporally aligned sequence for downstream transformer-based models, facilitating generalist event prediction and pseudo-labeling.

2. Compression and Storage Efficiency

The OWA Toolkit dramatically reduces storage requirements for large-scale behavioral datasets through dual-layer encoding:

• All synchronized event metadata is serialized into the MCAP container.
• Video frames and other heavy media are externally referenced and encoded with H.265/HEVC, yielding extreme compression ratios.

Empirically, converting the VPT desktop activity dataset from the raw JSONL representation (1.06 TiB) into OWAMcap format resulted in a reduction to 7.12 GiB, a 152× compression. The high efficiency is attributed to optimized video codec utilization (H.265 yields ≈217× compression for raw frames) and sharply reduced event encoding overhead. This scale of reduction is critical for internet-scale learning and enables transfer pipelines to operate without prohibitive I/O or memory bottlenecks.

3. Integration within the D2E Pipeline

The OWA Toolkit operates as the input data backbone to the entire D2E system (Choi et al., 7 Oct 2025). Over 335 hours of human gameplay episodes across 31 games are captured in robust alignment, enabling standardized access for both pretraining and pseudo-labeling. Subsequent modules—such as the Generalist-IDM inverse dynamics model—are pretrained using tokenized event sequences from OWA-collected data, supporting zero-shot generalization across new desktop domains.

Pseudo-labeled online gameplay data, generated via timestamp-based event prediction, is further appended to the human demonstration corpus, expanding training coverage to over 1,300 hours and facilitating robust transfer learning.

4. Technical Contributions: Schema, Timing, and Data Pipeline

A key technical innovation is the event tokenization and temporal alignment mechanism. By precisely synchronizing all inputs, the toolkit enables non-equidistant, multi-rate actions (such as mouse events or variable video frame rates) to be modeled without artificial resampling.

The generic token format:

<EVENT_START>{TYPE}{TIMESTAMP}{DETAIL}</EVENT_END>

permits flexible parsing for neural architectures, particularly decoder-only transformers operating over multimodal time series.

An additional contribution is the adaptive batch decoding strategy. This modification in the file I/O pipeline aggregates individual frame reads into coalesced batches, reducing random disk access and increasing throughput. Evaluation tables in the paper demonstrate higher efficiency in downstream model training when using the OWA-mcap storage format compared to prior JSONL or raw buffer arrangements.

5. Enabling Temporal Modeling and Event Prediction

The high-fidelity, lossless event synchronization established by the OWA Toolkit is critical for the next-event prediction (NEP) objective central to Generalist-IDM. This NEP formulation: $$\mathcal{L}_{\text{NEP}-\tau} = - \mathbb{E}_{o_{1:T},\,a_{1:T}\sim\mathcal{D}} \Bigg[ \sum_{t=1}^{T} \log P_{\theta} \left(a_t \mid o_{1:\min(t+\tau,T)},\,a_{1:t-1}\right) \Bigg ]$$ allows actions to be predicted with a temporal offset, exploiting natural rather than artificially fixed event rates. The dense and accurately timestamped multimodal data produced by the OWA Toolkit enables this form of conditioning, which is essential for large-scale pseudo-labeling and robust cross-domain generalization.

6. Transfer Performance in Embodied AI Domains

The downstream impact of the OWA Toolkit on physical task transfer is quantified by its contribution to LIBERO manipulation and CANVAS navigation benchmarks. Desktop-pretrained representations—enabled by OWA’s standardized and compressed corpus—achieve:

• 96.6% success rate on LIBERO manipulation.
• 83.3% success rate on CANVAS navigation, with marked improvements under ambiguous or misleading instruction scenarios.

This finding confirms that sensorimotor primitives embedded in digital desktop environments possess sufficient invariance for transfer across the virtual-physical boundary, validating the practical utility of desktop pretraining for embodied robotics.

7. Significance and Implications

The OWA Toolkit is not a generic data logger but a deliberate software and data engineering framework tailored to the requirements of cross-domain, internet-scale behavior modeling. Its core strengths—universal schema, aggressive compression, precise multimodal temporal alignment, and pipeline efficiency—establish the infrastructure necessary for the D2E paradigm. This approach supports standardized data acquisition for vision-action models and underlies the advances in transfer learning to robotics manipulation and navigation tasks, as supported by the high empirical success rates on LIBERO and CANVAS.