VideoCAD: A Large-Scale Video Dataset for Learning UI Interactions and 3D Reasoning from CAD Software

Abstract: Computer-Aided Design (CAD) is a time-consuming and complex process, requiring precise, long-horizon user interactions with intricate 3D interfaces. While recent advances in AI-driven user interface (UI) agents show promise, most existing datasets and methods focus on short, low-complexity tasks in mobile or web applications, failing to capture the demands of professional engineering tools. In this work, we introduce VideoCAD, the first attempt at engineering UI interaction learning for precision tasks. Specifically, VideoCAD is a large-scale synthetic dataset consisting of over 41K annotated video recordings of CAD operations, generated using an automated framework for collecting high-fidelity UI action data from human-made CAD designs. Compared to existing datasets, VideoCAD offers an order of magnitude higher complexity in UI interaction learning for real-world engineering tasks, having up to a 20x longer time horizon than other datasets. We show two important downstream applications of VideoCAD: learning UI interactions from professional precision 3D CAD tools and a visual question-answering (VQA) benchmark designed to evaluate multimodal LLMs' (LLM) spatial reasoning and video understanding abilities. To learn the UI interactions, we propose VideoCADFormer - a state-of-the-art model in learning CAD interactions directly from video, which outperforms multiple behavior cloning baselines. Both VideoCADFormer and the VQA benchmark derived from VideoCAD reveal key challenges in the current state of video-based UI understanding, including the need for precise action grounding, multi-modal and spatial reasoning, and long-horizon dependencies.

An Analytical Overview of the "VideoCAD" Paper

The paper introduces "VideoCAD," a large-scale video dataset aimed at advancing the AI-driven learning of user interface (UI) interactions specific to the domain of Computer-Aided Design (CAD). CAD applications are distinctively intricate, requiring precise and lengthy user manipulation of 3D environments and interfaces, an aspect significantly more demanding than those encountered in typical consumer applications like mobile and web platforms. This paper sets an ambitious goal to encapsulate the complexities of CAD interactions through VideoCAD, which comprises over 41,000 annotated video records of CAD operations.

Dataset Characteristics and Contributions

VideoCAD represents a substantial leap in terms of UI interaction complexity and time horizons compared to pre-existing datasets. The paper highlights VideoCAD's ability to encapsulate tasks up to 20 times longer in duration, with a focus on real-world professional settings for CAD tools. This rigorous and high-fidelity dataset is synthesized from human-authored designs using Onshape, a popular browser-based CAD platform. As such, VideoCAD provides a foundation for modeling the nuanced English-user interactions requisite for precision CAD tasks.

The dataset is posited to support numerous downstream applications, notably in learning CAD UI interactions and employing visual question-answering (VQA) benchmarks to assess multimodal LLMs' (LLM) spatial reasoning capabilities. For UI interaction learning, the research introduces VideoCADFormer, a novel transformer model that demonstrates superiority over other behavior cloning baselines.

Empirical Results and Theoretical Insights

VideoCADFormer outperforms existing models through its refined capacity for processing detailed UI interactions and long-term dependencies in CAD modeling. It achieves significant improvements, evidencing up to 20% better performance in understanding complex CAD tasks compared to counterparts. The strong numerical results underscore the need for precision-grounding and multi-modal reasoning critical in UI understanding.

A case study utilizing the VideoCAD VQA benchmark reveals existing gaps in LLMs' spatial reasoning and temporal comprehension. LLMs are shown to struggle significantly across tasks demanding high-level 3D understanding, such as extrusion depth comparison and frame sequencing, indicating limitations in their current architectural designs and training regimens.

Implications and Future Directions

The implications of VideoCAD extend beyond CAD, serving as a benchmark for broader AI-driven UI navigation, software automation, and CAD generation. The paper effectively bridges the gap between machine learning subfields such as computer vision and reinforcement learning with practical CAD applications.

Additionally, the large-scale annotated dataset introduces the potential for developing more sophisticated agents capable of executing complex tasks in engineering software environments. Future developments could involve enriching the dataset with human interaction data, expanding to more CAD features, and incorporating additional CAD platforms to enhance generalization capabilities.

Conclusion

The VideoCAD paper presents a robust contribution to the field of AI and CAD interactions, setting a foundational benchmark for advancing research in UI interface automation and CAD modeling. By offering a comprehensive and open-source dataset, VideoCAD fosters opportunities for significant breakthroughs across various domains of machine learning, providing insights into the challenges and future potential for AI applications in complex, real-world settings.