Scaling laws for 3D VLM pretraining and downstream robot control

Investigate and characterize the scaling laws that relate SPEAR-1’s downstream robot control performance to the quantity and quality of the 3D visual question answering pretraining data used to train SPEAR-VLM, in order to understand how data scale and data quality affect task performance.

Background

SPEAR-1 is built on SPEAR-VLM, a 3D-aware vision-LLM trained on non-robotic 2D images enriched with 3D annotations. The authors show that this pretraining improves downstream control but note that their experiments were constrained in data scale and diversity.

They explicitly state that the relationship between downstream performance and the quantity and quality of 3D pretraining data is not well understood, leaving open how increasing or curating such data would impact generalization and control accuracy.