Robot-Powered Data Flywheels: Deploying Robots in the Wild for Continual Data Collection and Foundation Model Adaptation (2511.19647v1)

Abstract: Foundation models (FM) have unlocked powerful zero-shot capabilities in vision and language, yet their reliance on internet pretraining data leaves them brittle in unstructured, real-world settings. The messy, real-world data encountered during deployment (e.g. occluded or multilingual text) remains massively underrepresented in existing corpora. Robots, as embodied agents, are uniquely positioned to close this gap: they can act in physical environments to collect large-scale, real-world data that enriches FM training with precisely the examples current models lack. We introduce the Robot-Powered Data Flywheel, a framework that transforms robots from FM consumers into data generators. By deploying robots equipped with FMs in the wild, we enable a virtuous cycle: robots perform useful tasks while collecting real-world data that improves both domain-specific adaptation and domain-adjacent generalization. We instantiate this framework with Scanford, a mobile manipulator deployed in the East Asia Library for 2 weeks. Scanford autonomously scans shelves, identifies books using a vision-LLM (VLM), and leverages the library catalog to label images without human annotation. This deployment both aids librarians and produces a dataset to finetune the underlying VLM, improving performance on the domain-specific in-the-wild library setting and on domain-adjacent multilingual OCR benchmarks. Using data collected from 2103 shelves, Scanford improves VLM performance on book identification from 32.0% to 71.8% and boosts domain-adjacent multilingual OCR from 24.8% to 46.6% (English) and 30.8% to 38.0% (Chinese), while saving an ~18.7 hrs of human time. These results highlight how robot-powered data flywheels can both reduce human effort in real deployments and unlock new pathways for continually adapting FMs to the messiness of reality. More details are at: https://scanford-robot.github.io

• The paper introduces a robot-powered data flywheel framework that uses mobile robots to autonomously capture domain-specific data for continual foundation model adaptation.
• It details an integrated approach combining vision-language models with LiDAR navigation and catalog-augmented curation to address real-world challenges in perception.
• Empirical results show that even brief robot deployments can boost book identification accuracy by nearly 40% while significantly reducing manual librarian effort.

Robot-Powered Data Flywheels for Continual Foundation Model Adaptation

Problem Motivation and Framework

The substantial progress in foundation models (FMs) for vision-language tasks has not closed the generalization gap in real-world, unstructured environments due to a lack of representative pretraining data. Most FMs are developed using internet-scale, clean image corpora, resulting in brittleness when deployed for challenging perception tasks such as reading faded book labels or recognizing multilingual titles under occlusion and variable lighting. The paper presents the "Robot-Powered Data Flywheel" (RPDF) framework, repurposing mobile robots as continual data generators that both assist with physical tasks and autonomously gather hard, in-the-wild data distributions, completing a closed adaptation loop with foundation models.

The RPDF paradigm is instantiated as the Library Data Flywheel (LDF), wherein a mobile manipulator is deployed in the East Asia Library. LDF autonomously scans bookshelves, leveraging a vision-LLM (VLM) for recognition. The key innovation is closing the loop: LDF uses a VLM for in-situ book identification, curates the resulting image-label pairs via the library’s catalog, and fine-tunes the VLM on this deployment-collected dataset. Each cycle accumulates domain-specific, label-rich data that simultaneously enhances in-domain task performance and the VLM's generalization to domain-adjacent, multilingual OCR scenarios.

Figure 1: LDF operates in a challenging library environment, capturing images, labeling via VLM and catalog, and using this data to iteratively fine-tune the VLM.

Challenges in In-the-Wild Robotic Deployment

Deploying LDF in the East Asia Library surfaces diverse robotic and perception challenges. The physical library context exhibits widely varying shelf geometries, nonstandard shelf heights, and complex lighting, which regularly confound standard SLAM and odometry. Labels on book spines are exposed to occlusions, damage, fading, and varying print scripts (Chinese, Japanese, Korean), directly limiting out-of-the-box VLM performance.

Figure 1: Illustration of real-world challenges, including nonuniform shelf morphology and numerous label degradation modes, which drive the need for continuous adaptation.

LDF addresses navigation drift by fitting planes to LiDAR point clouds, enabling realignment relative to aisle constraints. For perception, LDF combines VLM-based recognition with a retrieval-augmented pipeline, leveraging catalog-provided call number ranges to restrict candidate label sets, thus maximizing robustness while automating the curation pipeline. Data curation uses string similarity on the catalog-augmented VLM outputs, enforcing both content and order constraints for high-reliability annotation without manual labeling.

Figure 1: Shelving irregularities and degraded labels necessitate adaptive LiDAR-based navigation and advanced label curation.

Quantitative Model Improvements

Central to the RPDF hypothesis is that deployment-collected data sharply boosts both domain-specific and orthogonal generalization capabilities of the foundation model. Empirical results from two weeks of LDF deployment are stark: fine-tuning Qwen2.5-VL (7B) with the curated shelf data yielded a book identification accuracy increase from 32.4% to 71.8% (absolute gain: 39.4%) on a held-out library benchmark.

Figure 2: LDF-collected data fine-tuning dramatically boosts both library book identification and external multilingual OCR performance.

Further, the same fine-tuning regime produced improvements in challenging OCR benchmarks: English OCR accuracy on hard subsets rose from 24.8% to 46.6%, and Chinese—from 30.8% to 38.0%. Notably, the improvement curve saturates quickly, with most gains realized within the first 1.5 hours of data collection ($\sim$1,352 images), indicating that even modest deployments rapidly address previously underrepresented data regimes.

LDF's adaptive loop is not just beneficial in a vacuum but directly reduces operational burden: across 2,103 shelves, an estimated 18.7 hours of librarian effort were saved, with negligible human interventions (26 events over 10 days, each under 5 minutes).

Figure 3: LDF's autonomous deployment statistics, showing saved human labor and the operational stability (few, short human interventions).

Theoretical and Practical Implications

This work demonstrates that robot-powered, in-the-wild data acquisition can systematically mitigate undercoverage in FM pretraining distributions. Practically, the closed-loop RPDF system fuses robotic autonomy with FM adaptation, yielding rapid performance gains both in the task’s ambient domain and in previously problematic, generalizable perception axes (e.g., noisy multilingual OCR).

Theoretically, the RPDF instantiates a continual adaptation cycle, in which robotic action and FM updates co-adapt, iteratively sampling the open-world, hard negative regions of the real-world data manifold. This raises the prospect of scaling FM capabilities along frontiers that are out-of-reach for static internet collection regimes—anywhere physically embodied agents can be instrumented for data harvesting.

Additionally, the deployment framework pinpoints relevant design and engineering insights:

• Zone of Proximal Development: Task settings must balance tractability (enabling robot model self-supervision) with novelty (collecting data distinctly unavailable online).
• Curations Pipelines: Leveraging structured, domain-side knowledge (e.g., catalogs) is essential for scalable, high-quality automated annotation.

Future Directions and Limitations

The demonstrated LDF framework remains focused on vision-language adaptation, requiring moderate system-level engineering and currently falling short of full robustness. Extending the methodology to more expressive foundation model classes, such as Vision-Language-Action models (VLAs) and LLM-planned robotics (Intelligence et al., 22 Apr 2025), is an open avenue. Such extensions could further minimize hand-engineering and maximize the set of adaptation-relevant deployment domains.

Integrating robotic deployment data as an augmentation for global FM pretraining, rather than only for post-hoc fine-tuning, may yield broader and more stable advances. There is also scope for further automating the curation pipeline and for building open-world evaluation datasets specifically curated via RPDF cycles.

Conclusion

This work provides a rigorous demonstration that robot-powered data flywheels can close critical real-world representation gaps in vision-language foundation models, concurrently performing valuable deployments and bootstrapping FM robustness. Even brief robotic deployments in underrepresented settings yield substantial improvements, underscoring the promise of coupling autonomous agents and continual FM adaptation for scalable, physical-world generalization.

---

Reference: "Robot-Powered Data Flywheels: Deploying Robots in the Wild for Continual Data Collection and Foundation Model Adaptation" (2511.19647)