AutoData: Automating Data Lifecycle
- AutoData is a family of automated techniques that convert raw, heterogeneous inputs into AI-ready datasets while ensuring traceability and interoperability.
- In both academic and automotive settings, AutoData streamlines dataset discovery, telemetry standardization, and annotation through multi-stage pipelines.
- Its approaches leverage rule-based routing, LLM planning, and selective human oversight to achieve measurable improvements in efficiency and model performance.
AutoData is a recurring label in recent arXiv literature for automated or semi-automated handling of data across several layers of the data lifecycle. In these works, raw tabular inputs are converted into AI-ready datasets, newly released datasets are discovered directly from papers, open-web sources are turned into structured datasets from natural-language instructions, automotive telemetry is standardized into interoperable schemas and marketplaces, and autonomous-driving corpora are selected, annotated, and validated with varying degrees of human oversight (Wang et al., 9 Nov 2025, Yang et al., 7 Mar 2026, Ma et al., 21 May 2025, Hohl et al., 1 Oct 2025). The term therefore does not denote a single canonical system; rather, it names a family of approaches centered on automation, provenance, validation, interoperability, and reduced dependence on manual expertise.
1. Semantic range and recurrent design patterns
Recent uses of AutoData fall into several technically distinct but conceptually related categories. One strand treats AutoData as autonomous data engineering for tabular machine learning, exemplified by Dataforge’s Data Agent, which aims to transform raw tables “From Data to Better Data” through cleaning, routing, feature transformation, and validation (Wang et al., 9 Nov 2025). A second strand treats AutoData as automated dataset discovery, exemplified by AutoDataset’s paper-first arXiv monitoring and semantic retrieval pipeline (Yang et al., 7 Mar 2026). A third strand uses the term for automated open-web collection, where a multi-agent system converts a natural-language instruction into a dataset gathered from HTML pages and REST APIs (Ma et al., 21 May 2025).
A separate automotive strand emphasizes structured access to fragmented vehicle and mobility data. In this usage, AutoData includes domain-specific dataset indexes such as ad-datasets, cross-OEM harmonization via the Common Vehicle Information Model, taxonomy-backed metadata for systems engineering, and curated market or incident datasets such as DVM-CAR, SRNI-CAR, and AVOID (Bogdoll et al., 2022, Pillmann et al., 2018, Hohl et al., 1 Oct 2025, Huang et al., 2021, Ding et al., 2023, Zheng et al., 2023). In autonomous driving, the same label is also attached to selection, annotation, and verification pipelines, including AIDE for open-world object detection, metadata-driven validation-data selection, and DARTS’s semi-automated multisensor annotation workflow (Liang et al., 2024, Trinh et al., 2024, Gamalii et al., 31 Dec 2025).
Across these uses, a recurring architectural pattern is staged rather than monolithic automation. Rule-based routers constrain high-level decisions, compact LLM planners or role-specialized agents operate inside bounded action spaces, grounded validators or human reviewers filter unsafe or low-quality outputs, and the resulting artifacts are logged with explicit provenance for auditability and reuse (Wang et al., 9 Nov 2025, Ma et al., 21 May 2025, Gamalii et al., 31 Dec 2025).
2. Autonomous preparation and optimization of input data
In tabular machine learning, Dataforge’s Data Agent implements a six-stage pipeline organized around perception–planning–grounding–execution principles: input and cleaning; task-level routing; action-level routing; grounding and execution; evaluation with dual feedback loops; and experience replay and reporting (Wang et al., 9 Nov 2025). Users upload .csv, .parquet, or .hdf tables, optionally specify predictors and response columns, and select cleaning modes light, aggressive, or time-series. A rule-based router infers classification, regression, or unsupervised learning from schema metadata, after which a compact LLM planner proposes feature-level operations such as scaling, encoding, interaction terms, aggregation, dimensionality reduction, and selection. Grounded validation performs schema alignment, type checks, logical safety tests, leakage detection, and provenance logging before execution. The optimization logic is expressed with standard objectives such as
conceptual action scoring
and an outer-loop acceptance rule based on a user-defined Min Improvement threshold. On nine tabular datasets, the system completed all runs without interruption, converged within ~2 action retries on average, finished each workflow in ≈3.9 s, required zero training cost, and reported selected gains such as German Credit F1 rising from 74.20% on the original data to 79.60% for DataAgent and SPECTF Heart improving from 0.772 to 0.840 while reducing features from 44 to 20 (Wang et al., 9 Nov 2025).
A related but distinct formulation appears in automatic optimization of prompt inputs for LLMs. ADO defines two transformations over each record: content engineering , which performs imputation, pruning, and enrichment, and structural reformulation , which maps the transformed record to a prompt schema (Lin et al., 17 Feb 2025). The core constrained objective is
with search performed over data-optimization prompts that specify both and . The method uses Diverse Prompt Search with cosine-similarity and METEOR constraints, optional factual validation, and validation-set scoring. Empirically, its DPS variant outperformed APE or OPRO on 8/9 datasets for GPT-3.5 Turbo, 8/9 for Gemini-1.5 Flash, and 9/9 for Llama-3.1 70B. Mean performance improved from 0.443 to 0.536 for GPT-3.5 Turbo, from 0.439 to 0.539 for Gemini-1.5 Flash, and from 0.431 to 0.523 for Llama-3.1 70B, while combinations such as CoT + ADO and ICL + ADO also exceeded their non-ADO counterparts (Lin et al., 17 Feb 2025).
These two strands share a common technical intuition: the quality of downstream inference depends materially on transformations applied before model fitting or prompting. In one case the target is an AI-ready table with typed schema and validated feature operations; in the other, it is a compact, high-saliency prompt representation.
3. Automated discovery and collection of datasets
AutoDataset addresses dataset discovery by treating the literature itself as the primary source of truth (Yang et al., 7 Mar 2026). The system continuously monitors six arXiv computer-science categories—IR, DB, AI, CL, CV, and MA—and reflects the reported growth in dataset-releasing papers from roughly 90 per year in 2014 to 8,566 in 2025, a ≈95× increase. Its first stage, BERT-Gate, classifies concatenated title and abstract and achieves accuracy 0.97 and F1 = 0.94 at an average inference latency of 11 ms per paper. Positive papers are then parsed with GROBID, and BERT-Desc performs sentence-level extraction of dataset descriptions with accuracy 0.949, precision 0.800, recall 0.924, F1 0.858, and 9.39 ms per inference. Dataset URLs are recovered from PDFs with a fallback to LaTeX source analysis, and descriptions are indexed with a dense semantic retriever using Alibaba-NLP/gte-large-en-v1.5. In a pilot with three researchers and five fresh datasets, manual discovery required 4–9 minutes per query, compared with 0.5–2 minutes with AutoDataset, corresponding to ≈60–80% reduction in time to first valid link (Yang et al., 7 Mar 2026).
Open-web collection systems push the concept further by constructing datasets directly from online sources. The multi-agent AutoData system for web collection takes a single natural-language instruction and produces a dataset from live HTML pages or REST APIs through a research squad, a development squad, and a central manager (Ma et al., 21 May 2025). Its coordination mechanism is an oriented message hypergraph
with oriented hyperedges and agent context
augmented by OHCache, which stores large artifacts locally and broadcasts only cache identifiers. The stated token-cost objective is
0
On Instruct2DS, AutoData reported Academic F1/Prec/Rec = 91.85/93.74/90.51, Stock 96.75/98.37/94.17, and Sport 90.14/95.63/85.28, with mean implementation time 5.58 minutes versus 15.37 for Manus and ~187 minutes for humans, and mean expense 12.49 for Manus. Case studies on picture-book collection and BibTeX extraction from surveys likewise showed higher accuracy and lower cost than Manus (Ma et al., 21 May 2025).
Taken together, these systems define AutoData not merely as automated preprocessing but as automated acquisition of data assets themselves, whether the source is the research literature or the live web.
4. Automotive repositories, taxonomies, and interoperable data models
In autonomous driving, one major AutoData problem is discoverability. The ad-datasets platform is an online, open-source meta-collection that indexed 158 datasets as of Nov 25, 2021 and organizes them with 16 property categories selected through an expert survey and feasibility study (Bogdoll et al., 2022). These categories include Annotations, Benchmark, Frames, Last Update, Licensing, Location, Number of Scenes, Publish Date, Related Datasets, Related Paper, Sampling Rate [Hz], Scene Length [s], Sensor Types, Sensors – Details, Size [GB], and Size [h]. The tool supports filter-and-sort operations over datasets for tasks such as lane detection, motion forecasting, 2D/3D detection and tracking, SLAM, end-to-end driving, and adverse-condition analysis. The paper’s detailed analysis of 40 datasets reports, for example, cameras in 39 of 40 datasets, LiDAR in 23 of 40, radar in 8 of 40, a median Size [h] of 16.5 h, and a median scene count of 158 (Bogdoll et al., 2022).
A broader structuring effort appears in “Structuring Automotive Data for Systems Engineering: A Taxonomy-Based Approach,” which derives an inductive taxonomy from a structured literature review of 21,467 papers narrowed to 87 publications (Hohl et al., 1 Oct 2025). The taxonomy organizes automotive data by Source—real-world or synthetic, physical or non-physical, vehicle or environment—and by Application—requirements, model identification, training, and V&V, further subdivided by methods and system domains. Its reported synthesis highlights a growing emphasis on real-world driving and machine-learning applications, while identifying a “critical gap in data availability for requirements engineering.” The work explicitly positions the taxonomy as a business glossary and metadata backbone aligned with the CMMI Data Management Maturity model (Hohl et al., 1 Oct 2025).
At the interoperability layer, the AutoMat project’s Common Vehicle Information Model addresses fragmentation of OEM-proprietary telemetry by defining a brand-independent harmonization layer between raw signals and marketplace-ready data packages (Pillmann et al., 2018). CVIM distinguishes vehicle-specific signal level, harmonized measurement channels, and aggregated data packages, and supports three standard aggregation families: time-series, histogram-based, and map-based geo-tile aggregation. Its mathematical primitives are the expected ones for windowed summaries and histograms, such as
2
The associated Vehicle Big Data Marketplace is designed to broker, catalogue, and monetize aggregated vehicle data under brand-independent semantics (Pillmann et al., 2018).
Structured automotive data integration is not limited to fleet telemetry. DVM-CAR combines 1,451,784 images of 899 car models with prices, trim data, and UK annual sales over two decades, while SRNI-CAR integrates Chinese monthly sales, 217,292 aspect-structured owner reviews, and 83,590 labeled industry-news items across 2016–2022 (Huang et al., 2021, Ding et al., 2023). These datasets extend AutoData beyond perception and control into visual marketing, product aesthetics, demand forecasting, and policy analysis.
5. Selection, annotation, and safety-oriented autonomous-driving data loops
Open-world perception systems require continuous data improvement after deployment, and AIDE frames this as an automatic data engine (Liang et al., 2024). Its four-stage loop consists of automatic issue identification with Otter dense captioning, efficient data curation with BLIP-2 retrieval, model improvement through OWL-v2 box proposals plus CLIP zero-shot classification, and verification via ChatGPT-generated scenario descriptions with human review. In the reported benchmark, dense captioning achieved 70.2% precision for identifying missing categories, compared with 18.3% AP50 for direct OVOD discovery; BLIP-2 text retrieval reached 54.5% top-3 accuracy, versus 33.9% for CLIP text retrieval and 18.9% for image similarity; and two-stage pseudo-label filtering raised average pseudo-label precision from 21.4% for raw OWL-v2 predictions to 45.7%. With the Data Feeder enabled, AIDE reported average novel AP 12.0%, known AP 26.6%, and Forgetting -3.3%, while an additional short update with 10 human-labeled images raised average novel AP from 12.0% to 14.2% at a total added cost of $1.59 (Liang et al., 2024).
A complementary problem is selecting validation data rather than generating it. The metadata-driven selection method for assessment of autonomous vehicles formulates subset selection as distribution matching in metadata space, with objectives
$S(o \mid D) = \alpha\, safety(o,D) + \beta\, perf\_delta(o,D) + \gamma\, prior(o \mid meta(D), history),$4
and similarity measures
$S(o \mid D) = \alpha\, safety(o,D) + \beta\, perf\_delta(o,D) + \gamma\, prior(o \mid meta(D), history),$5
On BDD100K metadata derived from OpenStreetMap, the method outperformed the Diverse Complexity baseline at all reported keep ratios; for example, at $S(o \mid D) = \alpha\, safety(o,D) + \beta\, perf\_delta(o,D) + \gamma\, prior(o \mid meta(D), history),$6 it achieved <code>𝓢_c = 0.8887</code> versus <code>0.7269</code> and <code>𝓢_d = 0.4136</code> versus <code>0.2154</code>, and across ADAS-feature experiments the reported curves rose to <code>0.9337</code> for <code>𝓢_c</code> and <code>0.6200</code> for <code>𝓢_d</code> at <code>ρ=0.99</code> (<a href="/papers/2407.12065" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Trinh et al., 2024</a>).</p> <p>For annotation, the DARTS project describes a semi-automated, human-in-the-loop multisensor pipeline built around LiDAR 3D detection, Segments.ai review, Apache Airflow orchestration, and provenance-aware storage in a nuScenes-like schema (<a href="/papers/2512.24896" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Gamalii et al., 31 Dec 2025</a>). The detector uses DSVT on top of OpenPCDet, the pipeline includes temporal smoothing and ID assignment through 3D MOT, and privacy-preserving image processing is handled by RetinaFace and a YOLO-based license-plate detector. The central productivity measure is the Correction Acceleration Ratio,</p> <p>$S(o \mid D) = \alpha\, safety(o,D) + \beta\, perf\_delta(o,D) + \gamma\, prior(o \mid meta(D), history),$7
where correction time $S(o \mid D) = \alpha\, safety(o,D) + \beta\, perf\_delta(o,D) + \gamma\, prior(o \mid meta(D), history),$8 is compared with manual-from-scratch time $S(o \mid D) = \alpha\, safety(o,D) + \beta\, perf\_delta(o,D) + \gamma\, prior(o \mid meta(D), history),$9. Reported results show DSVT reaching AP values of about 0.94 on Zenseact and 0.87 on nuScenes for cars/vehicles, with CAR values about 0.93 and 0.87, respectively, implying up to a 93% reduction in manual time in similar domains (Gamalii et al., 31 Dec 2025).
These pipelines depend on foundational multisensor datasets. Mcity provides more than 50 hours and over 3,000 miles of naturalistic open-road driving together with 18 scenario types, 255 runs, multiple LiDARs, radar, in-cabin cameras, GNSS/IMU, CAN, and more than 17.5k images labeled on the front 60° camera (Dong et al., 2019). CARRADA contributes synchronized camera and 77 GHz FMCW radar with raw range–angle and range–Doppler annotations across 30 sequences, 12,666 frames, and 7,193 annotated frames, enabling radar-only learning and radar–camera fusion (Ouaknine et al., 2020). AVOID extends the notion of AutoData to incident analysis by merging more than 1,000 official crash reports with 52 global incidents, adding land-use and weather enrichment, and partitioning the processed data into ADS, ADAS, shuttle, and news files; among crashed vehicles excluding news reports, the reported engagement status was 58% ADAS-engaged, 25% ADS-engaged, and 17% disengaged (Zheng et al., 2023).
6. Privacy, authorization, and unresolved constraints
AutoData systems in vehicles confront a privacy problem as much as a data-engineering problem. In Android Automotive OS, PriDrive combines network inspection, static analysis, and dynamic analysis to characterize vehicle-property access across OEM and third-party apps (Gözübüyük et al., 2024). The study reports that vehicle speed is collected at roughly 25 Hz in the abstract and ≈20 Hz in dynamic traces for OEM A, versus ≈5 Hz for OEM B, while model info, climate and A/C settings, seat data, and other state are collected in a batch ≈30 seconds after vehicle startup. Policy-coverage analysis found OEM A disclosing 735/845 = 86.98% of observed properties, OEM B 41/67 = 61.19%, and OEM C 8/33 = 24.24%, with substantial gaps in climate/comfort, lighting, driving assistance, and audio/video disclosure. The same study notes that AAOS has been integrated into over 100M vehicles and that several telemetry flows terminate in Google-operated data centers (Gözübüyük et al., 2024).
The broader ADS literature frames these findings within a distributed architecture spanning sensors, ECUs, vehicle buses, V2X communication, EDR and DSSAD storage, edge processing, and cloud back ends (Alhabib et al., 2024). The review emphasizes tight latency budgets, including an example of 4 ms responses for safety-critical off-vehicle communication, and substantial raw data rates such as 20–40 MB per second for camera streams and 10–70 MB per second for LiDAR. It also highlights authorization, privacy, and forensic tensions: EDR data ownership is generally assigned to the vehicle owner in the U.S.; DSSAD requires consumer approval for recording and/or access; and stakeholders including regulators, OEMs, suppliers, insurers, and researchers consume different subsets of data under evolving legal constraints (Alhabib et al., 2024).
At an even lower level, raw CAN telemetry itself can be personally identifying. “Automatic Driver Identification from In-Vehicle Network Logs” shows that byte-level CAN time series, without reverse-engineering proprietary signal semantics, allow re-identification of 33 drivers with 75–85% accuracy from less than two minutes of driving (Remeli et al., 2019). Using per-byte time series, CNN–LSTM–attention models, and a mixture layer over the top 10 individual time-series models, the study reports mean 1-vs-all accuracies of 0.758 for 20 s windows, 0.829 for 60 s, and 0.847 for 120 s, demonstrating that high-dimensional vehicle telemetry constitutes sensitive personal data (Remeli et al., 2019).
The literature therefore converges on a double limitation. Technically, many systems still note failure modes under extreme noise, severe class imbalance, missing or obfuscated URLs, highly specialized domain constraints, complex multi-table dependencies, or sites requiring login, CAPTCHA, or heavy client-side rendering (Wang et al., 9 Nov 2025, Yang et al., 7 Mar 2026, Ma et al., 21 May 2025). Organizationally, the automotive domain still exhibits fragmented formats, incomplete interoperability, and a lack of explicit data support for requirements engineering, even as real-world driving and machine-learning applications expand (Hohl et al., 1 Oct 2025). This suggests that AutoData is best understood not as fully solved automation, but as a rapidly developing set of architectures for making heterogeneous data searchable, auditable, usable, and governable at scale.
