DataClaw0: Agentic Tailoring Multimodal Data from Raw Streams

Abstract: Massive unstructured multimodal streams suffer from high "data entropy," impeding both efficient human knowledge acquisition and high-quality AI post-training. Existing passive annotation paradigms, heavily reliant on heuristic rules or general VLMs, are costly, monotonous, and fail to unlock the deep procedural logic embedded in raw data. We elevate data processing to a learnable capability, proposing a paradigm shift towards Agentic Data Tailoring, which actively refining and structuring data to align with diverse user and downstream intents. To overcome the data scarcity bottleneck in training such high-order capabilities, we design a two-stage pipeline grounding generative semantic synthesis in deterministic Factual Anchors, yielding a large-scale dataset spanning five core physical and digital domains. Building upon this, $\text{DataClaw}_0$-9B model synergizes Supervised Fine-Tuning (SFT) with Group Relative Policy Optimization (GRPO), achieving robust alignment with complex refinement and tailoring intents. To systematically quantify this capability, we construct $\text{DataClaw}_0$-val, the first benchmark dedicated to data refinement. Crucially, we adopt downstream post-training as the ultimate validation touchstone. Evaluations on video generation, real-world VQA, and GUI navigation confirm that $\text{DataClaw}_0$ delivers high-information-density tailored data, facilitating efficient model adaptation to new tasks under limited training data regimes. Project page: https://czjdsg.github.io/MakeAnyData

• The paper introduces an agentic data tailoring framework that transforms raw, high-entropy multimodal streams into dense, schema-aligned outputs via a two-stage pipeline.
• It employs deterministic reward functions with GRPO and expert routing to ensure strong schema compliance, temporal grounding, and effective structured synthesis.
• Empirical results show significant improvements in downstream tasks like GUI navigation, video generation, and spatio-temporal VQA compared to conventional methods.

DataClaw0: Agentic Tailoring of Multimodal Data from Raw Streams

Motivation and Conceptual Framework

The paper introduces a paradigm shift in multimodal data engineering through agentic data tailoring, elevating data refinement from passive annotation or simple captioning to an actively learnable capability. The approach targets high-entropy multimodal streams—such as tutorial videos, robot trajectories, and GUI logs—where conventional data processing pipelines fail to distill task-critical supervision or procedural logic due to noise, redundancy, and weak structure. Existing passive annotation with general VLMs or heuristic rules is insufficient for extracting dense, intent-aligned training data from such sources, and often yields hallucinated or fragmented outputs.

$\text{DataClaw}_0$ proposes agentic tailoring: an intelligent agent filters redundant information, locates task-critical evidence, and reorganizes raw multimodal streams into dense, verifiable, schema-aligned data customized for specific user or downstream intents. The capability is formalized as intent-conditioned entropy reduction via structured data synthesis, repurposing raw streams to produce high-information-density assets for efficient post-training and model adaptation.

Figure 1: Representative tailoring cases in various domains, showing $\text{DataClaw}_0$ decomposing raw streams into structured outputs based on explicit user intents.

Two-Stage Automated Construction Pipeline

To resolve the data paradox—requiring refined data to train refiners—the paper introduces a scalable two-stage pipeline for data construction:

1. Factual Anchor Extraction: Lightweight domain experts and heuristic scripts extract deterministic anchors from raw streams, such as object states, trajectories, event boundaries, OCR text, and GUI actions. These anchors provide low-level grounding and reduce annotation hallucinations.
2. Semantic Synthesis with Strong VLMs: Building on extracted anchors and explicit domain intents, a vision-LLM synthesizes structured supervision by chaining multimodal reasoning traces. This top-down synthesis expands beyond simple captions or QA pairs and injects multi-dimensional logic, supporting five core domains: daily life, education, embodied intelligence, world models/AIGC, and GUI agents.

   Figure 2: Overview of the $\text{DataClaw}_0$ pipeline: anchor extraction followed by intent-conditioned synthesis, yielding large-scale structured data across domains.

Model Architecture and Training Paradigms

$\text{DataClaw}_0$ leverages Qwen3.5-9B as its backbone and fuses Supervised Fine Tuning (SFT) with Group Relative Policy Optimization (GRPO). SFT initializes instruction-following and structural adherence, while GRPO introduces reinforcement learning with deterministic, rule-driven reward functions—including schema compliance, anchor alignment, and reasoning efficiency.

The GRPO phase sidesteps neural reward models, utilizing AST/regex parsers for format checking and shape alignment metrics for spatio-temporal grounding. This yields more reliable, concise, and grounded structured outputs. Two complementary deployment paradigms are explored:

• Omni (O): A unified model trained across all domains, supporting diverse intents flexibly.
• Expert (E): A modular system routing tasks to domain-specific tailoring agents, favoring domain specialization and modular scalability.

Structured Benchmark and Hierarchical Evaluation

$\text{DataClaw}_0$-val, the first benchmark for agentic multimodal data refinement, is designed to stress the agent's structured synthesis ability, covering diverse domains and long-tail scenarios. Each sample provides high-entropy inputs and explicit intent instructions, requiring schema-aligned JSON outputs for downstream training.

Evaluation uses hierarchical metrics: hard JSON validity gating, field-level schema integrity, semantic alignment (embedding-based cosine), and trajectory-shape similarity for spatio-temporal outputs. A fuzzy-intent subset is included to assess capability under underspecified or colloquial user requests.

Figure 3: Overview of the $\text{DataClaw}_0$ system, showing data mixture, scaling curves, and feature-space visualization highlighting emergent diversity.

Empirical Results and Ablations

Agentic Tailoring: Specialist Versus Generalist

$\text{DataClaw}_0$-E consistently outperforms baselines in structured data synthesis, achieving overall Field scores of 97.53 (schema completeness), competitive Semantic scores (74.94), and Sequence scores (48.86)—notably dominating sequence-sensitive and fuzzy-intent tasks. Expert routing enables stable scaling and avoids negative transfer seen in the Omni variant, which suffers task interference within shared weights and exhibits performance oscillations.

Contradictory to general expectations, domain specialization is shown to be superior for structured data synthesis from heterogeneous multimodal streams. Specialist agents avoid catastrophic forgetting and gradient conflicts prevalent in unified architectures.

Figure 4: Qualitative visualization comparing robot manipulation and GUI task reconstruction; $\text{DataClaw}_0$ yields coherent, temporally grounded outputs, while baselines miss critical events and structure.

Downstream Adaptation: Efficient Targeted Refinement

Structured data refined by $\text{DataClaw}_0$, when used for SFT on downstream models, outperforms self-refinement baselines and matches or exceeds proprietary annotator performance in end-to-end metrics:

• GUI navigation: $\text{DataClaw}_0$ achieves 15.6% TSR, surpassing Gemini's 14.2%.
• Action video generation: Lower FVD and higher contact mAP, reflecting better visual quality and action-object interaction.
• Spatio-temporal VQA: $\text{DataClaw}_0$0 delivers 33.2% overall accuracy, outstripping Gemini.

These results demonstrate substantially higher training utility of compact, tailored subsets compared with conventional full-scale datasets, with pronounced gains in final task completion.

Figure 5: Qualitative example for GUI navigation, showing task decomposition and state tracking using refined supervision.

Figure 6: Comparative qualitative outputs for action video generation, highlighting temporal consistency and affordance.

Figure 7: Example visualization for spatio-temporal VQA, illustrating improved semantic and structural grounding.

Scaling Laws, Diversity, and Intent Comprehension

$\text{DataClaw}_0$1-E displays stable log-linear scaling, while Omni models reveal oscillatory dynamics due to negative transfer. Feature-space t-SNE visualization confirms that tailored outputs cover a broader semantic space versus base model or raw data, revealing emergent clusters and long-tail patterns. In fuzzy-intent benchmarks, $\text{DataClaw}_0$2 approaches the performance of proprietary MLLMs in intent comprehension and utility.

Failure Analysis and Limitations

While the approach substantially advances agentic data tailoring, temporal hallucinations and inconsistencies persist in long-horizon scenarios, particularly when spatial cues contradict chronological frame order. Strict temporal grounding remains a challenge for LLMs, as illustrated by error cases in daily-life navigation.

Figure 8: Successful world-model or video-generation tailoring case, showing accurate segment selection and description.

Figure 9: Successful daily-life tailoring case, demonstrating environment understanding and reasoning trace.

Figure 10: Failed daily-life tailoring case, illustrating temporal inconsistency in reasoning despite correct semantic content.

Practical and Theoretical Implications

The introduction of rule-driven GRPO for structured data tailoring and domain-decoupled expert routing advances multimodal alignment and supervision for foundation models. Practically, the framework enables efficient adaptation to new tasks under limited training data regimes by extracting task-relevant assets rather than scaling brute-force curation. Theoretically, agentic tailoring redefines data as a controllable, learnable capability, with downstream application as the ultimate validation of refinement quality.

The modular architecture and deterministic reward design provide a template for future scalable multimodal annotation systems, encouraging research into autonomous data construction, hierarchical agent routing, and deeper procedural logic extraction.

Conclusion

$\text{DataClaw}_0$3 delivers robust agentic tailoring of heterogeneous multimodal streams, combining deterministic anchor extraction and vision-language semantic synthesis with rule-driven GRPO and domain-specialist deployment. It achieves competitive or superior data refinement compared to proprietary annotators, demonstrates strong downstream transfer, and provides practical protocol for scalable multimodal data engineering. The work highlights the necessity of domain specialization, deterministic rewards, and emergent diversity in structured supervision, laying the groundwork for future agentic data construction and adaptive alignment of multimodal foundation models (2606.21337).