EvoTrain-180K: Dual Datasets for Agent Learning

• EvoTrain-180K is a dual-purpose dataset that provides both high-fidelity dynamic hand-object interaction trajectories and a rich multi-domain language retrieval corpus.
• The DynaHOI-Gym subset captures over 180K closed-loop hand-capture episodes with detailed pose, kinematic data, and stratified motion types for rigorous evaluation.
• The EvoEmbedding subset offers sequentially annotated segments optimized for joint memory evolution and context-sensitive retrieval, enhancing long-context learning.

EvoTrain-180K is an umbrella name used for two distinct, technically advanced datasets at the forefront of embodied agent learning and evolvable retrieval representation research. In the context of dynamic hand-object interaction, EvoTrain-180K refers to a set of 180,000 closed-loop hand-capture trajectories forming the most challenging subset of DynaHOI-10M, a synthetic benchmark for contact-rich manipulation with moving targets. In a complementary but independent line of work, EvoTrain-180K also denotes a 184,137-sample multi-domain language retrieval corpus powering joint optimization of latent memory and retrieval representations in EvoEmbedding. Both datasets are designed for end-to-end, sequence- and memory-centric agentic learning, but differ fundamentally in domain, structure, and supervised task (Hu et al., 12 Feb 2026, Nie et al., 19 Jun 2026).

1. Dataset Genesis and Scope

EvoTrain-180K, as instantiated in DynaHOI-Gym, covers high-fidelity, closed-loop "dynamic capture" episodes where dexterous virtual hands must localize and intercept dynamic objects under a richly parameterized taxonomy of movement types and trajectories. Each trajectory corresponds to a single object-capture episode, with full pose, kinematic, and action-state recording at 20 Hz. This results in approximately 10 million frames, encapsulating around 140 hours of varied egocentric footage. The full dataset structure encompasses a strict train/val/test split (80/10/10 stratified by category), guaranteeing proportional representation across 8 major and 22 subclassified motion types (Hu et al., 12 Feb 2026).

In the retrieval context of EvoEmbedding, EvoTrain-180K is constructed by aggregating high-diversity source corpora: web documents (FineWeb), synthetic multi-turn persona-driven dialogues, and extracted agentic or "memory" texts such as lists and timelines. All documents are segmented into sequential context units (segments); each sample is annotated with dynamic QA templates and labeled for retrieval (positive/negative) via LLM verification. The result is a challenging, multi-domain corpus tailored for learning retrieval representations dependent on contextualized, evolving memory states (Nie et al., 19 Jun 2026).

2. Data Structure and Annotation Pipeline

In DynaHOI-Gym:

• Each episode is parameterized by a JSON "motion script" that formalizes object paths using explicit physical parameters (e.g., direction, angular velocity, amplitude, elasticity), computed via a Unity-managed simulation environment.
• Modalities recorded include: egocentric RGB (256×256, 60° FOV), hand state (18-D), fingertip and global object poses (right-handed Unity coordinates), and corresponding ground truth for evaluation.
• The motion taxonomy includes straight-line, simple harmonic, circular-arc, projectile, pendulum, inclined-rolling, impact-response, and hybrid types, with respective subcategories refining path complexity (e.g., "large-radius clockwise arc", "double bounce projectile") (Hu et al., 12 Feb 2026).

In EvoEmbedding:

• Each record consists of up to 256 sequential segments (mean ≈20; mean segment ≈63 tokens), carefully windowed to a maximum context of 12,000 tokens.
• Automated annotation involves: (1) context example generation; (2) dynamic QA construction via diverse LLMs (covering coreference, temporal, and multi-hop factoid queries); (3) segment-level retrieval labeling (using Gemini-3.1-Pro-Preview for positive/negative evidence, answer verification, and hallucination filtering).
• Data distributions: mean full-sample context length = 1,289.8 tokens; ≈53% of samples under 512 tokens; mean negatives per query ≈19.5 (Nie et al., 19 Jun 2026).

Domain	Sample Count	Mean Length	Labels
DynaHOI-Gym	180,000	≈56 frames	Success/Failure, Poses
EvoEmbedding	184,137	1,289.8 tokens	Retrieval (pos/neg), QA

3. Task Definitions and Evaluation Protocols

Dynamic Capture in DynaHOI-Gym:

• The agent faces a Markov decision process with observations comprising RGB (Iₜ), hand pose (hₜ), finger joint angles (qₜ), and optional keypoints.
• At each timestep, an action is a vector of Cartesian and joint adjustments.
• Success is defined by localization (palm center within $d_\text{loc}=0.3$ units of object position at any frame), and grasp (all five fingers close in correct direction by at least 90% of the ground-truth rotation at first successful localization frame).
• Rollout-based metrics: location success ($S_\text{loc}$), grasp success ($S_\text{gra}$), localization/grasp error ($E_\text{loc}, E_\text{gra}$), as well as trajectory-level smoothness ($Q_\text{smooth}$), spatial linearity ($Q_\text{line}$), and completion speed ($R_\text{time}$).

Long-Context Retrieval and Memory in EvoEmbedding:

• The model jointly optimizes for: (a) memory evolution (summarization into latent tokens) and (b) context-sensitive segment retrieval.
• Each input segment $x_t$ updates a FIFO latent memory $M_{t-1}$, using specialized transformer LoRA adapters for memory and retrieval.
• The loss $L_\text{total}$ combines memory-generation (cross-entropy against ground-truth answers) and a length-weighted multi-positive contrastive retrieval term over positive and negative segment embeddings.
• Representative notation:
  • $S_\text{loc}$0
  • Loss functions: $S_\text{loc}$1 (memory), $S_\text{loc}$2 (retrieval), see details in source (Nie et al., 19 Jun 2026).

4. Training, Preprocessing, and Batching Strategies

Vision Manipulation:

• Preprocessing: RGB normalized/resized with optional color jitter/blur; hand position and joint angle deltas normalized and robustified via quantile clipping and random translation.
• Batching: Stratified by subcategory, uniform sampling per batch (size 32–64), zero-padded variable-length trajectories.
• Standard optimizer is AdamW with learning rate warmdown; frame dropping used to simulate variable framerate.
• Training: Up to 100 epochs, 4×A100 GPUs; inputs consist of stacked observation windows with proprioceptive tokens (Hu et al., 12 Feb 2026).

Text Retrieval and Memory:

• Model initialized from Qwen3 with LoRA ranks 64 per stream; batch size 16 samples.
• Dynamic segment batching aggregates as many consecutive context segments as fit within max token budget (2,048, including latent memory tokens), enabling measured 3.8× speedup (26.6 h vs. 101.4 h on 8×H800) with 1.9% accuracy improvement.
• Cosine scheduler, temperature $S_\text{loc}$3 for contrastive loss, total steps 11,509 (one epoch in reported experiments).
• Memory queue capacity is 512 latent tokens (L=32 steps, K=16 tokens per step) (Nie et al., 19 Jun 2026).

5. Baselines and Performance

ObAct Baseline (DynaHOI-Gym):

• Observe-before-act approach: s=5 frame contextual window with VLM backbone; spatiotemporal cross-attention; fused proprioception; diffusion-style policy head (adapted from GR00T-N1.5, 3B parameters).
• Trained end-to-end with MSE-based denoising diffusion loss plus optional L2 action regularization.
• Key reported result: $S_\text{loc}$4 (vs. 27.9% for GR00T diffusion baseline), representing an +8.1 percentage point improvement. Matches or exceeds baselines on $S_\text{loc}$5 and $S_\text{loc}$6 (Hu et al., 12 Feb 2026).

EvoEmbedding Training (EvoTrain-180K):

• Outperforms Qwen3-Embedding-8B and KaLM-Embedding-Gemma3-12B on long-context retrieval tasks.
• Demonstrates generalization to agentic workflows, competitive with or superior to specialized agentic memory systems when used in a naive RAG configuration (Nie et al., 19 Jun 2026).

6. Dataset Access, Utilities, and Extensibility

• DynaHOI-Gym (including EvoTrain-180K hand-object trajectories) is publicly released, complete with Python data loaders, Unity-Gym interface, metric computation scripts, rollout testers, and pretrained ObAct checkpoints (3B parameters). Fine-tuning is supported via encoder freezing and stratified sample extension, and new motion types can be added by appending scripts to EpisodeManager and retraining (Hu et al., 12 Feb 2026).
• EvoTrain-180K for EvoEmbedding is available with detailed examples, segment/chunking utilities, batching and grouping logic, and scripts for initialization and loss computation (see project page). The dataset supports agentic personalization use cases and can be grouped or sampled by sequence length for efficient large-scale training (Nie et al., 19 Jun 2026).

7. Significance and Trajectory

EvoTrain-180K defines a high-precision experimental platform in two orthogonal domains: embodied dynamic capture and context-evolvable language retrieval. It provides standardized, richly labeled, and extensible data regimes for model development, robust evaluation, and ablation in dynamic, multimodal, or long-context agentic tasks. Its engineered diversity and procedural annotation pipelines enable rigorous benchmarking of both closed-loop visuomotor models (DynaHOI-Gym) and next-generation, context-aware retrieval architectures (EvoEmbedding). A plausible implication is that datasets synthesized and annotated via these regimes will continue to serve as reference benchmarks for temporally and contextually adaptive agentic systems (Hu et al., 12 Feb 2026, Nie et al., 19 Jun 2026).