Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stack

Abstract: In this report, we present Hy-Embodied-0.5-VLA, abbreviated as HyVLA-0.5, an end-to-end system that spans the full robot learning stack: data collection, model design, continued pre-training and supervised fine-tuning, RL post-training, and real-world deployment. Each component serves a distinct role in this stack.

• The paper presents HyVLA, a 4-billion-parameter vision-language-action system that co-designs data collection, model architecture, training, adaptation, and deployment for real-world robots.
• HyVLA achieves 90.9% clean and 90.1% randomized success on 50 RoboTwin 2.0 tasks, while transferring from UMI demonstrations to different robot platforms without target-robot teleoperation.
• The paper introduces FlowPRO, a critic-free and reward-free offline RL method that reaches 98–99% success on long-horizon bimanual tasks and reduces completion times to 16–37 seconds.

Overview and Motivation

Hy-Embodied-0.5-VLA (HyVLA), developed by Tencent Robotics X in collaboration with the Tencent Hy team, is an end-to-end vision-language-action (VLA) system that spans the complete robot learning pipeline: data collection hardware, model architecture, large-scale pre-training, supervised fine-tuning (SFT), reinforcement learning (RL) post-training, and real-world deployment (2606.14409). The paper's central position is that a generalist robot cannot emerge from a single model in isolation; instead, the data, training, adaptation, and execution layers must be co-designed around real-hardware constraints. This framing motivates three coupled data-side challenges—teleoperation's lack of haptic feedback, coarse action labels from raw human video, and the embodiment/control/perception gaps in cross-embodiment transfer—as well as architectural bottlenecks including autoregressive action discretization, generalist visual backbones ill-suited to dense spatiotemporal reasoning, reward-model-dependent RL recipes, and deployment latency.

The system is built on four pillars. First, a custom fingertip UMI device with an optical motion-capture cage yields over 10,000 hours of egocentric demonstrations with sub-millimeter pose precision that double as post-training trajectories. Second, the Hy-Embodied-0.5 backbone—a 4B-parameter Mixture-of-Transformers (MoT) VLM—is extended with a flow-matching action expert, a compact memory encoder, and a delta-chunk action representation. Third, FlowPRO provides critic-free, reward-free offline RL post-training via Proximalized Preference Optimization (PRO). Fourth, an asynchronous inference pipeline with cubic Bézier chunk stitching enables high-frequency closed-loop control. The model, dataset, and code are publicly released.

Model Architecture

HyVLA formulates manipulation as goal-conditioned, chunk-level control: at each step $t$, the policy consumes multi-view RGB history $\mathcal{I}_t$ ($n{\times}K$ frames), a language instruction $\ell$, and proprioceptive state $\mathbf{s}_t$, and predicts an $H$-step action chunk via conditional flow matching. Both state and actions are expressed in the end-effector frame as 10-dimensional vectors per arm (3-D translation, 6-D continuous rotation, 1-D gripper command). The delta-chunk representation predicts incremental end-effector motion relative to the current state, which decouples policy learning from embodiment-specific kinematics and shrinks the optimization search space—a design choice that directly enables the cross-embodiment results described below.

Three architectural components merit attention:

Embodied-native MoT backbone. Hy-Embodied-0.5-MoT maintains non-shared QKV and FFN parameters for visual and textual streams, with cross-modal interaction restricted to shared self-attention. Visual encoding uses Hy-ViT 2.0, a native-resolution ViT distilled from a larger internal teacher. Auxiliary next-token prediction on VQA, 2D grounding, and 3D geometric tasks co-trains the shared backbone to preserve semantic and spatial representations.

Dual-tower flow-matching action expert. A separate generation-oriented tower consumes projected state and noisy action tokens, interacting with the VLM tower through shared attention under block-wise causal masking. The flow-matching objective regresses a velocity field transporting Gaussian noise to target chunks, with flow timesteps sampled from a Beta distribution skewed toward high-noise regimes. At inference, 10 Euler integration steps suffice because observation keys/values are cached across solver iterations. The action expert is a compact 370M-parameter module (hidden size scaled from 2048 to 1024).

Compact memory encoder. Extending Pi-MEM-style factorized attention (Torne et al., 4 Mar 2026), temporal causal attention is interleaved every four layers within the ViT, reusing the same QKV and output projections with fixed sinusoidal temporal encodings. In upper layers, only current-frame tokens are forwarded to the VLM, so token count matches single-frame inference while historical context remains baked into the representation. Critically, the design introduces zero new learnable parameters and reduces exactly to the pre-trained ViT when $K{=}1$, allowing direct initialization from Hy-Embodied-0.5 weights.

Data: Hy-UMI-10K

The pre-training corpus comprises more than 1M episodes and 10K hours across 70 tasks in six scene-based families (Laundry Room 28.5%, Kitchen 19.2%, Personal Care & Miscellaneous 13.8%, Dexterous/Tool-use 10.4%, Storage & Organization 10.0%, Cleaning 5.7%). Unlike standard UMI pipelines that recover gripper poses via on-board visual SLAM, the capture rig tracks each gripper with an external optical motion-capture system, producing 6-DoF trajectories at sub-millimeter precision in a globally consistent frame. Grippers follow the Changingtek CTAG2F90 industrial form factor, use rotary encoders for sub-millimeter openness measurement, attach ergonomically to the operator's fingers for direct force feedback, and some carry tip-mounted 6-axis force/torque sensors. The authors are explicit about the trade-off: this setup prioritizes label quality at the cost of inconvenient in-the-wild deployment, and the egocentric camera viewpoint still differs from robot-mounted cameras—an acknowledged limitation bearing on transfer.

Pre-training uses the full corpus with $K{=}1$, three camera views at 224×320 resolution, horizon $H{=}50$ at 10 Hz, 200K steps at global batch 1,024, and AdamW in bfloat16.

Supervised Fine-Tuning: Two Deployment Tracks

SFT initializes from the UMI pre-trained checkpoint with all parameters trainable and $K{=}6$. The evaluation protocol separates two tracks:

• Track A (intra-embodiment): teleoperated Dobot X-Trainer demonstrations (four tasks, 300 demos each, 18 hours) fine-tuned and evaluated on the same platform.
• Track B (cross-embodiment): task-specific UMI demonstrations only—no target-robot teleoperation—deployed on morphologically different robots: JAKA K1 (300 UMI demos, 1.2 hours) and Astribot S1 (200 UMI demos, 1.5 hours).

Simulation evaluation uses RoboTwin 2.0 (50 bimanual tasks, ~2.75K episodes, >6M frames). For humanoid deployment, a deterministic heuristic infers chassis, torso, and head poses from bimanual gripper targets rather than predicting them, keeping the learned 20-dimensional dual-arm interface unchanged. The appendix candidly notes that UMI data carries no intrinsic reachability guarantee for arbitrary morphologies; pre-deployment IK feasibility filtering and reachability bounding enforce distributional alignment without altering the policy.

Reinforcement Learning Post-Training: FlowPRO

FlowPRO addresses the failure modes of three existing post-training families: SFT/DAgger discards or weakly exploits negative rollouts; reward/value-based RL requires brittle dense-reward models for contact-rich manipulation; and preference-based methods such as Flow-DPO inherit plain-DPO reward hacking. Three design principles govern the method: exploit failures directly as per-state contrastive signals, avoid reward and critic models entirely, and anchor the implicit reward against magnitude explosion.

The RPRO loss uses the per-sample flow-matching regression loss as a tractable negative log-likelihood surrogate, yielding an implicit reward proxy $\mathcal{I}_t$0 substituted into the PRO pairwise objective. A symmetric proximal regularizer minimized at $\mathcal{I}_t$1 structurally forbids the reward-hacking pathology where the policy drifts from both preferred and dispreferred actions. A contrastive gradient-cancellation property—when $\mathcal{I}_t$2 the contrastive gradient vanishes—makes it safe to route SFT demonstrations through the same objective.

Preference pairs come from a teleoperated intervention-and-rollback pipeline: upon observing an error, the operator rewinds to state $\mathcal{I}_t$3, logs the executed segment as negative, and records a corrective demonstration as positive, yielding naturally paired trajectories sharing an initial state. Smooth Interpolation synthesizes missing counterpart actions via cubic Bézier positions, Slerp orientations, and linear gripper interpolation, converting sparse trajectory-level corrections into dense per-state tuples $\mathcal{I}_t$4. Batches mix newest preference pairs, historical pools, and SFT data at 70/15/15 proportions after round one. Total data scale is at most $\mathcal{I}_t$5 preference pairs per task.

Evaluation Results

RoboTwin 2.0 simulation. HyVLA achieves 90.9% Clean / 90.1% Randomized average success over 100 stochastic rollouts per task across the 50-task suite—the best reported among eight compared systems. It exceeds $\mathcal{I}_t$6 by 25.0 and 31.7 points respectively, $\mathcal{I}_t$7 by 8.2 and 13.3 points, and the strongest competitor JoyAI-RA by 0.4 and 0.8 points. Removal ablations attribute consistent gains to both components: dropping the memory encoder costs 2.1/1.5 points, and further dropping UMI pre-training costs another 0.7 points. The authors concede the UMI gain in simulation is modest given the visual domain gap, contrasting it with its larger real-world benefit.

Method	Clean	Randomized
$\mathcal{I}_t$8	65.9	58.4
ABot-M0	81.2	80.4
$\mathcal{I}_t$9	82.7	76.8
starVLA	88.2	88.3
JoyAI-RA	90.5	89.3
HyVLA	90.9	90.1

Per-task results reveal weak spots: hanging mug (37/33%), turn switch (50/49%), open microwave (72/64%), and place object cabinet (78/83%) fall well below the suite average, indicating that deformable-object hanging and certain contact-rich skills remain unsolved.

Real-world SFT. Across six bimanual tasks on three platforms, UMI pre-training shows its clearest value at precision-critical bottlenecks: on Fold and Store Glasses and Zip Up the Pen Case, failures concentrate at sub-centimeter positioning moments (folding temples without slipping, pinching the zipper slider), and pre-training sharpens predictions precisely at these steps. On Track B, where no target-robot data exists during fine-tuning, any gain must derive from the pre-training prior—and HyVLA markedly outperforms identically post-trained $n{\times}K$0 and $n{\times}K$1 baselines on both JAKA K1 and Astribot S1. This is the paper's strongest claim regarding cross-embodiment transfer: high-fidelity UMI pre-training equips the model with embodiment-agnostic action priors sufficient to make small UMI fine-tuning sets deployable on unseen platforms.

Force-modality validation. On a Unitree G1, augmenting the action expert with two lightweight TCN encoders (~2M parameters) over 50-step F/T windows enables reliable selection of the lighter of two boxes whose positions are randomized—ruling out spatial-memory shortcuts and confirming that tip force signals from the UMI rig provide actionable non-visual cues.

FlowPRO post-training. After three iterative rounds on four long-horizon bimanual tasks (Bottle, Cap, USB, Zip), RPRO reaches near-ceiling success rates with reduced completion times, dominating both DAgger and a re-implemented advantage-conditioned $n{\times}K$2* baseline on identical preference data:

Method	Bottle SR	Cap SR	USB SR	Zip SR
DAgger	93±2.1%	88±1.8%	86±2.4%	83±2.0%
$n{\times}K$3*	95±1.5%	95±1.2%	95±1.4%	89±1.6%
RPRO	99±0.6%	99±0.7%	98±0.9%	94±1.1%

RPRO also attains the shortest completion times on every task (16–37 s vs. 23–55 s for baselines). The authors attribute the margin over $n{\times}K$4* to direct injection of the preference signal into the action-generation loss rather than indirect conditioning-token pressure diluted by VLM context. Results are averaged over 3 training seeds with 100 randomized rollouts each.

Deployment

The deployment stack preserves the embodiment-agnostic contract through three components. Platform mapping composes relative SE(3) predictions with forward kinematics and solves IK per target robot, with a cached chassis transform and heuristic torso/head inference for floating-base humanoids. Asynchronous producer–consumer execution overlaps backbone inference with servo-rate command dispatch through a thread-safe action buffer. Latency-aware cubic Bézier stitching discards stale chunk prefixes (controlled by truncation ratio $n{\times}K$5), selects an interior reconnection point ($n{\times}K$6), and constructs a $n{\times}K$7-continuous connector whose inner control points align with historical and future tangents ($n{\times}K$8-scaled). The stitcher is policy-agnostic, training-free, and applicable to Cartesian and joint-space control—in contrast to learned refinement approaches such as Real-Time Chunking (Jana et al., 10 Mar 2025) and VLASH (Tang et al., 30 Nov 2025).

Limitations and Open Questions

The paper is explicit about several constraints. Motion-capture-based collection trades wild deployability for label precision, motivating exoskeleton-based collection as future work; the marginal value of label precision itself is unquantified, though the authors propose controlled noise-injection studies on Hy-UMI-10K. The egocentric-versus-robot-mounted camera gap remains unaddressed beyond noting the need for systematic augmentation studies. Execution cadence—task completion speed under safety constraints—is identified as insufficiently fast for practical deployment, likely requiring combined deployment-time adaptation and RL. Most notably, the authors state plainly that HyVLA does not study zero-shot generalization because current data scale does not support such claims, distinguishing their position from systems reporting early emergent zero-shot behavior. How to rigorously evaluate such capabilities, and how evaluation can drive model iteration, is left open. A 2,000-hour UMI subset will be released to support community study of these questions.

Conclusion

HyVLA demonstrates that treating the VLA as one component of a co-designed stack—high-fidelity hand-held data, an embodied-native MoT backbone with parameter-free temporal memory, a kinematics-decoupled delta-chunk interface, critic-free preference post-training, and asynchronous Bézier-smoothed execution—yields measurable gains at every stage: state-of-the-art RoboTwin 2.0 performance (90.9%/90.1%), UMI-only cross-embodiment transfer to unseen platforms including a humanoid, near-ceiling real-world success rates (98–99%) after reward-free RL refinement, and validated force-aware manipulation. The empirical evidence supports the paper's thesis that data quality, action representation, and deployment timing deserve first-class design status alongside model scale, while the conceded gaps in zero-shot capability, execution speed, and collection scalability define the concrete open problems the released assets are intended to help resolve.