RhinoVLA: Token-Efficient VLA for Real-Time Robotics

• RhinoVLA is a real-time vision–language–action system that integrates a token-efficient Qwen3-VL backbone with a continuous Action Expert to enable efficient robotic manipulation.
• It employs a unified 72D state–action interface with a standardized View Registry and robot-instance LoRA, aligning heterogeneous robots to a single policy framework.
• Through hardware-aware optimizations and mixed-precision execution, RhinoVLA achieves 11.69 Hz end-to-end inference on the Huixi R1 edge SoC, meeting real-time closed-loop control targets.

Searching arXiv for RhinoVLA and closely related VLA papers for accurate contextual citations. RhinoVLA is a vision–language–action (VLA) model explicitly designed for real-time robotic manipulation on edge hardware, with a particular focus on Huixi’s R1 edge SoC. It combines a token-efficient Qwen3-VL vision–language backbone with a continuous Action Expert, and it introduces a unified interface for cross-robot learning based on a View Registry, a 72D physical state–action slot space, and robot-instance LoRA (Intelligence et al., 5 Jun 2026). The system is presented as a deployment-oriented alternative to heavier VLA configurations whose latency is dominated not merely by parameter count but by multimodal token volume, especially in GEMM-dominated projection operators. In reported experiments, RhinoVLA achieves downstream performance comparable to $\pi_0.5$ at a similar parameter scale while reaching 11.69 Hz end-to-end inference on Huixi R1, thereby meeting the 10 Hz real-time closed-loop control target (Intelligence et al., 5 Jun 2026).

1. Concept and problem setting

RhinoVLA is situated within the recent VLA line of work in which a vision–LLM provides perceptual and instruction-conditioned context and a policy module produces robot actions. The RhinoVLA report frames the central deployment bottleneck as token-induced latency: when model dimensions are fixed, the cost of GEMM-dominated projection operators scales linearly with the number of input tokens, making visual and context tokens a major obstacle to real-time inference on edge platforms (Intelligence et al., 5 Jun 2026).

The technical report makes this observation explicit using the linear layer formulation

$Y = XW,$

with

$$X \in \mathbb{R}^{B \times S \times d_{\text{in}}}, \quad W \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}, \quad Y \in \mathbb{R}^{B \times S \times d_{\text{out}}},$$

and approximate FLOPs

$\text{FLOPs} = 2 B S d_{\text{in}} d_{\text{out}}.$

Under fixed layer widths and depth, computation therefore grows linearly with the sequence length $S$, i.e. the total number of visual and language tokens (Intelligence et al., 5 Jun 2026). This motivates a design strategy that prioritizes token efficiency rather than model shrinkage alone.

The report further places RhinoVLA against the background of contemporary VLAs such as $\pi_0$, $\pi_0.5$, GR00T N1, OpenVLA, CoT-VLA, SpatialVLA, WorldVLA, NORA, SmolVLA, and RDT-1B (Intelligence et al., 5 Jun 2026). In that framing, RhinoVLA’s distinguishing emphasis is not only generalist manipulation but explicit algorithm–system co-design for edge deployment. A plausible implication is that the work is less a pure architecture paper than a joint study of model structure, control interface design, and hardware execution constraints.

2. Architecture and token-efficient design

RhinoVLA follows a two-module decomposition. The first module is a Qwen3-VL-2B backbone used for perception and instruction understanding; the second is a continuous Action Expert of approximately 0.40B parameters that generates continuous actions via flow matching over a unified 72-dimensional physical action/state space (Intelligence et al., 5 Jun 2026). The Qwen3-VL component processes task text together with one or more images carrying explicit view tags, and it exposes the key–value cache of its last 18 layers to the Action Expert (Intelligence et al., 5 Jun 2026).

A central architectural choice is the use of a token-efficient Qwen3-VL backbone. Under 224×224 or 256×256 inputs, Qwen3-VL uses aggressive spatial merging, yielding approximately 64 merged visual tokens per image, whereas PaliGemma-224, used by $\pi_0.5$, uses 256 image tokens (Intelligence et al., 5 Jun 2026). The report states that RhinoVLA therefore reduces visual tokens by 4× for each image relative to PaliGemma-based configurations at comparable scale (Intelligence et al., 5 Jun 2026). Because multi-view robotic perception compounds token count across cameras, this reduction directly targets the dominant MLP projection cost identified in the latency analysis.

The Action Expert is described as a Qwen-style Transformer with 18 layers, hidden size 1024, MLP size 3072, 16 attention heads, and 8 KV heads using GQA (Intelligence et al., 5 Jun 2026). At each denoising step it conditions on the VLM context, the current 72D robot state, state and action masks, a noisy action chunk, the flow-matching time $t$, and a robot-instance identifier (Intelligence et al., 5 Jun 2026). The predictive map is written as

$$\hat{v}_{\theta} = f_{\theta}(x_t, t, s, m_s, m_a, c_{\mathrm{vlm}}, r),$$

where the target is the flow direction from Gaussian noise to the clean action chunk (Intelligence et al., 5 Jun 2026).

The interpolation process is

$Y = XW,$0

with clean action chunk $Y = XW,$1 and noise $Y = XW,$2 (Intelligence et al., 5 Jun 2026). The masked flow-matching loss is given by

$Y = XW,$3

so that only active action slots contribute to supervision (Intelligence et al., 5 Jun 2026). The report additionally notes direct supervision on the base Action Expert prediction and regularization on adapter residuals, to ensure that the shared policy captures common structure while robot-instance LoRA supplies residual corrections (Intelligence et al., 5 Jun 2026).

This design is presented not as capability pruning but as a backbone-family substitution: fewer visual tokens are used without discarding pretrained multimodal capability, and the Action Expert retains access to rich vision–language context through the last-18-layer KV cache (Intelligence et al., 5 Jun 2026).

3. Unified cross-robot interface

A major contribution of RhinoVLA is a unified interface intended to align heterogeneous robots under a shared policy (Intelligence et al., 5 Jun 2026). The report identifies three main sources of heterogeneity: camera layouts and naming, state/action vector formats, and differences in controllable degrees of freedom and control conventions. RhinoVLA addresses these through the View Registry, the unified 72D physical state–action slot space with masks, and robot-instance LoRA (Intelligence et al., 5 Jun 2026).

The View Registry standardizes camera roles across datasets by mapping each camera field at preprocessing time to a role–modality tag from a fixed vocabulary, such as [head|rgb], [front|rgb], [left_wrist|rgb], or [head|depth] (Intelligence et al., 5 Jun 2026). These structured tags are presented together with images and task text to Qwen3-VL before tokenization (Intelligence et al., 5 Jun 2026). The purpose is to make camera identity explicit rather than inferred from dataset-specific ordering, so that the model can learn consistent functional distinctions between, for example, wrist views for fine manipulation and head or front views for scene context.

The 72D physical slot space gives each action/state dimension a fixed physical meaning shared across robots. The report enumerates the slot groups as follows: D0–D6 for Arm 0 canonical joints, D7–D13 for Arm 1 canonical joints, D14–D15 for parallel grippers, D16–D31 for Hand 0 active DoF, D32–D47 for Hand 1 active DoF, D48–D50 for head/neck roll–pitch–yaw, D51–D52 for torso pitch and lift, D53–D54 for folded-leg mechanism joints, D55–D57 for waist roll–pitch–yaw, D58–D60 for base velocity command $Y = XW,$4, and D61–D71 as reserved currently inactive slots (Intelligence et al., 5 Jun 2026). State and action share this same slot layout, with state representing current values in canonical units and action representing desired targets over a chunk horizon (Intelligence et al., 5 Jun 2026).

Masking is integral to the interface. Binary masks $Y = XW,$5 indicate valid state dimensions and valid action labels respectively (Intelligence et al., 5 Jun 2026). Missing DoF are masked rather than encoded as zeros, and only positions with $Y = XW,$6 enter the flow-matching loss (Intelligence et al., 5 Jun 2026). This preserves semantic alignment even when robots differ substantially in embodiment.

Robot-instance LoRA addresses residual embodiment effects that remain even after semantic alignment, such as calibration differences, controller behavior, joint limits, or end-effector mechanics (Intelligence et al., 5 Jun 2026). Each robot instance has a low-rank LoRA adapter inserted into the feed-forward network of each Action Expert layer, while the base Action Expert parameters remain shared (Intelligence et al., 5 Jun 2026). At training time, the instance identifier $Y = XW,$7 activates the relevant adapter. The report argues that this preserves a shared 72D output interface, maintains a single deployment graph, and permits low-cost extension to new robots by adding a new LoRA and normalization statistics without modifying the backbone or slot definitions (Intelligence et al., 5 Jun 2026).

A concise summary of the interface components is given below.

Component	Function	Reported role
View Registry	Standardizes camera roles via role–modality tags	Aligns heterogeneous visual observations
72D slot space + masks	Provides shared physical meaning for state/action dimensions	Aligns heterogeneous action schemas
Robot-instance LoRA	Supplies robot-specific residual adaptation inside the Action Expert	Captures embodiment-dependent corrections

The report’s analysis of LoRA residual similarity versus action-mask distance suggests that these adapters encode embodiment-dependent residual structure rather than merely dataset identifiers (Intelligence et al., 5 Jun 2026). This suggests that the interface is intended as a physically grounded coordination layer between heterogeneous datasets and a single policy.

4. Training data and learning setup

RhinoVLA’s cross-robot pretraining uses real robot data from AgiBotWorld Beta, AgiBotWorld 2026, and joint-space subsets of Open-X Embodiment (Intelligence et al., 5 Jun 2026). The covered platforms include AgiBot G1 and G2 robots as well as single-arm, dual-arm, parallel-gripper, and dexterous-hand configurations across many manipulation tasks (Intelligence et al., 5 Jun 2026). Every demonstration is converted into the unified interface through camera tagging, semantic mapping of state/action variables into the 72D slot space, unit conversion, and application of masks (Intelligence et al., 5 Jun 2026). The report states that only reliably mappable fields are used and that leg or foot joints are currently not supervised (Intelligence et al., 5 Jun 2026).

Training jointly optimizes the Qwen3-VL LoRA parameters, the shared Action Expert parameters, and the robot-instance LoRA parameters within the Action Expert (Intelligence et al., 5 Jun 2026). Dataset sampling follows a $Y = XW,$8-style power-law balancing rule:

$Y = XW,$9

where $$X \in \mathbb{R}^{B \times S \times d_{\text{in}}}, \quad W \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}, \quad Y \in \mathbb{R}^{B \times S \times d_{\text{out}}},$$0 is the size of dataset $$X \in \mathbb{R}^{B \times S \times d_{\text{in}}}, \quad W \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}, \quad Y \in \mathbb{R}^{B \times S \times d_{\text{out}}},$$1 (Intelligence et al., 5 Jun 2026). This procedure is intended to balance contributions from differently sized sources during pretraining.

The flow-matching objective operates on action chunks in the unified 72D space, with the masked loss ensuring that invalid or unavailable dimensions do not contribute (Intelligence et al., 5 Jun 2026). Because the VLM backbone and Action Expert are factorized, the report describes post-training adaptation as lightweight: for a target robot and task, the VLM backbone and VLM LoRA are frozen, most Action Expert weights are frozen, and only the target robot’s instance LoRA is fine-tuned on a small task-specific dataset (Intelligence et al., 5 Jun 2026). This constitutes the principal adaptation mechanism reported for transferring the shared policy to downstream robotic settings.

The report’s framing implies a separation between shared multimodal semantics and robot-specific control residuals. A plausible implication is that this decomposition is intended to improve sample efficiency in downstream adaptation, although the report chiefly documents the mechanism rather than providing a generalized sample-efficiency law.

5. Deployment on Huixi R1

RhinoVLA is co-designed with Huixi R1, a 7 nm edge SoC with 500 TOPS INT8 compute, an 8-core SIMT architecture, and approximately 200 GB/s memory bandwidth (Intelligence et al., 5 Jun 2026). The deployment stack combines hardware-aware compilation, mixed-precision execution, and parallel visual encoding (Intelligence et al., 5 Jun 2026). The report emphasizes that these systems optimizations are not ancillary but necessary to reach the target control frequency.

Compilation optimization is described at three levels. At the operator level, the report adapts FlashAttention-style tiling to R1’s software-managed on-chip scratchpad memory, keeping K, Q, V tiles, online softmax statistics, and partial outputs on chip; this is reported to achieve greater than 80% of theoretical peak compute throughput on representative shapes (Intelligence et al., 5 Jun 2026). At the graph level, Transformer-block operations such as normalization, linear layers, bias, activation, and residual steps are fused, with small parameters and intermediate activations kept in scratchpad memory to reduce off-chip traffic (Intelligence et al., 5 Jun 2026). At the runtime level, kernels are decomposed into smaller tasks and scheduled across compute units according to compute intensity, scratchpad usage, and data movement so as to reduce stalls from small or memory-bound operations (Intelligence et al., 5 Jun 2026).

Mixed-precision execution uses INT8 weights and FP16 activations, i.e. W8A16 (Intelligence et al., 5 Jun 2026). The report notes that naïve W8A16 can be inefficient because of dequantization overhead, so it implements a fused W8A16 GEMM in which sub-matrix tasks are distributed across 8 cores, offline weight layout is reorganized to balance DDR channel usage, and weight loading, INT8-to-FP16 conversion, and FP16 MAC are pipelined in a single kernel with per-channel scaling for accuracy (Intelligence et al., 5 Jun 2026). On a key up_proj layer of $$X \in \mathbb{R}^{B \times S \times d_{\text{in}}}, \quad W \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}, \quad Y \in \mathbb{R}^{B \times S \times d_{\text{out}}},$$2, the paper reports 191 $$X \in \mathbb{R}^{B \times S \times d_{\text{in}}}, \quad W \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}, \quad Y \in \mathbb{R}^{B \times S \times d_{\text{out}}},$$3s for W16A16 and 113 $$X \in \mathbb{R}^{B \times S \times d_{\text{in}}}, \quad W \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}, \quad Y \in \mathbb{R}^{B \times S \times d_{\text{out}}},$$4s for the custom W8A16 kernel, corresponding to a 1.69× speedup and approximately 50.6% compute utilization (Intelligence et al., 5 Jun 2026).

Parallel visual encoding targets a deployment configuration with three camera streams: one head camera and two wrist cameras (Intelligence et al., 5 Jun 2026). Rather than running single-image ViT inference sequentially as in a $$X \in \mathbb{R}^{B \times S \times d_{\text{in}}}, \quad W \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}, \quad Y \in \mathbb{R}^{B \times S \times d_{\text{out}}},$$5-style baseline, RhinoVLA packs all three images into a single batch (Intelligence et al., 5 Jun 2026). The measured vision latency decreases from 34.52 ms sequentially to 24.31 ms in the batched mode (Intelligence et al., 5 Jun 2026).

The final optimized end-to-end breakdown reported on R1 is 24.31 ms for the three-view vision encoder, 20.78 ms for the VLM backbone, 36.71 ms for the Action Expert, and 3.74 ms for other components, totaling 85.54 ms or 11.69 Hz (Intelligence et al., 5 Jun 2026). The report explicitly states that this satisfies and exceeds the 10 Hz target for real-time closed-loop control (Intelligence et al., 5 Jun 2026).

6. Empirical results

The report presents results spanning pretraining diagnostics, held-out action prediction, simulation benchmarking on LIBERO, and real-robot experiments (Intelligence et al., 5 Jun 2026). In pretraining diagnostics, the global flow-matching loss is reported to decrease steadily, while per-instance losses for AgiBot G1 and G2 show different magnitudes and decay patterns, which the authors interpret as embodiment-specific residual structure captured by instance LoRA (Intelligence et al., 5 Jun 2026).

In an ablation comparing the baseline without instance LoRA to the full model on held-out action prediction, the reported metrics change as follows: Masked FM Loss from 0.0192 to 0.0191, Arm MAE from 0.0446 to 0.0440, Yaw-rate MAE from 0.0195 to 0.0194, Gripper MAE from 0.1064 to 0.1056, and Base velocity MAE from 0.0187 to 0.0188 (Intelligence et al., 5 Jun 2026). The report characterizes the gains as modest but consistent, strongest in arm and gripper dimensions where embodiment differences are largest (Intelligence et al., 5 Jun 2026).

On the LIBERO benchmark, RhinoVLA reports 93.0% on Spatial, 91.0% on Object, 93.4% on Goal, 82.4% on Long, and 90.0% average (Intelligence et al., 5 Jun 2026). The technical report states that this outperforms many VLA and direct-policy baselines including Diffusion Policy, Octo, MDT, OpenVLA, SpatialVLA, WorldVLA, CoT-VLA, $$X \in \mathbb{R}^{B \times S \times d_{\text{in}}}, \quad W \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}, \quad Y \in \mathbb{R}^{B \times S \times d_{\text{out}}},$$6-FAST, $$X \in \mathbb{R}^{B \times S \times d_{\text{in}}}, \quad W \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}, \quad Y \in \mathbb{R}^{B \times S \times d_{\text{out}}},$$7, and NORA, while trailing $$X \in \mathbb{R}^{B \times S \times d_{\text{in}}}, \quad W \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}, \quad Y \in \mathbb{R}^{B \times S \times d_{\text{out}}},$$8, which obtains a 96.9% average (Intelligence et al., 5 Jun 2026). It also highlights LIBERO-Long, where RhinoVLA’s 82.4% exceeds $$X \in \mathbb{R}^{B \times S \times d_{\text{in}}}, \quad W \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}, \quad Y \in \mathbb{R}^{B \times S \times d_{\text{out}}},$$9 by 9.4 points and $\text{FLOPs} = 2 B S d_{\text{in}} d_{\text{out}}.$0-FAST by 22.2 points (Intelligence et al., 5 Jun 2026). The report notes that, unlike $\text{FLOPs} = 2 B S d_{\text{in}} d_{\text{out}}.$1, RhinoVLA’s LIBERO setup initializes from Qwen3-VL-2B-Instruct but trains the Action Expert from scratch on LIBERO only, without extra robot-specific pretraining beyond LIBERO (Intelligence et al., 5 Jun 2026).

The real-robot experiments cover Galbot G1 short-horizon single-arm pick-and-place, AgiBot G2 long-horizon pick-and-place sequences, and AgiBot G1 bimanual towel folding (Intelligence et al., 5 Jun 2026). Two evaluation settings are used: “Seen,” where adaptation and evaluation occur on the same robot and workspace, and “Unseen,” where evaluation is performed on a different instance, workspace, or object distribution from adaptation (Intelligence et al., 5 Jun 2026). Reported results include 100% success for both $\text{FLOPs} = 2 B S d_{\text{in}} d_{\text{out}}.$2 and RhinoVLA on a Galbot G1 red-bag-to-far-bin unseen task; 58% on AgiBot G2 seen three-step long sequence for RhinoVLA; 24% for RhinoVLA versus 18% for $\text{FLOPs} = 2 B S d_{\text{in}} d_{\text{out}}.$3 on the AgiBot G2 unseen setting; and 67% seen and 43% unseen for RhinoVLA on AgiBot G1 towel folding (Intelligence et al., 5 Jun 2026). The report presents these as evidence that a single pretrained policy, adapted via instance LoRA, can be deployed across multiple robots and tasks, including deformable object manipulation (Intelligence et al., 5 Jun 2026).

The performance–efficiency claim of the paper is therefore twofold: downstream capability comparable to $\text{FLOPs} = 2 B S d_{\text{in}} d_{\text{out}}.$4 at similar parameter scale, and edge deployment at 11.69 Hz on Huixi R1 (Intelligence et al., 5 Jun 2026). This suggests that RhinoVLA is best understood as a compromise point in the accuracy–latency design space rather than as a purely accuracy-maximizing system.

7. Position within the VLA literature and limitations

Within the broader VLA literature, the report situates RhinoVLA among systems such as $\text{FLOPs} = 2 B S d_{\text{in}} d_{\text{out}}.$5, $\text{FLOPs} = 2 B S d_{\text{in}} d_{\text{out}}.$6, OpenVLA, CoT-VLA, SpatialVLA, WorldVLA, NORA, SmolVLA, GR00T N1, and RDT-1B (Intelligence et al., 5 Jun 2026). Its reported novelty lies in three linked emphases: token-efficiency as a first-class design objective, cross-robot unification through a physically grounded 72D interface, and deep algorithm–hardware co-design for a non-GPU edge SoC (Intelligence et al., 5 Jun 2026). In that sense, RhinoVLA is not merely a smaller or faster baseline but a system-level attempt to reconcile generalist VLA structure with closed-loop execution constraints.

Several limitations are explicitly noted in the report. First, the work demonstrates inference on R1, while training on R1 remains future work; the authors state a plan to enable training on R1 to support on-device reinforcement learning and online improvement (Intelligence et al., 5 Jun 2026). Second, the unified interface presently does not supervise leg or foot joints, and the fixed 72D space leaves D61–D71 reserved but inactive; future work may extend slot definitions and assign explicit semantics to the reserved slots (Intelligence et al., 5 Jun 2026). Third, although RhinoVLA is competitive on LIBERO, $\text{FLOPs} = 2 B S d_{\text{in}} d_{\text{out}}.$7 still outperforms it overall, which the report partly attributes to richer multi-source pretraining for $\text{FLOPs} = 2 B S d_{\text{in}} d_{\text{out}}.$8 (Intelligence et al., 5 Jun 2026). The report also indicates an intention to follow newer Qwen-series VLMs to further explore the efficiency–capability trade-off (Intelligence et al., 5 Jun 2026).

The technical report further states that the project will be open-sourced at https://github.com/HuixiAI/RhinoVLA, with code for the Qwen3-VL plus Action Expert architecture, the training pipeline with the unified interface, and deployment kernels with R1 integration (Intelligence et al., 5 Jun 2026). Taken together, these elements define RhinoVLA as a deployment-oriented VLA framework in which token-efficient multimodal representation, physically grounded cross-robot alignment, and hardware-aware execution are treated as mutually dependent parts of a single system design (Intelligence et al., 5 Jun 2026).