---
title: 'RhinoVLA: Token-Efficient VLA for Real-Time Robotics'
url: https://www.emergentmind.com/topics/rhinovla
type: topic
---

# RhinoVLA: Token-Efficient VLA for Real-Time Robotics

Searching arXiv for RhinoVLA and closely related VLA papers for accurate contextual citations.
RhinoVLA is a vision–language–action (VLA) model explicitly designed for real-time robotic manipulation on edge hardware, with a particular focus on Huixi’s R1 edge SoC. It combines a token-efficient Qwen3-VL vision–language backbone with a continuous Action Expert, and it introduces a unified interface for cross-robot learning based on a View Registry, a 72D physical state–action slot space, and robot-instance LoRA [2606.07383]. The system is presented as a deployment-oriented alternative to heavier VLA configurations whose latency is dominated not merely by parameter count but by multimodal token volume, especially in GEMM-dominated projection operators. In reported experiments, RhinoVLA achieves downstream performance comparable to $\pi_0.5$ at a similar parameter scale while reaching 11.69 Hz end-to-end inference on Huixi R1, thereby meeting the 10 Hz real-time closed-loop control target [2606.07383].

## 1. Concept and problem setting

RhinoVLA is situated within the recent VLA line of work in which a vision–language model provides perceptual and instruction-conditioned context and a policy module produces robot actions. The RhinoVLA report frames the central deployment bottleneck as token-induced latency: when model dimensions are fixed, the cost of GEMM-dominated projection operators scales linearly with the number of input tokens, making visual and context tokens a major obstacle to real-time inference on edge platforms [2606.07383].

The technical report makes this observation explicit using the linear layer formulation
$$
Y = XW,
$$
with
$$
X \in \mathbb{R}^{B \times S \times d_{\text{in}}}, \quad
W \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}, \quad
Y \in \mathbb{R}^{B \times S \times d_{\text{out}}},
$$
and approximate FLOPs
$$
\text{FLOPs} = 2 B S d_{\text{in}} d_{\text{out}}.
$$
Under fixed layer widths and depth, computation therefore grows linearly with the sequence length $S$, i.e. the total number of visual and language tokens [2606.07383]. This motivates a design strategy that prioritizes token efficiency rather than model shrinkage alone.

The report further places RhinoVLA against the background of contemporary VLAs such as $\pi_0$, $\pi_0.5$, GR00T N1, OpenVLA, CoT-VLA, SpatialVLA, WorldVLA, NORA, SmolVLA, and RDT-1B [2606.07383]. In that framing, RhinoVLA’s distinguishing emphasis is not only generalist manipulation but explicit algorithm–system co-design for edge deployment. A plausible implication is that the work is less a pure architecture paper than a joint study of model structure, control interface design, and hardware execution constraints.

## 2. Architecture and token-efficient design

RhinoVLA follows a two-module decomposition. The first module is a Qwen3-VL-2B backbone used for perception and instruction understanding; the second is a continuous Action Expert of approximately 0.40B parameters that generates continuous actions via flow matching over a unified 72-dimensional physical action/state space [2606.07383]. The Qwen3-VL component processes task text together with one or more images carrying explicit view tags, and it exposes the key–value cache of its last 18 layers to the Action Expert [2606.07383].

A central architectural choice is the use of a token-efficient Qwen3-VL backbone. Under 224×224 or 256×256 inputs, Qwen3-VL uses aggressive spatial merging, yielding approximately 64 merged visual tokens per image, whereas PaliGemma-224, used by $\pi_0.5$, uses 256 image tokens [2606.07383]. The report states that RhinoVLA therefore reduces visual tokens by 4× for each image relative to PaliGemma-based configurations at comparable scale [2606.07383]. Because multi-view robotic perception compounds token count across cameras, this reduction directly targets the dominant MLP projection cost identified in the latency analysis.

The Action Expert is described as a Qwen-style Transformer with 18 layers, hidden size 1024, MLP size 3072, 16 attention heads, and 8 KV heads using GQA [2606.07383]. At each denoising step it conditions on the VLM context, the current 72D robot state, state and action masks, a noisy action chunk, the flow-matching time $t$, and a robot-instance identifier [2606.07383]. The predictive map is written as
$$
\hat{v}_{\theta} = f_{\theta}(x_t, t, s, m_s, m_a, c_{\mathrm{vlm}}, r),
$$
where the target is the flow direction from Gaussian noise to the clean action chunk [2606.07383].

The interpolation process is
$$
x_t = (1-t)a + tz,\quad t \in [0,1],
$$
with clean action chunk $z \in \mathbb{R}^{H \times 72}$ and noise $a \sim \mathcal{N}(0,I)$ [2606.07383]. The masked flow-matching loss is given by
$$
\mathcal{L}_{\mathrm{FM}} =
\frac{\sum_{h,d} m_a(d)\, w(h,d)\,
\left\|\hat{v}_{\theta}(h,d) - (z(h,d)-a(h,d))\right\|_2^2}
{\sum_{h,d} m_a(d) w(h,d) + \epsilon},
$$
so that only active action slots contribute to supervision [2606.07383]. The report additionally notes direct supervision on the base Action Expert prediction and regularization on adapter residuals, to ensure that the shared policy captures common structure while robot-instance LoRA supplies residual corrections [2606.07383].

This design is presented not as capability pruning but as a backbone-family substitution: fewer visual tokens are used without discarding pretrained multimodal capability, and the Action Expert retains access to rich vision–language context through the last-18-layer KV cache [2606.07383].

## 3. Unified cross-robot interface

A major contribution of RhinoVLA is a unified interface intended to align heterogeneous robots under a shared policy [2606.07383]. The report identifies three main sources of heterogeneity: camera layouts and naming, state/action vector formats, and differences in controllable degrees of freedom and control conventions. RhinoVLA addresses these through the View Registry, the unified 72D physical state–action slot space with masks, and robot-instance LoRA [2606.07383].

The View Registry standardizes camera roles across datasets by mapping each camera field at preprocessing time to a role–modality tag from a fixed vocabulary, such as `[head|rgb]`, `[front|rgb]`, `[left_wrist|rgb]`, or `[head|depth]` [2606.07383]. These structured tags are presented together with images and task text to Qwen3-VL before tokenization [2606.07383]. The purpose is to make camera identity explicit rather than inferred from dataset-specific ordering, so that the model can learn consistent functional distinctions between, for example, wrist views for fine manipulation and head or front views for scene context.

The 72D physical slot space gives each action/state dimension a fixed physical meaning shared across robots. The report enumerates the slot groups as follows: D0–D6 for Arm 0 canonical joints, D7–D13 for Arm 1 canonical joints, D14–D15 for parallel grippers, D16–D31 for Hand 0 active DoF, D32–D47 for Hand 1 active DoF, D48–D50 for head/neck roll–pitch–yaw, D51–D52 for torso pitch and lift, D53–D54 for folded-leg mechanism joints, D55–D57 for waist roll–pitch–yaw, D58–D60 for base velocity command $(v_x, v_y, \text{yaw rate})$, and D61–D71 as reserved currently inactive slots [2606.07383]. State and action share this same slot layout, with state representing current values in canonical units and action representing desired targets over a chunk horizon [2606.07383].

Masking is integral to the interface. Binary masks $m_s, m_a \in \{0,1\}^{72}$ indicate valid state dimensions and valid action labels respectively [2606.07383]. Missing DoF are masked rather than encoded as zeros, and only positions with $m_a(d)=1$ enter the flow-matching loss [2606.07383]. This preserves semantic alignment even when robots differ substantially in embodiment.

Robot-instance LoRA addresses residual embodiment effects that remain even after semantic alignment, such as calibration differences, controller behavior, joint limits, or end-effector mechanics [2606.07383]. Each robot instance has a low-rank LoRA adapter inserted into the feed-forward network of each Action Expert layer, while the base Action Expert parameters remain shared [2606.07383]. At training time, the instance identifier $r$ activates the relevant adapter. The report argues that this preserves a shared 72D output interface, maintains a single deployment graph, and permits low-cost extension to new robots by adding a new LoRA and normalization statistics without modifying the backbone or slot definitions [2606.07383].

A concise summary of the interface components is given below.

| Component | Function | Reported role |
|---|---|---|
| View Registry | Standardizes camera roles via role–modality tags | Aligns heterogeneous visual observations |
| 72D slot space + masks | Provides shared physical meaning for state/action dimensions | Aligns heterogeneous action schemas |
| Robot-instance LoRA | Supplies robot-specific residual adaptation inside the Action Expert | Captures embodiment-dependent corrections |

The report’s analysis of LoRA residual similarity versus action-mask distance suggests that these adapters encode embodiment-dependent residual structure rather than merely dataset identifiers [2606.07383]. This suggests that the interface is intended as a physically grounded coordination layer between heterogeneous datasets and a single policy.

## 4. Training data and learning setup

RhinoVLA’s cross-robot pretraining uses real robot data from AgiBotWorld Beta, AgiBotWorld 2026, and joint-space subsets of Open-X Embodiment [2606.07383]. The covered platforms include AgiBot G1 and G2 robots as well as single-arm, dual-arm, parallel-gripper, and dexterous-hand configurations across many manipulation tasks [2606.07383]. Every demonstration is converted into the unified interface through camera tagging, semantic mapping of state/action variables into the 72D slot space, unit conversion, and application of masks [2606.07383]. The report states that only reliably mappable fields are used and that leg or foot joints are currently not supervised [2606.07383].

Training jointly optimizes the Qwen3-VL LoRA parameters, the shared Action Expert parameters, and the robot-instance LoRA parameters within the Action Expert [2606.07383]. Dataset sampling follows a $\pi$-style power-law balancing rule:
$$
p_i = \frac{N_i^{0.43}}{\sum_j N_j^{0.43}},
$$
where $N_i$ is the size of dataset $i$ [2606.07383]. This procedure is intended to balance contributions from differently sized sources during pretraining.

The flow-matching objective operates on action chunks in the unified 72D space, with the masked loss ensuring that invalid or unavailable dimensions do not contribute [2606.07383]. Because the VLM backbone and Action Expert are factorized, the report describes post-training adaptation as lightweight: for a target robot and task, the VLM backbone and VLM LoRA are frozen, most Action Expert weights are frozen, and only the target robot’s instance LoRA is fine-tuned on a small task-specific dataset [2606.07383]. This constitutes the principal adaptation mechanism reported for transferring the shared policy to downstream robotic settings.

The report’s framing implies a separation between shared multimodal semantics and robot-specific control residuals. A plausible implication is that this decomposition is intended to improve sample efficiency in downstream adaptation, although the report chiefly documents the mechanism rather than providing a generalized sample-efficiency law.

## 5. Deployment on Huixi R1

RhinoVLA is co-designed with Huixi R1, a 7 nm edge SoC with 500 TOPS INT8 compute, an 8-core SIMT architecture, and approximately 200 GB/s memory bandwidth [2606.07383]. The deployment stack combines hardware-aware compilation, mixed-precision execution, and parallel visual encoding [2606.07383]. The report emphasizes that these systems optimizations are not ancillary but necessary to reach the target control frequency.

Compilation optimization is described at three levels. At the operator level, the report adapts FlashAttention-style tiling to R1’s software-managed on-chip scratchpad memory, keeping K, Q, V tiles, online softmax statistics, and partial outputs on chip; this is reported to achieve greater than 80% of theoretical peak compute throughput on representative shapes [2606.07383]. At the graph level, Transformer-block operations such as normalization, linear layers, bias, activation, and residual steps are fused, with small parameters and intermediate activations kept in scratchpad memory to reduce off-chip traffic [2606.07383]. At the runtime level, kernels are decomposed into smaller tasks and scheduled across compute units according to compute intensity, scratchpad usage, and data movement so as to reduce stalls from small or memory-bound operations [2606.07383].

Mixed-precision execution uses INT8 weights and FP16 activations, i.e. W8A16 [2606.07383]. The report notes that naïve W8A16 can be inefficient because of dequantization overhead, so it implements a fused W8A16 GEMM in which sub-matrix tasks are distributed across 8 cores, offline weight layout is reorganized to balance DDR channel usage, and weight loading, INT8-to-FP16 conversion, and FP16 MAC are pipelined in a single kernel with per-channel scaling for accuracy [2606.07383]. On a key up\_proj layer of $\pi_0.5$, the paper reports 191 $\mu$s for W16A16 and 113 $\mu$s for the custom W8A16 kernel, corresponding to a 1.69× speedup and approximately 50.6% compute utilization [2606.07383].

Parallel visual encoding targets a deployment configuration with three camera streams: one head camera and two wrist cameras [2606.07383]. Rather than running single-image ViT inference sequentially as in a $\pi_0.5$-style baseline, RhinoVLA packs all three images into a single batch [2606.07383]. The measured vision latency decreases from 34.52 ms sequentially to 24.31 ms in the batched mode [2606.07383].

The final optimized end-to-end breakdown reported on R1 is 24.31 ms for the three-view vision encoder, 20.78 ms for the VLM backbone, 36.71 ms for the Action Expert, and 3.74 ms for other components, totaling 85.54 ms or 11.69 Hz [2606.07383]. The report explicitly states that this satisfies and exceeds the 10 Hz target for real-time closed-loop control [2606.07383].

## 6. Empirical results

The report presents results spanning pretraining diagnostics, held-out action prediction, simulation benchmarking on LIBERO, and real-robot experiments [2606.07383]. In pretraining diagnostics, the global flow-matching loss is reported to decrease steadily, while per-instance losses for AgiBot G1 and G2 show different magnitudes and decay patterns, which the authors interpret as embodiment-specific residual structure captured by instance LoRA [2606.07383].

In an ablation comparing the baseline without instance LoRA to the full model on held-out action prediction, the reported metrics change as follows: Masked FM Loss from 0.0192 to 0.0191, Arm MAE from 0.0446 to 0.0440, Yaw-rate MAE from 0.0195 to 0.0194, Gripper MAE from 0.1064 to 0.1056, and Base velocity MAE from 0.0187 to 0.0188 [2606.07383]. The report characterizes the gains as modest but consistent, strongest in arm and gripper dimensions where embodiment differences are largest [2606.07383].

On the LIBERO benchmark, RhinoVLA reports 93.0% on Spatial, 91.0% on Object, 93.4% on Goal, 82.4% on Long, and 90.0% average [2606.07383]. The technical report states that this outperforms many VLA and direct-policy baselines including Diffusion Policy, Octo, MDT, OpenVLA, SpatialVLA, WorldVLA, CoT-VLA, $\pi_0$-FAST, $\pi_0$, and NORA, while trailing $\pi_0.5$, which obtains a 96.9% average [2606.07383]. It also highlights LIBERO-Long, where RhinoVLA’s 82.4% exceeds $\pi_0$ by 9.4 points and $\pi_0$-FAST by 22.2 points [2606.07383]. The report notes that, unlike $\pi_0.5$, RhinoVLA’s LIBERO setup initializes from Qwen3-VL-2B-Instruct but trains the Action Expert from scratch on LIBERO only, without extra robot-specific pretraining beyond LIBERO [2606.07383].

The real-robot experiments cover Galbot G1 short-horizon single-arm pick-and-place, AgiBot G2 long-horizon pick-and-place sequences, and AgiBot G1 bimanual towel folding [2606.07383]. Two evaluation settings are used: “Seen,” where adaptation and evaluation occur on the same robot and workspace, and “Unseen,” where evaluation is performed on a different instance, workspace, or object distribution from adaptation [2606.07383]. Reported results include 100% success for both $\pi_0.5$ and RhinoVLA on a Galbot G1 red-bag-to-far-bin unseen task; 58% on AgiBot G2 seen three-step long sequence for RhinoVLA; 24% for RhinoVLA versus 18% for $\pi_0.5$ on the AgiBot G2 unseen setting; and 67% seen and 43% unseen for RhinoVLA on AgiBot G1 towel folding [2606.07383]. The report presents these as evidence that a single pretrained policy, adapted via instance LoRA, can be deployed across multiple robots and tasks, including deformable object manipulation [2606.07383].

The performance–efficiency claim of the paper is therefore twofold: downstream capability comparable to $\pi_0.5$ at similar parameter scale, and edge deployment at 11.69 Hz on Huixi R1 [2606.07383]. This suggests that RhinoVLA is best understood as a compromise point in the accuracy–latency design space rather than as a purely accuracy-maximizing system.

## 7. Position within the VLA literature and limitations

Within the broader VLA literature, the report situates RhinoVLA among systems such as $\pi_0$, $\pi_0.5$, OpenVLA, CoT-VLA, SpatialVLA, WorldVLA, NORA, SmolVLA, GR00T N1, and RDT-1B [2606.07383]. Its reported novelty lies in three linked emphases: token-efficiency as a first-class design objective, cross-robot unification through a physically grounded 72D interface, and deep algorithm–hardware co-design for a non-GPU edge SoC [2606.07383]. In that sense, RhinoVLA is not merely a smaller or faster baseline but a system-level attempt to reconcile generalist VLA structure with closed-loop execution constraints.

Several limitations are explicitly noted in the report. First, the work demonstrates inference on R1, while training on R1 remains future work; the authors state a plan to enable training on R1 to support on-device reinforcement learning and online improvement [2606.07383]. Second, the unified interface presently does not supervise leg or foot joints, and the fixed 72D space leaves D61–D71 reserved but inactive; future work may extend slot definitions and assign explicit semantics to the reserved slots [2606.07383]. Third, although RhinoVLA is competitive on LIBERO, $\pi_0.5$ still outperforms it overall, which the report partly attributes to richer multi-source pretraining for $\pi_0.5$ [2606.07383]. The report also indicates an intention to follow newer Qwen-series VLMs to further explore the efficiency–capability trade-off [2606.07383].

The technical report further states that the project will be open-sourced at `https://github.com/HuixiAI/RhinoVLA`, with code for the Qwen3-VL plus Action Expert architecture, the training pipeline with the unified interface, and deployment kernels with R1 integration [2606.07383]. Taken together, these elements define RhinoVLA as a deployment-oriented VLA framework in which token-efficient multimodal representation, physically grounded cross-robot alignment, and hardware-aware execution are treated as mutually dependent parts of a single system design [2606.07383].

Source: https://www.emergentmind.com/topics/rhinovla