Bi-VLA: Bilateral Control Imitation Learning

• Bi-VLA is a multi-task framework in robotic imitation learning that integrates force feedback from bilateral control, visual observations, and language instructions.
• It employs a Transformer-driven CVAE with FiLM-based fusion to combine joint states, RGB images from dual cameras, and natural language commands for precise action generation.
• Empirical evaluations show that using language to resolve visual ambiguities can boost task success rates up to 90% in challenging pick-and-place scenarios.

Bi-VLA is a name used in several recent robotics and multimodal-learning contexts, but in its exact 2025 usage it denotes “Bilateral Control-Based Imitation Learning via Vision-Language Fusion for Action Generation,” a framework that extends bilateral control-based imitation learning to handle more than one task within a single model by combining robot joint angle, velocity, and torque with visual observations and natural-language instructions (Kobayashi et al., 23 Sep 2025). The same label has also been used for a modular bimanual vision-language-action system for household dexterous manipulation (Gbagbe et al., 2024) and, in a different sense, for a bi-directional VLA model–expert collaboration framework (Xiang et al., 6 Mar 2025). This suggests that “Bi-VLA” is not a fully standardized designation across the literature, and its meaning depends on the research lineage in which it appears.

1. Terminology and scope

In the bilateral-control literature, Bi-VLA is defined as a framework for multi-task execution within a single model that fuses robot state from bilateral control, visual observations, and natural-language instructions (Kobayashi et al., 23 Sep 2025). In this sense, “Bi” refers to bilateral control, not to bidirectional transformers, bi-level optimization, or generic two-stream multimodality. The method is explicitly positioned as an extension of bilateral-control-based imitation learning rather than as an internet-scale generalist VLA.

Other papers use closely related but distinct names and meanings. A 2024 paper titled “Bi-VLA: Vision-Language-Action Model-Based System for Bimanual Robotic Dexterous Manipulations” uses Bi-VLA for a modular dual-arm household manipulation system in which an LLM generates Python code over predefined robot APIs and a VLM checks ingredient availability (Gbagbe et al., 2024). A 2025 paper on “VLA Model-Expert Collaboration for Bi-directional Manipulation Learning” uses Bi-VLA in the sense of a bi-directional learning loop between a VLA model and an expert policy (Xiang et al., 6 Mar 2025). Related terms also appear nearby in the literature without being literal matches: Bi-VLGM is a bi-level vision-language design for medical image segmentation rather than robotics (Wenting et al., 2023), and BiliVLA is a scene-aware vision-language-action model for autonomous biliary endoscopic navigation rather than a bilateral-control manipulation framework (Lin et al., 22 Jun 2026). A broader conceptual relative is KineVLA, which introduces a bi-level action decomposition for kinematics-aware VLA but does not name itself Bi-VLA (Han et al., 18 Mar 2026).

2. Problem setting and motivation

The 2025 Bi-VLA framework arises from robot imitation learning for manipulation, particularly contact-rich manipulation where force matters, not just motion (Kobayashi et al., 23 Sep 2025). Its immediate precursor is bilateral-control-based imitation learning, in which a human manipulates a leader robot and a follower robot mirrors the motion while feeding back interaction forces to the operator. This differs from unilateral teleoperation settings because demonstrations include not only motion trajectories but also force-related signals.

The motivating limitation is that conventional bilateral control methods, and even stronger recent variants such as Bi-ACT, are described as restricted to single-task execution. In practice, this implies a separate model per task. The paper treats this as a serious obstacle for service robots and collaborative robots expected to switch among tasks in dynamic environments. Vision and language are introduced as complementary modalities to overcome that restriction: vision grounds the policy in the current scene, while language specifies the intended task, especially when the scene itself is ambiguous.

The experiments are constructed to expose two distinct regimes. In the Two-Target task, vision alone is insufficient because the initial scene is identical for two different outcomes, so the instruction must disambiguate whether the robot should place the ball Up or Down. In the Two-Source task, vision alone is sufficient because the ball begins at one of two visible source positions, so language should not be necessary for correct source selection. This task design is central to the paper’s claim that language should help when visual information underdetermines the task and should remain largely neutral when vision already suffices (Kobayashi et al., 23 Sep 2025).

3. Architecture and multimodal representation

Bi-VLA fuses three input sources: follower joint states, RGB images, and a natural-language instruction (Kobayashi et al., 23 Sep 2025). The robot-side state uses joint angle, joint velocity, and torque from bilateral control. Each robot has 5 joints and 3 measurements per joint, so each robot contributes 15 features, and leader plus follower together yield a 30-dimensional state vector during logging. At inference, the model receives the most recent follower joint states together with images and language, and the supervised target is the leader robot joint trajectories including angles, velocities, and torques.

The visual input consists of RGB images from two cameras: an overhead camera and a gripper-mounted camera. Images are captured at 640 × 360 resolution and 100 Hz. Visual features are extracted by EfficientNet. The language instruction is encoded into a fixed-length embedding using SigLIP, and the paper compares this design against a variant using DistilBERT. Vision and language are fused through FiLM-based modulation, after which the fused representation is combined with robot state into a unified latent space.

The policy backbone is described as a Transformer-driven Conditional Variational Autoencoder (CVAE) that outputs action chunks (Kobayashi et al., 23 Sep 2025). The stepwise pipeline is: multimodal input acquisition; text encoding with SigLIP; visual encoding with EfficientNet; vision-language fusion using FiLM; fusion with robot state; Transformer encoder-decoder processing within a CVAE; and action chunk prediction. The model predicts leader joint trajectories over a chunk, including angles, velocities, and torques, and these predicted trajectories are converted into current commands by the bilateral control system and applied to the follower robot in real time. The reported architectural hyperparameters are 4 encoder layers and 7 decoder layers.

The sensing stack is also unusual relative to standard visuomotor imitation learning. Joint angles come from encoders, while external torques are estimated using a disturbance observer (DOB) and a reaction force observer (RFOB), so no dedicated force/torque sensor is required (Kobayashi et al., 23 Sep 2025). This preserves the force-aware character of bilateral-control imitation learning while adding scene grounding and instruction conditioning.

4. Bilateral control formulation and data pipeline

The demonstration system uses a four-channel bilateral control system governed by two explicit relations (Kobayashi et al., 23 Sep 2025):

$\theta_l - \theta_f = 0,$

$\tau_l + \tau_f = 0,$

where $\theta$ denotes joint angle, $\tau$ denotes torque, and the subscripts $l$ and $f$ denote leader and follower. The first equation expresses position synchronization between leader and follower, while the second expresses the action-reaction law through equal-and-opposite torques. The practical consequence is that the operator can command the follower while feeling contact forces in real time.

The control loop runs at 1000 Hz, while images are captured at 100 Hz. During collection, robot state, images, and language instructions are temporally aligned. The paper uses DABI to synchronize sensing rates and enlarge the dataset: robot signals are reduced from 1000 Hz to 100 Hz, then augmented. This expands the training data from 6 demonstrations to 60 for task-specific models and from 4 demonstrations to 40 for the mixed multi-task model.

The language instructions in the experiments are short task commands. For the Two-Target task they are “put ball upward” and “put ball downward.” For the Two-Source task they are “pick ball upward” and “pick ball downward.” In the standard task-specific setting, the paper collects six demonstrations per task, specifically 3 for Up and 3 for Down. In the multi-task reduced-data setting, Bi-VLA (SigLIP-Mix) is trained jointly across Two-Target and Two-Source using only four raw demonstrations total: one each for Target-Up, Target-Down, Source-Up, and Source-Down (Kobayashi et al., 23 Sep 2025).

The training objective is described as supervised reconstruction of action chunks, specifically reconstruction of ground-truth leader trajectories. However, the paper does not specify the exact loss form, does not print a CVAE ELBO, and does not provide the optimizer, learning rate, batch size, or latent dimensionality. This omission is one of the central reproducibility constraints of the method as published (Kobayashi et al., 23 Sep 2025).

5. Experimental evaluation and empirical findings

Evaluation is performed on real-robot pick-and-place tasks using ROBOTIS OpenManipulator-X arms, with 10 independent evaluation trials per condition (Kobayashi et al., 23 Sep 2025). A trial is counted as successful only if Pick, Move, and Place are all completed and there is no unintended object drop outside the target area. The principal comparisons are Bi-ACT, Bi-VLA (DistilBERT), Bi-VLA (SigLIP), and Bi-VLA (SigLIP-Mix).

On the Two-Target task, where language is essential because the visual scene is initially identical for both outcomes, the results are sharply differentiated. Bi-ACT achieves 100% for Up and 0% for Down, for an average overall success of 50%. Bi-VLA (DistilBERT) achieves 100% for Up and 20% for Down, averaging 60%. Bi-VLA (SigLIP) achieves 80% for Up and 100% for Down, averaging 90%. Bi-VLA (SigLIP-Mix) achieves 70% for Up and 70% for Down, averaging 70% (Kobayashi et al., 23 Sep 2025). The paper interprets these results as evidence that language is indispensable when vision is underdetermined and that the choice of language encoder materially affects grounding quality.

On the standard Two-Source task, where vision alone is sufficient, the vision-only baseline remains strongest but the multimodal model stays competitive. Bi-ACT achieves 90% for Up and 100% for Down, averaging 95%. Bi-VLA (SigLIP) achieves 100% for Up and 80% for Down, averaging 90%, and Bi-VLA (SigLIP-Mix) also achieves 100% for Up and 80% for Down, averaging 90% (Kobayashi et al., 23 Sep 2025). The paper treats this as a control condition showing that adding language does not catastrophically interfere when vision already disambiguates the task.

A more demanding generalization test evaluates the Two-Source policy in an unlearned 3-ball environment. Here Bi-ACT again collapses to a biased strategy, with 100% for Up and 0% for Down, averaging 50%. Bi-VLA (SigLIP) achieves 100% for Up and 50% for Down, averaging 75%, while Bi-VLA (SigLIP-Mix) achieves 90% for Up and 60% for Down, also averaging 75% (Kobayashi et al., 23 Sep 2025). The authors interpret this as evidence that language grounding can improve robustness when visual saliency degrades because of distractors.

The comparative pattern across these experiments supports three narrower claims. First, language is necessary when the task is visually ambiguous. Second, SigLIP is more effective than DistilBERT in this framework. Third, a single policy jointly trained across tasks can retain acceptable performance even under reduced data, though the mixed model does not match the best task-specific model on the most language-dependent condition (Kobayashi et al., 23 Sep 2025).

6. Significance, limitations, and related uses of the name

Within bilateral-control imitation learning, Bi-VLA is significant because it connects two previously separate emphases: force-aware bilateral demonstrations and vision-language task conditioning (Kobayashi et al., 23 Sep 2025). Relative to prior bilateral approaches, the main advance is not the replacement of force-sensitive control, but the move from single-task, task-specific models toward a single multimodal policy that can switch tasks using scene information and language commands. This suggests a path by which bilateral-control methods may become more semantically flexible without abandoning torque-aware state representations.

The published method also has clear limitations. The experiments cover only a small set of tasks and environments; the tasks are relatively simple pick-and-place tasks; more complex multi-step manipulation and more diverse object categories are left for future work; and the framework has not yet been validated on multiple robot platforms (Kobayashi et al., 23 Sep 2025). Reproducibility is additionally constrained because the paper does not provide the EfficientNet version, the SigLIP variant, embedding dimensions, latent dimension of the CVAE, action chunk size, observation window length, transformer hidden size, number of heads, exact loss formula, KL weighting, optimizer, learning rate, batch size, number of epochs, normalization details, inference latency, or the exact conversion from predicted trajectories to follower commands.

The broader literature shows that the string “Bi-VLA” now spans several neighboring meanings. In one direction, it names a modular bimanual household manipulation system built from recipe retrieval, Qwen-VL-based ingredient verification, LLM planning, and API-level execution rather than bilateral imitation learning (Gbagbe et al., 2024). In another, it refers to a bi-directional VLA model–expert collaboration framework in which a VLA and an expert alternate control and the interaction data are used to fine-tune the model (Xiang et al., 6 Mar 2025). Closely related work on bi-level and kinematics-aware VLA, such as KineVLA, separates goal-level invariance from kinematics-level variability through bi-level action representations and reasoning tokens (Han et al., 18 Mar 2026). Meanwhile, names such as BiliVLA designate still more specialized domains, including biliary endoscopic navigation (Lin et al., 22 Jun 2026). Taken together, these usages indicate that Bi-VLA functions less as a single canonical architecture name than as a context-sensitive label whose interpretation must be anchored to the specific paper or subfield under discussion.