Vision-Language-Kinematics: Bridging Perception & Control

• VLK is a framework that bridges visual and language understanding with kinematic data to generate physically feasible robot actions.
• It employs diverse representation strategies—ranging from implicit motion vectors to explicit pose estimation—to address tasks in manipulation, navigation, and control.
• Architectures leveraging VLK integrate multimodal encoders, attention mechanisms, and runtime corrections to enhance motion prediction and ensure actionable execution.

Vision-Language-Kinematics (VLK) is a class of formulations that couple visual perception and natural-language conditioning with kinematic structure, so that multimodal models do not merely predict actions but do so through representations grounded in motion, pose, geometry, or physically feasible trajectories. Across recent work, the term spans several distinct but related uses: implicit motion-based temporal grounding for manipulation, camera-space or world-space waypoint prediction, SE(3) keyframe generation from multi-view RGB-D, kinematics-aware decoding and control acceleration, spatio-temporal reasoning over metric object motion in video, and whole-body trajectory prediction for humanoid loco-manipulation (Lin et al., 10 Dec 2025, Argus et al., 2 Jul 2025, Singh et al., 1 Jun 2025, Zheng et al., 2 Mar 2026, Ko et al., 25 Mar 2025, Wang et al., 29 Jun 2026). Survey treatments place VLK as the bridge between high-level multimodal understanding and embodiment-specific coordinate frames, poses, joint states, Jacobians, and constraint sets required for execution, while also distinguishing “pure VLA” pipelines, where kinematics is largely implicit, from hybrid systems that invoke explicit IK, planners, or safety layers (Zhang et al., 23 Sep 2025, Din et al., 14 Jul 2025).

1. Conceptual scope and definitions

VLK does not denote a single architecture. In survey form, it is the representational and algorithmic bridge between vision-language understanding and executable robot kinematics, including coordinate frames, SE(3) poses, joint-space models, Jacobians, and constraints (Zhang et al., 23 Sep 2025). A closely related review frames VLK as a kinematics-centric view of VLA in which joint positions $q$, velocities $\dot{q}$, end-effector pose $T \in SE(3)$, gripper state, contact state, and force/torque signals are first-class alongside images and language (Din et al., 14 Jul 2025).

Recent primary papers instantiate that bridge in markedly different ways. HiF-VLA treats motion itself as a compact kinematic proxy, using MPEG-4 motion vectors rather than explicit joint states to connect past and future visual states for long-horizon manipulation (Lin et al., 10 Dec 2025). cVLA reframes VLA as a camera-space VLK problem by predicting two robot end-effector keyposes directly in image-frame coordinates and delegating embodiment-specific conversion to a robot-side IK and planning layer (Argus et al., 2 Jul 2025). OG-VLA outputs quasi-static 6-DoF end-effector keyframes in the scene/world frame after canonical orthographic rendering of multi-view RGB-D point clouds (Singh et al., 1 Jun 2025). KERV uses kinematics not primarily as the policy output space, but as a runtime rectifier and regulator for speculative decoding in token-domain VLAs (Zheng et al., 2 Mar 2026). ST-VLM extends the term beyond control: it defines VLK as the capacity of a multimodal model to infer traveled distance, speed, movement direction, and relative motion from raw videos in real-world units (Ko et al., 25 Mar 2025). VLK for humanoids goes further by mapping egocentric RGB, language, and current kinematic state to short-horizon whole-body trajectories and wrist-object contact labels (Wang et al., 29 Jun 2026). AerialVLA similarly treats language-grounded navigation as direct prediction of continuous 3-DoF kinematic commands plus an intrinsic landing signal (Xu et al., 15 Mar 2026).

A common implication across these formulations is that VLK is less a narrow model family than a design principle: multimodal policies become more effective when action generation is mediated by a representation that preserves physical structure, temporal continuity, or geometric consistency. The specific kinematic substrate may be implicit motion, camera-space pose, canonical SE(3) keyframes, joint-space trajectories, or metric video motion, depending on the task domain.

2. Representation strategies: implicit motion, explicit pose, and whole-body state

A central axis of variation in VLK is the choice of kinematic representation. HiF-VLA adopts an implicit representation: MPEG-4 Motion Vectors (MVs) extracted from adjacent frames with 16×16 macroblocks. For consecutive frames $o_{t-1}$ and $o_t$, the motion vector at macroblock position $(x,y)$ is

$$MV_{t-1:t}(x,y) = \big(x_t - x_{t-1},\ y_t - y_{t-1}\big),$$

and the Group of Pictures window is

$$GOP = [MV_{t-m:t-m+1}, \dots, MV_{t-2:t-1}, MV_{t-1:t}, o_t].$$

This representation explicitly excludes robot joint states, forward or inverse kinematics, optical flow, scene flow, depth, and 3D keypoints; kinematics is treated as visual macroblock displacement that summarizes inter-state change while filtering static redundancy (Lin et al., 10 Dec 2025). This makes HiF-VLA a notable counterexample to the misconception that VLK always requires explicit robot kinematic variables.

cVLA uses a more explicit formulation centered on end-effector waypoints in the camera frame. The model outputs two keyposes, each represented by discretized position tokens and orientation tokens; positions are absolute image-frame coordinates $(u,v,z)$, where $u$ and $\dot{q}$0 are normalized width and height and $\dot{q}$1 is distance from the camera (Argus et al., 2 Jul 2025). With camera intrinsics $\dot{q}$2, the standard back-projection is

$\dot{q}$3

followed by camera-to-base transformation

$\dot{q}$4

and, for full pose composition,

$\dot{q}$5

This decouples the learned policy from the robot’s kinematic chain: a robot-specific IK/planning layer handles embodiment-dependent execution (Argus et al., 2 Jul 2025).

OG-VLA also uses explicit geometry, but in a world-frame, multi-view setting. Multi-view RGB-D observations are fused into a point cloud and rendered from canonical orthographic views—front, left, right, and top—so that the policy input and the action representation inhabit the same coordinate space (Singh et al., 1 Jun 2025). Orthographic projections are defined as

$\dot{q}$6

The model then decodes a 6-DoF end-effector pose $\dot{q}$7 by combining translation and rotation heatmaps across views. Position is estimated through a product-of-heatmaps objective,

$\dot{q}$8

while orientation is read from axis-specific Gaussian cues relative to the translation point (Singh et al., 1 Jun 2025).

Humanoid VLK uses an even richer state. The policy in reconstructed 3DGS scenes predicts an $\dot{q}$9-step whole-body trajectory with $T \in SE(3)$0 frames, where each future frame is encoded as

$T \in SE(3)$1

Here $T \in SE(3)$2 are heading-normalized root displacements, $T \in SE(3)$3 is root height, $T \in SE(3)$4 is a 6D root orientation representation, $T \in SE(3)$5 encode all joint angles, and $T \in SE(3)$6 are wrist-object contact labels (Wang et al., 29 Jun 2026). This is a substantially more explicit kinematic representation than that used in manipulation-only VLAs.

Across the broader literature, surveyed action spaces include joint angles, end-effector poses, gripper scalars, discretized action tokens, waypoints, and dynamics-aware whole-body commands (Zhang et al., 23 Sep 2025, Din et al., 14 Jul 2025). This suggests that VLK is best understood not by a single data type but by the role the representation plays: it must preserve the geometric or dynamical information needed to connect multimodal intent to executable motion.

3. Architectural patterns for coupling vision, language, and kinematics

HiF-VLA couples vision-language inference with bidirectional temporal reasoning. DINOv2 and SigLIP provide hybrid image embeddings, while a Prismatic-7B VLM initialized from OpenVLA processes instructions and current images. A 3D convolution followed by a 4-layer ViT encodes the hindsight motion sequence into compact tokens $T \in SE(3)$7, and the VLM emits foresight motion latents $T \in SE(3)$8 and action latents $T \in SE(3)$9 from learnable query tokens (Lin et al., 10 Dec 2025). Fusion occurs in a single hindsight-modulated joint expert rather than a mixture-of-experts:

$o_{t-1}$0

with hindsight conditioning realized through Adaptive Layer Normalization,

$o_{t-1}$1

This explicitly addresses temporal myopia by conditioning future motion-action predictions on compact summaries of past dynamics (Lin et al., 10 Dec 2025).

cVLA instead uses a lightweight next-token trajectory generator. PaliGemma2 serves as the VLM backbone; inputs include a live RGB image at 224×224, optional depth rendered through the Matplotlib “viridis” colormap, current end-effector pose, and either a task description or a demonstration pair (Argus et al., 2 Jul 2025). The model trains only the attention layers and emits the two keyposes in one shot as a short token sequence, reusing localization tokens for positions and segmentation tokens for orientations. The low temporal resolution is deliberate: a separate low-level planner inserts approach and retreat segments and executes the resulting waypoints with IK (Argus et al., 2 Jul 2025).

OG-VLA combines a frozen ImageBind visual encoder, a Vicuna-7B v1.5 LLM via X-VILA, and Stable Diffusion 1.5 as an image generator (Singh et al., 1 Jun 2025). The LLM outputs four image action tokens that, together with CLIP-encoded instruction residuals and per-view visual features, condition the diffusion model to generate orthographic action images aligned with canonical views. This architecture is notable because it does not regress action vectors directly; it generates images whose geometry is then decoded back to a 6-DoF pose (Singh et al., 1 Jun 2025).

KERV overlays a kinematic runtime layer on top of a token-domain VLA policy. OpenVLA is the verifier, a single LLaMA block is the draft model, and a Kalman filter supplies kinematic-domain correction (Zheng et al., 2 Mar 2026). Each action slice has 7 DoFs—$o_{t-1}$2—and when speculative decoding encounters a first erroneous DoF index $o_{t-1}$3, the verified prefix is preserved while the remaining suffix is filled by the Kalman predictor:

$o_{t-1}$4

The same runtime layer also adjusts the relaxed acceptance threshold using a kinematic variability signal $o_{t-1}$5, rather than a fixed threshold (Zheng et al., 2 Mar 2026).

KineVLA introduces a different architectural decomposition. It splits action representation into a goal-level codebook $o_{t-1}$6 and a kinematics-level codebook $o_{t-1}$7 learned through residual vector quantization, and pairs them with two supervised reasoning streams: coarse goal reasoning and fine-grained kinematics reasoning (Han et al., 18 Mar 2026). The RVQ-VAE iteratively quantizes residuals,

$o_{t-1}$8

to produce bi-level action tokens, while policy finetuning aligns reasoning and action with cross-entropy and a symmetric InfoNCE-style conditional mutual information loss (Han et al., 18 Mar 2026). This architecture explicitly separates “what to do” from “how to do it.”

The broader survey taxonomy groups such systems into early-fusion transformers, late-fusion modular designs, token-based autoregressive policies, diffusion policies, and hierarchical planners with explicit control modules (Din et al., 14 Jul 2025). In that taxonomy, VLK appears either as a latent property of the learned policy or as an explicit interface—IK, planners, Jacobian control, SE(3)-aware heads, or safety projection layers—coupled to multimodal reasoning (Zhang et al., 23 Sep 2025).

4. Training objectives and inference-time control mechanisms

HiF-VLA jointly trains future motion and action prediction over a foresight horizon $o_{t-1}$9 using L1 losses,

$o_t$0

and

$o_t$1

with $o_t$2 selected by ablation (Lin et al., 10 Dec 2025). The paper reports that removing the action branch slows and destabilizes motion-loss convergence, supporting the “think-while-acting” claim that motion and action prediction are mutually constraining (Lin et al., 10 Dec 2025).

cVLA uses standard autoregressive next-token cross-entropy for the two-keypose output sequence (Argus et al., 2 Jul 2025). It also studies several inference-time techniques that materially affect executability: image cropping around task-relevant regions, greedy decoding, sampling, beam search, and Beam-search-NMS. On DROID-hard, the reported mean trajectory L1 values are 34.44 for greedy, 34.31 for sampling, 34.17 for beam, and 33.42 for Beam-NMS in Top-1 evaluation; for Top-3 best-of, Beam-NMS reaches 25.00 versus 33.94 for sampling and beam (Argus et al., 2 Jul 2025). For a distributional metric denoted $o_t$3, Beam-NMS improves from 0.11 to 0.31 without crop and from 0.16 to 0.45 with crop size 700 (Argus et al., 2 Jul 2025). These figures show that, in camera-space VLK, decoding strategy is not peripheral but part of the action-quality interface.

KERV is primarily about inference-time control. Its Kalman filter uses a constant-velocity state model over pose and gripper variables and replaces speculative decoding re-inference with one-step prediction. The threshold update is

$o_t$4

followed by

$o_t$5

clamped by $o_t$6 and $o_t$7 (Zheng et al., 2 Mar 2026). This replaces task-dependent fixed threshold tuning with a physically motivated adjustment based on kinematic discrepancy.

ST-VLM, although not a control policy, is trained with standard autoregressive next-token cross-entropy over textual answers to kinematic questions (Ko et al., 25 Mar 2025). Its importance for VLK lies in the supervision pipeline: metric pseudo-labels are produced by MonST3R, Metric3Dv2, and Grounded-SAM2, with traveled distance, speed, and direction computed in a canonicalized world frame. For example,

$o_t$8

This establishes a data-construction paradigm in which metric kinematic supervision is generated at scale even when native labels are unavailable (Ko et al., 25 Mar 2025).

VLK for humanoids uses flow matching rather than token prediction. Given clean future kinematics $o_t$9, noise $(x,y)$0, and $(x,y)$1,

$(x,y)$2

and the policy predicts the clean trajectory with loss

$(x,y)$3

Auxiliary losses penalize foot-contact mismatch, accumulated root inconsistency, forward-kinematics errors at ankles and wrists, and foot skating, combined as

$(x,y)$4

(Wang et al., 29 Jun 2026). This is one of the clearest examples of VLK as explicit physical regularization of multimodal prediction.

Mechanistic steering work introduces yet another control mechanism: activation-level intervention. For selected FFN neurons aligned with semantic concepts such as “slow,” “fast,” or “up,” activations are overwritten by a scalar $(x,y)$5,

$(x,y)$6

which induces a residual shift $(x,y)$7 and modulates action-token distributions without any finetuning or reward optimization (Häon et al., 30 Aug 2025). In the VLK framing of that work, kinematic semantics are internal linear directions in the transformer rather than explicit state variables.

5. Application domains and empirical results

Long-horizon robotic manipulation is the most developed VLK application. HiF-VLA reports 94.4% average success rate on LIBERO-Long with third-view and 96.4% with multi-view (Lin et al., 10 Dec 2025). On CALVIN ABC-D, it reports average length 4.08 in third view versus 3.80 for UniVLA and up to 3.65–3.92 for other strong baselines, and 4.35 in multi-view, matching or exceeding top baselines such as VPP at 4.33 (Lin et al., 10 Dec 2025). Efficiency measurements are integral to the claim: foresight-only inference is 82.7 ms versus 72.9 ms baseline with 31.8 GB versus 30.8 GB memory; hindsight plus foresight is 121.6 ms and 32.2 GB, while a history-frames baseline reaches 229.5 ms and 63.6 GB (Lin et al., 10 Dec 2025). In real-world tests on an AgileX Piper with a 6-DoF arm, 1-DoF gripper, Intel RealSense D435, and wrist-mounted USB camera, the system is reported to detect subtle state transitions such as pressed versus unpressed buttons; the OpenVLA-OFT baseline attains only 17.4% on the button-order task (Lin et al., 10 Dec 2025).

Camera-space waypoint prediction is evaluated differently. cVLA is trained in simulation with ManiSkill3 and tested on DROID subsets and a Franka Panda with a wrist-mounted StereoLabs ZED2i (Argus et al., 2 Jul 2025). Its claims center on efficiency, embodiment agnosticism, and sim-to-real transfer rather than a single aggregate robotics benchmark score. Demonstration-conditioned one-shot imitation on CLEVR simulations yields 70% success on easy and 18% on hard when trained on CLEVR-easy, versus 44% and 28% when trained on CLEVR-hard, with the harder training improving generalization metrics such as mean trajectory L1 on DROID-easy from 16.37 to 11.56 (Argus et al., 2 Jul 2025). Real-robot evaluation consists of zero-shot rollouts on 15 tabletop tasks with everyday objects, including behaviors such as placing a spatula on a cutting board and a mango on a plate (Argus et al., 2 Jul 2025).

OG-VLA targets 3D-aware manipulation under camera and scene variation. On Arnold, it reports overall 37.7% on the Novel Pose split at 100k iterations versus 34.0% for PerAct, with reported overall relative improvements of 20.0% at 30k and 46.5% at 100k on other generalization splits (Singh et al., 1 Jun 2025). On Colosseum, it reports 10.5% task-averaged success with a 45.8% relative improvement over baselines including R3M, MVP, 3DDA, RVT, and PerAct (Singh et al., 1 Jun 2025). In real-world adaptation with 3–5 demonstrations per task on a Franka Panda, the reported seen performance is Pickup 100%, Put-in-Drawer 90%, Open Drawer 60%, and Close Drawer 90%; novel object performance is 80%, 70%, 30%, and 50%; novel scene performance is 90%, 80%, 50%, and 90% (Singh et al., 1 Jun 2025).

KERV evaluates VLK in the decoding loop rather than the control space. Across LIBERO-Goal, LIBERO-Object, LIBERO-Spatial, and LIBERO-Long, it reports 1.48×–1.57× acceleration versus naive VLA+SD with essentially unchanged success rates (Zheng et al., 2 Mar 2026). For example, on Goal it reports 75.6% SR at 1.54× speed versus 76.2% SR at 1.00× for naive VLA+SD; on Spatial, 83.7% at 1.57× versus 82.8% at 1.00× (Zheng et al., 2 Mar 2026). Against SpecVLA with fixed relaxed thresholds, the reported gain is 27%–37% faster while maintaining similar or better success rate (Zheng et al., 2 Mar 2026).

ST-VLM demonstrates VLK in video reasoning. On STKit-Bench, ST-VLM-7B reports 59.8% average accuracy across seven tasks, with 49.5% for traveled distance, 42.0% for traveling speed, 32.0% for movement direction, 69.0% for direction timestamp, 75.5% for traveled-distance comparison, 76.5% for speed comparison, and 74.0% for movement-direction comparison (Ko et al., 25 Mar 2025). GPT-4V reports 28.5% average and GPT-4o 26.8%, while the untuned LLaVA-OneVision-7B baseline reports 27.4% (Ko et al., 25 Mar 2025). The same model also improves over LLaVA-OneVision-7B on PerceptionTest, MVBench, VideoMME, MLVU, and NExT-QA (Ko et al., 25 Mar 2025).

Whole-body VLK has also reached physical deployment. The humanoid VLK system trained from synthetic interactions in reconstructed scenes produces 48,000 paired trajectories and is evaluated on a Unitree G1 (Wang et al., 29 Jun 2026). In simulation, success counts out of 1000 include 994 for Walk To and 731 for Pick (Floor) in the lab scene, and 948 and 749 respectively in the apartment scene (Wang et al., 29 Jun 2026). In the real world, reported success counts out of 20 include lab: Walk To 20/20, Turn Around 20/20, Pick (Floor) 16/20, Put (Floor) 20/20, Pick (Surface) 11/20, Put (Surface) 8/20; apartment: 19/20, 18/20, 18/20, 20/20, 13/20, and 15/20 (Wang et al., 29 Jun 2026). A contact-label ablation reduces Pick (Floor) to 0/5 in both scenes, underscoring the role of contact prediction (Wang et al., 29 Jun 2026).

AerialVLA shows that VLK extends beyond manipulation. On TravelUAV, it reports state-of-the-art seen performance and strong unseen generalization: on Unseen Map, NE 67.42, SR 37.58%, OSR 52.92%, SPL 28.22%, compared with LongFly’s NE 108.32, SR 11.27%, OSR 30.27%, SPL 9.32% (Xu et al., 15 Mar 2026). The paper characterizes this as nearly three times the success rate of leading baselines in unseen scenarios (Xu et al., 15 Mar 2026).

KineVLA evaluates specifically whether language-specified kinematic details are followed. It reports kinematics success rates of 76.5% on LIBERO-Goal-Relabeled, 70.4% on Kine-LIBERO, and 65.0% on Kine-Realman-75 (Han et al., 18 Mar 2026). Replacing predicted reasoning tokens with random or mismatched ones drops those numbers to 52.4%, 48.6%, and 42.4%, while goal completion remains largely unaffected (Han et al., 18 Mar 2026). This is a direct empirical demonstration that in fine-grained VLK, kinematic compliance and task completion are separable evaluation dimensions.

6. Limitations, misconceptions, and research directions

A recurrent misconception is that VLK necessarily means explicit robot kinematics. HiF-VLA contradicts that assumption by using MPEG-4 motion vectors as an implicit kinematic bridge and by explicitly not modeling joint angles, link lengths, or dynamics constraints (Lin et al., 10 Dec 2025). Conversely, cVLA, OG-VLA, and humanoid VLK rely on explicit pose or joint representations and downstream IK or tracking (Argus et al., 2 Jul 2025, Singh et al., 1 Jun 2025, Wang et al., 29 Jun 2026). The literature therefore supports a broader interpretation: VLK can be explicit or implicit, provided the model uses a physically meaningful intermediate representation to link perception and control.

Another misconception is that better multimodal reasoning alone guarantees kinematic fidelity. KineVLA’s intervention study shows that goal success can remain high while kinematics success degrades sharply if fine-grained reasoning tokens are corrupted (Han et al., 18 Mar 2026). Mechanistic steering work similarly indicates that kinematic attributes such as speed, direction, and transport height are encoded as manipulable internal semantic directions, but those directions remain indirect control levers rather than formal guarantees (Häon et al., 30 Aug 2025). This suggests that semantic competence and kinematic adherence should be evaluated separately.

The main technical limitations are representation noise, incomplete geometry, and missing constraints. HiF-VLA notes sensitivity of motion-vector extraction in highly dynamic or textureless scenes, lack of explicit safety or joint-limit modeling, and failures such as premature opening, insufficient lift, or shallow descent that reflect missing 3D geometry and constraint awareness (Lin et al., 10 Dec 2025). cVLA identifies weaker orientation prediction on real data, rigidity of one-step decoding, and limited transfer beyond quasi-static tabletop pick-and-place with top-down grasps (Argus et al., 2 Jul 2025). OG-VLA depends on accurate depth, posed multi-view RGB-D, and relatively expensive diffusion-based inference, and remains out of scope for dynamic manipulation (Singh et al., 1 Jun 2025). KERV assumes short-horizon smoothness and shows that larger prediction length degrades success quickly; its reported safe defaults are PL=1, AC=10, and $(x,y)$8 (Zheng et al., 2 Mar 2026). ST-VLM depends on camera calibration, metric scale recovery, tracking quality, and reconstruction accuracy, with failure cases under heavy occlusion, abrupt viewpoint changes, and complex 3D sports trajectories (Ko et al., 25 Mar 2025). Humanoid VLK remains limited to single-object transport and depends on scene reconstruction plus manual scene annotation (Wang et al., 29 Jun 2026). AerialVLA is reactive and can struggle with global backtracking in repetitive environments (Xu et al., 15 Mar 2026).

The surveys identify broader open problems: more diverse kinematic datasets aligned with language; stronger treatment of contact, compliance, and dynamics rather than kinematics alone; learning directly on manifolds such as $(x,y)$9 and $$MV_{t-1:t}(x,y) = \big(x_t - x_{t-1},\ y_t - y_{t-1}\big),$$0; interpretable runtime safety mechanisms; and co-design of efficient inference with physically grounded controllers (Zhang et al., 23 Sep 2025, Din et al., 14 Jul 2025). Several concrete integration avenues are already suggested in the primary papers. HiF-VLA proposes adding explicit joint-state tokens $$MV_{t-1:t}(x,y) = \big(x_t - x_{t-1},\ y_t - y_{t-1}\big),$$1, constraint-aware heads, and depth or scene flow to augment motion tokens (Lin et al., 10 Dec 2025). KERV shows how kinematic prediction can be inserted as a drop-in decoding guardrail around an existing OpenVLA-style model (Zheng et al., 2 Mar 2026). KineVLA suggests that future VLK systems may need bi-level or hierarchical representations whenever goal equivalence and trajectory variability must be disentangled (Han et al., 18 Mar 2026).

Taken together, these works indicate that VLK is becoming a unifying language for physically grounded multimodal intelligence. The concept now covers temporal reasoning in manipulation, camera-space and world-space control interfaces, kinematics-aware decoding, metric video understanding, and whole-body humanoid behavior. A plausible implication is that future embodied foundation models will be judged less by whether they output “actions” in the abstract, and more by how explicitly and reliably they encode the kinematic structure that makes those actions executable, controllable, and verifiable.