Sensor-Space Imitation Kinematic Control

• SS-ILKC is a framework that learns sensor-space representations to define actions, decoupling perception from embodiment-specific kinematic execution.
• It integrates diverse learning methods—including reinforcement learning, imitation, and energy-based models—to address complex robotic tasks.
• Empirical results show improved generalization, reduced joint errors, and higher manipulation success rates compared to end-to-end joint-space control.

Sensor-Space Imitation Learning Kinematic Control (SS-ILKC) denotes a family of architectures in which policy learning is carried out in a sensor-, object-, task-, or shape-space representation, while embodiment-specific execution is delegated to a separate kinematic layer. In the literature, the term is used explicitly for redundant soft manipulator control, where a sensor-space policy outputs reference sensor signals tracked by a low-level controller, and closely related rigid- and soft-robot systems instantiate the same separation between learned skill representation and kinematic realization: virtual end-effector policies followed by whole-body quadratic programming, state-space primitive predictors followed by inverse kinematics, or proprioceptive shape policies grounded by low-level shape-conditioned control (Meng et al., 19 Jul 2025, Lu et al., 2023, Xie et al., 2020, Yoo et al., 3 Mar 2025).

1. Conceptual basis and scope

SS-ILKC is defined by a specific factorization of the control problem: learn what to do in a low-dimensional representation tied to sensed geometry, and solve how to do it with a kinematic realization mechanism. In articulated-object manipulation, this appears as a disembodied virtual manipulator that learns skill dynamics in task space while a whole-body controller solves a quadratic program with robotic singularity and kinematic constraints; the reinforcement-learning policy never sees or controls joint angles, and instead controls a low-dimensional end-effector in a task/sensor/object-centric space (Lu et al., 2023). In hierarchical imitation for bimanual manipulation, the learned policy predicts future gripper and object states in continuous pose space, and a separate inverse kinematics solver converts predicted gripper poses into robot joint commands (Xie et al., 2020). In soft manipulation, the same pattern appears in a different guise: policies operate on spring-sensor geometry or on strain-derived shape estimates, while pressure or tendon actuation is handled by an inner controller (Meng et al., 19 Jul 2025, Yoo et al., 3 Mar 2025).

The “sensor-space” qualifier is broader than raw vision alone. Across the cited systems it includes object-centric world-frame poses and point-cloud-derived state estimates for articulated objects, RGB images fused with proprioception, embedded spring frequencies or lengths, strain-derived mesh vertices, and even sparse macroscopic diagnostic histories in partially observed PDE control (Lu et al., 2023, Ganapathi et al., 2022, Meng et al., 19 Jul 2025, Yoo et al., 3 Mar 2025, Xia et al., 6 May 2026). The common design choice is that the policy reasons over a representation closer to sensed task geometry than to raw actuator coordinates.

A common misconception is that SS-ILKC implies a single learning paradigm. The literature does not support that restriction. The decoupled articulated-object framework learns the skill layer with pure Soft Actor-Critic and uses imitation learning only for baselines; the soft-manipulator SS-ILKC framework combines multi-goal reinforcement learning with generative adversarial imitation learning; HDR-IL is imitation-centric and hierarchical; KineSoft uses diffusion-based imitation; and implicit kinematic policies use energy-based behavioral cloning with differentiable forward kinematics (Lu et al., 2023, Meng et al., 19 Jul 2025, Xie et al., 2020, Yoo et al., 3 Mar 2025, Ganapathi et al., 2022).

2. Observation spaces, action spaces, and kinematic targets

The observation and action abstractions used in SS-ILKC are diverse, but they share a consistent reduction of embodiment-specific dimensionality.

System	Sensor/task representation	Kinematic realization
Decoupled articulated-object manipulation	$s_t=[s_{o,t},s_{ee,t}]$, $a_t=[v_{h,d,t},q_{f,d,t}]$	whole-body QP tracking desired end-effector motion
HDR-IL	continuous state of grippers and objects, policy outputs future states	inverse kinematics to joint commands
IKP	RGB images + proprioception, joint candidates with FK-derived Cartesian actions	differentiable FK inside an implicit policy
Soft-manipulator SS-ILKC	$s=(P_{\text{S2R}},g,\tilde e)$, $a=f_{\text{sensor}}$	inner PID pressure tracking from sensor references
KineSoft	12 strain sensors mapped to mesh vertices $V_t$, plus RGB-D and pose	shape-conditioned controller projecting mesh error to servo increments

In the articulated-object setting, the policy state is explicitly object-centric: $$s_t = [s_{o,t}, s_{ee,t}], \qquad s_o = [s_{cab}, s_{link}, s_{hdl}, s_{size}],$$ and the policy action is

$$a_t = [v_{h,d,t}, q_{f,d,t}], \quad v_{h,d,t}\in\mathbb{R}^6,\; q_{f,d,t}\in\mathbb{R}^2,\quad a_t\in(-1,1)^8.$$

The important point is that the action is end-effector velocity plus gripper commands, independent of robot embodiment (Lu et al., 2023).

In the soft-manipulator SS-ILKC formulation, the policy state is

$$s = (P_{\text{S2R}}, g, \tilde e)\in\mathbb{R}^{18},$$

where $P_{\text{S2R}}\in\mathbb{R}^6$ is the S2R-corrected end-effector pose, $g\in\mathbb{R}^6$ is the goal pose, and $a_t=[v_{h,d,t},q_{f,d,t}]$0 is the scaled pose error; the action is the 9-dimensional sensor command $a_t=[v_{h,d,t},q_{f,d,t}]$1, represented as reference sensor-space values or equivalent spring lengths (Meng et al., 19 Jul 2025).

KineSoft moves the representation even further from conventional joint coordinates. Its raw sensor space is the 12-dimensional resistance vector $a_t=[v_{h,d,t},q_{f,d,t}]$2, one from each embedded strain sensor, but the control-relevant state is a learned shape representation $a_t=[v_{h,d,t},q_{f,d,t}]$3 consisting of mesh vertex positions inferred from strain. The high-level policy then outputs

$a_t=[v_{h,d,t},q_{f,d,t}]$4

that is, desired shape changes together with end-effector pose changes (Yoo et al., 3 Mar 2025).

A related but distinct line is kinematic-space sensor fusion. FusePose converts multi-view vision output into local joint rotations through an inverse kinematics layer, calibrates IMU quaternions into the same local representation, and performs fusion through NaiveFuse, KineFuse, or AdaDeepFuse. Although FusePose addresses human pose estimation rather than robot control, it demonstrates a general SS-ILKC front-end pattern: heterogeneous sensors are first aligned in kinematic space, and only then fused (Bao et al., 2022).

3. Learning formulations: behavioral cloning, energy models, diffusion, reinforcement learning, and hybrid training

SS-ILKC is not identified with a single objective, but with where the objective acts. In the decoupled rigid-manipulation formulation, manipulation is posed as an MDP

$a_t=[v_{h,d,t},q_{f,d,t}]$5

with the standard objective

$a_t=[v_{h,d,t},q_{f,d,t}]$6

The skill policy is trained with Soft Actor-Critic using a dense reward that stages the manipulation into approach, grasp, open, and stabilize phases. Crucially, there is no imitation component in the skill learning itself; the policy over the virtual manipulator is pure reinforcement learning (Lu et al., 2023).

The soft-manipulator SS-ILKC framework uses a dual-learning strategy. In open space it trains a goal-conditioned policy with Truncated Quantile Critics on the reward

$a_t=[v_{h,d,t},q_{f,d,t}]$7

and improves sample efficiency with HER-style relabeling. In confined spaces it augments reinforcement learning with GAIL. The discriminator $a_t=[v_{h,d,t},q_{f,d,t}]$8 is trained with

$a_t=[v_{h,d,t},q_{f,d,t}]$9

and supplies the imitation-derived reward

$s=(P_{\text{S2R}},g,\tilde e)$0

This explicitly couples sensor-space imitation to saturation-aware kinematic control (Meng et al., 19 Jul 2025).

In imitation-centric systems, the same sensor-space emphasis remains. IKP formulates policy learning as implicit behavioral cloning with an energy function $s=(P_{\text{S2R}},g,\tilde e)$1 and an InfoNCE-style loss over positive and negative actions. Its key construction is a multi-action-space energy model

$s=(P_{\text{S2R}},g,\tilde e)$2

so that the policy can exploit both joint and Cartesian structure without hand-selecting a single action representation (Ganapathi et al., 2022).

HDR-IL decomposes demonstrations into primitives and learns, for each primitive, a recurrent graph neural network that predicts future state sequences in continuous pose space. A separate high-level planner selects which primitive to execute next. KineSoft, by contrast, uses a diffusion policy whose state includes a shape embedding, a point-cloud embedding, and end-effector pose, and whose action space is shape and pose deltas. The diffusion policy is therefore grounded in the same proprioceptive geometry that the low-level controller later tracks (Xie et al., 2020, Yoo et al., 3 Mar 2025).

A broader theoretical framing is provided by the DSIL survey, which places SS-ILKC naturally within policies of the form

$s=(P_{\text{S2R}},g,\tilde e)$3

or, more generally, sensor-conditioned autonomous and non-autonomous dynamical systems. That survey connects sensor-conditioned DS policies to Lyapunov stability, contraction theory, diffeomorphism mapping, and policy-improvement layers such as RL, deep RL, and evolutionary strategies (Hu et al., 2024).

4. Kinematic realization layers

The kinematic layer is the defining second half of SS-ILKC. In the decoupled articulated-object system, the learned policy produces a virtual end-effector trajectory, and the robot follows it through a quadratic program over desired joint velocities: $s=(P_{\text{S2R}},g,\tilde e)$4 subject to

$s=(P_{\text{S2R}},g,\tilde e)$5

Here $s=(P_{\text{S2R}},g,\tilde e)$6 is adapted by a lookahead inverse-kinematics feasibility test: with $s=(P_{\text{S2R}},g,\tilde e)$7, $s=(P_{\text{S2R}},g,\tilde e)$8, and $s=(P_{\text{S2R}},g,\tilde e)$9, base motion is penalized when fixed-base IK succeeds and encouraged when fixed-base IK fails. Fingers are not in the QP; the Panda finger joints directly take $a=f_{\text{sensor}}$0 from the policy (Lu et al., 2023).

Inverse kinematics is the most classical realization mechanism, but the literature shows several variants. HDR-IL uses an external IK solver to turn predicted gripper poses into Baxter joint commands (Xie et al., 2020). A complementary comparison of refinement in joint and Cartesian spaces argues that Cartesian-space policy search benefits substantially from an approximate inverse kinematic solver that returns the closest feasible pose, rather than failing on unreachable commands. The solver minimizes a configurable pose metric

$a=f_{\text{sensor}}$1

which allows position and orientation to be weighted differently (Fabisch, 2019).

A stricter notion of kinematic realization appears in IKP. Rather than using a separate external controller, IKP embeds the kinematic chain as a differentiable forward-kinematics module inside the policy. The learned energy function therefore evaluates each action simultaneously in joint space and in FK-derived Cartesian space, and a residual formulation

$a=f_{\text{sensor}}$2

lets the model absorb small encoder offsets (Ganapathi et al., 2022).

Soft-robot instantiations replace rigid-body IK with geometry-tracking layers. In KineSoft, for each finger $a=f_{\text{sensor}}$3, the current estimated mesh $a=f_{\text{sensor}}$4 and desired mesh $a=f_{\text{sensor}}$5 define a shape residual $a=f_{\text{sensor}}$6. The controller projects that residual onto two servo directions and computes

$a=f_{\text{sensor}}$7

understood in context as aggregating contributions from vertices. The controller is explicitly quasi-static and geometric rather than dynamic (Yoo et al., 3 Mar 2025). In the pneumatically actuated soft-manipulator SS-ILKC system, the outer policy outputs desired sensor references and an inner PID loop converts those references to chamber pressures; this shifts the hard actuation physics below the policy layer while retaining sensor-space control (Meng et al., 19 Jul 2025).

5. Empirical performance, generalization, and robustness

The most direct rigid-manipulation evidence for SS-ILKC-style decoupling comes from articulated-object manipulation on ManiSkill. Across three seeds and 100 episodes each, the end-to-end joint-space baselines BC-robot, BCQ-robot, and SAC-robot achieve train/test success rates of $a=f_{\text{sensor}}$8, $a=f_{\text{sensor}}$9, and $V_t$0, respectively. The decoupled variants achieve $V_t$1 for Ours-ee-s, $V_t$2 for Ours-robot-s, $V_t$3 for Ours-ee-v, and $V_t$4 for Ours-robot-v, with shorter episode lengths throughout. The same study reports that the QP-tracked policy produces much lower joint velocities and avoids the singular “stretching” or “curling” postures observed with end-to-end joint policies (Lu et al., 2023).

Implicit kinematic policies provide a complementary empirical argument about action representation. On simulated bimanual sweeping, implicit Cartesian imitation reaches $V_t$5, implicit joint imitation $V_t$6, and IKP $V_t$7. On flipping, the pattern reverses: implicit Cartesian $V_t$8, implicit joints $V_t$9, and IKP $$s_t = [s_{o,t}, s_{ee,t}], \qquad s_o = [s_{cab}, s_{link}, s_{hdl}, s_{size}],$$0. Under joint encoder offsets on UR5e, the gap is sharper: in sweeping with 1000 demonstrations, explicit joints obtain $$s_t = [s_{o,t}, s_{ee,t}], \qquad s_o = [s_{cab}, s_{link}, s_{hdl}, s_{size}],$$1, implicit Cartesian $$s_t = [s_{o,t}, s_{ee,t}], \qquad s_o = [s_{cab}, s_{link}, s_{hdl}, s_{size}],$$2, implicit joints $$s_t = [s_{o,t}, s_{ee,t}], \qquad s_o = [s_{cab}, s_{link}, s_{hdl}, s_{size}],$$3, and IKP $$s_t = [s_{o,t}, s_{ee,t}], \qquad s_o = [s_{cab}, s_{link}, s_{hdl}, s_{size}],$$4; in L-block insertion, IKP reaches $$s_t = [s_{o,t}, s_{ee,t}], \qquad s_o = [s_{cab}, s_{link}, s_{hdl}, s_{size}],$$5 with 100 demonstrations and $$s_t = [s_{o,t}, s_{ee,t}], \qquad s_o = [s_{cab}, s_{link}, s_{hdl}, s_{size}],$$6 with 1000 demonstrations (Ganapathi et al., 2022).

Hierarchical state-space imitation with separate kinematic execution also improves generalization in bimanual tasks. For table-lift, a GRU-GRU baseline reaches $$s_t = [s_{o,t}, s_{ee,t}], \qquad s_o = [s_{cab}, s_{link}, s_{hdl}, s_{size}],$$7 success, a ResInt single-model $$s_t = [s_{o,t}, s_{ee,t}], \qquad s_o = [s_{cab}, s_{link}, s_{hdl}, s_{size}],$$8, and HDR-IL $$s_t = [s_{o,t}, s_{ee,t}], \qquad s_o = [s_{cab}, s_{link}, s_{hdl}, s_{size}],$$9 across 127 test configurations. For peg-in-hole, the reported success rates are $$a_t = [v_{h,d,t}, q_{f,d,t}], \quad v_{h,d,t}\in\mathbb{R}^6,\; q_{f,d,t}\in\mathbb{R}^2,\quad a_t\in(-1,1)^8.$$0, $$a_t = [v_{h,d,t}, q_{f,d,t}], \quad v_{h,d,t}\in\mathbb{R}^6,\; q_{f,d,t}\in\mathbb{R}^2,\quad a_t\in(-1,1)^8.$$1, and $$a_t = [v_{h,d,t}, q_{f,d,t}], \quad v_{h,d,t}\in\mathbb{R}^6,\; q_{f,d,t}\in\mathbb{R}^2,\quad a_t\in(-1,1)^8.$$2 on 281 test starts. The same work attributes this to relational modeling plus modular primitive decomposition, with IK left as the deterministic embodiment-specific layer (Xie et al., 2020).

Soft-robot evidence is equally strong, but on different metrics. In the redundant soft-manipulator SS-ILKC framework, circular path following in open space yields average translation errors of $$a_t = [v_{h,d,t}, q_{f,d,t}], \quad v_{h,d,t}\in\mathbb{R}^6,\; q_{f,d,t}\in\mathbb{R}^2,\quad a_t\in(-1,1)^8.$$3 mm without S2R and demonstrations, $$a_t = [v_{h,d,t}, q_{f,d,t}], \quad v_{h,d,t}\in\mathbb{R}^6,\; q_{f,d,t}\in\mathbb{R}^2,\quad a_t\in(-1,1)^8.$$4 mm with S2R only, and $$a_t = [v_{h,d,t}, q_{f,d,t}], \quad v_{h,d,t}\in\mathbb{R}^6,\; q_{f,d,t}\in\mathbb{R}^2,\quad a_t\in(-1,1)^8.$$5 mm for full SS-ILKC; the method then maintains mean translation error around $$a_t = [v_{h,d,t}, q_{f,d,t}], \quad v_{h,d,t}\in\mathbb{R}^6,\; q_{f,d,t}\in\mathbb{R}^2,\quad a_t\in(-1,1)^8.$$6 mm under payloads of $$a_t = [v_{h,d,t}, q_{f,d,t}], \quad v_{h,d,t}\in\mathbb{R}^6,\; q_{f,d,t}\in\mathbb{R}^2,\quad a_t\in(-1,1)^8.$$7, $$a_t = [v_{h,d,t}, q_{f,d,t}], \quad v_{h,d,t}\in\mathbb{R}^6,\; q_{f,d,t}\in\mathbb{R}^2,\quad a_t\in(-1,1)^8.$$8, $$a_t = [v_{h,d,t}, q_{f,d,t}], \quad v_{h,d,t}\in\mathbb{R}^6,\; q_{f,d,t}\in\mathbb{R}^2,\quad a_t\in(-1,1)^8.$$9, $$s = (P_{\text{S2R}}, g, \tilde e)\in\mathbb{R}^{18},$$0, and $$s = (P_{\text{S2R}}, g, \tilde e)\in\mathbb{R}^{18},$$1 g, despite training under no load. In confined pick-and-place, average translation errors relative to the demonstrated path are $$s = (P_{\text{S2R}}, g, \tilde e)\in\mathbb{R}^{18},$$2 mm in $$s = (P_{\text{S2R}}, g, \tilde e)\in\mathbb{R}^{18},$$3, $$s = (P_{\text{S2R}}, g, \tilde e)\in\mathbb{R}^{18},$$4 mm in $$s = (P_{\text{S2R}}, g, \tilde e)\in\mathbb{R}^{18},$$5, and $$s = (P_{\text{S2R}}, g, \tilde e)\in\mathbb{R}^{18},$$6 mm in $$s = (P_{\text{S2R}}, g, \tilde e)\in\mathbb{R}^{18},$$7, and the learned reward transfers to a smaller pipe of diameter $$s = (P_{\text{S2R}}, g, \tilde e)\in\mathbb{R}^{18},$$8 mm after retraining the RL policy (Meng et al., 19 Jul 2025).

KineSoft shows how a shape-space SS-ILKC pipeline can outperform raw strain matching. Its model-based strain-to-mesh estimate achieves $$s = (P_{\text{S2R}}, g, \tilde e)\in\mathbb{R}^{18},$$9 mm unidirectional Chamfer error, compared with $P_{\text{S2R}}\in\mathbb{R}^6$0 mm for DeepSoRo and $P_{\text{S2R}}\in\mathbb{R}^6$1 mm for constant curvature. In shape tracking, the mesh-based controller reports $P_{\text{S2R}}\in\mathbb{R}^6$2 mm error, versus $P_{\text{S2R}}\in\mathbb{R}^6$3 mm for a strain-tracking baseline. In real manipulation, the strain-matching policy achieves $P_{\text{S2R}}\in\mathbb{R}^6$4 on cap unscrewing and $P_{\text{S2R}}\in\mathbb{R}^6$5 on container unlidding, whereas KineSoft reaches $P_{\text{S2R}}\in\mathbb{R}^6$6 and $P_{\text{S2R}}\in\mathbb{R}^6$7, respectively (Yoo et al., 3 Mar 2025).

These results collectively support a recurring empirical pattern: when the policy is expressed in a representation aligned with sensed task geometry, and the embodiment-specific realization is offloaded to a structured kinematic layer, generalization to new objects, loads, or embodiments improves relative to direct end-to-end control in raw joint space.

6. Theoretical framing, misconceptions, limitations, and research directions

From a control-theoretic standpoint, SS-ILKC sits at the intersection of partial observation, structured motion representation, and embodied kinematic realization. The DSIL survey explicitly places such systems within sensor-conditioned dynamical systems, including forms such as $P_{\text{S2R}}\in\mathbb{R}^6$8, and organizes the relevant stability machinery around Lyapunov stability, contraction theory, and diffeomorphism mapping (Hu et al., 2024). A separate theoretical result in partially observed Vlasov–Poisson control shows how a full-information stabilizing expert can be distilled into a sensor-space policy operating only on macroscopic measurements, with closed-loop stability bounded by the best achievable behavior-cloning loss under the observation constraints: $P_{\text{S2R}}\in\mathbb{R}^6$9 and with the irreducible approximation term characterized by a resolution-entropy quantity $g\in\mathbb{R}^6$0 (Xia et al., 6 May 2026). This suggests that a rigorous SS-ILKC theory can be built around three ingredients: an explicit observation operator, a policy class over sensor histories, and a perturbation analysis of the closed loop relative to a full-state expert.

Several misconceptions are corrected by the existing literature. First, SS-ILKC is not synonymous with pure imitation learning: pure SAC skill learning, GAIL-augmented RL, implicit BC, hierarchical supervised imitation, and diffusion imitation all appear in systems with the same architectural split (Lu et al., 2023, Meng et al., 19 Jul 2025, Ganapathi et al., 2022, Xie et al., 2020, Yoo et al., 3 Mar 2025). Second, “sensor space” is not limited to RGB images; it includes point clouds, object-centric states, IMU-aligned kinematic parameters, spring frequencies, and strain-derived meshes (Lu et al., 2023, Bao et al., 2022, Meng et al., 19 Jul 2025, Yoo et al., 3 Mar 2025). Third, the kinematic layer is not reducible to exact inverse kinematics; the literature uses QP tracking, approximate IK, differentiable forward kinematics, PID tracking of sensor references, and shape-conditioned controllers (Fabisch, 2019, Lu et al., 2023, Ganapathi et al., 2022, Meng et al., 19 Jul 2025, Yoo et al., 3 Mar 2025).

The limitations are equally consistent. IKP requires a known kinematic model and studies rigid industrial arms rather than soft or continuum robots (Ganapathi et al., 2022). HDR-IL assumes access to accurate 6-DoF poses of relevant objects and grippers and relies on hand-defined primitives and labels (Xie et al., 2020). The soft-manipulator SS-ILKC framework is quasi-static, assumes a known and static pipe geometry, and exhibits saturation near workspace boundaries under extreme loads such as $g\in\mathbb{R}^6$1 kg (Meng et al., 19 Jul 2025). KineSoft is also quasi-static, has no explicit force/torque sensing, and is affected by sensor creep and hysteresis (Yoo et al., 3 Mar 2025). FusePose, while relevant as a sensor-fusion front-end, is purely per-frame and has no explicit temporal modeling (Bao et al., 2022).

The research directions proposed in the cited works converge on a broader agenda for SS-ILKC. IKP points toward integrating joint torques and velocities through forward and inverse dynamics, and toward learning link parameters or handling non-linear joint drift (Ganapathi et al., 2022). The soft-manipulator SS-ILKC paper calls for better saturation modeling, dynamic control integration, co-design and sensor placement optimization, richer perception, and multi-arm coordination (Meng et al., 19 Jul 2025). The DSIL survey highlights obstacle-aware dynamical systems, stable manifold-valued policies, and RL- or ES-based refinement under explicit stability structure (Hu et al., 2024). Taken together, these directions indicate that SS-ILKC is evolving from a useful architectural pattern into a more general research program: learn in a sensor-grounded representation, preserve geometric and kinematic structure in execution, and make the interface between the two increasingly analyzable, robust, and transferable.