TK-STGN: Tactile-Kinesthetic Graph Networks

• TK-STGN is a graph-learning framework that integrates tactile and kinesthetic feedback to represent hand sensorimotor states for multi-fingered grasping.
• It employs graph convolutions, bidirectional LSTMs, and temporal attention to efficiently model the spatial and temporal dynamics of hand movements.
• A force-position hybrid mapping converts predicted node-level states into robotic actuation commands, enabling cross-platform generalization and reliable grasp performance.

Tactile-Kinesthetic Spatio-Temporal Graph Networks (TK-STGN) denote a multimodal spatio-temporal graph-learning framework for representing and predicting hand-centered sensorimotor states from tactile and kinesthetic signals. In current arXiv usage, the term is most directly associated with the grasp-transfer architecture in "Grasp Like Humans: Learning Generalizable Multi-Fingered Grasping from Human Proprioceptive Sensorimotor Integration," where a hand is encoded as a graph with polar-coordinate kinematics and tactile contact features, processed by graph convolutions, bidirectional LSTM layers, and temporal attention to predict node-wise desired joint states for multi-fingered robotic grasping (Guo et al., 10 Sep 2025). A related, but task-distinct, spatio-temporal graph formulation appears in multi-point tactile perceptual-importance prediction, where mask-passing attention, dynamic spatial attention, and self-supervision are used for haptic media modeling rather than grasp control (He et al., 2024).

1. Problem setting and conceptual scope

TK-STGN addresses the sensorimotor integration problem for dexterous robotic hands: mapping human-style tactile and kinesthetic feedback to robotic motor actions that achieve generalizable multi-fingered grasping, covering rigid, slippery, irregular, and deformable objects, without reliance on vision once a pre-grasp pose is established (Guo et al., 10 Sep 2025). The operational targets are explicit: coordinate fingers to avoid unbalanced forces and slippage, manage contact forces to minimize deformation or damage, and generalize across objects and across different robotic hands, including underactuated variants.

This places TK-STGN at the intersection of imitation learning, proprioceptive control, tactile sensing, and graph-based representation learning. Its defining premise is that sparse full-hand tactile measurements and joint-level kinesthetic measurements should not be treated as independent channels; instead, they should be encoded on an anatomical graph that preserves biomechanical topology and temporal evolution. This suggests a sensorimotor model closer to human proprioceptive-tactile integration than to vision-dominant grasp pipelines.

A common misconception is to treat TK-STGN as primarily a vision-guided grasp planner. The reported formulation is instead centered on tactile and kinesthetic feedback after a pre-grasp pose has been established. Another misconception is to treat it as merely a graph-classification architecture. Its reported role is predictive control: it produces node-wise desired joint angle and force states that are subsequently mapped to actuator commands.

2. Glove-mediated sensing and embodiment alignment

A custom glove-mediated framework provides the measurement substrate for TK-STGN. The glove combines a flexible tactile sensor array with an IMU-based motion capture system and is worn by both human demonstrators and robotic hands, yielding isomorphic, standardized input data formats for training and execution (Guo et al., 10 Sep 2025). This alignment allows training on natural human demonstrations and deployment to diverse robotic hands via a downstream hybrid mapping, without recollection or retraining when the hand changes.

Component	Reported specification	Function
Tactile sensing	25 pads across phalanges, metacarpals, lower palm	Contact force sensing
Motion capture	PN3 Pro; 20 joints; $\sim 70$ Hz	Joint-level kinesthetic capture
Demonstrations	30 grasps per object; 10 s; 15 Hz	Training sequences

The dataset spans 20 diverse objects, with 5 seen for training and 15 unseen for generalization tests, covering variation in mass, hardness, shape, and elastic modulus, and including deformable, slippery, and irregular examples such as pitaya, carambola, and jar. Each 10 s sequence is divided functionally into 5 s of closing to stable contact and 5 s of lifting while maintaining grasp. Data are collected from both left and right hands and from multiple demonstrators with different hand lengths, including female and male hands. This diversity is part of the reported generalization strategy rather than an incidental data-collection detail.

The glove-mediated alignment is central because it standardizes node labels and raw data format across human and robotic embodiments. A plausible implication is that the framework reduces embodiment-specific preprocessing burdens that often obstruct imitation-based transfer in dexterous manipulation.

3. Unified graph representation and polar-coordinate encoding

The hand is represented as a graph

$\mathcal{G}=(\mathcal{V},\mathcal{E}),$

with $N=25$ nodes and a fixed topology during grasping (Guo et al., 10 Sep 2025). The tactile layout includes 19 bone-aligned tactile pads and 6 palm pads, namely Palm1-Palm4, InHandThumb, and Hand, while kinematic measurements cover 20 joints with aligned labels across pads and joints. Nodes on fingers are connected sequentially; nodes in the palm are fully connected, reflecting stronger proprioceptive coupling in the palm and relatively independent finger motion. For a node $v_i$, the 1-hop neighborhood is

$\mathcal{N}(v_i),$

and $k$-hop subgraphs collect all nodes reachable within at most $k$ edges.

Each node carries fused kinesthetic and tactile features. The kinesthetic component is defined in polar coordinates as

$$\mathbf{x}_{ki} \triangleq [\, r_i,\ \theta_i,\ \dot{\theta}_i \,]^\top,$$

where $r_i$ is bone length, $\theta_i$ is the joint angle in the principal motion plane, and $\mathcal{G}=(\mathcal{V},\mathcal{E}),$0 is angular velocity. For fingers, the principal plane is flexion/extension; for Thumb1, it is abduction/adduction. Tactile input is the scalar pressure or contact-force magnitude $\mathcal{G}=(\mathcal{V},\mathcal{E}),$1, reported in an approximate $\mathcal{G}=(\mathcal{V},\mathcal{E}),$2 range. The time-indexed node sample is

$\mathcal{G}=(\mathcal{V},\mathcal{E}),$3

The graph shift operator is the symmetrically normalized adjacency

$\mathcal{G}=(\mathcal{V},\mathcal{E}),$4

with $\mathcal{G}=(\mathcal{V},\mathcal{E}),$5 the adjacency matrix and $\mathcal{G}=(\mathcal{V},\mathcal{E}),$6 the degree matrix. Because topology is assumed fixed during grasp, $\mathcal{G}=(\mathcal{V},\mathcal{E}),$7 is constant over time.

Two representational choices are particularly characteristic. First, morphology is encoded explicitly through $\mathcal{G}=(\mathcal{V},\mathcal{E}),$8, rather than treated as an unmodeled domain shift. Second, the kinematic representation is reduced to grasp-critical degrees of freedom and velocities rather than full pose parameterizations. The reported interpretation is that this improves generalization and reduces over-parameterization. The lower palm pads provide only tactile signals; their associated kinematics are negligible, but carpometacarpal motions are assigned to that region to maintain feature alignment.

4. Architecture and learning objective

TK-STGN is composed of $\mathcal{G}=(\mathcal{V},\mathcal{E}),$9 graph convolution blocks for spatial encoding, followed by bidirectional LSTM layers for temporal dynamics, a multi-head attention module for task-critical temporal weighting, and an MLP output head for node-wise state prediction (Guo et al., 10 Sep 2025). With $N=25$0 the maximum subgraph order, spatial convolution is defined as

$N=25$1

where $N=25$2 is the node-feature matrix and $N=25$3 are learned shared filter weights. Layerwise propagation is

$N=25$4

After spatial encoding, bidirectional LSTM layers process the sequence: $N=25$5

$N=25$6

$N=25$7

This captures both causal and anti-causal temporal dependencies, which are reported as important for anticipating contact transitions.

Temporal attention is then applied over the sequence: $N=25$8

$N=25$9

An MLP predicts node-wise desired states: $v_i$0 where $v_i$1 is next-step desired joint angle, $v_i$2 is desired contact force, and $v_i$3 is a steady-state indicator at the current time. The target is

$v_i$4

Training minimizes a weighted MSE objective over a prediction horizon $v_i$5: $v_i$6

$v_i$7

The steady-state term $v_i$8 is annotated but not included in the explicit loss formula.

The reported implementation uses $v_i$9, $\mathcal{N}(v_i),$0, $\mathcal{N}(v_i),$1, $\mathcal{N}(v_i),$2, Adam with learning rate $\mathcal{N}(v_i),$3, batch size 32, PyTorch, Ubuntu 20.04, and an NVIDIA GeForce RTX 3060 Ti GPU. No explicit curriculum or regularizers are reported; generalization is attributed to the polar representation, multimodal fusion, and demonstration diversity.

5. Force-position hybrid mapping and morphology transfer

TK-STGN predicts desired node-level states in a topology aligned with the human hand, but robotic hands differ in actuation layout, coupling, and underactuation. The bridge from predicted states to executable hand commands is a force-position hybrid mapping (Guo et al., 10 Sep 2025). The final command vector $\mathcal{N}(v_i),$4, where $\mathcal{N}(v_i),$5 is the number of active degrees of freedom, is

$\mathcal{N}(v_i),$6

with

$\mathcal{N}(v_i),$7

$\mathcal{N}(v_i),$8

The matrices $\mathcal{N}(v_i),$9 are experimentally determined sparse matrices. For fully actuated hands they are reported as near diagonal. For underactuated hands, one actuator can aggregate signals from multiple correlated joints or pads, such as Index1-3, InHandIndex, and Palm1-2. These matrices are tuned per hardware configuration but kept invariant across objects.

This mapping is one of the main mechanisms of cross-platform deployment. Morphological differences are already partly encoded in the node features through bone length $k$0; the hybrid controller then transfers predicted human-topology states into robot-specific action space without retraining. This suggests a separation of concerns: the learned model handles embodiment-invariant sensorimotor prediction, while $k$1 and $k$2 handle embodiment-specific actuation.

6. Empirical performance, ablations, and robustness

Evaluation uses success rate (SR), force metrics (FEM-AT, FEM-KE, FEM-MAE), time to force steady-state (S-Time), processing latency from perception to command output (P-Time), and deformation Chamfer distance (DCD) between pre- and post-grasp meshes (Guo et al., 10 Sep 2025). The experimental platform uses a UR3 arm and multiple robotic hands: Inspire left/right, RY-H1 left, Ti5 right, and Ti5 variants with blocked fingers.

Method	SR	Selected reported values
TK-STGN	91.67%	FEM-AT $k$3; FEM-KE $k$4; FEM-MAE $k$5; S-Time $k$6; P-Time $k$7; DCD $k$8
Human Operation	100%	FEM-AT $k$9; FEM-KE $k$0; FEM-MAE $k$1; S-Time $k$2
Admittance Control	84%	FEM-AT $k$3; FEM-KE $k$4; FEM-MAE $k$5; S-Time $k$6
Teleoperation	70.33%	FEM-AT $k$7; FEM-KE $k$8; FEM-MAE $k$9; S-Time $$\mathbf{x}_{ki} \triangleq [\, r_i,\ \theta_i,\ \dot{\theta}_i \,]^\top,$$0
Modified ACT	66.67%	FEM-AT $$\mathbf{x}_{ki} \triangleq [\, r_i,\ \theta_i,\ \dot{\theta}_i \,]^\top,$$1; P-Time $$\mathbf{x}_{ki} \triangleq [\, r_i,\ \theta_i,\ \dot{\theta}_i \,]^\top,$$2
Modified MULSA	51%	FEM-AT $$\mathbf{x}_{ki} \triangleq [\, r_i,\ \theta_i,\ \dot{\theta}_i \,]^\top,$$3; P-Time $$\mathbf{x}_{ki} \triangleq [\, r_i,\ \theta_i,\ \dot{\theta}_i \,]^\top,$$4
GenDexGrasp	59%	FEM-AT $$\mathbf{x}_{ki} \triangleq [\, r_i,\ \theta_i,\ \dot{\theta}_i \,]^\top,$$5; P-Time $$\mathbf{x}_{ki} \triangleq [\, r_i,\ \theta_i,\ \dot{\theta}_i \,]^\top,$$6

The reported comparison emphasizes three points. First, TK-STGN approaches human operation in force management and steady-state time while substantially outperforming non-human baselines in success rate. Second, its latency, approximately $$\mathbf{x}_{ki} \triangleq [\, r_i,\ \theta_i,\ \dot{\theta}_i \,]^\top,$$7, is compatible with high-frequency closed-loop proprioceptive control and is far below the reported $$\mathbf{x}_{ki} \triangleq [\, r_i,\ \theta_i,\ \dot{\theta}_i \,]^\top,$$8 of the vision-optimization baseline GenDexGrasp. Third, DCD results indicate lower deformation on deformable objects, including clay specimens, which is consistent with the reported force-control objective.

Ablation studies isolate the roles of modality fusion, polar encoding, and temporal modeling. T-GCN, using tactile only, reaches SR $$\mathbf{x}_{ki} \triangleq [\, r_i,\ \theta_i,\ \dot{\theta}_i \,]^\top,$$9; K-GCN, using kinesthetic only, reaches SR $r_i$0; TA-GCN, using axis-angle rather than polar coordinates, reaches SR $r_i$1; TK6-GCN, using 6D joint pose input without polar simplification, reaches SR $r_i$2 and exhibits oscillatory behavior with poor steady-state, $r_i$3-Time $r_i$4; TK-GCN, which removes LSTM and attention, reaches SR $r_i$5. The reported summary is that adding temporal modeling improves SR by $r_i$6 and reduces force relative to TK-GCN.

Robustness tests are reported over five seen and five unseen objects, with 150 trials per setup. Baseline SR is $r_i$7 for TK-STGN versus $r_i$8 for Modified ACT. Under random invalid tactile pads with three off channels, SR becomes $r_i$9 for TK-STGN and $\theta_i$0 for ACT. Under random approach angles it becomes $\theta_i$1 versus $\theta_i$2; under random initial postures, $\theta_i$3 versus $\theta_i$4; under random placement positions, $\theta_i$5 versus $\theta_i$6. Cross-robot generalization is likewise strong: Inspire Left $\theta_i$7, Inspire Right $\theta_i$8, RY-H1 Left $\theta_i$9, Ti5 Right $\mathcal{G}=(\mathcal{V},\mathcal{E}),$00, Ti5 Right with pinky blocked $\mathcal{G}=(\mathcal{V},\mathcal{E}),$01, and Ti5 Right with pinky and ring blocked $\mathcal{G}=(\mathcal{V},\mathcal{E}),$02, all exceeding the corresponding ACT results.

7. Related formulations, distinctions, and limitations

TK-STGN is not identical to standard ST-GCN or GAT formulations. Its reported design separates spatial graph encoding from temporal modeling: spatial aggregation is performed by weighted $\mathcal{G}=(\mathcal{V},\mathcal{E}),$03-hop powers of a fixed normalized adjacency, whereas temporal dependencies are modeled by BiLSTM plus multi-head attention (Guo et al., 10 Sep 2025). The attention mechanism operates over temporal sequence features after spatial encoding, not over graph neighborhoods. This differs from graph-attention approaches in which edge weights are learned directly over spatial neighborhoods.

A technically adjacent spatio-temporal graph model appears in "Self-supervised Spatio-Temporal Graph Mask-Passing Attention Network for Perceptual Importance Prediction of Multi-point Tactility" (He et al., 2024). There, the input is a frequency-time tensor

$\mathcal{G}=(\mathcal{V},\mathcal{E}),$04

and the model predicts node-wise perceptual-importance probabilities $\mathcal{G}=(\mathcal{V},\mathcal{E}),$05. Its encoder uses temporal-spectral mask-passing attention,

$\mathcal{G}=(\mathcal{V},\mathcal{E}),$06

together with a learned global spatial attention matrix $\mathcal{G}=(\mathcal{V},\mathcal{E}),$07, a TCN decoder, GATv2, and a self-supervised masked branch. Reported performance is Acc $\mathcal{G}=(\mathcal{V},\mathcal{E}),$08, AUC $\mathcal{G}=(\mathcal{V},\mathcal{E}),$09, and F1 $\mathcal{G}=(\mathcal{V},\mathcal{E}),$10. That model does not explicitly model kinesthetic signals as a separate input branch; the reported extension path is to add kinesthetic channels to each node or to define a secondary kinesthetic node set with cross-modal edges. This suggests a broader family of tactile-kinesthetic graph models whose objectives may range from grasp control to perceptual compression.

The main reported limitations of the grasp-oriented TK-STGN are fourfold (Guo et al., 10 Sep 2025). First, tactile sensing is sparse and limited to normal forces; richer tactile channels such as shear, vibration, and temperature are absent. Second, motion representation emphasizes flexion/extension plus thumb abduction/adduction, omitting other dexterous degrees of freedom for efficiency and robustness. Third, the hybrid mapping matrices $\mathcal{G}=(\mathcal{V},\mathcal{E}),$11 and $\mathcal{G}=(\mathcal{V},\mathcal{E}),$12 require manual per-hand calibration. Fourth, evaluation focuses on pre-positioned grasping rather than in-hand manipulation, arm-hand coordination, or multi-hand tasks. The reported future directions are therefore high-density tactile sensors such as BioTac and GelSight, hierarchical graph encoders, full-dexterity motion representations, adaptive learned mappings, and integrated tactile-feedback manipulation with arm motion planning.