TouchThinker: Scaling Tactile Commonsense Reasoning to the Open World with Large-scale Data and Action-aware Representation

Abstract: Touch is a key modality for embodied agents to understand the physical world. Although recent work has incorporated tactile signals into language systems for tactile commonsense reasoning, scaling such systems to realistic open-world settings remains challenging due to two key bottlenecks: (1) current tactile reasoning datasets remain limited in format and scale, providing insufficient supervision for reasoning from tactile observations to physical commonsense and hindering the learning of transferable tactile commonsense; (2) Tactile signals are inherently redundant and action-specific, yet existing methods often overlook these properties, resulting in inefficient representations with limited semantic expressiveness. To address these limitations, we propose TouchThinker, a tactile-language framework that scales tactile commonsense reasoning to the open world from both data and representation perspectives. First, we construct TouchThinker-1M, a million-scale, multi-source tactile reasoning dataset covering \textbf{415} objects, \textbf{8} scenarios, and \textbf{7} sensor types, providing a solid data foundation for open-world generalization. We further introduce TouchThinker-Bench, an open-world benchmark with more realistic and diverse tasks. Then, we propose action-aware modeling mechanism to improve tactile representation efficiency and enable efficient reasoning. Experimental results demonstrate that TouchThinker achieves competitive performance against state-of-the-art models across multiple datasets. Our code and dataset will be made available at: https://github.com/lvkailin0118/TouchThinker.

• The paper presents an action-aware tactile encoder that fuses question guidance with temporal tactile embeddings using a Gaussian Mixture-of-Experts mechanism.
• It introduces the TouchThinker-1M dataset and TouchThinker-Bench benchmark for robust, sensor-invariant tactile commonsense reasoning across unseen sensors and objects.
• It employs a two-stage tactile-language alignment strategy that yields significant improvements in open-ended tactile QA and physical reasoning under domain shifts.

Scaling Tactile Commonsense Reasoning: An Analysis of "TouchThinker"

Motivation and Problem Formulation

The integration of tactile perception with large-scale multimodal reasoning is a key challenge in embodied AI, as tactile cues provide critical physical and material information inaccessible to vision alone. Prior approaches have identified two critical bottlenecks: insufficient tactile data diversity and scale, and inefficient, redundancy-prone representations that fail to capture the action-specific nature of tactile signals. Unlike visual streams, tactile data is characterized by high redundancy, strong dependence on sensor type and action modality, and sparse query-relevant evidence, underlining the need for architectural innovations.

Scalable Tactile Data: TouchThinker-1M Dataset

To address data limitations, the authors constructed TouchThinker-1M, a multi-source visuotactile dataset spanning over 1M frames, 415 objects, eight scenarios, and seven sensor types. This dataset systematically unifies annotation schemas into a comprehensive four-dimensional attribute space (hardness, protrusion, elasticity, friction) and extends supervision to both chain-of-thought (CoT) and open-ended reasoning formats, supporting robust and transferable tactile understanding.

Figure 1: Data and schema statistics for TouchThinker-1M and TouchThinker-Bench, highlighting unprecedented object, sensor, and sample scale as well as detailed attribute distributions.

TouchThinker-1M standardizes input across heterogeneous source datasets, enforces consistent temporal formatting, and broadens the QA paradigm from static template answering to more sophisticated CoT and open-ended tactile interaction. Such systematic curation directly enables the evaluation of generalization to both novel sensors and previously unseen object classes.

Open-World Evaluation: TouchThinker-Bench

The TouchThinker-Bench benchmark is introduced to rigorously assess open-world tactile reasoning, comprising splits on unseen sensors and objects, as well as a comprehensive task taxonomy targeting basic tactile attribute understanding, standard commonsense reasoning (e.g., SFD, SOI, OSC, TSA tasks), and open-ended natural language explanation. It incorporates data from additional sensors (e.g., DuraGel, GelSight Mini, GelSight Var. 3) that are unseen during training, resulting in challenging domain shifts and testing sensor-invariant representation learning.

Figure 3: Representative examples and overview of TouchThinker-Bench with explicit separation of sensor and object-based generalization splits.

Action-Aware Tactile Representation Learning

The core methodological contribution is an action-aware tactile encoder that fuses question-guided information with video-stream tactile embeddings and employs a Gaussian Mixture-of-Experts (MoE) framework for temporal attention. This mechanism localizes question-relevant action segments and minimizes the inclusion of temporally redundant or irrelevant tactile frames, thereby maximizing representational efficiency and semantic expressiveness.

The model pipeline proceeds as follows:

• Framewise tactile features are extracted using a ViFi-CLIP-initialized backbone.
• Questions are embedded via a frozen language encoder and projected into the tactile feature space.
• Cross-modal attention aligns temporal tactile tokens with word-level question features, followed by self-attention.
• A temporal Gaussian MoE module adaptively allocates expert attention to segments of the stream with action-specific relevance (e.g., pressing for hardness, sliding for friction).

  Figure 4: Overview of the TouchThinker framework and its two-stage training including question-guided action-aware tactile modeling.

Action-aware attention weights are predicted in a fully differentiable manner from the question encoding, allowing the network to modulate its focus on interaction windows dictated by the semantics of the inquiry.

Figure 5: Visualization of action-aware temporal weights, showing dynamic expert focus corresponding to question specificity and action relevance.

Two-Stage Tactile-LLM Training

TouchThinker leverages a two-stage tactile-language alignment paradigm:

1. Tactile-Text Alignment: The tactile encoder is aligned to the frozen LLM embedding space using a lightweight adapter, supervised on attribute question answering.
2. End-to-End Instruction Fine-Tuning: LoRA adapters are trained for chain-of-thought and open-ended instruction tasks, reinforcing evidence-driven, physically consistent reasoning.

This progressive alignment strategy ensures both cross-modal consistency and utilization of LLM generalization, leading to coherent, informative tactile explanations.

Empirical Results and Ablations

TouchThinker consistently surpasses prior tactile-LLMs (e.g., Octopi, VTV-LLM) on both VTV-150K and the challenging TouchThinker-Bench. Notable empirical findings include:

• Strong Numerical Improvements: On the VTV-150K benchmark, TouchThinker-7B achieves an average accuracy of 67.4%, outperforming VTV-LLM-7B (+7.0%), and even exceeding VTV-LLM-14B with fewer parameters.
• For open-ended tactile QA tasks, TouchThinker demonstrates substantial gains in both automatic (METEOR) and LLM-based evaluation metrics.
• Under domain shift (unseen sensor, unseen object) in TouchThinker-Bench, TouchThinker maintains high robustness (average score 58.6%) while other models show pronounced degradation.

Ablation studies confirm that removing the action-aware modeling or either training stage leads to marked performance drop, validating the critical role of temporal attention and progressive tactile-language supervision.

Figure 6: Qualitative comparison of reasoning responses across models, with TouchThinker producing more accurate, physically grounded, and logically consistent open-ended answers.

Theoretical and Practical Implications

The findings demonstrate the necessity and efficacy of large-scale, heterogeneity-aware training for sensor-invariant tactile representations. The action-aware framework advances tactile-language alignment by explicitly incorporating interaction semantics, setting a new standard for open-world tactile QA systems and embodied agents.

Practically, this enables more robust deployment in manipulation, assistive robotics, and industrial QA, particularly where adaptation to new sensors or object classes is nontrivial. Theoretically, this work motivates further study into scalable representation learning for highly redundant, modality-specific sensor streams and the role of action-guided fusion in multimodal foundation models.

Limitations and Future Directions

The present system’s attribute schema, although comprehensive, does not exhaust the entire space of real-world tactile phenomena (omitting, e.g., malleability, thermal conductivity). The average interaction window is constrained to 6–7 seconds, leaving long-horizon, high-complexity manipulation tasks as open research problems. Furthermore, reliance on large LLM backbones (7B, 14B parameters) poses computational challenges for edge and robotic deployment. Future work includes attribute expansion, support for prolonged tactile scenarios, and design of lightweight tactile-language architectures.

Conclusion

TouchThinker introduces a systematic framework for open-world tactile commonsense reasoning by integrating scalable, multi-source data curation with an action-aware, temporally adaptive representation learning mechanism. Experimental evidence establishes its superiority in open-ended tactile QA, robust sensor and object generalization, and faithful physical reasoning. By releasing large-scale benchmarks and raising the standard for tactile-language performance, the work provides a foundation for future advances in embodied tactile intelligence.