TouchThinker: Open-World Tactile Commonsense

• TouchThinker is a tactile-language framework for open-world tactile commonsense reasoning that infers physical properties, affordances, and outcomes using visuotactile data and an LLM.
• It employs an action-aware tactile encoder with question-guided token fusion and Gaussian temporal MoE to focus on key tactile dynamics from interaction videos.
• The framework leverages large-scale multi-source datasets and a two-stage training pipeline to generalize across unseen objects, sensors, and complex scenarios.

TouchThinker is a tactile-language framework for open-world tactile commonsense reasoning that combines large-scale visuotactile data, an action-aware tactile encoder, and a LLM to infer physical properties, affordances, and outcomes from touch. It is designed to operate beyond closed attribute sets and rigid QA templates, generalizing to unseen objects, unseen sensors, and unseen scenarios and tasks, while answering open-ended natural language questions grounded in tactile evidence (Lyu et al., 10 Jun 2026).

1. Definition, scope, and problem setting

TouchThinker addresses tactile commonsense reasoning, defined as the ability to infer physical properties, affordances, and outcomes from touch. The motivating examples include questions such as whether a surface is suitable for wiping spills, which object will give a more stable grip, what material is being touched, and what might happen if an agent pushes harder or slides faster. In this formulation, touch complements vision because properties such as local friction, compliance, protrusions, and contact state are often hard to estimate reliably from images alone but are naturally revealed by tactile interaction (Lyu et al., 10 Jun 2026).

The framework is motivated by two bottlenecks in prior tactile-language systems. First, existing public tactile reasoning datasets are described as small and narrow, often limited to 1–3 sensor types, restricted action types, and interaction scenarios, with a heavy reliance on fixed question templates and predefined attribute labels. Second, tactile signals are characterized as both redundant and action-specific: many frames in a tactile video are static or transitional, while only short segments around key actions such as maximal compression or onset of sliding carry most task-relevant information; moreover, different actions expose different properties, with pressing linked to hardness and compliance, sliding to friction and surface roughness, and rotation or twisting to texture patterns and edge structures (Lyu et al., 10 Jun 2026).

In the TouchThinker formulation, open-world tactile commonsense reasoning means operating beyond the closed attribute sets and rigid QA templates of training, and generalizing to unseen objects and categories, unseen sensors with different imaging mechanisms and appearance statistics, and unseen scenarios and tasks, including free-form explanations, affordances, and commonsense reasoning. The framework is therefore organized around two scaling axes: data and representation (Lyu et al., 10 Jun 2026).

2. Data foundation: TouchThinker-1M and TouchThinker-Bench

The data backbone is TouchThinker-1M, a large-scale, multi-source visuotactile dataset constructed for open-world tactile commonsense reasoning. The paper reports approximately 1,001,344 tactile frames, 415+ unique objects after deduplication, 7 tactile sensor types, 8 scenarios, and 9 source datasets, with interaction actions including pressing, sliding, rotation/twisting, and related manipulations. Annotation formats include a 4D unified tactile attribute space, template-based QA, chain-of-thought reasoning, and open-ended tactile QA (Lyu et al., 10 Jun 2026).

The benchmark companion is TouchThinker-Bench, which is designed for evaluation rather than training. It stresses unseen objects, unseen sensors, and diverse task types including free-form QA. Its cross-object split withholds objects at train time using an object-level 6:1 split. Its cross-sensor split includes three sensors unseen in training: DuraGel from TacQuad, commercial GelSight Mini (self-collected with UR5 arm), and GelSight Var. 3 from VisGel. The benchmark overall covers 10 tactile sensors, 200 object categories in the main benchmark, and 82+ test objects across both splits (Lyu et al., 10 Jun 2026).

Resource	Role	Key reported coverage
TouchThinker-1M	Training / pretraining corpus	~1,001,344 tactile frames; 415+ unique objects; 7 tactile sensor types; 8 scenarios; 9 source datasets
TouchThinker-Bench	Evaluation benchmark	unseen objects; unseen sensors; 10 tactile sensors; 200 object categories; 82+ test objects across both splits

A central contribution of TouchThinker-1M is the 4D unified tactile attribute space: Hardness, Protrusion, Elasticity, and Friction. Existing labels from source datasets are mapped into this schema, while unannotated data are manually assigned attributes based on object appearance together with tactile observations and deformation patterns in the tactile video. The annotation process uses multiple annotators per sample, cross verification, disagreement resolution, and removal of samples with insufficient tactile evidence. This creates a semantically consistent attribute space across nine source datasets and seven sensors (Lyu et al., 10 Jun 2026).

The source datasets include VTV-150K, PhysiCLEAR / Octopi, Touch and Go, TacQuad, Touch-Slide, YCB-Slide, FeelSight-Real, ObjectFolder-Real, and HaTT, spanning sensor families such as GelSight Mini, DIGIT, Tac3D, GelSight17 var.1, GelSight17 var.2, GelSight var.1, and GelSight var.2. To standardize irregular raw data, the authors convert all inputs to a unified video format, keep valid contact intervals, remove non-contact or noisy segments, group static images into short sequences when needed, and apply contact region cropping, frame rate resampling and interpolation, and temporal truncation and length normalization. Typical clips are reported as about 6–8 seconds (Lyu et al., 10 Jun 2026).

The instruction side goes beyond templated QA. Template-based tasks include Tactile Feature Analysis (TFA), Surface Feature Distinction (SFD), Surface Optimality Identification (SOI), Object Sensation Correlation (OSC), and Tactile Scenario Analysis (TSA). The dataset also adds tactile chain-of-thought supervision with outputs of the form:

$f_{\mathrm{tac}(\cdot)}$6

and around 5,000 high-quality tactile instruction-following samples for open-ended tactile QA, obtained by generating dialogues with DeepSeek-V4, extracting single-turn QA pairs, and manually reviewing them for consistency and grounding (Lyu et al., 10 Jun 2026).

3. Architecture and action-aware representation

TouchThinker is a modular tactile-language framework whose components are specified as follows: a ViFi-CLIP-derived tactile backbone, pretrained on tactile attribute prediction; a frozen question encoder from Qwen2.5; a tactile-language adapter; and an LLM backbone based on Qwen2.5-7B or Qwen2.5-14B with LoRA adapters (Lyu et al., 10 Jun 2026).

The key modeling innovation is the action-aware tactile encoder, which combines question-guided token fusion with an action-aware Gaussian temporal mixture-of-experts (MoE). The intent is to make tactile features question-aware, reduce redundancy, and emphasize question-relevant action segments. The framework begins from a tactile video $V = \{ I_t \}_{t=1}^{T}$ and a question $q$. A base tactile encoder $f_{\mathrm{tac}(\cdot)}$ is initialized from ViFi-CLIP, fine-tuned with a tactile attribute classifier using cross-entropy, then frozen to produce frame-level features

$$F = [f_1, \ldots, f_T] \in \mathbb{R}^{T \times d}.$$

The question is encoded with a frozen text encoder $f_{\mathrm{txt}(\cdot)}$ to obtain word-level features $Q_w \in \mathbb{R}^{L \times d_t}$ and a sentence-level feature $q_s \in \mathbb{R}^{d_t}$, which are projected into the tactile feature space:

$\tilde{Q}_w = Q_w W_w,\quad \tilde{q}_s = q_s W_s.$

Question-guided token fusion then applies cross-attention and self-attention:

$$F_{\mathrm{qa}} = \mathrm{SelfAttn}\Bigl( \mathrm{CrossAttn}(F, \tilde{Q}_w, \tilde{Q}_w) \Bigr).$$

This yields tactile features conditioned on the question, which suppress question-irrelevant temporal segments such as pre-contact frames when the question concerns friction (Lyu et al., 10 Jun 2026).

The second stage is the action-aware Gaussian temporal MoE. Given

$$F_{\mathrm{qa}} = [f_{\mathrm{qa},1}, \ldots, f_{\mathrm{qa},T}],$$

the sentence-level question representation generates expert routing weights

$q$0

Each expert is assigned a Gaussian temporal window

$q$1

where $q$2 is normalized time and $q$3 are produced from the question embedding. The final aggregated tactile representation is

$q$4

The paper interprets this as a soft combination of experts over time and over experts, with each expert specializing in a phase of the interaction such as contact onset, max compression, or sliding phase. The reported effect is that temporal pooling is no longer uniform but becomes both question-dependent and segment-aware (Lyu et al., 10 Jun 2026).

The architectural rationale is explicit. Redundancy reduction arises because many tactile frames carry little incremental information. Action specificity follows from the observation that friction questions align with sliding segments, while elasticity questions align with deep press or relax segments. Better semantic alignment results from question-guided fusion plus temporal MoE. A plausible implication is that the model’s compression is not merely temporal subsampling but a semantics-conditioned latent reweighting of contact dynamics (Lyu et al., 10 Jun 2026).

4. Training procedure and task formulation

TouchThinker uses a two-stage tactile–language training pipeline. In Stage I, the objective is Tactile–text alignment: tactile embeddings are mapped into the LLM text token embedding space while the LLM is frozen. Let $q$5 denote the action-aware tactile encoder applied to video $q$6 with prompt $q$7, and let $q$8 be the tactile-language adapter. The tactile embedding is

$q$9

Given a ground-truth answer sequence $f_{\mathrm{tac}(\cdot)}$0, the paper defines the autoregressive cross-entropy objective

$f_{\mathrm{tac}(\cdot)}$1

where $f_{\mathrm{tac}(\cdot)}$2 is the attribute QA subset from TouchThinker-1M and the LLM weights remain frozen (Lyu et al., 10 Jun 2026).

In Stage II, the model performs end-to-end supervised fine-tuning (SFT) for instruction following, chain-of-thought, and open-ended tactile QA. The instruction data may include attribute QA, tactile CoT targets of the form > ... <answer> ... </answer>, and free-form descriptive answers. The loss is

$f_{\mathrm{tac}(\cdot)}$3

In this stage, the tactile encoder is frozen, the tactile-language adapter is trainable, and the LLM is equipped with LoRA adapters, with only LoRA parameters updated (Lyu et al., 10 Jun 2026).

The implementation details reported in the paper are specific: LoRA rank: 128, scaling factor: 256, optimizer: AdamW, lr $f_{\mathrm{tac}(\cdot)}$4, batch size 16, and 10,000 steps. The design principle for multi-source training is to abstract away sensor-specific details by standardizing all tactile inputs as videos, unifying attribute labels, and using question-guided fusion and MoE to focus on interaction dynamics and attributes rather than raw appearance (Lyu et al., 10 Jun 2026).

Evaluation tasks are divided into three categories. Basic Tactile Property Understanding predicts hardness, roughness as related to protrusion, elasticity, and friction, using accuracy. Basic Tactile Reasoning uses SFD, SOI, OSC, and TSA, again with accuracy. Open-ended Tactile Commonsense Reasoning uses three subtypes—Touch Attribute Understanding (TAU), Touch Interaction Understanding (TIU), and Touch Knowledge Reasoning (TKU)—evaluated with METEOR and two LLM judges, GPT-5 and DeepSeek-V4, each scoring answers from 1–10 on semantic correctness, tactile consistency, commonsense/reasoning plausibility, information completeness, and language quality. The final score per judge is

$f_{\mathrm{tac}(\cdot)}$5

(Lyu et al., 10 Jun 2026)

5. Empirical performance and open-world generalization

On VTV-150K, the paper compares TouchThinker against general MLLMs and tactile-specific models including VTV-LLM-7B and VTV-LLM-14B. For TouchThinker-7B, reported Tactile Feature Analysis results are 79.1% hardness, 80.8% protrusion, 75.3% elasticity, 63.4% friction, and 40.7% combined. Reported Basic reasoning results are 78.9% SFD, 64.2% SOI, 50.7% OSC, and 74.0% TSA, with an overall average of 67.4% versus 60.4% for VTV-LLM-7B. TouchThinker-14B is reported at 69.5% average, exceeding VTV-LLM-14B at 62.1% (Lyu et al., 10 Jun 2026).

On TouchThinker-Bench open-ended QA, the paper reports that TouchThinker-7B achieves the highest METEOR in all three subtasks. For TAU, the scores are 34.06 METEOR, 8.17 GPT-5, and 8.33 DeepSeek-V4. For TIU, they are 28.71 METEOR, 7.49 GPT-5, and 7.21 DeepSeek-V4, with the paper noting that the DeepSeek-V4 score is slightly below Octopi-13B’s 7.79. For TKU, the scores are 27.43 METEOR, 7.87 GPT-5, and 7.81 DeepSeek-V4 (Lyu et al., 10 Jun 2026).

On the unseen sensors and unseen objects split of TouchThinker-Bench, reported average accuracies are 34.7% for Octopi-7B, 38.0% for Octopi-13B, 49.3% for VTV-LLM-7B, and 58.6% for TouchThinker-7B. The paper also reports 68.3% hardness, 69.7% protrusion, 70.6% elasticity, 51.7% friction, 37.2% combined, 68.2% SFD, 52.1% SOI, 47.2% OSC, and 62.0% TSA for TouchThinker-7B on this setting (Lyu et al., 10 Jun 2026).

Setting	Metric	Reported TouchThinker result
VTV-150K	Overall average	67.4% for TouchThinker-7B; 69.5% for TouchThinker-14B
TouchThinker-Bench open-ended QA	TAU	34.06 METEOR; 8.17 GPT-5; 8.33 DeepSeek-V4
TouchThinker-Bench unseen sensors/objects	Average accuracy	58.6% for TouchThinker-7B

The ablation studies attribute a substantial portion of the gains to the action-aware mechanism and the two-stage training. On VTV-150K, removing action-aware modeling yields 61.6% average over SFD, SOI, OSC, and TSA, compared with 67.0% for the baseline; removing Stage I (alignment) yields 58.3% average; removing Stage II (SFT) yields 53.3% average. Appendix ablations further state that removing Question-Guided Token Fusion or Gaussian Temporal MoE each hurts performance, and that scaling the LLM from 7B to 14B improves reasoning-heavy tasks further (Lyu et al., 10 Jun 2026).

The paper interprets these results as evidence of open-world generalization across unseen objects, unseen sensors, and unseen scenarios and tasks. The specific argument is that strong transfer to DuraGel, commercial GelSight Mini, and GelSight Var.3 suggests a focus on contact deformation patterns and dynamics rather than color/appearance. This suggests that the framework is learning sensor-invariant tactile semantics rather than merely fitting sensor-specific appearance biases (Lyu et al., 10 Jun 2026).

6. Relation to embodied intelligence, nomenclature, and limitations

TouchThinker is a tactile-language system, but it also sits within a broader family of embodied reasoning models. A separate paper, "Thinker: A vision-language foundation model for embodied intelligence" (Pan et al., 29 Jan 2026), presents Thinker as a 10B-parameter VLM foundation model for robotics with unified handling of images, videos, and text, using a joint whole-video plus last-frame input scheme, a two-stage training strategy, and robotics-tailored datasets for ego-view videos, visual grounding, spatial reasoning, and chain-of-thought planning. In that comparison, “TouchThinker can be understood as ‘Thinker, plus touch’”, meaning that Thinker supplies a visual-linguistic backbone for embodied planning while TouchThinker adds tactile reasoning and tactile-specific representations (Pan et al., 29 Jan 2026). This is a conceptual bridge rather than an identity claim: the published TouchThinker framework itself is specified as a tactile-language architecture built around a tactile encoder and Qwen2.5 backbones (Lyu et al., 10 Jun 2026).

A common naming misconception arises because other arXiv works use closely related titles. "THiNK: Can LLMs Think-aloud?" (Yu et al., 26 May 2025) is a multi-agent, feedback-driven evaluation framework grounded in Bloom’s Taxonomy for assessing higher-order thinking in LLMs, and "The Art of Tool Interface Design" (Wu et al., 26 Mar 2025) presents an agentic framework, Thinker, for customer service scenarios with State-Machine Augmented Generation (SMAG), LLM-powered tools, and Adaptive Context Management. These systems are unrelated in modality and objective to tactile commonsense reasoning, despite the lexical similarity in names (Yu et al., 26 May 2025, Wu et al., 26 Mar 2025).

The TouchThinker paper states several limitations. Its attribute schema is restricted to four core attributes: hardness, protrusion, elasticity, and friction, whereas tactile perception also includes properties such as malleability, prickliness, stickiness, and temperature. Its tactile clips are typically 6–8 seconds, so the framework does not address long-horizon manipulation, multi-step tasks, tool use, or dynamic regrasping. The main models use 7B and 14B LLMs, which the paper identifies as challenging for resource-constrained robots, and although cross-sensor generalization is evaluated, real-world safety-critical contexts still require further testing (Lyu et al., 10 Jun 2026).

The future-work directions are correspondingly specific: expand tactile attribute coverage, extend to long-horizon tactile manipulation and planning, develop more lightweight or on-device models, and increase real-world deployments and feedback loops, including active data collection. A plausible implication is that TouchThinker supplies a foundation for tactile commonsense reasoning analogous to how vision-LLMs have served as backbones for embodied perception, but its present scope remains centered on short-horizon tactile understanding and open-ended reasoning rather than closed-loop control (Lyu et al., 10 Jun 2026).