Octopi-1.5: Interactive VTLM

• Octopi-1.5 is a visual-tactile-language model that integrates multi-part tactile inputs with visual and language tokens to generate detailed object descriptions and rankings.
• It employs a CLIP-style tactile encoder paired with a Qwen2-VL backbone, and utilizes retrieval-augmented generation to boost identification accuracy from 73% to 95% after teaching.
• Leveraging a handheld tactile interface and lightweight RAG, Octopi-1.5 supports interactive, real-time tactile reasoning and incremental learning of new object properties.

Searching arXiv for Octopi and Octopi-1.5 to ground the article in the cited papers. arXiv search query: "Octopi tactile-LLM" Octopi-1.5 is a visual-tactile-LLM (VTLM) designed to let users and robots reason about objects using touch, vision, and language. It was introduced as a demonstration system that extends Octopi-1 along three axes: multi-part tactile processing, a simple retrieval-augmented generation (RAG) module, and a handheld tactile-enabled interface (TMI) that allows real-time interaction without requiring a robot. Its core architecture couples a CLIP-style tactile encoder to a Qwen2-VL backbone, so that tactile tokens, language tokens, and, when available, visual tokens can be jointly processed to generate descriptions, rankings, and object guesses (Yu et al., 14 Jul 2025).

1. Lineage and problem setting

Octopi-1.5 is the successor to Octopi, a system introduced for object property reasoning with large tactile-LLMs. The original Octopi addressed physical reasoning challenges that are hard or impossible to resolve through vision and language alone, especially when properties such as hardness, roughness, and bumpiness are occluded or visually ambiguous. It combined tactile sensing with language by aligning a fine-tuned tactile encoder to Vicuna-7b v1.5 and Vicuna-13b v1.5, and it used intermediate object property descriptions as an explicit reasoning scaffold for comparison, superlative selection, object matching, and scenario reasoning (Yu et al., 2024).

Octopi-1.5 preserves the basic premise that touch provides direct contact-based information for latent physical properties, but it broadens the formulation in three specific ways. First, it is explicitly trained to ingest and reason over tactile signals corresponding to different parts of the same object, such as a hairbrush handle and bristles. Second, it introduces a lightweight tactile-to-text RAG layer that augments generated descriptions with similar-object context drawn from a tactile embedding corpus. Third, it is paired with the TMI, a portable interface equipped with GelSight and TAC-02 tactile sensors, so tactile interaction no longer depends on a robot arm (Yu et al., 14 Jul 2025).

This evolution changes the operational emphasis. Octopi-1 centered on property-guided reasoning over a curated tactile-language benchmark, whereas Octopi-1.5 emphasizes live interaction, multi-part description and ranking, and incremental teaching of new items through retrieval. A plausible implication is that the newer system shifts from a benchmark-oriented tactile-language LVLM toward a more deployable interactive VTLM.

2. Architecture and multimodal data flow

The language and vision backbone of Octopi-1.5 is Qwen2-VL 7B. The text states that, compared to the LLaMA-based Octopi-1, Qwen2-VL improves interaction quality and commonsense grounding; however, some experiments in the results table are reported as “Octopi-1.5 (8B).” The standard image module of Qwen2-VL processes scene images when available, although demonstrations can be vision-free during tactile steps (Yu et al., 14 Jul 2025).

The tactile pathway is built around a CLIP-style visual encoder trained on tactile images from GelSight Mini sensors. Input to this encoder is not an entire long clip; instead, Octopi-1.5 selects salient frames from a short tactile video segment. The heuristic keeps the top 10 frames with the largest frame-to-frame differences, which functions as a motion- and texture-saliency filter. Each selected frame $I_i$ is encoded by the tactile CLIP encoder $f_t(I_i)$, and part-level embeddings are pooled by a mean operation:

$e_{\text{part}} = \frac{1}{N} \sum_i f_t(I_i).$

A learned projection $P$ then maps the pooled tactile embedding into VLM-consumable tokens,

$Z_t = P(e_{\text{part}}),$

and Visual Prompt Tuning (VPT) can insert trainable prompt tokens during end-to-end fine-tuning (Yu et al., 14 Jul 2025).

The multimodal flow is sequential. GelSight Mini produces high-resolution tactile images of surface deformation, while TAC-02 produces pressure and taxel readings; for training and most demonstrations, GelSight is the primary modality because more data are available. Frames are batched by object and by part, such as “Object 1.1” and “Object 1.2.” Each part’s embedding enters the VLM as its own tactile token group. Tactile tokens are then concatenated with language tokens and, if used, visual tokens, and the combined sequence is processed by Qwen2-VL’s transformer to generate text outputs, including descriptions, rankings, and guesses (Yu et al., 14 Jul 2025).

The distinctive representational commitment is that tactile evidence from different object parts remains explicitly separable until fusion. This enables part-wise description and ranking, such as “Object parts ranked in decreasing hardness: 1.1, 1.2,” rather than collapsing the entire object into a single tactile summary.

3. Tactile representation learning and training objectives

Octopi-1.5 uses a two-stage training procedure. Stage 1 is tactile CLIP pretraining; Stage 2 is VTLM fine-tuning with the tactile encoder frozen and the projection layer plus VLM adapted through Visual Prompt Tuning on instruction-style tasks that include multi-part descriptions and ranking (Yu et al., 14 Jul 2025).

In Stage 1, the tactile encoder is trained with both regression and contrastive objectives. The regression objective targets human-annotated hardness and roughness scores:

$$L_{\text{reg}} = \sum_j \|\hat{h}_j - h_j\|^2 + \|\hat{r}_j - r_j\|^2,$$

where $h_j$ and $r_j$ are annotator scores and $\hat{h}_j$ and $\hat{r}_j$ are predictions. The contrastive component aligns embeddings across object and part identity by cosine similarity with temperature $f_t(I_i)$0:

$f_t(I_i)$1

The InfoNCE term uses positives from the same object or part and negatives from other objects or parts, with negatives sampled using the same GelSight pad type, marker or markerless, to control nuisance variation:

$f_t(I_i)$2

The overall tactile pretraining objective is

$f_t(I_i)$3

CLIP was trained for 30 epochs, and the best encoder was selected by validation loss on six unseen objects (Yu et al., 14 Jul 2025).

Stage 2 freezes the CLIP module and trains the projection layer and VLM with standard next-token cross-entropy on demonstration-style outputs, with tactile tokens conditioning generation. The available description does not specify batch sizes, learning rates, optimizer, prompt parameter counts, or compute resources for this fine-tuning stage. It does specify the dataset basis for training: PhysiCLeAR-Plain with 100 objects and 2689 tactile videos, PhysiCLeAR-Dotted with 68 objects and 1939 tactile videos, the Hardness dataset with 210 objects and 1860 videos, and ObjectFolder-Real with 100 objects and 3550 videos. Hardness and roughness supervision follow a 0–10 guidance rubric for annotators (Yu et al., 14 Jul 2025).

This training design differs materially from Octopi-1. The predecessor used a CLIP ViT-L/14-based encoder adapted to tactile videos, salient-frame selection based on the top 30% in total pixel intensity change, and a three-stage procedure consisting of encoder fine-tuning, tactile feature alignment, and end-to-end fine-tuning with LoRA. Its property supervision was categorical over hardness, roughness, and bumpiness, whereas Octopi-1.5’s tactile pretraining combines continuous hardness and roughness regression with contrastive alignment (Yu et al., 2024).

4. Multi-part tactile reasoning and retrieval-augmented generation

A defining feature of Octopi-1.5 is explicit multi-part tactile processing. Parts are recorded sequentially or in separate clips, and each part’s embedding is introduced into the VLM as its own tactile token group. This lets the model describe and rank parts independently and then reason across parts, rather than treating an object as tactilely homogeneous. The example given in the technical description is a hairbrush, for which the handle and bristles can be represented as different tactile entities (Yu et al., 14 Jul 2025).

The RAG component operates over a corpus of tactile embeddings from the training datasets, specifically PhysiCLeAR, Hardness, and ObjectFolder-Real, together with associated textual labels and descriptions collected during dataset creation and model training. The index is a simple embedding store keyed by averaged tactile embeddings per sample or per part; no specialized ANN index is described. Given a current grasp, the query embedding is the mean-pooled tactile embedding of the saliency-selected frames. Retrieval uses cosine similarity,

$f_t(I_i)$4

followed by top-5 retrieval. Retrieved results are aggregated by unique object and ranked by how many samples matched, producing a prioritized list (Yu et al., 14 Jul 2025).

The retrieved context is not fused as raw tactile exemplars. Instead, Octopi-1.5 appends a textual line to the generated description containing “Most similar objects…” and the corresponding labels and characteristics. That augmented text then guides downstream guessing and sorting. The same mechanism supports on-the-fly learning: after a session, averaged embeddings for new item or part samples are inserted into the index with a label, and no retraining is needed. Demonstrations reported that, in the Guessing Game, unseen accuracy improved from 73.17% with RAG before teaching to 95.12% after teaching (Yu et al., 14 Jul 2025).

The retrieval layer is therefore lightweight in two senses. First, it does not require retraining to incorporate new instances. Second, it augments text rather than modifying the transformer’s tactile tokenization or attention mechanism. A plausible implication is that the system prioritizes rapid extensibility and operational simplicity over tighter end-to-end integration of retrieval with multimodal representation learning.

5. Evaluation, task structure, and empirical behavior

The reported evaluation emphasizes interactive tactile inference tasks rather than the benchmark suite used in Octopi-1. The principal tasks are tactile inference through descriptions, a Guessing Game for object identification, tactile-based sorting by material properties, and RAG-based learning of new items. In the Guessing Game, a set of candidate objects can be presented through language or vision, the user performs a tactile-only grasp, and the model describes tactile properties and selects the most likely object while explaining the decision with commonsense knowledge. Sorting asks for tactile descriptions and rankings by hardness or roughness (Yu et al., 14 Jul 2025).

Split	Octopi-1.5 no RAG	Octopi-1.5 with RAG
Balls	56.00	96.00
Fruits	57.69	100.00
Unseen	41.46	73.17
Unseen (teaching)	—	95.12

These values are average accuracies in percent for the Guessing Game. The corresponding sample counts were 25 ball samples, 26 fruit samples, and 41 unseen samples. The same study reports an encoder-only cosine-similarity baseline, “Encoder-1.5,” with Balls 80.00, Fruits 100.00, Unseen N/A, and Unseen (teaching) 89.02. Octopi-1 baselines without RAG were lower: Octopi-1 (7B, no RAG) achieved Balls 44.00, Fruits 42.31, Unseen 43.90, while Octopi-1 (13B, no RAG) achieved Balls 48.00, Fruits 34.62, Unseen 53.66. The text characterizes the backbone upgrade and enlarged tactile training as yielding modest gains without RAG, while RAG substantially boosts performance on seen categories and improves generalization to unseen items, especially after on-the-fly teaching (Yu et al., 14 Jul 2025).

For sorting by hardness, the reported accuracies were Balls 100%, Fruits 93.18%, and Unseen 43.33%. The same description notes that the model struggles more on unseen items and on properties like roughness, although no roughness-sorting numbers are reported. The observed failure modes include confusions among unseen items with overlapping tactile signatures, such as a hairbrush handle and certain plastics, and more error-prone roughness sorting (Yu et al., 14 Jul 2025).

Relative to Octopi-1, the evaluation emphasis changes from explicit object property description, property comparison, property superlative selection, property-object matching, and property scenario reasoning on PhysiCLeAR to a more demonstration-driven set of tactile inference and retrieval tasks. Octopi-1 had shown that intermediate property descriptions could markedly improve downstream reasoning, for example raising Octopi-13b property superlative selection from 39.88% without OPD to 84.00% with OPD, and that tactile grounding could support zero-shot robot avocado ripeness classification at 63.00% in pairwise selection (Yu et al., 2024). Octopi-1.5 does not discard this reasoning orientation; rather, it relocates it into multi-part prompting, Qwen2-VL interaction, and RAG-augmented tactile description.

6. TMI interface, operation, limitations, and positioning

The TMI is a handheld tactile-enabled interface that modifies the Universal Manipulation Interface with tactile sensor mounts. One finger houses a GelSight Mini optical tactile sensor, with compartments designed to fit either marker or markerless pads, and the other finger houses a TAC-02 piezoresistive unit with 64 taxels and up to 1 kHz sampling. Electronics integrate camera capture for optional vision, sensor streaming, and a laptop client. The operational setup is portable: TMI plus a laptop, with Octopi-1.5 running on a remote workstation or server accessed via stable internet, including a mobile hotspot, and assembly time is under 15 minutes (Yu et al., 14 Jul 2025).

The interface is designed for live tactile interaction. The user grasps items, the client selects tactile frames and sends them to the server, and Octopi-1.5 returns descriptions, guesses, and sorting decisions. Safety and handling guidance includes avoiding excessive force on delicate items, keeping GelSight pad type consistent within a session, cleaning sensor surfaces, and recognizing that GelSight pads can wear. For multi-part objects, distinct clips should be recorded per part and labeled accordingly, such as “Object 1.1 handle” and “Object 1.2 bristles” (Yu et al., 14 Jul 2025).

Demonstration scenarios include a Guessing Game and tactile sorting. Illustrative outputs involve tactile descriptions such as “hard, firm, glossy, smooth,” followed by commonsense-based selection among objects such as apple, orange, and kiwi. RAG can append context such as “Most similar objects: unpeeled, ripe apple…,” reinforcing the final decision. The described prompt pattern is “description first, decision second,” which encourages property-based reasoning and mirrors the earlier Octopi design choice to make physical properties explicit before answering (Yu et al., 14 Jul 2025).

The limitations are also explicit. Performance drops on unseen objects without RAG; tactile descriptions are sensitive to pad type, contact conditions, and sensor noise; roughness is harder to estimate reliably; and language descriptions of continuous properties can be subjective. Optical tactile sensors have relatively low sampling rates, so salient-frame selection is used to reduce latency and computation, but full video processing remains heavier than single-frame vision. Marker versus markerless pads introduce domain shifts, and the system currently generates text only: no action output is produced, and coupling to manipulation policies, described as VTLA, is planned future work (Yu et al., 14 Jul 2025).

Within the recent VTLM literature, Octopi-1.5 is positioned by three stated differentiators: explicit multi-part tactile reasoning, a Qwen2-VL backbone, and simple tactile-to-text RAG for both performance and incremental learning. The trade-off is also stated directly: the RAG and CLIP-based encoding are computationally light and easy to reproduce, while more advanced fusion, such as cross-attention between raw tactile sequences and language, and richer retrieval would likely improve performance at higher complexity (Yu et al., 14 Jul 2025).