Papers
Topics
Authors
Recent
Search
2000 character limit reached

FTP-1: A Generalist Foundation Tactile Policy Across Tactile Sensors for Contact-Rich Manipulation

Published 11 Jun 2026 in cs.RO | (2606.13102v1)

Abstract: Despite the success of vision-based generalist robotic policies, existing tactile-based policies remain tied to fixed embodiments and sensor setups. This is because tactile signals are highly heterogeneous across hardware, making cross-sensor generalization difficult. We present FTP-1,the first generalist foundation tactile policy pretrained to acquire transferable tactile manipulation abilities across diverse sensors and embodiments. FTP-1 supports varied tactile inputs, including image-, array-, and state-based signals, by using heterogeneous encoders to project them into unified morphology-aware latent tokens that are jointly modeled by a shared tactile Transformer expert. Pretrained on around 3,000 hours of tactile manipulation data aggregated from 26 data sources, spanning human and robot demonstrations across 21 sensors, FTP-1 learns tactile skills that transfer beyond the sensors seen during pretraining. Across downstream finetuning experiments spanning 5 hardware configurations, FTP-1 improves contact-rich manipulation on seen sensor setups by +17.2% and, surprisingly, transfers to two previously unseen tactile-sensor setups, achieving a +31% gain in success rate. FTP-1 establishes the first unified foundation baseline for tactile manipulation, providing future tactile policies with a shared model-level starting point. Pretrained models, datasets, training code and more visualization at https://ftp1-policy.github.io.

Summary

  • The paper introduces FTP-1, a unified tactile policy model employing a Morphology-Aware Tactile Token Space to abstract heterogeneous tactile signals.
  • The methodology uses modality-specific encoders and a 300M-parameter Transformer, pretrained on 3,000 hours of data across 21 diverse tactile sensors.
  • Results show a +17.2% gain on seen sensors and a +31.6% improvement on unseen tactile hardware, demonstrating strong transferability and robust performance.

FTP-1: Foundation Tactile Policy for Cross-Sensor Contact-Rich Manipulation

Introduction

FTP-1 introduces a generalist foundation model for tactile robotic policy learning, addressing a key barrier in the scaling and generalization of tactile manipulation: sensor heterogeneity. Prior systems are typically tied to device-specific signal modalities, tactile morphologies, and robot embodiments, which severely limits transfer learning and deployment flexibility. FTP-1 establishes, for the first time, a unified framework supporting transfer, adaptation, and pretraining across diverse tactile sensors—including image, array, and state-based modalities—enabling effective manipulation on both familiar and novel hardware.

Architecture and Pretraining Strategy

FTP-1's modeling approach is underpinned by the Morphology-Aware Tactile Token Space (MTTS), which abstracts tactile observations from heterogeneous sources into a constant set of 24 functional-area tokens. Diverse tactile inputs are projected into MTTS via modality-dependent encoders: lightweight ViTs for images, CNNs for arrays, and Fourier-encoded MLPs for state signals. These tokens are augmented with task- and morphology-aligned embeddings, enforcing functionality-preserving representation sharing across hardware. The tactile tokens are then processed by a dedicated 300M-parameter Transformer ("tactile expert"), decoupled from vision-language and proprioceptive branches, which supports both effective cross-sensor pretraining and modular policy adaptation.

The scale and diversity of tactile experience is a core design feature. FTP-1 is pretrained on a dataset comprising approximately 3,000 hours of robot and human demonstrations, aggregated from 26 data sources and spanning 21 tactile sensors (7 image, 5 array, 9 state). Dataset imbalance is managed through targeted sampling ratios, enforcing coverage across human, dexterous robotic hand, and simpler gripper domains. All tactile signals are normalized to the common MTTS interface, facilitating end-to-end heterogeneous policy training.

Downstream Evaluation: Generalization and Transfer

Comprehensive finetuning and transfer experiments are conducted across five robotic embodiments, including both simulation (UniVTAC) and real-world robot platforms, involving both seen (pretraining-covered) and entirely novel (unseen) tactile hardware. Performance is measured in terms of task success rates on contact-rich manipulation benchmarks featuring in-hand adjustment, precision insertion, force-controlled pressing, dexterous assembly, and long-horizon behaviors.

Results on Seen Sensors

On benchmarks where pretraining and downstream sensors overlap (UniVTAC, Sharpa North, Sharpa&Dexmate setups), FTP-1 establishes a clear lead over all baselines—including vision-language-action models without tactile input [pi0.5, (Intelligence et al., 22 Apr 2025)] and prior tactile fusion designs like Tactile-VLA (Huang et al., 12 Jul 2025), surpassing them by +17.2% absolute gain in average success rate. Notably, vision-centric models can partially compensate for the absence of tactile cues on some manipulation tasks, but their performance degrades significantly for tasks requiring reactive force control or resilience to subtle contact dynamics. Models with naïve tactile integration (i.e., adapters into vision-language backbones) are not robust and sometimes degrade overall performance, highlighting the necessity of modular tactile expert design as in FTP-1.

Transfer to Unseen Sensors

A key contribution is the demonstration of strong transfer to previously unseen tactile sensors and hardware. FTP-1’s architecture supports initializing only the sensor-specific encoder while leveraging a pretrained tactile expert. This yields an average gain of +31.6% in success rate on entirely novel sensor setups (e.g., FlexivXense, TactileUMI), more than doubling the task success rate relative to the best baseline. Detailed analysis reveals substantial improvements in stability, data efficiency, and real-time feedback exploitation: on fine-grained insertion (Insert Hanoi, Insert USB), FTP-1 policies dynamically modulate insertion speed and contact forces using tactile cues, a behavior not emergent in non-pretrained or adapter-based systems.

Ablation studies further show that these gains are not merely due to data distribution alignment but derive from truly transferable manipulation knowledge encoded in the tactile branch. Models with no tactile branch pretraining (NTP-1) achieve intermediate results, while FTP-1 consistently outperforms, especially on novel sensor domains.

Implications and Limitations

FTP-1 establishes a robust foundation for sensor-agnostic tactile policy pretraining, analogous to “foundation models” in vision and language. The MTTS framework and modular tactile expert not only unlock direct transfer capabilities but set the stage for scalable tactile world modeling, prediction-based control, and cross-embodiment skill sharing. Practically, this enables rapid system adaptation as new tactile hardware is incorporated, lowers the demonstration sample complexity for novel setups, and provides a reusable initialization point for future tactile policy research.

However, the current instantiation does not yet tackle tactile-based closed-loop servoing or prediction. The diversity and scale of the FTP-1 dataset, while large relative to past tactile work, remains modest compared to vision- or language-scale foundation models, suggesting room for further scaling and improvement. Extensions to tactile world models, prediction-augmented policy architectures, and open-ended tactile data aggregation are natural directions.

Conclusion

FTP-1 represents a significant step toward generalist tactile manipulation policies by introducing a unified morphology-aware token framework, multi-modality transformation and fusion pipelines, and large-scale cross-sensor pretraining. The approach establishes new baselines for contact-rich manipulation, demonstrates robust generalization to unseen tactile sensors, and provides a replicable infrastructure for future tactile policy research. These results chart a path toward more versatile, adaptive, and sensor-agnostic tactile robotic systems.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

Explaining “FTP-1: A Generalist Foundation Tactile Policy Across Tactile Sensors for Contact-Rich Manipulation”

Overview

This paper is about teaching robots to “feel” and use touch well, no matter what kind of tactile sensor (touch sensor) they have. The authors created a big, general model called FTP-1 that can understand many types of touch signals and help robots do careful, contact-heavy tasks—like twisting a bottle cap, inserting a USB, or wiping a surface—without being tied to one specific sensor or robot.

Key Objectives and Questions

The paper set out to answer a simple question:

  • Can we build one touch-based robot policy (a decision-making model) that learns from many different touch sensors and still works well on new sensors and robots it has never seen before?

To get there, they aimed to:

  • Combine very different touch inputs (images, numbers in grids, or simple force readings) into one shared format.
  • Pretrain (practice) on a very large and mixed dataset so the model gains “general” touch skills.
  • Test if those skills help on both familiar and brand-new sensors and hardware.

Methods and Approach

Think of tactile sensors like different kinds of “gloves” robots wear to feel the world. Some gloves “see” the touch (image-based), some measure pressure points in a grid (array-based), and some just give overall force numbers (state-based). These differences make it hard to train one model that understands all touch types.

The authors’ approach has three main ideas:

  • Morphology-Aware Tactile Token Space (MTTS)
    • Imagine a map of the robot’s hands and wrists broken into 24 “functional areas” (like thumb tip, index fingertip, wrist torque, etc.).
    • No matter which sensor you use, its touch data gets converted into a small set of shared “tokens” for these areas—like translating many languages into the same set of simple words.
    • This lets the model handle touch from different hardware using the same interface.
  • Heterogeneous Tactile Encoders
    • Because sensors are different, they use the right tool for each type:
    • Image-type touch (like GelSight) uses a small vision transformer (a kind of image-understanding network).
    • Array-type touch uses CNNs (networks good at seeing patterns in grids).
    • State-type touch (force/torque numbers) uses simple MLPs after a smart “Fourier” encoding (a way to make repeating patterns easier to learn).
    • All these encoders turn their sensor inputs into the shared MTTS tokens.
  • A Shared Tactile Expert (plus vision and language)
    • The model includes a large “tactile expert” (a Transformer, which is like a powerful pattern-finding coach) that learns general touch skills from the tokens.
    • It also reads camera images and language instructions (like “insert USB”) using a vision-language expert, and the robot’s own joint states (“proprioception”—like knowing where your arm is) to decide actions.
    • The action expert looks at everything and outputs what the robot should do next.
    • Important design choice: tactile has its own expert rather than being squeezed into the vision module. This keeps touch learning strong and avoids confusion with vision-language knowledge.

Pretraining data:

  • They built a huge dataset: about 3,000 hours from 26 sources, across 21 different tactile sensors (7 image-type, 5 array-type, 9 state-type), including human and robot demos.
  • All data was converted to the MTTS format so the model learns consistent “touch skills” across many situations.

Main Findings and Why They Matter

  • Better performance on familiar sensors:
    • When finetuned (lightly retrained) on setups similar to what it saw before, FTP-1 improved manipulation success by about +17.2% compared to strong baselines.
  • Surprisingly strong transfer to new, unseen sensors:
    • On totally new sensors and robot setups (never seen during pretraining), FTP-1 still boosted success by about +31.6%.
    • Example tasks included Insert Hanoi (a small ring onto a peg), Insert USB, and Wipe Board. FTP-1 showed smoother, more stable actions and better “reactive control”—slowing down or adjusting force based on touch when things were misaligned.
  • Why this matters:
    • Most older touch-based policies only work with one specific sensor. FTP-1 shows you can learn general touch skills that carry over to different hardware, making robot learning more flexible and practical.

Extra insights:

  • In some simulations, vision alone did well on easy “lifting” tasks, but for trickier, contact-rich tasks, FTP-1’s touch made a clear difference.
  • Ablation tests (controlled comparisons) show the gains come from the tactile pretraining itself, not just from having similar data. In other words, FTP-1 really learns reusable “touch knowledge.”

Implications and Potential Impact

  • A shared starting point for touch:
    • FTP-1 acts like a “foundation model” for touch, giving future researchers and engineers a solid base that already understands tactile signals across different sensors.
  • Faster adaptation:
    • Teams with new hardware can plug their sensors into the MTTS format, train a small encoder, and benefit from the pretrained tactile expert—saving time and data.
  • Better contact-rich manipulation:
    • From industry assembly to household robots, reliable touch is crucial for safety and precision. FTP-1 helps robots adjust force, handle misalignment, and make smoother decisions when physically interacting with objects.

Looking ahead:

  • The authors note limitations: they focused on perception and policy rather than detailed force control (like low-level “servoing”). Future work could scale the dataset even more and add advanced force-aware control, making robots even better at delicate tasks.

In short, FTP-1 shows that robots can learn general touch skills from lots of diverse experiences and use those skills on new sensors and tasks—bringing us closer to versatile, safe, and capable robots that “feel” as they work.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

Below is a concise, actionable list of what remains missing, uncertain, or unexplored in the paper, to guide future research:

  • Generality of MTTS mapping: How robust is the 24 functional-area schema across radically different morphologies (e.g., soft grippers, whole-hand skins, suction tools)? Automated discovery/learning of functional areas and the impact of mis-specified mappings remain untested.
  • Spatial compression trade-offs: Each functional area is represented by a single token; the effect of this compression on tasks requiring fine-grained contact geometry or shear/texture cues is not analyzed. Alternatives (multi-token per area, hierarchical tokens) are unexplored.
  • Modality timing and synchronization: The handling of variable sampling rates, asynchrony, and latency jitter across tactile, proprioceptive, and visual streams is unspecified; robustness to timestamp misalignment and dropped frames is not evaluated.
  • Fusion design choices: The action expert attends to tactile expert outputs without reciprocal attention; there is no ablation on bidirectional attention, gating, or cross-modal conditioning to test if better fusion improves transfer or stability.
  • Transfer boundaries across sensor types: Unseen-sensor evaluation covers one image-type (Xense) and one array-type (Contactile); transfer to unseen state-type (force/torque) sensors and to full-hand tactile skins is untested.
  • Dataset scale and diversity: ~3,000 hours across 21 sensors is modest for a “foundation” model; scaling laws with respect to data volume, sensor variety, and task diversity are absent, and material/contact diversity is not quantified.
  • Simulation vs. real tactile fidelity: High success on “lift” tasks without touch in simulation suggests limited reliance on simulated tactile cues; the sim tactile realism and sim-to-real transfer for touch are not investigated.
  • Control and action space coverage: The Unified Action Space’s support for position/velocity vs. impedance/force control is not compared; tactile- or force-based servoing and hybrid force–position policies are acknowledged as future work but not addressed.
  • When touch helps vs. hurts: Negative transfer cases and conditions where tactile inputs degrade performance (e.g., noisy or misaligned touch) are not characterized; strategies to adaptively weight or disable touch are absent.
  • Language grounding quality: The extent and quality of language annotations in the aggregated tactile data are unclear; the paper does not analyze how instruction grounding affects tactile policy behavior.
  • Sample efficiency characterization: While some tasks use 50–100 demonstrations, systematic learning curves across sensors/tasks to quantify data requirements for adapting new sensors are missing.
  • Safety and contact constraints: There is no evaluation of peak forces, contact safety, or slip/overforce detection; mechanisms to enforce safe interaction during finetuning/deployment are not discussed.
  • Real-time performance: Inference latency and compute footprint of a 300M-parameter tactile expert in closed-loop control are not reported; feasibility at high control frequencies is unknown.
  • Calibration and drift: Robustness to calibration differences, sensor wear, and drift across tactile hardware is untested; no procedures for cross-sensor recalibration or adaptation are provided.
  • Environmental/material robustness: Generalization across surface materials, lubricants, temperature/humidity changes, and contaminants that affect tactile signals is not evaluated.
  • Vision-degraded settings: The policy is always vision-enabled; performance with occlusions, poor lighting, or tactile-only operation is not assessed.
  • Cross-embodiment breadth: Evaluations span limited hands/grippers; transfer to drastically different embodiments (e.g., anthropomorphic hands with dense skins, bimanual whole-arm contact) remains open.
  • Minimal adaptation protocol for new sensors: Best practices and minimal data budgets for bringing up a new tactile sensor (encoder initialization, data augmentations, calibration steps) are not established.
  • Task complexity and multi-contact dexterity: Benchmarks emphasize insertion, pressing, and some in-hand tasks; capabilities for complex regrasping with simultaneous multi-digit contacts and non-convex contact transitions are not probed.
  • Interpretability of tactile representations: What the tactile expert encodes and how tokens influence actions is not analyzed (e.g., saliency, attention maps, causal ablations).
  • Standardization pipeline ablations: The effects of the data normalization choices (coordinate frames, resampling, filtering) used to aggregate 26 sources are not validated via ablations.
  • Baseline breadth: Comparisons omit several recent tactile-fusion and force-aware MoE approaches and strong closed-loop force controllers; head-to-head evaluations would clarify gains attributable to FTP-1’s architecture vs. pretraining.
  • Continual/online adaptation: There is no mechanism or study of on-the-fly adaptation to changing contact dynamics or sensors, nor analysis of catastrophic forgetting during finetuning.
  • Zero-shot capabilities: FTP-1 requires finetuning on unseen sensors; true zero-shot or ultra–few-shot (<10 demos) transfer across sensors and tasks is not evaluated.
  • Evaluation metrics beyond success rate: Force smoothness, contact stability, energy, path efficiency, and user preference metrics are not reported, limiting insights into control quality.
  • Data governance and reproducibility: The 26-source aggregation’s licensing, reproducibility of curation, and potential privacy constraints are not detailed; how easily others can reproduce MTTS mappings for new datasets is unclear.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

The following applications can be deployed now by leveraging FTP-1’s released checkpoints, dataset, MTTS interface, and training code, with modest fine-tuning and systems integration.

  • Sensor-agnostic upgrades for contact-rich industrial insertion
    • Sectors: manufacturing, electronics assembly, automotive
    • Use case: Rapidly fine-tune FTP-1 to perform peg-in-hole, USB/connector insertion, and fastening on existing arms with diverse tactile sensors; expect higher stability and recovery behaviors with limited new demos.
    • Tools/workflows/products: Pretrained FTP-1 checkpoint, MTTS-based tactile drivers, small in-factory demo collection (tens–hundreds of trajectories), fine-tuning scripts integrated into existing VLA pipelines.
    • Assumptions/dependencies: Availability of tactile sensors (image, array, or F/T), MTTS mapping for the hardware, real-time control stack; low-level hybrid force/position control still required for best performance.
  • Retrofitting fleets with heterogeneous tactile sensors under a unified policy
    • Sectors: robotics OEMs, system integrators, sensor vendors
    • Use case: Add tactile capabilities to deployed robots (e.g., swapping GelSight to Contactile) by training only a sensor-specific encoder while reusing the shared tactile expert; reduce per-sensor policy engineering.
    • Tools/workflows/products: MTTS-compliant sensor SDKs, encoder templates (ViT/CNN/MLP), calibration and tokenization utilities.
    • Assumptions/dependencies: Accurate functional-area mapping and calibration; sufficient compute for inference; driver reliability in production environments.
  • Improved force-aware pressing and wiping in service and cleaning robots
    • Sectors: facilities/janitorial robotics, hospitality, food service, lab automation
    • Use case: Maintain stable contact and force during wiping/pressing tasks (e.g., boards, dishes, lab plates) with fewer oscillations or loss of contact.
    • Tools/workflows/products: Fine-tuned FTP-1 policy with task-specific demos; monitoring dashboards for force stability.
    • Assumptions/dependencies: Access to compliant hardware or low-level controller for force regulation; safety interlocks; task-specific surfaces and environmental variability.
  • Dexterous household tasks with reliable tactile feedback
    • Sectors: consumer robotics, assistive technologies
    • Use case: Robustly twist caps, align and place small objects, flip pages/books, and perform deformable-object interactions (e.g., balloon drawing or delicate wiping) with higher success rates.
    • Tools/workflows/products: Home-robot skill library initialized from FTP-1; minimal task-specific demos; MTTS-based integration with low-cost tactile sensors.
    • Assumptions/dependencies: Cost-effective, durable tactile sensors; latency constraints on embedded compute; user safety and fail-safes.
  • Tactile-augmented VLA pipelines for robotics software stacks
    • Sectors: robotics software platforms, system software vendors
    • Use case: Drop-in tactile expert branch alongside vision-LLMs to improve contact-rich task performance without degrading visual perception modules.
    • Tools/workflows/products: FTP-1 tactile expert module, adapters for existing VLA models, unified action space (UAS) integration.
    • Assumptions/dependencies: Model serving infrastructure for multi-expert architectures; scheduling and real-time QoS management.
  • Data-efficient onboarding of new tactile sensors
    • Sectors: tactile sensor startups, research labs
    • Use case: Achieve competitive performance on new sensors with limited downstream data (50–100 demos) by reusing shared tactile expert and training only the encoder.
    • Tools/workflows/products: Encoder bootstrapping templates; benchmarking with UniVTAC + small real-world suites.
    • Assumptions/dependencies: Domain shift management; careful sensor-to-functional-area mapping; periodic recalibration.
  • Academic benchmarking and curriculum for tactile generalization
    • Sectors: academia, education
    • Use case: Use FTP-1-Dataset and MTTS to teach cross-sensor tactile policy design and run standardized evaluations across image/array/state sensors.
    • Tools/workflows/products: Released dataset, pretraining/fine-tuning code, evaluation protocols across seen/unseen sensors.
    • Assumptions/dependencies: Hardware access for practical labs; compute grants or shared clusters for training.
  • OEM sensor drivers and MTTS compliance kits
    • Sectors: tactile hardware vendors
    • Use case: Ship MTTS-compliant drivers and tokenizers to make sensors “foundation-ready,” lowering customers’ integration costs.
    • Tools/workflows/products: MTTS mapping guides, reference encoder implementations, calibration tools.
    • Assumptions/dependencies: Collaboration with end-users to validate mappings; maintenance of firmware/software updates.
  • Teleoperation stabilization with tactile-aware policies
    • Sectors: remote operations, hazardous environments
    • Use case: Reduce action jitter and recover from partial failures in contact-rich teleop by using FTP-1 as a stabilizing assistive policy.
    • Tools/workflows/products: Assistive shared-autonomy layer conditioned on operator input; real-time tactile streaming and tokenization.
    • Assumptions/dependencies: Low-latency bidirectional links; safety oversight; UI for human-in-the-loop arbitration.

Long-Term Applications

These applications require further research, scaling, dataset expansion, or deeper systems integration (e.g., low-level force servoing, standardization, certification).

  • Industry-wide tactile interface standardization (MTTS as a common layer)
    • Sectors: standards bodies, robotics industry consortia
    • Use case: Establish MTTS-like functional-area standards for tactile data interchange to enable plug-and-play policies across vendors.
    • Tools/workflows/products: Open specifications, compliance test suites, conformance certification programs.
    • Assumptions/dependencies: Multi-stakeholder buy-in; alignment with existing force/torque and safety standards; IP/licensing clarity.
  • General-purpose assembly lines with fast SKU switching
    • Sectors: manufacturing, electronics
    • Use case: Reduce reprogramming time by transferring tactile manipulation skills across fixtures, grippers, and sensor stacks with minimal data collection.
    • Tools/workflows/products: Skill graphs initialized from foundation tactile policies; rapid task adapters; auto-calibration workflows.
    • Assumptions/dependencies: Broader pretraining coverage, robust domain adaptation, and integration with MES/QA systems.
  • Hybrid force-position control with predictive tactile world models
    • Sectors: robotics controls, advanced manufacturing
    • Use case: Combine FTP-1’s perception with predictive tactile models and low-level controllers for fine-grained servoing (e.g., force-aware micro-assembly).
    • Tools/workflows/products: Learned tactile dynamics modules; compliant controllers; safety-verified control stacks.
    • Assumptions/dependencies: Research advances in tactile prediction and control; formal safety validation.
  • Household generalist robots for delicate manipulation
    • Sectors: consumer robotics
    • Use case: Reliable in-home tasks (e.g., inserting chargers, opening containers, dish stacking) across varied tactile hardware and surfaces.
    • Tools/workflows/products: Consumer-grade tactile suites, model compression/distillation for edge devices, self-calibration workflows.
    • Assumptions/dependencies: Robust low-cost sensors, on-device inference budgets, rigorous safety and user privacy frameworks.
  • Assistive and healthcare manipulation
    • Sectors: healthcare, eldercare, rehabilitation
    • Use case: Dressing aid, feeding assistance, and mobility support using tactile-aware policies that adapt to personalized tactile sensors (gloves, sleeves).
    • Tools/workflows/products: MTTS-mapped wearable tactile arrays, certified assistive robots, clinician-in-the-loop training tools.
    • Assumptions/dependencies: Medical device regulation, human safety certification, extensive clinical validation.
  • Surgical and micro-manipulation robotics
    • Sectors: surgical robotics, life sciences
    • Use case: Sub-millimeter, force-sensitive tasks leveraging generalized tactile perception fused with high-resolution vision.
    • Tools/workflows/products: Sterilizable tactile end-effectors, high-frequency low-latency controllers, validated simulators for pretraining.
    • Assumptions/dependencies: Sensor innovation at surgical scale, stringent regulatory pathways, explainability and auditability.
  • Human-robot collaboration safety via tactile reflexes
    • Sectors: industrial HRC, logistics
    • Use case: Use generalized tactile cues to trigger safe, context-aware responses (e.g., compliant yielding, micro-adjustments) in shared workspaces.
    • Tools/workflows/products: Safety supervisors integrated with tactile expert; certified behaviors; real-time monitoring.
    • Assumptions/dependencies: Standardized risk assessments, fail-safes, certification under ISO/TS standards.
  • Tactile skill marketplaces and cross-vendor portability
    • Sectors: robotics platforms, SI ecosystem
    • Use case: Distribute and deploy “skill packs” (e.g., insertion, cap twisting) across embodiments and sensors using MTTS-compliant encoders.
    • Tools/workflows/products: Skill catalogs, deployment toolchains, telemetry and A/B testing for skill updates.
    • Assumptions/dependencies: Licensing and IP models, versioning standards, cross-platform validation.
  • Edge deployment via model compression and distillation
    • Sectors: embedded/edge AI, hardware accelerators
    • Use case: Run tactile experts on robot controllers (or micro-controllers) with strict real-time constraints.
    • Tools/workflows/products: Distilled/quantized tactile experts; compiler/runtime support for multi-expert policies.
    • Assumptions/dependencies: Maintain tactile performance under compression; co-design with hardware accelerators.
  • Policy and regulatory frameworks for tactile-AI safety and data governance
    • Sectors: public policy, regulation, standards
    • Use case: Define evaluation protocols and guardrails for tactile-driven robots (e.g., contact force limits, fail-safe behaviors) and tactile data privacy.
    • Tools/workflows/products: Conformance tests using MTTS-formatted datasets, incident reporting standards, certification pathways.
    • Assumptions/dependencies: Collaboration across regulators, industry, and academia; harmonization with existing machine safety and AI governance norms.
  • Cross-embodiment transfer for dexterous hands and humanoids
    • Sectors: humanoid robotics, advanced manipulators
    • Use case: Transfer tactile skills between multi-fingered hands and parallel grippers by exploiting MTTS’s functional-area mapping and unified action spaces.
    • Tools/workflows/products: Embodiment adapters, simulation-to-real pipelines, broader pretraining datasets with human and robot demos.
    • Assumptions/dependencies: Greater dataset diversity, improved embodiment-agnostic control abstractions, robust sim2real methods.

Notes on feasibility and dependencies across applications

  • Real-time constraints: The multi-expert architecture (including a ~300M-parameter tactile expert) must meet control-loop latencies; compression and optimized serving may be necessary for embedded deployment.
  • Low-level control: FTP-1 focuses on perception and high-level action chunking; best performance on force-critical tasks likely requires integration with hybrid force/position controllers.
  • Sensor mapping: Accurate MTTS functional-area mapping and calibration are critical; errors degrade transfer and stability.
  • Data and compute: Fine-tuning requires task demos and compute resources; on-device inference may need distillation.
  • Safety: Contact-rich tasks entail force limits and collision handling; regulatory compliance is mandatory in industrial and medical settings.
  • Generalization limits: Performance benefits (+17.2% on seen setups; +31.6% on unseen) depend on similarity to pretraining distribution and the quality of the sensor encoder.
View Paper Prompt  View All Prompts 

Glossary

  • Ablation studies: controlled experiments that remove or modify components to isolate their impact on performance. "Ablation studies further confirm that these gains stem from FTP-1's transferable tactile manipulation skills."
  • Action chunk: a contiguous sequence of future actions predicted in one step. "FTP-1 predicts an action chunk At:t+H-1 = TO(l, It, St, Xt)"
  • Action horizon: the number of future timesteps in the predicted action sequence. "H is the action horizon, and D is the di- mension of a predefined Unified Action Space (UAS) [13]"
  • Adaptive RMSNorm: a variation of Root Mean Square normalization adapted to improve fusion of modalities. "We further find that fusing proprioceptive state with adaptive RMSNorm [5] improves performance and generalization, as detailed in App. B.3."
  • Array-type inputs: tactile observations represented as structured arrays rather than images or scalar states. "For array-type inputs (e.g., Contactile [19]), we use a CNN [20] to cap- ture spatial tactile structure and compress each functional area into one token."
  • CNN: Convolutional Neural Network, used to process grid-structured tactile arrays. "we use a CNN [20] to cap- ture spatial tactile structure and compress each functional area into one token."
  • Contact-rich manipulation: robotic tasks requiring sustained, feedback-driven physical contact. "tactile sensing plays an essential role in contact-rich, fine-grained manipula- tion."
  • Dexterous manipulation: fine-grained, multi-finger manipulation requiring precise control. "including in-hand adjustment, force-controlled pressing, insertion and extraction, and long-horizon fine-grained dexterous manipulation."
  • Embodiment: the specific robot hardware configuration and morphology a policy operates on. "transferable tactile manipulation abilities across diverse sensors and embodiments"
  • Flow-matching Transformer action expert: an action-generation Transformer trained with flow-matching objectives to attend to perception tokens. "whose outputs are attended by a flow-matching Transformer action expert [1]."
  • Fourier-encode: map raw signals into sinusoidal feature spaces to capture periodicity and scale. "For state-type inputs (e.g., force-torque), we Fourier-encode the raw state [21] and process it with a lightweight MLP."
  • Functional areas: predefined sensor/hand regions used to organize heterogeneous tactile signals into tokens. "MTTS organizes tactile sig- nals into 24 functional areas [14], each represented by one token; the full definition is shown in Fig. 3."
  • Functional-area embedding: learnable vectors indicating the functional region associated with each tactile token. "we add a learnable functional-area embedding, shared across all sensors, be- fore feeding the tokens into the subsequent architecture."
  • GelSight: a vision-based, high-resolution tactile sensor that produces image-type tactile inputs. "For image-type inputs (e.g., GelSight [16]), we use a lightweight sensor-specific ViT [17] followed by a shared pretrained T3. Transformer tactile encoder [18] across sensors"
  • Heterogeneous tactile encoders: modality-specific encoders used to project diverse tactile inputs into a unified token space. "use heterogeneous tactile encoders [15] to tokenize sensor-specific inputs into MTTS."
  • Image-type inputs: tactile observations captured as images from vision-based tactile sensors. "For image-type inputs (e.g., GelSight [16]), we use a lightweight sensor-specific ViT [17]"
  • Latent tokens: compressed, learned token representations that unify heterogeneous tactile modalities. "by using heterogeneous encoders to project them into unified morphology-aware latent tokens that are jointly mod- eled by a shared tactile Transformer expert."
  • Mixture-of-Experts (MoE): a modular architecture that routes inputs to multiple experts for specialized processing. "More complex modality fusion designs e.g. MoE-based fusion [25, 24, 26], did not yield consistent gains in our experiment, so we adopt the simplest multi-expert design."
  • Morphology-Aware Tactile Token Space (MTTS): a unified tactile interface organizing signals by functional areas and embeddings to enable sensor-agnostic modeling. "we introduce Morphology-Aware Tactile Token Space (MTTS) as a unified inter- face across sensors."
  • Multi-expert architecture: a design that uses separate, specialized experts (e.g., tactile, vision-language, action) for modality fusion. "FTP-1 adopts a multi-expert architecture [1] to fuse tactile tokens with vision-language perception for action generation, as shown in Fig. 2."
  • Pretraining: large-scale prior training that provides a strong initialization for downstream tasks. "pretrain- ing provides powerful initialization for downstream manipulation tasks."
  • Proprioception: internal sensing of robot states such as joint positions or forces. "Given a language instruction l, (multi-view) RGB observations It, proprioception st, and tactile observations At, FTP-1 predicts an action chunk"
  • Sensor-specific ViT: a Vision Transformer tailored to a particular tactile sensor’s image modality. "we use a lightweight sensor-specific ViT [17] followed by a shared pretrained T3. Transformer tactile encoder [18] across sensors"
  • State-type inputs: low-dimensional tactile signals such as force/torque readings treated as scalar state observations. "For state-type inputs (e.g., force-torque), we Fourier-encode the raw state [21] and process it with a lightweight MLP."
  • Tactile manipulation: robotic manipulation that leverages tactile sensing for perception and control. "transferable tactile manipulation abilities across diverse sensors and embodiments"
  • Tactile-VLA: a tactile-augmented vision-language-action architecture that injects tactile tokens into the VLM expert without a separate tactile expert. "Tactile-VLA [9]: a tactile-based VLA architecture adapted from Huang et al. [9], which injects tokenized tactile inputs into the VLM expert without a separate tactile expert."
  • Tokenize: convert raw sensor inputs into discrete tokens for Transformer-based modeling. "use heterogeneous tactile encoders [15] to tokenize sensor-specific inputs into MTTS."
  • Transformer: an attention-based neural network architecture used for modeling sequences and multimodal tokens. "The tactile expert is chosen as a 300M-parameter Transformer [12]."
  • Unified Action Space (UAS): a common, predefined control space that normalizes action representations across heterogeneous embodiments. "D is the di- mension of a predefined Unified Action Space (UAS) [13] that handles heterogeneity of control signal"
  • Vision-language Transformer expert: the Transformer module that processes visual observations and language instructions. "processes image observations and language instructions with a pretrained vision-language Transformer expert [22]"
  • Vision-Language-Action (VLA) model: an integrated model that maps vision and language inputs to actions. "TT0.5 [5]: a SOTA open-source VLA model without tactile input."
  • ViT: Vision Transformer, a Transformer architecture adapted for image inputs. "we use a lightweight sensor-specific ViT [17]"
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 9 tweets with 281 likes about this paper.

Sign Up for Free