MLLM-Fabric: Robotic Fabric Sorting

• MLLM-Fabric is a multimodal robotic framework that integrates RGB, visuotactile, and pressure sensing to perform pairwise fabric comparisons for sorting and selection.
• It employs a two-stage training pipeline with direct supervision and explanation-guided knowledge distillation, merging modalities into a shared token space for robust decision-making.
• Experimental results show that Fabric-Llama-90B outperforms baseline vision-language models in ranking accuracy and selection reliability, though fine-grained adjacent-level discrimination remains challenging.

MLLM-Fabric is a multimodal LLM-driven robotic framework for fabric sorting and selection in textile manufacturing, apparel production, and smart retail. It combines a robotic arm, an RGB-D camera, a visuotactile sensor, and a pressure sensor, and it uses supervised fine-tuning together with multimodal explanation-guided knowledge distillation to classify and rank fabric properties. The reported system is centered on pairwise comparison and request-conditioned selection—for example, selecting the fabric that best satisfies a prompt such as “maximize softness”—and is accompanied by a dataset of 220 unique fabric samples with RGB images and synchronized visuotactile and pressure data. In the reported experiments, the resulting Fabric-Llama-90B model outperforms pretrained vision-language baselines in both property ranking accuracy and selection reliability (Wang et al., 6 Jul 2025).

1. Task formulation and operational scope

The framework addresses a concrete decision problem: choosing the right fabric to meet functional and quality requirements in robotic applications. Rather than treating fabric understanding as a purely visual recognition problem, MLLM-Fabric formulates sorting and selection through multimodal comparison over four ordinal attributes: elasticity, softness, thickness, and texture. Each fabric is annotated on an ordinal scale $\{0,1,2\}$, corresponding to low, moderate, and high levels, and the learning setup emphasizes extreme-level discrimination by excluding moderate samples from extreme-level training (Wang et al., 6 Jul 2025).

Operationally, the system is organized as a sequence of modules. The Vision Module captures tuples $(F_i, P_i, I_i)$ consisting of fabric ID, 3D pose, and segmented RGB image. The Tactile Module synchronizes each GelSight image frame $x^{gel}_t$ with a force reading $f_t$, forming sequences $\{x^{gel}_t, f_t\}_{t=1\ldots T}$. All modalities are archived per fabric ID. A Reasoning Module then fuses visual, tactile, and force embeddings into a shared token space for downstream comparison. Finally, an Execution Module receives a user request, performs pairwise comparisons, and instructs the robotic arm to pick the chosen sample via its stored ID and pose (Wang et al., 6 Jul 2025).

This organization makes the framework a selection system rather than only a classifier. A plausible implication is that the ranking objective is structurally aligned with downstream robotic manipulation: the model is not merely asked to assign an attribute label, but to support a final pick under an explicit task request.

2. Robotic platform and sensing stack

The hardware platform comprises a Franka Emika Panda robotic arm, an Intel RealSense D435 camera, a GelSight Mini visuotactile sensor, and a planar strain-gauge pressure sensor (HZC-30B). The Panda manipulates and positions fabrics. The RealSense D435, mounted on the arm, provides RGB and depth for fabric localization. ArUco tags provide unique IDs and 3D poses, and the Segment Anything Model (SAM) segments each sample into $224\times224$ RGB crops. On the tactile side, the GelSight Mini records high-resolution deformation frames at 25 Hz, while the HZC-30B records normal contact force in the 0–10 kg range at 25 Hz (Wang et al., 6 Jul 2025).

Subsystem	Component	Reported role
Manipulation	Franka Emika Panda	Manipulates and positions fabrics
Vision	Intel RealSense D435 + ArUco + SAM	Localization, ID/pose recovery, $224\times224$ RGB crops
Visuotactile sensing	GelSight Mini	High-resolution deformation frames at 25 Hz
Pressure sensing	HZC-30B	Normal contact force, 0–10 kg range, at 25 Hz

The sensing design is explicitly synchronized and multimodal. Per sample, the archived metadata are stored as $(F_i, I_i, \{x^{gel}_t, f_t\})$, with RGB and GelSight normalized, SAM-based segmentation applied to RGB, and tactile and force streams time-synchronized (Wang et al., 6 Jul 2025). This suggests that the framework treats contact mechanics and surface appearance as jointly informative, rather than assigning tactile sensing a purely auxiliary role.

3. Model architecture and multimodal representation

The model backbone is Llama3.2-Vision-90B, referred to in the framework as “Fabric-Llama-90B,” described as a 90 B-parameter multimodal LLM with a vision encoder pre-aligned to the Transformer (Wang et al., 6 Jul 2025). Adaptation is performed with LoRA adapters inserted into the key/value projections of the Transformer. The reported weight update is

$W' = W + \Delta W,\qquad \Delta W = AB,$

with

$$A \in \mathbb{R}^{d\times r},\qquad B \in \mathbb{R}^{r\times d},\qquad r \ll d.$$

Multimodal fusion proceeds through tokenization of all input streams into a single shared sequence. RGB crops and GelSight frames are processed by a frozen visual encoder into sequences of patch tokens. Scalar force readings $(F_i, P_i, I_i)$0 are converted into text-like tokens such as “force=3.5 N” and embedded via an input projection. All tokens are then concatenated and fed into the multimodal Transformer for early fusion and cross-modal attention (Wang et al., 6 Jul 2025).

This fusion strategy is central to the framework’s reported behavior. The paper’s ablation results indicate that the full modality combination—RGB, GelSight sequence, and force—is crucial, and that removing force drops performance by approximately 30 points (Wang et al., 6 Jul 2025). A plausible implication is that the model relies not only on static tactile images but also on the temporal coupling between deformation and applied force.

4. Training procedures and supervision design

The training pipeline has two stages: Directly Supervised Fine-Tuning (D-SFT) and multimodal explanation-guided knowledge distillation (MEKD). In the seen-fabric setting, 200 fabrics produce 24,000 extreme-level $(F_i, P_i, I_i)$1 pairs, with 4,000 pairs used for training and 400 for validation under equal attribute balance. A separate unseen-fabric setting holds out 20 fabrics for generalization tests. Each pair is labeled “left better,” “right better,” or “inconclusive” (Wang et al., 6 Jul 2025).

For D-SFT, the objective is cross-entropy over the three classes for each property $(F_i, P_i, I_i)$2:

$(F_i, P_i, I_i)$3

Inference uses

$(F_i, P_i, I_i)$4

MEKD adds rationale supervision. GPT-4o serves as a teacher that generates post hoc explanations $(F_i, P_i, I_i)$5 for each ground-truth comparison, after which human experts refine those explanations. Fabric-Llama-90B serves as the student and generates ad hoc reasoning $(F_i, P_i, I_i)$6. Distillation enforces agreement between student rationales and refined teacher rationales using cross-entropy,

$(F_i, P_i, I_i)$7

and the total loss is

$(F_i, P_i, I_i)$8

The distinction between post hoc teacher explanations and ad hoc student reasoning is methodologically notable. The reported formulation suggests that explanation supervision is used not as a separate interpretability layer, but as an optimization target coupled to the ranking objective itself (Wang et al., 6 Jul 2025).

5. Dataset composition and evaluation protocol

The released dataset contains 220 real-world fabric samples, each annotated on the four ordinal attributes $(F_i, P_i, I_i)$9. For each sample, the dataset includes a segmented RGB image of size $x^{gel}_t$0, a GelSight sequence of 250 frames over 10 s at 25 Hz, and aligned force readings $x^{gel}_t$1 at 25 Hz (Wang et al., 6 Jul 2025).

Evaluation uses four reported measures. Property-wise Accuracy (ACC) is defined as

$x^{gel}_t$2

Mean Accuracy (mean ACC) is the average ACC over the four attributes. Prediction Skewness (SK) quantifies class bias:

$x^{gel}_t$3

where $x^{gel}_t$4 is the fraction of “class 0” predictions, and lower values are better. Real-world Selection Reliability is defined as the fraction of scenarios in which the model’s final pick matches expert choice (Wang et al., 6 Jul 2025).

These metrics make explicit that the framework is evaluated not only on average correctness but also on class bias and end-to-end robotic selection. The inclusion of SK is particularly relevant because pairwise comparison systems can achieve superficially strong results while collapsing toward one decision class.

6. Empirical results, limitations, and relation to adjacent textile AI

In the principal multimodal setting using RGB, GelSight sequence, and force, the reported seen-fabric results are as follows (Wang et al., 6 Jul 2025):

Training regime	mean ACC	SK
Llama3.2-Vision-90B (no fine-tuning)	47%	5.5
+ D-SFT	93%	1.0
Fabric-Llama-90B (MEKD, frozen vision encoder)	94%	0.0

On unseen fabrics, Fabric-Llama-90B reaches mean ACC $x^{gel}_t$5 and SK $x^{gel}_t$6, compared with a baseline of $x^{gel}_t$7 (Wang et al., 6 Jul 2025). In ten real-world selection scenarios, Fabric-Llama-90B achieves 8/10 correct (80%), outperforming GPT-4o, reported at 0.3 mean sorting ACC, and modular CNN-LSTM + frozen MLLM pipelines, reported at 50% (Wang et al., 6 Jul 2025). The paper also reports that frozen encoder + LoRA outperforms full fine-tuning under limited data, with mean ACC 94% versus 73%, and that sorting four fabrics takes approximately 2–3 minutes on an A800 GPU while enabling “chain-of-thought” style stepwise justifications (Wang et al., 6 Jul 2025).

The same report documents clear failure modes. Fine-grained comparisons between adjacent ordinal levels—$x^{gel}_t$8 and $x^{gel}_t$9—remain near chance, with mean ACC approximately 50% and SK greater than 50. Multi-attribute scenarios involving three properties are more error-prone, which the paper interprets as indicating a need for finer attribute fusion or hierarchical reasoning. The stated future directions are duplicate-aware training, finer ordinal supervision, and real-time parallel inference (Wang et al., 6 Jul 2025). A common misconception would be to equate strong extreme-level ranking with robust ordinal reasoning across the entire label space; the reported near-chance adjacent-level performance directly contradicts that interpretation.

Within the broader textile-AI landscape, MLLM-Fabric belongs to a family of systems that use large models to structure fabric-related tasks, but its objective is distinct from generative textile modeling. For example, FabricGen addresses woven fabric generation for rendering by decomposing material into a macro-scale albedo/texture map and a micro-scale geometric model, and by using a specialized WeavingLLM to produce weaving drafts and fabric parameters (Tang et al., 7 Mar 2026). The contrast is substantive: MLLM-Fabric is an embodied robotic framework for sorting and selection, whereas FabricGen is a generation-and-rendering framework. This suggests that “fabric AI” currently spans at least two technically separate directions—multimodal robotic decision-making and microstructure-aware material synthesis—despite sharing large-model components (Wang et al., 6 Jul 2025).