Fabric-Llama-90B: Multimodal Fabric Selection Model

• Fabric-Llama-90B is a multimodal model fine-tuned for property-specific fabric comparisons, leveraging synchronized RGB images, GelSight tactile data, and pressure readings.
• It employs pairwise comparison methodology to rank fabric properties such as elasticity, softness, thickness, and texture, with an abstain option for inconclusive cases.
• The model, optimized via MEKD and LoRA with a frozen vision encoder, achieves a mean ACC of 94% on seen-fabric extreme pairs in robotic fabric selection tasks.

Fabric-Llama-90B is a multimodal LLM proposed within the MLLM-Fabric robotic framework for fabric sorting and selection. It is obtained by fine-tuning Llama3.2-Vision-90B for pairwise comparison, ranking, and selection of fabrics using synchronized RGB images, visuotactile GelSight sequences, and scalar pressure values. The model is trained to answer property-conditioned comparisons such as elasticity, softness, thickness, and texture, and it may abstain when differences are inconclusive. In the reported experiments, the recommended variant—MEKD-fine-tuned with a frozen vision encoder—achieves a mean ACC of 94% and SK of 0 on seen-fabric extreme pairs, and it is embedded in a robotic system that performs perception, reasoning, and pick-and-place execution (Wang et al., 6 Jul 2025).

1. Definition and task formulation

Fabric-Llama-90B is the proposed multimodal model obtained by fine-tuning Llama3.2-Vision-90B, with approximately 90B parameters, for robotic fabric selection. Its central task is not generic fabric classification but property-specific pairwise comparison: given two fabrics and an instruction such as “Which fabric is softer?”, the model predicts one of three outcomes—fabric A, fabric B, or abstain. Supervision is explicitly organized around four semantic physical properties: elasticity, softness, thickness, and texture (Wang et al., 6 Jul 2025).

This task formulation is significant because it recasts material selection as context-aware decision-making rather than category labeling. The model is trained to determine which of two fabrics better satisfies a requested property, and this pairwise setup is also used as the basis for ranking and downstream selection. The paper describes this as enabling fabric selection beyond static labels and toward functional decision-making in robotics. A plausible implication is that the abstention mechanism is intended to mitigate overconfident behavior in ambiguous comparisons, especially when fabrics are near-duplicates or differ only weakly along the queried dimension.

The model operates inside the broader MLLM-Fabric system, which includes a robotic arm, a camera, a visuotactile sensor, and a pressure sensor. Within that system, Fabric-Llama-90B is the reasoning component that consumes multimodal evidence and produces pairwise judgments and, during distillation, natural-language explanations. For selection tasks involving multiple constraints, it performs stepwise reasoning over multiple attributes before the robot retrieves the target fabric.

2. Multimodal architecture and input representation

Fabric-Llama-90B uses three input modalities: segmented RGB images, visuotactile images from GelSight Mini sequential frames, and scalar pressure values from a strain-gauge sensor. RGB and GelSight images are processed by a pretrained visual encoder tied to Llama3.2-Vision. In the recommended configuration, the visual encoder is frozen and only the language backbone is adapted via LoRA. Pressure signals are transformed into a “language-compatible” representation and embedded through the model’s input projection layers, after which they are treated as tokens within the multimodal transformer (Wang et al., 6 Jul 2025).

Fusion is performed by embedding each modality into a shared token space and jointly processing the resulting tokens through early fusion. The description emphasizes “token-level reasoning” rather than a separate cross-attention module. Pressure is therefore not handled by an external side network; instead, it is injected through the model’s input projections and co-reasoned with image tokens and instruction text. This architectural choice places force information on the same inferential path as the visual inputs.

The input format is instruction-conditioned and pairwise. Each prompt contains two fabrics, the target property, and the abstain option. This design couples multimodal perception with explicit comparison semantics. The architecture is thus specialized for relational judgments—determining which item better satisfies a query—rather than standalone captioning or VQA. The reported ablations further suggest that the tri-modal formulation is central to performance: RGB plus GelSight sequential frames plus force is the strongest modality combination, whereas unlabeled GelSight sequences alone perform poorly.

3. Training objectives and adaptation strategy

Training is organized around two pipelines: supervised fine-tuning, denoted D-SFT, and Multimodal Explanation-Guided Knowledge Distillation, denoted MEKD. In D-SFT, the pairwise comparison problem is cast as 3-way classification over A, B, and abstain. The cross-entropy objective is reported as

$$L_{\text{CE}} = -\sum_i \sum_{c \in \{A,B,\text{abstain}\}} y_{i,c}\log p_{i,c}.$$

The paper also gives the compact training form

$$L_{\text{train}} = \sum_{i=1}^{N}\sum_{q \in Q'} L_{\text{CE}}\big(g(z_i,\phi), y_{i,q}\big),$$

where $g(z_i,\phi)$ is the predicted distribution for property $q$, $y_{i,q}$ is the ground-truth label, and $z_i$ contains the multimodal inputs for the pair (Wang et al., 6 Jul 2025).

Parameter-efficient adaptation is implemented with LoRA:

$W' = W + \Delta W, \quad \Delta W = AB,$

with $A \in \mathbb{R}^{d \times r}$, $B \in \mathbb{R}^{r \times d}$, and $r \ll d$. Two strategies were explored: full fine-tuning of vision and language modules, and language-only fine-tuning with the vision encoder frozen. The best results were obtained with frozen vision encoders, indicating that under the reported data regime, constraining adaptation to the language backbone is empirically preferable.

MEKD distills GPT-4o’s post hoc explanations into the student model. Explanations are generated for labeled comparisons, refined by experts, and then used to align the student’s ad hoc reasoning. The teacher explanation is expressed as

$$L_{\text{train}} = \sum_{i=1}^{N}\sum_{q \in Q'} L_{\text{CE}}\big(g(z_i,\phi), y_{i,q}\big),$$0

where $$L_{\text{train}} = \sum_{i=1}^{N}\sum_{q \in Q'} L_{\text{CE}}\big(g(z_i,\phi), y_{i,q}\big),$$1 is the multimodal pair input, $$L_{\text{train}} = \sum_{i=1}^{N}\sum_{q \in Q'} L_{\text{CE}}\big(g(z_i,\phi), y_{i,q}\big),$$2 is the instruction, and $$L_{\text{train}} = \sum_{i=1}^{N}\sum_{q \in Q'} L_{\text{CE}}\big(g(z_i,\phi), y_{i,q}\big),$$3 is the expert-validated ranking. Student alignment is trained with

$$L_{\text{train}} = \sum_{i=1}^{N}\sum_{q \in Q'} L_{\text{CE}}\big(g(z_i,\phi), y_{i,q}\big),$$4

The paper specifies cross-entropy for explanation alignment and does not report KL-divergence, temperature-based KD, ranking losses, or weighting coefficients for a combined loss. In practice, “Fabric-Llama-90B” refers to the MEKD-fine-tuned Llama3.2-Vision-90B with the vision encoder frozen.

Reported training details are partial. Pairs are created from 200 training fabrics at extreme attribute levels, yielding 24,000 possible pairs, of which 4,000 are sampled for training and 400 for validation, balanced across the four properties. Llama3.2-Vision-90B is trained on an NVIDIA A800 with 80 GB for approximately 6.5 hours. Optimizer, batch size, learning rate, epochs, instruction-tuning details, and KD temperature are not disclosed.

4. Dataset, sensing apparatus, and annotation protocol

The accompanying dataset contains 220 unique real-world fabric samples with RGB images, GelSight sequential frames, and synchronized pressure values. GelSight and pressure are sampled at 25 Hz, and each GelSight frame is aligned to a corresponding scalar pressure reading. Data collection uses two presses per fabric, each of approximately 10 seconds, with a moderate approach and slow closure. The sensing stack consists of a Franka arm with gripper, an Intel RealSense D435 for RGB/depth, a GelSight Mini mounted on the left fingertip, and an HZC-30B strain-based planar pressure sensor mounted under a rigid plane at the right fingertip (Wang et al., 6 Jul 2025).

Visual preprocessing uses ArUco tags as unique identifiers and for pose tracking on a hanger. SAM segments the fabrics, and images are resized to $$L_{\text{train}} = \sum_{i=1}^{N}\sum_{q \in Q'} L_{\text{CE}}\big(g(z_i,\phi), y_{i,q}\big),$$5. The resulting tuples are $$L_{\text{train}} = \sum_{i=1}^{N}\sum_{q \in Q'} L_{\text{CE}}\big(g(z_i,\phi), y_{i,q}\big),$$6 for fabric ID, 3D position, and segmented image. The pressure sensor is specified with rated range 0–10 kg, sensitivity 1.0–1.5 mV/V, nonlinearity $$L_{\text{train}} = \sum_{i=1}^{N}\sum_{q \in Q'} L_{\text{CE}}\big(g(z_i,\phi), y_{i,q}\big),$$7 F.S., creep $$L_{\text{train}} = \sum_{i=1}^{N}\sum_{q \in Q'} L_{\text{CE}}\big(g(z_i,\phi), y_{i,q}\big),$$8 F.S., operating range $$L_{\text{train}} = \sum_{i=1}^{N}\sum_{q \in Q'} L_{\text{CE}}\big(g(z_i,\phi), y_{i,q}\big),$$9 to $g(z_i,\phi)$0, and overload 200% F.S.

Annotation is organized around a four-dimensional attribute vector

$g(z_i,\phi)$1

corresponding to elasticity, softness, thickness, and surface texture, where 0 denotes low, 1 moderate, and 2 high. To reduce ambiguity, supervised training uses only extreme labels 0 and 2; moderate labels are discarded for training. Experts validate both labels and rationales. The seen-fabric setting uses 200 fabrics, while the unseen-fabric setting reserves 20 fabrics excluded from training, with pairs constructed similarly under strict isolation.

A distinctive aspect of the sensing design is the use of Newton’s third law for force labeling. Rather than calibrating GelSight to estimate force, the system uses the pressure reading as the normal contact force on the GelSight surface. This provides aligned force labels for the tactile stream and anchors the multimodal representation in measured contact mechanics rather than inferred force proxies.

5. Evaluation protocol and reported performance

Evaluation uses property-wise Accuracy (ACC), mean ACC across the four attributes, and Prediction Skewness (SK). ACC is defined as

$g(z_i,\phi)$2

and SK as

$g(z_i,\phi)$3

where $g(z_i,\phi)$4 is the proportion predicted as “class 0.” Lower SK indicates more balanced outputs. Spearman’s $g(z_i,\phi)$5 and Kendall’s $g(z_i,\phi)$6 are not used. Selection reliability is reported as selection correctness across ten expert scenarios rather than through a separate formal top-$g(z_i,\phi)$7 metric (Wang et al., 6 Jul 2025).

On seen-fabric extreme pairs, the main reported comparison is as follows:

Model/configuration	mean ACC	SK
Llama3.2-Vision-90B (no fine-tune)	47%	2.1
Llama3.2-Vision-90B + D-SFT	93%	1
Fabric-Llama-90B (MEKD, frozen vision encoder)	94%	0
Fabric-Llama-90B* (MEKD, full fine-tuning of vision+language)	73%	4.5

For Fabric-Llama-90B, the reported property-wise ACC values are Elasticity 90, Thickness 100, Texture 97, and Softness 89. These results indicate that MEKD provides a modest improvement over D-SFT at 90B scale, while full fine-tuning of the vision encoder degrades performance under the reported data conditions.

Model scaling and alternative backbones show weaker behavior. Llama3.2-Vision-11B + D-SFT reaches mean ACC 47 with SK 31.4, and Fabric-Llama-11B reaches mean ACC 53 with SK 33.1. Janus-Pro-7B (no fine-tune) reports mean ACC 39 with SK 84.4; Janus-Pro-7B + D-SFT reports mean ACC 50 with SK 100; and Fabric-Janus-7B reports mean ACC 47 with SK 5.5. The reported pattern is that larger models at 90B scale learn more robust pairwise property judgments and that MEKD substantially improves prediction balance.

Generalization results are mixed. On the original seen extreme pairs, Fabric-Llama-90B reports mean ACC 94 and SK 0. On the fine-grained test using adjacent levels 0 vs 1 and 1 vs 2, which are not used in training, D-SFT reports mean ACC 51.7 with SK 55.5 and Fabric-Llama-90B reports mean ACC 52.2 with SK 54.5. On unseen fabrics, the non-fine-tuned Llama3.2-Vision-90B reports mean ACC 42.8 with SK 13.6, D-SFT reports mean ACC 55.7 with SK 7.4, and Fabric-Llama-90B reports mean ACC 65.7 with SK 1.8. The reported results therefore support moderate coarse-level generalization to held-out fabrics but do not support strong fine-grained discrimination.

The modality ablation further emphasizes multimodal grounding. The best modality combination is RGB plus GelSight sequential frames plus force, reaching 65.2% accuracy for GPT-4o and 47.6% for the Llama3.2 baseline in the ablation setting. Unlabeled GelSight sequences perform poorly, at approximately 19.4% for GPT-4o and 30.8% for the Llama3.2 baseline. In the real-world selection benchmark, Fabric-Llama-90B correctly selects 8 of 10 scenarios, corresponding to 80% selection reliability, while non-fine-tuned MLLMs and interaction trajectory pipelines using CNN-LSTM followed by a frozen MLLM achieve 50% or 5 of 10 selections. One reported scenario, “Lightweight smooth fabric for formal shirts,” reaches 90% sorting accuracy.

Several evaluation boundaries are explicit. Statistical significance is not reported. Baselines such as CLIP, BLIP-2, Flamingo, and LLaVA are not included. The primary baselines are Llama3.2-Vision, DeepSeek Janus-Pro-7B, GPT-4o as the teacher for MEKD, and a CNN-LSTM tactile classifier.

6. Robotic pipeline, execution profile, and deployment constraints

The end-to-end robotic pipeline comprises visual perception, tactile perception, MLLM reasoning, and task execution. Visual perception uses RealSense and ArUco for identification and pose tracking, together with SAM-based segmentation and image resizing. Tactile perception uses GelSight Mini sequences and synchronized pressure values at 25 Hz. Fabric-Llama-90B then processes pairwise multimodal inputs under property-specific instructions, outputs a ranking among A, B, and abstain, and, in distillation-related settings, generates reasoning. Task execution uses the stored fabric ID and 3D position to retrieve the selected fabric via grasp and pick-and-place (Wang et al., 6 Jul 2025).

The paper does not detail grasp policies beyond the press protocol and tracked-position retrieval. Nevertheless, the integration claim is precise: perception, reasoning, and execution are unified in a single framework for deformable object selection. The system also archives fabric data for efficient decision-making, indicating that selection is not performed from raw live sensing alone but from stored multimodal representations aligned with identity and pose metadata.

Deployment constraints are substantial. Sorting four fabrics takes approximately 2–3 minutes on an A800 GPU, largely due to multimodal inference and reasoning generation. Llama3.2-Vision-90B requires large GPU memory; the reported setup uses an 80 GB A800. The paper explicitly notes that real-time operation is constrained by model size and inference time, and it identifies future parallelization and token optimization as likely routes to higher throughput. A plausible implication is that the current configuration is better suited to accuracy-oriented or moderate-throughput industrial workflows than to strict real-time manipulation loops.

The reported robustness profile is similarly bounded. Visual preprocessing with ArUco and SAM supports stable localization, and the model generalizes moderately to unseen fabrics at coarse attribute levels. However, explicit assessments of lighting variability, deformation variability, or other environmental perturbations are not reported.

7. Limitations, safety, and reproducibility

The paper identifies several limitations. Dataset scale and diversity remain restricted to 220 fabrics, and supervised training uses only extreme labels, which limits fine-grained generalization; adjacent-level tests remain near chance. Performance depends strongly on synchronized GelSight and pressure sensing, and removing force labels or sequential tactile cues reduces accuracy. Unseen fabrics produce moderate performance drops, and selection becomes more difficult when multiple attributes must be satisfied simultaneously. The abstention mechanism is also imperfect: models abstain on fewer than 50% of curated duplicate pairs, whereas GPT-4o abstains on more than 90%, indicating that duplicate-aware or uncertainty-aware training remains underdeveloped (Wang et al., 6 Jul 2025).

Interpretability is another bounded area. Explanations are distilled from GPT-4o and refined by experts, but mechanistic attribution to modality-specific evidence is reported as limited. A common overinterpretation would be to treat explanation alignment as full interpretability; the reported evidence supports explanation-guided supervision, not a complete account of how modality-specific signals drive decisions internally.

Safety considerations are practical rather than formalized. Slow closure and moderate approach are intended to mitigate the risk of damaging fabrics or hardware, and overload protection is specified for the pressure sensor. The paper recommends maintaining standard robot safety practices such as collision checks and force limits. On the decision side, the abstain option and multi-attribute reasoning are presented as measures that reduce overconfident decisions, while further redundant sensing or uncertainty thresholds are suggested as possible improvements.

Reproducibility is partial. Dataset and code are stated to be planned for release on HuggingFace and a project website upon publication, and a demonstration video accompanies the paper. Pretrained weights for Fabric-Llama-90B are also planned for release, but exact repository URLs and licensing terms are not provided. The reported reproduction guidance includes attaching ArUco tags, capturing RGB/depth with RealSense, segmenting with SAM, resizing to $g(z_i,\phi)$8, collecting synchronized GelSight sequences and pressure at 25 Hz with two presses per fabric, and fine-tuning Llama3.2-Vision-90B with LoRA and a frozen vision encoder. However, optimizer, learning rate, batch size, epochs, and several instruction-tuning details are not disclosed, which constrains exact replication.

Within those boundaries, Fabric-Llama-90B is positioned as a multimodal model for physically grounded pairwise reasoning over material properties. Its reported contribution lies in combining RGB, visuotactile, and force inputs with D-SFT and MEKD to produce abstention-aware comparisons for robotic fabric selection. The strongest evidence supports coarse-grained property ranking and moderate generalization to unseen fabrics; the weakest evidence concerns fine-grained distinctions, duplicate handling, and fully real-time deployment.