FORGE:Fine-grained Multimodal Evaluation for Manufacturing Scenarios

Abstract: The manufacturing sector is increasingly adopting Multimodal LLMs (MLLMs) to transition from simple perception to autonomous execution, yet current evaluations fail to reflect the rigorous demands of real-world manufacturing environments. Progress is hindered by data scarcity and a lack of fine-grained domain semantics in existing datasets. To bridge this gap, we introduce FORGE. Wefirst construct a high-quality multimodal dataset that combines real-world 2D images and 3D point clouds, annotated with fine-grained domain semantics (e.g., exact model numbers). We then evaluate 18 state-of-the-art MLLMs across three manufacturing tasks, namely workpiece verification, structural surface inspection, and assembly verification, revealing significant performance gaps. Counter to conventional understanding, the bottleneck analysis shows that visual grounding is not the primary limiting factor. Instead, insufficient domain-specific knowledge is the key bottleneck, setting a clear direction for future research. Beyond evaluation, we show that our structured annotations can serve as an actionable training resource: supervised fine-tuning of a compact 3B-parameter model on our data yields up to 90.8% relative improvement in accuracy on held-out manufacturing scenarios, providing preliminary evidence for a practical pathway toward domain-adapted manufacturing MLLMs. The code and datasets are available at https://ai4manufacturing.github.io/forge-web.

• The paper introduces FORGE, a benchmark using real-world 2D images and 3D point clouds to evaluate LLMs on fine-grained manufacturing tasks.
• It rigorously assesses tasks such as workpiece verification, surface inspection, and assembly validation with precise domain annotations.
• The study reveals that current MLLMs struggle with domain-specific reasoning, highlighting the need for targeted fine-tuning and hybrid architectures.

FORGE: Fine-Grained Multimodal Evaluation for Manufacturing LLMs

Motivation and Benchmark Design

FORGE ("Fine-Grained Multimodal Evaluation for Manufacturing Scenarios" (2604.07413)) systematically addresses the lack of rigorous evaluation resources for multimodal LLMs (MLLMs) in manufacturing settings. Modern manufacturing operations, characterized by extensive heterogeneity in data and stringent requirements for semantic and geometric precision, are not adequately captured by existing general-purpose multimodal benchmarks or by simulation-focused, low-diversity datasets. FORGE directly responds to these limitations by curating a high-quality dataset composed of aligned 2D images and 3D point clouds from real-world manufacturing components, complete with fine-grained domain annotations (e.g., precise model numbers).

Figure 1: The main pipeline of FORGE, showing the progression from data curation (images and point clouds) through annotation, scenario construction, and model evaluation.

The benchmark defines three cognitively challenging tasks representative of real-world manufacturing requirements:

1. Workpiece Verification (WorkVeri): Requires discrimination at both the workpiece and model-number level to detect mismatched or out-of-spec components within an assembly.
2. Structural Surface Inspection (SurfInsp): Involves fine-grained defect identification and type classification (e.g., crack, deformation) from 3D point clouds.
3. Assembly Verification (AssyVeri): Demands complex logical reasoning about assembly compatibility, evaluating both high-level assembly rules and subtle specification constraints.

   Figure 2: Task descriptions, input modalities, and example MCQ-style dialogues for all three FORGE tasks.

Dataset and Task Construction

The FORGE dataset is derived from authentic industry parts and categorized into two major subsets:

• 3D Point Cloud Subset: 14 workpiece categories × 90 model numbers, supporting all three task types.
• Image Subset: ∼3,000 images from four manufacturing scenarios, enabling robust testing of MLLMs on both vision-only and multi-view representations.

Importantly, the 3D point cloud data is rendered into three orthogonal 2D projections (front/side/top), enabling use with standard visual encoders in general-purpose MLLMs, due to the lack of widespread 3D LLM backbones.

For synthetic or challenge conditions (e.g., defect simulation in SurfInsp), the authors employ morphology-preserving algorithms and non-rigid deformation grounded in domain best practices.

Figure 3: Data examples from the WorkVeri task, demonstrating scenario diversity and model-number granularity.

Experimental Evaluation and Findings

FORGE comprehensively benchmarks 18 state-of-the-art MLLMs—including both open-weight and proprietary models—under zero-shot, reference-conditioned, and in-context demonstration regimes. Representative models include Qwen3-VL-235B, GPT-5, Gemini-3/Flash, Claude-4.5, and advanced open-source alternatives.

Key quantitative outcomes are as follows:

• Macroscopic vs. Microscopic Perception: While leading MLLMs perform comparably well on coarse workpiece identification (up to 80–100% in some regimes), fine-grained tasks—particularly SurfInsp and model-number-level discrimination—reveal that models, even at frontier scale, typically plateau ≤50% accuracy, only slightly above chance. The best open-weight models (Qwen3-VL-235B, Mistral-3-Large) consistently lag closed-source alternatives in precise error analysis and assembly logic.

  Figure 4: Head-to-head performance comparison of open- vs. closed-source MLLMs, split by workpiece vs. model-number discrimination, revealing persistent deficits on the latter.

• Modality and Conditioning Effects: Introduction of reference images (Ref-Cond) does not yield robust performance gains and can even degrade results, especially for three-view (3V) data. Only the richer in-context demonstration (ICD) setting realizes trajectory improvements, indicating that reference-based visual grounding is not the key bottleneck—task understanding and reasoning remain limiting factors.
• Domain Knowledge Bottleneck: Dedicated bottleneck analyses, including explicit visual grounding and missing-part detection, show that almost all strong MLLMs can localize and annotate 2D/3D components at near-perfect rates, and only fail where domain-specific reasoning and context integration are required.
• Raw 3D Input Feasibility: When fed with serialized raw coordinate tables, even LLMs operate at near-chance, confirming that visual rendering (multi-view projection) remains a necessary adaptation interface for general-purpose MLLMs in manufacturing.

Error Analysis and Qualitative Results

The error taxonomy highlights two archetypal failure patterns:

• Material Hallucination and Over-Reliance: MLLMs often overemphasize inferred or hallucinated material properties (e.g., hypothesizing plastic vs. metal) in tasks not demanding such discrimination, leading to reasoning errors downstream.

Figure 5: Typical error case—in AssyVeri, the model chooses the wrong part due to over-inference of material, demonstrating insufficient domain knowledge integration.

• Partial Successes: Even in incorrect responses, advanced MLLMs occasionally demonstrate nuanced understanding of workpiece degradation, such as wear or chipping, indicating the emergence of latent capabilities useful for predictive maintenance.

Training Resource Potential and Transferability

A significant and practical contribution is the demonstration that FORGE is not just a static benchmark but an actionable fine-tuning resource. When Qwen2.5-VL-3B-Instruct is fine-tuned via supervised domain adaptation on FORGE, performance on held-out categories/situations surges by up to 90.8% (relative improvement), bringing compact open-models within range of much larger proprietary alternatives:

(Figure 1, SFT illustration; described in text as Figure training_results)

Figure training_results: Training curve showing relative improvements for a compact 3B-parameter model post-SFT, validating the dataset's value for domain adaptation.

This out-of-distribution evaluation protocol rules out memorization, showing that the structured, semantically rigorous FORGE annotations successfully encode transferable manufacturing knowledge.

Theoretical and Practical Implications

FORGE reveals that, for industrial AI deployment in smart factories, the dominant limitation for generalist MLLMs is not low-level visual grounding but a deficit in domain-aware, contextually grounded reasoning capabilities. This result is in contrast to many prior domain benchmarks, which overemphasize visual pattern recognition.

Implications include:

• Transfer Learning and Domain Gap: Fine-grained industrial data is necessary for MLLMs to achieve operationally relevant precision and reliability, especially for safety and quality compliance.
• Benchmarking Beyond Simulation: Standard simulation or synthetic benchmarks fail to expose the critical knowledge gaps present in real-world shop-floor deployments.
• Architecture and Training: Neither increased model size nor pure visual pretraining closes the gap in fine-grained manufacturing tasks—domain adaptation and task-specific logic integration are required.

Recommendations for Future Research

• Development of MLLMs that can natively process 3D geometric data, possibly via hybrid vision-language-geometry architectures, is warranted.
• Automated extraction and integration of manufacturing ontologies or standards into the LLM's training and inference cycles may be necessary to achieve consistent fine-grained performance.
• Extension of FORGE-style benchmarks to include sensor data (e.g., force/torque, process control logs) would further support holistic evaluation towards autonomous execution systems.

(Figure 6, 12, 13, 14, 15, 17, 19, 20, 21, 22)

Figures 9, 12, 13, 14, 15, 17, 19, 20, 21, 22: Random examples displaying the range of workpiece types and defect presentations, underscoring the complexity and diversity in real-world industrial scenarios.

Conclusion

FORGE advances the field by establishing a high-fidelity, comprehensive, and domain-authentic benchmark for MLLMs in manufacturing, providing both a platform for scientific evaluation and a resource for domain specialization. The benchmark's design and analysis reveal that the primary bottleneck for high-stakes deployment is domain reasoning, not perception. Structured fine-tuning using FORGE yields significant, transferable gains even in data-constrained settings.

FORGE thus constitutes a rigorous reference point and a practical foundation for future advancements in AI-driven manufacturing quality assurance and autonomy.