MultModLM: A multi-modal benchmark for Large-Language Model based hardware schematic generation

Abstract: Recently, LLMs find application in several fields. This extends to hardware definition and synthesis. However, most works at the intersection of LLMs and hardware generation focus on text-based tasks, creating a gap for multi-modal LLMs for RTL design. In this work, we introduce MultModLM, a benchmark for evaluating LLMs on the task of generating hardware schematics from RTL (Register Transfer Level) descriptions. The dataset consists of 99 diverse RTL modules spanning arithmetic, control, and state-based designs. To address the challenges of non-unique schematic representations, we propose a multi-stage evaluation framework combining rubric-based scoring, self-evaluation, cross-model assessment, blind evaluation, and human validation to enable exhaustive evaluation. Through experiments on state-of-the-art LLMs, we observe that while models can generate visually interpretable schematics, their functional correctness remains constrained. Furthermore, we find that LLM-based evaluators exhibit near-zero agreement with human raters, revealing, as a key finding, that LLM-as-a-judge paradigms are unreliable in structurally precise domains. These findings suggest that reliable evaluation of multi-modal hardware outputs remains an open challenge, motivating the need for more robust and domain-aware evaluation methodologies, as well as tools for structural evaluation, so as to enable formal equivalence checkers.

• The paper introduces a benchmark to evaluate LLMs' ability to generate hardware schematics from RTL designs.
• It employs a multi-stage evaluation framework including self-, cross-, and human assessments to measure schematic accuracy.
• The results reveal that current LLM evaluations misalign with expert judgments, underscoring the need for domain-specific evaluators.

MultModLM: A Multi-Modal Benchmark for LLM-Based Hardware Schematic Generation

Motivation and Problem Statement

The rapid integration of LLMs into the electronic design automation (EDA) landscape has catalyzed advances across text-based tasks, from RTL code generation and debugging to explainability. However, the lack of benchmarks for evaluating LLMs on multi-modal tasks, particularly hardware schematic generation from RTL descriptions, presents a clear bottleneck for progress in automating structural circuit design workflows. Existing EDA tools for schematic generation, such as Synopsys or Cadence, are expensive, closed-source, and require substantial domain expertise, limiting accessibility and reproducibility for the broader research community.

The central objective of MultModLM is to fill this evaluative gap by systematically probing multi-modal LLMs as hardware schematic generators, addressing both technical feasibility and methodological evaluation of their outputs. The non-uniqueness of schematic representations—many structurally distinct schematics are functionally equivalent—complicates automated evaluation, demanding nuanced metrics for correctness that move beyond simple visual comparison or direct simulation.

Benchmark Design and Dataset

MultModLM provides a curated dataset comprising 99 RTL modules, selected to cover arithmetic datapaths, finite-state machines, numeric property checkers, control logic, and counters. The selected designs encapsulate algorithmic complexity and span both combinational and sequential logic circuits, thereby stress-testing LLM schematic generation across a spectrum of hardware constructs.

No canonical ground-truth schematics are provisioned, given the one-to-many mapping from RTL to legal schematic realizations. Instead, correctness is interpreted along three main axes: functional consistency with RTL description, structural fidelity to specified datapaths and controls, and schematic validity/legibility (including labeling, clocking, and signal semantics).

Multi-Stage Evaluation Framework

Given the open-ended and visually semantic nature of the task, the paper instantiates a multi-stage evaluation pipeline, summarized below and depicted for clarity.

Figure 1: The evaluation pipeline for comparing schematic generation capabilities of GPT and Gemini, including prompt structure, schematic generation, and layered evaluation protocols.

1. Self-Evaluation: LLMs rate their own schematics against a rigid rubric, focusing on key hardware-centric criteria expressed as Boolean outcomes (e.g., "correct datapath representation," "clocking and timing accuracy"). This stage provides initial introspective measurement and quantifies self-assessment bias.
2. Grounded Cross-Evaluation: The schematic generated by one LLM is rated by another, using access to both the RTL prompt and the generated image. This supports cross-model cross-validation.
3. Blind Cross-Evaluation: The evaluating LLM is shown only the image, without the RTL, to probe for internal visual coherence.
4. Human Evaluation: Domain experts provide independent judgments according to the established rubric, serving as the primary reference for alignment analysis.
5. Rubric Structure: Eight criteria—spanning from structural fidelity through semantic accuracy and visual clarity—are scored true/false, facilitating objective aggregation and inter-method comparisons.

Empirical Results

Evaluation leverages recent state-of-the-art multi-modal LLMs: GPT-5.2 (Go) and Gemini 3 (Flash Pro). For each design, both models independently generate schematics and undergo the complete evaluation cascade.

Inter-Rater Reliability: Quantification via Cohen’s $\kappa$ reveals that LLM-as-a-judge frameworks fail to reliably align with human evaluators ($\kappa_{\text{GPT vs Human}} \approx 0.013$, $\kappa_{\text{Gemini vs Human}} = 0$). This near-zero agreement robustly establishes that the latent reasoning underpinning LLM ratings diverges substantially from hardware expert assessments, confirming the unreliability of current LLM-based evaluation for structurally precise, symbolic visual domains.

Functional and Visual Accuracy: Accuracy (fraction of rubric criteria satisfied) as adjudicated by model cross-evaluation hovers between 0.314 (Gemini rated by GPT) and 0.297 (GPT rated by Gemini), underscoring significant room for improvement. Notably, Gemini achieves marginally higher accuracy across most semantic/functional criteria, while GPT scores higher on visual-oriented attributes Figure 2:

Figure 2: Rubric-based comparative accuracy across semantic, structural, and visual schematic criteria for GPT and Gemini, illustrating divergent strengths.

Win/Loss Analysis: Gemini outperforms GPT in 37 out of 99 modules, GPT wins in 33, with 29 ties, confirming no single model manages consistent dominance.

Theoretical and Practical Implications

The empirical findings surface several critical conclusions for both LLM deployment and automatic evaluation methodology in EDA-related multi-modal tasks:

• Limitation of LLM-as-a-Judge: LLMs exhibit dramatically unreliable evaluation of hardware schematics compared to human experts, even when provided the RTL grounding. Despite their success as evaluators for natural language or code, their capabilities do not generalize to hardware-centric, structurally rigorous visual domains.
• Structural vs Visual Evaluation: Functionally, Gemini demonstrates a bias towards semantic/structural fidelity, whereas GPT better satisfies subjective visual organization and clarity. However, high visual clarity scores rarely imply accurate hardware realization, underscoring a disconnect between visual aesthetics and semiconductor design semantics.
• Need for Domain-Aware Tools: Automated, scalable evaluation of schematic correctness—particularly via formal equivalence checking to RTL—remains unsolved and is not tractable with current visual LLM pipelines. Human-in-the-loop protocols or domain-specific structural validators remain essential.
• Generalizability: The systematic limitations documented are likely to extend to other multi-modal generation benchmarks for domains characterized by strong symbolic logic and non-executable visual output—biological pathway diagrams, circuit designs, or architectural blueprints—where correctness depends on structural and functional, not merely visual, criteria.

Conclusion

MultModLM addresses a core bottleneck in LLM-driven hardware design flows by providing an open multimodal benchmark, detailed rubric, and an empirical baseline for RTL-to-schematic generation performance. The results demonstrate that while current LLMs can produce visually plausible hardware schematics, their outputs frequently lack semantic and functional fidelity. Critically, LLM-based evaluators—central to the recent "LLM-as-a-judge" paradigm—show negligible alignment with human ground truth for these tasks.

The work underscores the urgent need for domain-aware, formally grounded evaluation protocols for multi-modal hardware generation. Future developments in this arena should explore integration with programmatic equivalence checkers, graph-based schematic analysis, and expanded datasets to facilitate model fine-tuning with rigorous structural supervision. These directions will be mandatory to enable reliable, scalable LLM application in EDA pipelines and related symbolic design tasks.