CircuitSense: Hierarchical Circuit Benchmark

• CircuitSense is a hierarchical benchmark that assesses multimodal models' ability to translate visual circuit diagrams into formal symbolic representations across perception, analysis, and design tasks.
• It leverages synthetic generation to produce circuit schematics and block diagrams with auto-derived ground-truth symbolic labels, including transfer functions and nodal equations.
• Empirical results reveal a stark contrast between high visual recognition accuracy and low symbolic derivation performance, underscoring the challenge of true engineering competence.

Searching arXiv for the benchmark paper and closely related circuit-understanding works.

arXiv search query: "CircuitSense hierarchical benchmark visual symbolic circuit understanding (Akbari et al., 26 Sep 2025)"

CircuitSense is a hierarchical circuit-system benchmark for evaluating whether multimodal models can move from visual circuit representations to formal symbolic reasoning across the engineering design process. It spans component-level schematics through system-level block diagrams, organizes tasks into Perception, Analysis, and Design, and places particular emphasis on deriving symbolic equations such as transfer functions and nodal equations from visual inputs. The benchmark was introduced to expose a recurrent failure mode in multimodal LLMs: strong visual recognition does not translate into strong circuit analysis or synthesis, with closed-source models exceeding 85\% on perception while falling below 19\% on symbolic derivation and analytical reasoning (Akbari et al., 26 Sep 2025).

1. Definition and problem setting

CircuitSense is structured around the observation that engineering design proceeds through hierarchical abstraction, moving from system specifications to subsystem blocks and component-level schematics. At each level, the operative task is not merely recognition of symbols or topology, but translation of visual structure into symbolic models that support analysis of stability, noise, impedance, sensitivity, and performance. CircuitSense therefore evaluates the full workflow as Perception, Analysis, and Design, rather than restricting evaluation to component recognition or multiple-choice reasoning (Akbari et al., 26 Sep 2025).

Its technical focus is the visual-to-symbolic step. In the benchmark formulation, Perception covers component detection, connection identification, and function classification; Analysis requires derivation of $s$-domain equations from diagrams using KCL, KVL, or Modified Nodal Analysis (MNA); and Design requires synthesis from specifications or performance targets. The benchmark formalizes standard impedances as $Z_R = R$, $Z_L = sL$, and $Z_C = \dfrac{1}{sC}$, and uses canonical circuit relations such as the voltage-divider law

$$V_{\text{out}} = V_{\text{in}} \,\frac{R_2}{R_1 + R_2}$$

and the first-order RC low-pass transfer function

$$H(s) = \frac{V_{\text{out}}(s)}{V_{\text{in}}(s)} = \frac{1}{1 + sRC}.$$

For general linear circuits, the benchmark adopts the MNA form

$(G + sC)\,v = b.$

These definitions make symbolic derivation the core competency rather than an auxiliary formatting exercise (Akbari et al., 26 Sep 2025).

This framing also distinguishes CircuitSense from benchmarks that reward answer selection without requiring equation formation. A central claim of the benchmark is that engineering competence depends on deriving and manipulating symbolic models from diagrams, not solely on parsing them (Akbari et al., 26 Sep 2025).

2. Hierarchical organization and corpus composition

CircuitSense comprises 8,006 problems spanning six hierarchy levels and three task categories. The hierarchy runs from elementary passive networks to control-oriented system diagrams, and the benchmark explicitly aligns these levels with engineering abstraction (Akbari et al., 26 Sep 2025).

Level	Description	Samples
Level 0	Resistive networks (DC analysis)	1,777
Level 1	RLC circuits (frequency-domain)	3,147
Level 2	Small-signal circuits	537
Level 3	Transistor circuits	795
Level 4	Block-level	559
Level 5	System block diagrams	228

The task categories are distributed as 806 Perception problems, 7,043 Analysis problems, and 157 Design problems. CircuitSense combines 2,986 curated problems from canonical textbooks and university courses with 5,020 synthetically generated circuits that include guaranteed ground-truth symbolic labels (Akbari et al., 26 Sep 2025).

Within Analysis, the reported subcategories are Transient Response (3,811), Transfer Function Analysis (1,736), Small-Signal Analysis (915), Power/Energy Analysis (222), Frequency Response (184), Noise/Jitter Analysis (121), and CMR/PSRR (54). The benchmark therefore covers both low-level algebraic derivation and discipline-specific analytical tasks that are standard in analog, mixed-signal, and control-oriented workflows (Akbari et al., 26 Sep 2025).

The hierarchy is paired with explicit labels. Perception tasks use component lists, netlists, and function tags. Analysis tasks use transfer functions and nodal equations as gold labels. Design tasks evaluate topology and parameter choices against target behavior. Each problem folder includes an image, question text, ground-truth answer, multiple-choice options when applicable, correct option, and a step-by-step derivation (Akbari et al., 26 Sep 2025).

3. Synthetic generation and symbolic ground truth

A defining feature of CircuitSense is its hierarchical synthetic generation pipeline. At the schematic level, a grid-based generator places components on an $m \times n$ grid, samples parameters within feasible ranges, and imposes validity constraints such as no floating nodes, node degree $\ge 2$, exactly one voltage source, avoidance of shorts, and proper control links for dependent sources. The component library includes 18 types, including $R$, $Z_R = R$0, $Z_R = R$1, independent sources, controlled sources (VCVS, VCCS, CCVS, CCCS), and ideal op-amp templates (Akbari et al., 26 Sep 2025).

The synthesis pipeline converts grid topology into a SPICE-compatible netlist via node labeling and element enumeration, renders the schematic image, and then validates the instance through topological checks, ngspice DC/AC sanity checks, and symbolic analysis in Lcapy. The reported ground-truth labels include transfer functions and MNA nodal equations, verified both symbolically and numerically. When direct symbolic equivalence is intractable, the validation procedure evaluates candidate and ground-truth expressions at 100 random complex $Z_R = R$2 points (Akbari et al., 26 Sep 2025).

At the system level, the block diagram generator constructs a main forward path of $Z_R = R$3 blocks, augments it with feedback and feedforward paths, and derives exact symbolic labels. It uses the standard block-diagram relations

$Z_R = R$4

$Z_R = R$5

and

$Z_R = R$6

For multi-loop graphs it uses Mason’s gain formula,

$Z_R = R$7

and

$Z_R = R$8

This makes the benchmark unusual among multimodal datasets: symbolic labels are not human-authored approximations but auto-derived algebraic objects tied to rendered circuit structure (Akbari et al., 26 Sep 2025).

The benchmark’s symbolic pipeline is therefore not limited to answer checking. It operationalizes the full chain from graph extraction to exact equation derivation. A plausible implication is that CircuitSense is as much an evaluation framework for representation fidelity as it is a benchmark of multimodal reasoning.

4. Evaluation protocol and empirical results

CircuitSense evaluates six multimodal models: Gemini-2.5-Pro, GPT-4o, Claude-Sonnet-4, InternVL3-78B, Qwen2.5-VL-72B-Instruct, and GLM-4.5V. Multiple-choice answers are scored by normalized exact match; open-ended numeric answers use LLM-as-a-judge with Gemini-2.5-Flash; symbolic equations are checked with a SymPy-based parser and simplifier, with fallback numerical validation on 100 random complex $Z_R = R$9 samples; and design tasks requiring simulation are verified with ngspice (v43) using the SkyWater SKY130 PDK (Akbari et al., 26 Sep 2025).

The headline result is the gap between visual parsing and symbolic reasoning. Closed-source models achieve high perception accuracy: Gemini-2.5-Pro scores 100 on Component Detection, 100 on Connection Identification, and 95 on Function Classification; GPT-4o scores 100, 94, and 95; Claude-Sonnet-4 scores 100, 88, and 86. Open-source models lag substantially, especially on function classification, where InternVL3-78B scores 12, Qwen2.5-VL scores 20, and GLM-4.5V scores 26 (Akbari et al., 26 Sep 2025).

Performance falls sharply on symbolic derivation. On curated multiple-choice problems, Gemini-2.5-Pro reaches 80.71\% overall, Claude-Sonnet-4 69.67\%, and GPT-4o 45.07\%. On curated open-ended problems, Gemini-2.5-Pro reaches 70.32\%, while the remaining models are at or below 35\%. On the synthetic split, where symbolic derivation is required, Gemini-2.5-Pro reaches 19.06\% overall and the other models fall in the 3–6\% range. The reported drop of more than 60 points from curated multiple-choice to synthetic derivation is presented as evidence that current models rely heavily on pattern matching rather than de novo equation extraction (Akbari et al., 26 Sep 2025).

Analysis subtasks further sharpen this diagnosis. Gemini-2.5-Pro reports 83\% on Frequency Response, 13\% on Transient, 38\% on Transfer Function, 74\% on Small Signal, 77\% on CMR/PSRR, 90\% on Noise/Jitter, and 87\% on Power/Energy. Design performance is also modest: for Gemini-2.5-Pro, schematic-level design reaches 36.38\%, block-level 67.27\%, and hierarchical 51.35\% (Akbari et al., 26 Sep 2025).

These results support the benchmark’s central thesis: symbolic reasoning, rather than object recognition, is the limiting factor for engineering-capable multimodal systems.

5. Error structure and technical interpretation

CircuitSense includes a fine-grained failure analysis of transfer-function derivation. In a sample of 100 Gemini-2.5-Pro attempts, the model achieved 97\% on component identification and 95\% on impedance assignment, but only 81\% on total input impedance, 8\% on output impedance derivation, 39\% on impedance ratio formation, and 55\% on transfer-function simplification. The paper identifies output-impedance derivation as the primary bottleneck (Akbari et al., 26 Sep 2025).

Typical errors include mis-parsing topology through incorrect node merging or missing ground references, misinterpreting output impedance through confusion between Thevenin and Norton views, making algebraic manipulation mistakes in partial fraction decomposition or rational simplification, and mixing time-domain parameters into $Z_L = sL$0-domain algebra. In block-diagram tasks, performance drops further when models move from abstract symbolic blocks to explicit rational functions; Claude-Sonnet-4, for example, scores 28.51\% on abstract-symbol block diagrams but 7.89\% on exact rational forms (Akbari et al., 26 Sep 2025).

This error profile rules out a simple “OCR problem” interpretation. The models often recognize the right components and even assign standard impedances correctly, but fail when composing those local recognitions into a globally correct analytical object. This suggests that current multimodal systems are comparatively competent at syntactic parsing yet weak at constrained symbolic elimination, impedance composition, and algebraic normalization.

A related misconception addressed by the benchmark is that high VQA-style circuit performance implies readiness for engineering assistance. CircuitSense shows the opposite: component recognition and topology identification can be strong while transfer-function derivation and design remain poor. In the benchmark’s terms, Perception is not a reliable proxy for engineering competence (Akbari et al., 26 Sep 2025).

6. Relation to adjacent research, limitations, and significance

CircuitSense occupies a position between image understanding, graph extraction, and symbolic reasoning. Adjacent work has addressed these layers separately. A public dataset for handwritten circuit diagrams provides 1,152 images of 144 circuits by 12 drafters, annotated with bounding boxes across 45 class labels plus helper symbols such as junctions, crossovers, terminals, and text, and reports a Faster R-CNN baseline with 52\% mAP (Thoma et al., 2021). In a lower-resource recognition setting, a two-stage pipeline based on selective search and an ultra lightweight CNN reports 93.4\% end-to-end accuracy for circuit component recognition (Ju et al., 2020). Semantic post-processing has also been studied through RDF-based circuit representations and Apache Jena forward-chaining rules for generating functional component descriptions such as flyback diodes and pull-up resistors from CAE-native or image-derived graphs (Bayer et al., 2022).

Against that background, CircuitSense extends the problem from detection or graph-level semantic labeling to formal visual-to-symbolic derivation across multiple abstraction levels. It differs from prior multimodal circuit benchmarks by emphasizing auto-labeled symbolic equations via MNA and Mason’s formula, and by evaluating the full workflow from Perception to Analysis to Design (Akbari et al., 26 Sep 2025).

The benchmark also has explicit limitations. Component coverage is concentrated on linear elements and ideal op-amps; transistor-level symbolic generation is limited because nonlinear devices hinder exact symbolic labels. The synthetic pipeline currently emphasizes transfer-function and nodal analysis, with planned extension to noise, frequency response, jitter, and related categories. Symbolic derivation becomes intractable beyond roughly 12–15 components. Real-world handwritten, noisy, or proprietary diagrams may require domain-specific OCR and symbol normalization, and richer parameter distributions and tolerance modeling remain open directions (Akbari et al., 26 Sep 2025).

Its broader significance lies in how it operationalizes competence. CircuitSense argues that the decisive capability for multimodal engineering systems is not captioning a schematic or selecting a plausible answer, but deriving the mathematical object that engineers actually use. That position is supported by the benchmark’s empirical result that models with stronger symbolic derivation consistently achieve higher design-task accuracy (Akbari et al., 26 Sep 2025). This suggests that future progress in multimodal circuit intelligence will depend less on incremental gains in visual recognition than on integrating diagram parsing with exact symbolic manipulation, equation checking, and domain-constrained reasoning.