CircuitFormer: Transformer Models for Circuits

• CircuitFormer is a family of transformer-based models tailored for circuit design, encompassing point-cloud placement evaluation, natural language-to-netlist translation, and modular circuit interpretability.
• In physical design, it leverages geometric point-cloud representations to predict routing congestion and DRC violations directly from raw component coordinates.
• For analog design and interpretability, it employs a connectivity-aware tokenizer and binary mask extraction, enabling accurate SPICE netlist generation and causal model insights.

CircuitFormer is a name associated with multiple transformer-based formulations of circuit problems rather than a single canonical architecture. In physical design, it denotes a Transformer-based point-cloud perception model that treats a placed circuit as a set of 2D points and predicts grid-level congestion and design rule check (DRC) maps directly from raw geometry (Zou et al., 2023). In analog design, it denotes a 511M-parameter encoder–decoder Transformer that translates natural-language specifications into SPICE topologies using a circuit-specific tokenizer (Islam et al., 7 May 2026). The name is also used, in details motivated by mechanistic-interpretability research, for a framework that extracts, validates, and composes transformer circuits as binary masks over mediators (Mondorf et al., 2024). A closely related but officially distinct name is “Circuit Transformer,” which addresses exact Boolean-circuit synthesis with equivalence-preserving decoding (Li et al., 2024).

1. Terminological scope

The label “CircuitFormer” spans at least three technically distinct research directions. One concerns routability-aware physical design; another concerns modular circuit masks inside LLMs; a third concerns natural-language-to-netlist generation for analog circuits. The resulting ambiguity is substantive rather than merely nominal, because the underlying objects differ: geometric point sets, mediator masks in transformer computation graphs, and graph-tokenized circuit netlists.

Usage	Domain	Defining features
CircuitFormer (Zou et al., 2023)	Physical design placement evaluation	Point-cloud Transformer over raw $(x,y,w,h)$ geometry; predicts congestion and DRC heatmaps
CircuitFormer (Islam et al., 7 May 2026)	Analog topology generation	511M encoder–decoder Transformer; Circuit Tokenizer (CKT); natural language to SPICE netlists
CircuitFormer motivated by “Circuit Compositions” (Mondorf et al., 2024)	Mechanistic interpretability	Circuit library of reusable binary masks, set algebra, faithfulness and overlap metrics
“Circuit Transformer” (Li et al., 2024)	Logic synthesis	Officially distinct name; equivalence-preserving decoding for AIG generation

A plausible implication is that the term functions as a family label for transformer systems whose primary inductive bias is tailored to circuit structure, rather than as a single stable model class.

2. CircuitFormer as point-cloud placement evaluation

In the physical-design setting, CircuitFormer is a Transformer-based point-cloud perception model for fast placement evaluation after global placement (Zou et al., 2023). The central problem is that routing congestion and DRC violations degrade manufacturability and power, performance, and area, yet accurate routability maps are typically available only after time-consuming placement-and-routing. The model addresses this by inferring grid-level routability indicators directly from placement geometry, enabling pre-routing evaluation.

The defining abstraction is to treat all placed components uniformly as points in a pseudo-2D point cloud. The input set is

$P=\{P_i=(x_i,y_i,w_i,h_i): i=1,\dots,n\},$

where $(x_i,y_i)$ are center coordinates and $(w_i,h_i)$ are width and height. Coordinates are normalized to

$C=\{C_i=(\tilde{x}_i,\tilde{y}_i): i=1,\dots,n\},$

with $\tilde{x}_i,\tilde{y}_i \in [0,1]$ under die-size normalization. This representation avoids hand-crafted raster features such as RUDY or pin density and also avoids explicit graph reconstruction. The paper positions this as an alternative to two prior families: CNN-based image methods that depend on rasterized engineered features, and GNN-based methods that require graph construction over hundreds of thousands of nodes and millions of nets.

Labels are continuous-valued grid maps in $[0,1]$. Congestion is defined as overflow of routing demand over available resources, and DRC labels denote rule-violating locations. Points are assigned to multi-scale grids by

$$G_i=\left(\left\lfloor \frac{\tilde{x}_i}{d_x}\right\rfloor,\left\lfloor \frac{\tilde{y}_i}{d_y}\right\rfloor\right),$$

with $d_x=d_y \in \{1,2,4,8\}$. Scatter-sum reduction then aggregates point features into grid-aligned tensors. The representation is variable-size on the point side, while a fixed-size learnable latent code of length $k$ with $P=\{P_i=(x_i,y_i,w_i,h_i): i=1,\dots,n\},$0 provides compression through cross-attention.

This formulation is explicitly end-to-end from raw component geometry. Only raw $P=\{P_i=(x_i,y_i,w_i,h_i): i=1,\dots,n\},$1 and die size are required, and the reported preprocessing consists of reading these values and normalizing coordinates. A plausible implication is that the model shifts feature engineering away from EDA-specific raster design toward learned geometric aggregation.

3. Architectural structure, optimization, and empirical profile in physical design

The physical-design CircuitFormer consists of a Transformer-based point-cloud encoder followed by a segmentation decoder (Zou et al., 2023). The encoder stacks four multi-scale Grid-based Attention (GA) modules, one for each grid scale. The decoder is a standard image-segmentation stack, reported in the main configuration as a ResNet-18 backbone with a UNet++ head. The encoder produces a feature image $P=\{P_i=(x_i,y_i,w_i,h_i): i=1,\dots,n\},$2 aligned to the label raster, and the decoder refines it into per-pixel predictions for congestion or DRC.

Attention follows the standard Transformer form,

$P=\{P_i=(x_i,y_i,w_i,h_i): i=1,\dots,n\},$3

CircuitFormer uses learnable latent codes as queries for cross-attention compression, then performs point-to-grid aggregation, sparse image assembly, ConvFFN refinement with two depth-wise convolutions, broadcast back to points, and a second cross-attention expansion. This replaces quadratic self-attention complexity $P=\{P_i=(x_i,y_i,w_i,h_i): i=1,\dots,n\},$4 with encoder complexity $P=\{P_i=(x_i,y_i,w_i,h_i): i=1,\dots,n\},$5. The GA modules combine latent compression, scatter-sum grid pooling, local convolutional exchange across grids, and pointwise redistribution, thereby mixing global and multi-scale geometric information.

Training is end-to-end with AdamW for 100 epochs on a single NVIDIA RTX 3090 GPU, with learning rate $P=\{P_i=(x_i,y_i,w_i,h_i): i=1,\dots,n\},$6, cosine annealing, and a 10-epoch warmup. The work uses CircuitNet for congestion and DRC, and ISPD2015 for congestion. The largest CircuitNet design reported is RISCY-FPU-a with 77,707 nodes, while ISPD2015 includes “superblue” designs with more than one million nodes. Because labels are long-tailed continuous values, the loss is a re-weighted MSE using Label Distribution Smoothing (LDS), where effective label density is smoothed by Gaussian-kernel convolution and inverse-density weights are applied.

The reported results are state-of-the-art on congestion prediction on both CircuitNet and ISPD2015, and on DRC violation prediction on CircuitNet. On CircuitNet congestion, CircuitFormer attains Pearson/Spearman/Kendall correlations of $P=\{P_i=(x_i,y_i,w_i,h_i): i=1,\dots,n\},$7, compared with $P=\{P_i=(x_i,y_i,w_i,h_i): i=1,\dots,n\},$8 for UNet++ with Gpdl features and $P=\{P_i=(x_i,y_i,w_i,h_i): i=1,\dots,n\},$9 for CircuitGNN. On CircuitNet DRC violation, it attains $(x_i,y_i)$0; this yields the best Spearman and Kendall, while Pearson remains lower than the UNet++ baseline that uses congestion as an input feature. On ISPD2015 congestion, it reports $(x_i,y_i)$1 on the full set, and $(x_i,y_i)$2 on the subset without superblue designs.

Ablation studies indicate that the four-stage GA stack is materially important: reducing from four stages to one lowers congestion performance from $(x_i,y_i)$3 to $(x_i,y_i)$4 while reducing memory from 2475 MB to 811 MB. Decoder comparisons show the default ResNet-18 + UNet++ configuration balances accuracy and latency, with 32.02 ms latency on CircuitNet. LDS improves both congestion and DRC. Against point-cloud baselines on CircuitNet, CircuitFormer exceeds PointPillars, DSVT, Point-Transformer, and Msg-Transformer on congestion, and is competitive or best on DRC.

The paper also states clear limitations. Net connectivity is not explicitly encoded, so the model captures long-range geometric relationships through attention but lacks an explicit graph prior. The authors suggest Graphormer-style positional biases as a future direction. The DRC results likewise indicate that higher-level inputs such as congestion can help that task, especially in Pearson correlation.

4. CircuitFormer as a modular circuit-mask framework in mechanistic interpretability

In a different usage, the details attached to “CircuitFormer” describe a modular interpretability stack motivated by “Circuit Compositions: Exploring Modular Structures in Transformer-Based LLMs” (Mondorf et al., 2024). Here a circuit is a minimal computational subgraph causally responsible for behavior on a task, instantiated as a binary mask $(x_i,y_i)$5 over a mediator space $(x_i,y_i)$6. A mask entry $(x_i,y_i)$7 marks mediator $(x_i,y_i)$8 as causally relevant; $(x_i,y_i)$9 marks it as irrelevant.

Three criteria organize the framework: faithfulness, minimality, and completeness. Faithfulness asks whether the masked subnetwork reproduces the full model’s behavior on task $(w_i,h_i)$0 when all other nodes are ablated to fixed values. Minimality prefers small masks under the $(w_i,h_i)$1 norm. Completeness is acknowledged as difficult to guarantee. Rather than exhaustive node-by-node patching, circuit extraction is framed as continuous sparsification over activations. A continuous mask $(w_i,h_i)$2 approximates the binary mask via annealing, and the objective balances cross-entropy to the full model’s cached output distribution against $(w_i,h_i)$3 sparsity. Upon convergence, the circuit is recovered as $(w_i,h_i)$4.

Faithfulness is evaluated with normalized Jensen–Shannon divergence, defining

$(w_i,h_i)$5

Overlap between two circuits is measured by Intersection over Union and Intersection over Minimum,

$(w_i,h_i)$6

Because circuits are binary masks, they can be composed by set operations, especially union,

$(w_i,h_i)$7

with an ablation asymmetry under mean ablation because ablated values are taken from the first circuit’s reference task.

The experimental context is a transformer trained on PCFG SET, a controlled compositional dataset of ten modular string-edit operations: six unary functions—copy, echo, repeat, reverse, swap, shift—and four binary functions—append, prepend, remove_first, remove_second. The base model is a 6-layer encoder and 6-layer decoder Transformer with GLU feed-forward blocks and 8 attention heads per layer. On isolated subtasks, unary-task accuracies exceed 95% except repeat at 94.3%, while binary tasks range from 83.2% to 99.9%.

The extracted circuits exhibit both sparsity and modular overlap. Reverse, swap, and shift form a cluster with IoU around 0.36–0.42 and IoM around 0.60. Echo, repeat, and swap each have IoM around 0.42–0.56 with the copy circuit, indicating that roughly half of copy’s nodes are embedded in those larger circuits. Global sparsity analysis identifies prepend as the largest circuit, retaining about 39% of activations, and copy as the smallest. Local module analysis shows that the copy circuit retains a large fraction of decoder cross-attention activations while pruning most decoder feed-forward and self-attention activations, consistent with a routing-dominant mechanism.

Composition by set union yields nontrivial functional transfer. The union of repeat and reverse yields echo with 78% accuracy even though neither base circuit alone solves echo. The union of swap and reverse improves shift accuracy to 33%. Other unions also generate intermediate or novel behaviors, which supports the claim that circuits can be reused and combined through set operations. At the same time, the framework identifies substantial caveats: completeness is not guaranteed; the discovered circuits depend on the ablation choice; and cross-task faithfulness can look high when averaged over all output tokens but collapse when evaluated only on token positions where two tasks actually differ. Deterministic masks are reported as highly stable across random seeds, with IoU and IoM near 1.0 in tests such as five independently extracted prepend circuits.

5. CircuitFormer as a circuit LLM for analog topology design

The 2026 model titled “CircuitFormer: A Circuit LLM for Analog Topology Design from Natural Language Prompt” is a 511M-parameter encoder–decoder Transformer for translating natural-language specifications into analog circuit topologies in SPICE netlist form (Islam et al., 7 May 2026). Its central premise is that circuit behavior is defined by graph connectivity rather than textual line order, so general-purpose tokenizers such as BPE or SentencePiece induce spurious order sensitivity and poor sample efficiency when applied directly to netlists.

The work addresses both dataset scarcity and serialization mismatch. It releases a dataset of 31,341 netlist–natural language pairs spanning all major analog circuit classes, drawn from 62 analog and RF textbooks and reference volumes. The corpus contains 20 major SPICE element types and 328,565 total components, with circuit sizes from 1 to 139 components. A filtered subset of 18,011 valid netlists, excluding subcircuit “X” calls, is used for model training, split 94% train and 6% validation. The extraction pipeline rasterizes PDFs, detects circuit figures with DocLayout-YOLO, filters images with a VLM, converts schematic images into SPICE netlists and descriptions using Gemini 3, verifies syntax and convergence with Ngspice, and discards failing netlists. On a 50-instance manual audit, the reported image-to-netlist accuracy is 95% and element-wise accuracy is 99.25%.

A core contribution is the Circuit Tokenizer (CKT), a connectivity-aware graph tokenizer. CKT first canonicalizes netlists into topology-only graphs by anonymizing instance names, remapping nodes to contiguous integers, mapping ground to 0, stripping directives, and inserting dummy nodes for controlling branches of dependent sources such as F, H, and K. It then mines frequent subcircuits using Graph Re-Pair over a disjoint union graph of training circuits. Candidate digrams are scored by the MDL-style gain

$(w_i,h_i)$8

and are merged only if the gain is positive. Macro tokens are capped to a fixed target vocabulary, such as 512 total tokens in the reported “Optimal Vocab” configuration.

The tokenizer’s formal scaling claim is that prior instance-specific tokenizers have vocabulary growth $(w_i,h_i)$9, whereas CKT has

$C=\{C_i=(\tilde{x}_i,\tilde{y}_i): i=1,\dots,n\},$0

and therefore $C=\{C_i=(\tilde{x}_i,\tilde{y}_i): i=1,\dots,n\},$1 with respect to $C=\{C_i=(\tilde{x}_i,\tilde{y}_i): i=1,\dots,n\},$2 when the cap is fixed. Empirically, the 512-token configuration reports a compression ratio of 2.54 and sequence-length reduction of 57.1% versus BPE and 59.1% versus SentencePiece; the abstract summarizes this as 57% sequence-length reduction and 2.3× compression with a fixed vocabulary of size 512.

The architecture uses an MPNet/Sentence-BERT family encoder, specifically a 110M MPNet encoder with wordpiece tokenization, and a GPT-2-style decoder with cross-attention, 24 layers, model dimension $C=\{C_i=(\tilde{x}_i,\tilde{y}_i): i=1,\dots,n\},$3, and 12 heads. Training proceeds in three stages: decoder pretraining with a causal language-model objective, a cross-attention alignment phase with encoder and decoder frozen, and end-to-end finetuning after unfreezing all parameters. Optimization uses AdamW, 10% linear warmup, learning rate $C=\{C_i=(\tilde{x}_i,\tilde{y}_i): i=1,\dots,n\},$4 for 100 epochs of decoder pretraining, 25 epochs of alignment, and $C=\{C_i=(\tilde{x}_i,\tilde{y}_i): i=1,\dots,n\},$5 for 75 epochs of full finetuning. All experiments are reported on a single NVIDIA H100 with 96 GB memory. Line-shuffle augmentation is applied every epoch to enforce permutation invariance over netlist line order.

Evaluation is conducted on CircuitBench-100, a 100-prompt benchmark spanning amplifiers, filters, oscillators, rectifiers and power electronics, RF structures, bias/reference circuits, logic/digital building blocks, and sensors/opto. CircuitFormer achieves 100% syntactic correctness, 83% functional success, and MMD 0.232. Baselines reported include GPT-OSS at 69% success and 90% validity, DeepSeek-V3.2 at 63% success and 94% validity, Llama 3.3 at 31% success and 60% validity, Mistral Large 2 at 32% success and 85% validity, and Gemma 3 at 23% success and 64% validity. The paper states that CircuitFormer outperforms GPT-OSS by 14% in success and 10% in validity while using about 240× fewer parameters. Ablations further show that replacing CKT with BPE lowers success from 83% to 70%, and disabling line-shuffle lowers success to 23% while validity remains 98%.

The model’s limitations are explicit. The dataset depends on VLM parsing of textbook schematics and may retain edge cases. Functional evaluation relies on expert review corroborated by an LLM judge rather than a fully automated specification-level checker. Parameter values are not optimized; the focus is topology generation. Extremely large hierarchical circuits with “X” blocks were filtered out during training. Future directions include SPICE-in-the-loop reinforcement learning, hierarchical subcircuits, richer device models, and layout-aware or PDK-specific constraints.

6. Related naming, misconceptions, and adjacent work

A frequent point of confusion is the relation between “CircuitFormer” and “Circuit Transformer.” The paper “Circuit Transformer: A Transformer That Preserves Logical Equivalence” explicitly states that it does not use the term “CircuitFormer”; if that label appears in connection with the work, it is “almost certainly an informal shorthand or a conflation” rather than the official method name (Li et al., 2024). The distinction matters because the technical objective is different from the other CircuitFormer usages.

Circuit Transformer addresses combinational logic synthesis with exact functional preservation. It operates on And-Inverter Graphs (AIGs), serializes them through a memory-less depth-first traversal, and uses tree-aware positional encodings. Its defining innovation is an equivalence-preserving decoder that masks any candidate token which would cause the partially constructed circuit to contradict the target truth table. The masking uses Boolean cutoff properties such as $C=\{C_i=(\tilde{x}_i,\tilde{y}_i): i=1,\dots,n\},$6 and $C=\{C_i=(\tilde{x}_i,\tilde{y}_i): i=1,\dots,n\},$7, thereby converting equivalence from a post-hoc verification problem into a hard decoding constraint. The model can also be cast as an MDP and combined with Monte-Carlo Tree Search to optimize compactness while keeping equivalence exact.

The reported model has about 88.2M parameters and is trained on 20 million randomly generated 8-input, 2-output AIGs with targets produced by ABC’s resyn2 flow. On 960 held-out AIGs, Circuit Transformer alone achieves an average AND count of 9.519 versus 9.578 for resyn2, and Circuit Transformer plus MCTS reaches 9.127 under the best reported setting ($C=\{C_i=(\tilde{x}_i,\tilde{y}_i): i=1,\dots,n\},$8, $C=\{C_i=(\tilde{x}_i,\tilde{y}_i): i=1,\dots,n\},$9), all with zero equivalence violations. This work is therefore adjacent to, but not synonymous with, CircuitFormer.

Taken together, these threads indicate that “CircuitFormer” has become a polymorphic label across EDA and transformer research. One usage centers on pseudo-2D geometric perception for routability, another on connectivity-aware language modeling for analog netlists, and another on reusable circuit masks for mechanistic interpretability. A plausible implication is that the common denominator is not a shared implementation but a shared design principle: transformer architectures are adapted so that circuit structure—geometric, topological, or causal—becomes the primary computational object.