VeriFormer: Structure-Aware Soft Prompts

• VeriFormer is a structure-aware soft prompt synthesizer that maps hardware graph embeddings into LLM-compatible representations for improved Verilog generation.
• It integrates retrieved graph embeddings from circuit data path graphs and aligns them with code generation through a two-stage training pipeline.
• Empirical results demonstrate that structure-aware prompting significantly reduces syntactic errors and enhances the functional correctness of generated Verilog designs.

VeriFormer is the structure-aware soft prompt synthesizer inside the VeriGRAG framework for LLM-based Verilog code generation. Its function is to align retrieved graph embeddings of hardware structure with the code embedding space of a LLM, and to transform those embeddings into continuous prompts that are prepended to the model input during decoding. In the formulation reported for VeriGRAG, VeriFormer is motivated by the observation that Verilog encodes hardware structure—module hierarchies, netlist connectivity, and dataflow—beyond plain token sequences, and that conventional LLMs trained mostly on text often miss such structure and thus produce syntactic or functional errors such as incorrect bit-widths, missing sensitivity lists, and improper always block semantics (Zhao et al., 27 Sep 2025).

1. Position within VeriGRAG

VeriFormer operates after structural graph extraction, graph encoding, and graph retrieval. VeriGRAG first constructs circuit data path graphs from Verilog with Yosys, using the command sequence read_verilog; hierarchy -check; proc; opt; fsm. These graphs are then encoded by a GNN, and a multimodal retriever selects graph embeddings relevant to a natural-language hardware specification. VeriFormer receives those retrieved graph embeddings and synthesizes structure-aware soft prompts that condition the downstream LLM (Zhao et al., 27 Sep 2025).

Within this pipeline, VeriFormer is the module that performs the graph-to-code alignment step. The retrieved graph embeddings are not directly usable as LLM prefixes; they first need to be mapped into a representation that is compatible with code generation. VeriFormer addresses this by aligning the GNN graph embedding space with the LLM’s code embedding space and then producing continuous prompt vectors inserted as a prefix to the LLM input. The LLM is thereby conditioned to attend to hardware structure during decoding, with the stated goal of improving both syntactic correctness and functional behavior (Zhao et al., 27 Sep 2025).

A central implication of this design is that VeriFormer is not a standalone retrieval component and not merely a prompt-tuning layer. It is the cross-modal adapter that converts retrieved structural evidence into a form that can bias token generation. The paper explicitly positions this as distinct from appending text-only contexts, emphasizing alignment between graph topology and code modality (Zhao et al., 27 Sep 2025).

2. Upstream representations and the need for alignment

The structural inputs to VeriFormer originate in data path graphs extracted from Verilog. In the reported setup, nodes represent modules or signals and edges represent interconnections (wires). Node features include operation types, I/O types, I/O signal ports, and parameter lists, and these node attributes are encoded using a pretrained BERT encoder. Edge features capture normalized wire bit-widths, intended to represent structural relationships and signal flow (Zhao et al., 27 Sep 2025).

These graphs are embedded with GINEConv, described as Graph Isomorphism Network with Edge features, modified from GINConv to better incorporate edge features in message passing. The node update is

$$x_i' = h_\Theta\left((1+\epsilon)\cdot x_i + \sum_{j \in N(i)} \mathrm{ReLU}(x_j + e_{j,i})\right).$$

The graph encoder is trained with unsupervised contrastive learning on two stochastic augmented views of a graph, generated by edge deletion and feature perturbation. If the encoder is denoted by $f_\theta$, the two embeddings are $z_i^{(1)} = f_\theta(\tilde{G}^{(1)})$ and $z_i^{(2)} = f_\theta(\tilde{G}^{(2)})$, with InfoNCE objective

$$\mathrm{Loss} = - \frac{1}{B} \sum_{i=1}^B \log \left[ \frac{\exp(\mathrm{sim}(z_i^{(1)}, z_i^{(2)})/\tau)} {\sum_{j=1}^B \exp(\mathrm{sim}(z_i^{(1)}, z_j^{(2)})/\tau)} \right],$$

where $\mathrm{sim}$ is cosine similarity and $\tau$ is the temperature (Zhao et al., 27 Sep 2025).

Retrieval is also cross-modal. The query $q$ is a natural-language hardware description, and the database contains precomputed graph embeddings $g$. A teacher cross-attention encoder is trained with InfoNCE, but cannot cache graph embeddings; a student dual-encoder is then trained by distillation for fast inference. The student similarity is

$$S(q, g) = \frac{\langle f_q(q), f_g(g)\rangle}{\|f_q(q)\| \cdot \|f_g(g)\|},$$

and the distillation term is

$f_\theta$0

At inference, graph embeddings are pre-encoded and indexed with FAISS, while the query is encoded on the fly by the student text encoder and top-$f_\theta$1 similar graphs are retrieved efficiently (Zhao et al., 27 Sep 2025).

This upstream design explains why VeriFormer is needed. Retrieval returns graph embeddings that are relevant to a specification, but those embeddings live in a structural representation space rather than the code generation space of the LLM. VeriFormer is the module that converts retrieved structural evidence into prompt vectors compatible with the downstream decoder.

3. Architecture and prompt synthesis

VeriFormer is described as inspired by BLIP-2 style lightweight adapters but specialized for Verilog graphs. Architecturally, it contains two Transformer submodules that share self-attention weights: a graph transformer and a code transformer. It also introduces learnable query tokens that serve two linked purposes: they attend to graph embeddings through cross-attention layers to extract structure-informative features, and they interact with code representations through shared self-attention layers to achieve cross-modal alignment (Zhao et al., 27 Sep 2025).

In the reported experiments, VeriFormer is initialized from CodeBERT and uses 96 query tokens, each of dimension 768, matching the VeriFormer hidden size. Let $f_\theta$2 denote the $f_\theta$3 retrieved graph embeddings, and let the learnable query tokens be $f_\theta$4 with $f_\theta$5 and $f_\theta$6. For each retrieved graph representation $f_\theta$7, the graph transformer computes

$f_\theta$8

$f_\theta$9

where $z_i^{(1)} = f_\theta(\tilde{G}^{(1)})$0, $z_i^{(1)} = f_\theta(\tilde{G}^{(1)})$1, and $z_i^{(1)} = f_\theta(\tilde{G}^{(1)})$2 are projection matrices (Zhao et al., 27 Sep 2025).

If multiple graphs are retrieved, the query outputs may be aggregated by attention over the retrieved items or by concatenation plus a Transformer block. The attention formulation reported is

$z_i^{(1)} = f_\theta(\tilde{G}^{(1)})$3

with $z_i^{(1)} = f_\theta(\tilde{G}^{(1)})$4 exemplified as dot-product with a learned vector, and the aggregate is

$z_i^{(1)} = f_\theta(\tilde{G}^{(1)})$5

This aggregate is then projected to the LLM embedding space via a learned linear projection $z_i^{(1)} = f_\theta(\tilde{G}^{(1)})$6:

$z_i^{(1)} = f_\theta(\tilde{G}^{(1)})$7

where $z_i^{(1)} = f_\theta(\tilde{G}^{(1)})$8 denotes the soft prompt vectors inserted as a continuous prefix to the LLM input (Zhao et al., 27 Sep 2025).

After stage 1 of training, the code transformer is discarded and only the graph transformer and the query tokens are retained. Their outputs are projected to the LLM embedding dimension and prepended to the token embeddings:

$z_i^{(1)} = f_\theta(\tilde{G}^{(1)})$9

where $z_i^{(2)} = f_\theta(\tilde{G}^{(2)})$0 is the embedding sequence corresponding to the natural-language specification or partial code. The LLM parameters remain frozen while VeriFormer synthesizes $z_i^{(2)} = f_\theta(\tilde{G}^{(2)})$1 conditioned on retrieved structure (Zhao et al., 27 Sep 2025).

In functional terms, the architecture allows structural graph information to influence decoding without modifying the LLM weights. The paper states that the prepended continuous vectors alter the attention stack at the first layer by modifying keys, queries, and values, thereby biasing token probabilities toward structurally consistent Verilog, including correct sensitivity lists, reset behavior, and module I/O agreement (Zhao et al., 27 Sep 2025).

4. Objectives and training procedure

VeriFormer is trained in two stages. In stage 1, the graph and code transformers, together with the learnable query tokens, are trained while the GNN is frozen. The objective combines contrastive alignment, generation, and graph-code matching:

$z_i^{(2)} = f_\theta(\tilde{G}^{(2)})$2

The contrastive alignment term compares each query output $z_i^{(2)} = f_\theta(\tilde{G}^{(2)})$3 to the pooled code representation $z_i^{(2)} = f_\theta(\tilde{G}^{(2)})$4 of a Verilog example. The positive similarity is defined by the maximum similarity across query tokens,

$z_i^{(2)} = f_\theta(\tilde{G}^{(2)})$5

and the batchwise InfoNCE objective is

$z_i^{(2)} = f_\theta(\tilde{G}^{(2)})$6

where $z_i^{(2)} = f_\theta(\tilde{G}^{(2)})$7 is computed between the $z_i^{(2)} = f_\theta(\tilde{G}^{(2)})$8-th code and the $z_i^{(2)} = f_\theta(\tilde{G}^{(2)})$9-th graph’s query outputs by the same max-over-queries scheme, and $$\mathrm{Loss} = - \frac{1}{B} \sum_{i=1}^B \log \left[ \frac{\exp(\mathrm{sim}(z_i^{(1)}, z_i^{(2)})/\tau)} {\sum_{j=1}^B \exp(\mathrm{sim}(z_i^{(1)}, z_j^{(2)})/\tau)} \right],$$0 is cosine similarity (Zhao et al., 27 Sep 2025).

The graph-code generation objective trains VeriFormer to generate the target Verilog from graph embeddings through the code transformer pathway using next-token cross-entropy:

$$\mathrm{Loss} = - \frac{1}{B} \sum_{i=1}^B \log \left[ \frac{\exp(\mathrm{sim}(z_i^{(1)}, z_i^{(2)})/\tau)} {\sum_{j=1}^B \exp(\mathrm{sim}(z_i^{(1)}, z_j^{(2)})/\tau)} \right],$$1

The graph-code matching objective is binary. Each query output is passed through a linear classifier, the logits are averaged, and the prediction is

$$\mathrm{Loss} = - \frac{1}{B} \sum_{i=1}^B \log \left[ \frac{\exp(\mathrm{sim}(z_i^{(1)}, z_i^{(2)})/\tau)} {\sum_{j=1}^B \exp(\mathrm{sim}(z_i^{(1)}, z_j^{(2)})/\tau)} \right],$$2

with loss

$$\mathrm{Loss} = - \frac{1}{B} \sum_{i=1}^B \log \left[ \frac{\exp(\mathrm{sim}(z_i^{(1)}, z_i^{(2)})/\tau)} {\sum_{j=1}^B \exp(\mathrm{sim}(z_i^{(1)}, z_j^{(2)})/\tau)} \right],$$3

These three terms jointly train the model to align structural and code modalities, to preserve generation utility, and to discriminate matched from mismatched graph-code pairs (Zhao et al., 27 Sep 2025).

In stage 2, the code transformer is removed, the graph transformer and query tokens are retained, and a projection to the LLM embedding space is added. The LLM is frozen, and the training objective becomes

$$\mathrm{Loss} = - \frac{1}{B} \sum_{i=1}^B \log \left[ \frac{\exp(\mathrm{sim}(z_i^{(1)}, z_i^{(2)})/\tau)} {\sum_{j=1}^B \exp(\mathrm{sim}(z_i^{(1)}, z_j^{(2)})/\tau)} \right],$$4

The distribution calibration term is introduced to mitigate mismatch between the LLM’s native embedding outputs $$\mathrm{Loss} = - \frac{1}{B} \sum_{i=1}^B \log \left[ \frac{\exp(\mathrm{sim}(z_i^{(1)}, z_i^{(2)})/\tau)} {\sum_{j=1}^B \exp(\mathrm{sim}(z_i^{(1)}, z_j^{(2)})/\tau)} \right],$$5 and VeriFormer’s projected prompt outputs $$\mathrm{Loss} = - \frac{1}{B} \sum_{i=1}^B \log \left[ \frac{\exp(\mathrm{sim}(z_i^{(1)}, z_i^{(2)})/\tau)} {\sum_{j=1}^B \exp(\mathrm{sim}(z_i^{(1)}, z_j^{(2)})/\tau)} \right],$$6:

$$\mathrm{Loss} = - \frac{1}{B} \sum_{i=1}^B \log \left[ \frac{\exp(\mathrm{sim}(z_i^{(1)}, z_i^{(2)})/\tau)} {\sum_{j=1}^B \exp(\mathrm{sim}(z_i^{(1)}, z_j^{(2)})/\tau)} \right],$$7

The paper identifies prompt length $$\mathrm{Loss} = - \frac{1}{B} \sum_{i=1}^B \log \left[ \frac{\exp(\mathrm{sim}(z_i^{(1)}, z_i^{(2)})/\tau)} {\sum_{j=1}^B \exp(\mathrm{sim}(z_i^{(1)}, z_j^{(2)})/\tau)} \right],$$8 as the default, with $$\mathrm{Loss} = - \frac{1}{B} \sum_{i=1}^B \log \left[ \frac{\exp(\mathrm{sim}(z_i^{(1)}, z_i^{(2)})/\tau)} {\sum_{j=1}^B \exp(\mathrm{sim}(z_i^{(1)}, z_j^{(2)})/\tau)} \right],$$9 controlling contrastive alignment sharpness and $\mathrm{sim}$0 controlling prompt distribution calibration (Zhao et al., 27 Sep 2025).

The full training pipeline includes GNN pretraining, retriever training with teacher-student distillation, VeriFormer stage 1, and VeriFormer stage 2. Hyperparameters reported for VeriFormer are: stage 1 for 3 epochs with learning rate $\mathrm{sim}$1, minimum learning rate $\mathrm{sim}$2, on $\mathrm{sim}$3V100 32GB, global batch size 400; stage 2 for 1 epoch with learning rate $\mathrm{sim}$4, minimum learning rate $\mathrm{sim}$5, early stopping, and global batch size 64. VeriFormer is trained in a system that uses 276,627 Verilog examples from OriGen and PyraNet after MinHash/Jaccard deduplication and filtering (Zhao et al., 27 Sep 2025).

5. Integration with base LLMs and observed generation behavior

The reported integration is prefix-based. The synthesized soft prompt $\mathrm{sim}$6 is prepended to the token embeddings of the input specification or partial code, while the parameters of the base LLM remain frozen. The method was tested with DeepSeek-Coder-7B-Instruct, Qwen2.5-Coder-7B-Instruct, and Qwen2.5-Coder-14B-Instruct, and decoding temperatures of 0.2, 0.5, and 0.8 were evaluated (Zhao et al., 27 Sep 2025).

The claimed mechanism is that the prompt vectors inject architectural constraints—bit-widths, module connectivity, and dataflow—directly into the attention stack at the first layer. During generation, the LLM attends to these prompt vectors, which biases token probabilities toward structurally consistent Verilog. The paper further states that explicit syntax guards or post-processing constraints are not required, because structure-aware prompting reduces syntax errors and wrong module wiring natively (Zhao et al., 27 Sep 2025).

Two qualitative examples are used to illustrate the kinds of problems VeriFormer-conditioned prompting resolves. In a parameterized up-counter with synchronous active-high reset and enable, the baseline output is described as using always @(posedge clk or posedge rst) and therefore as missing synchronous reset semantics, with incorrect bit-width propagation and erroneous sensitivity-list behavior. The VeriFormer-conditioned output instead uses always @(posedge clk), parameterized zero initialization, and a width-consistent increment constant, which the paper interprets as proper structural alignment (Zhao et al., 27 Sep 2025).

In a producer-consumer valid-ready handshake example, the baseline output is described as transferring data on valid only and ignoring ready. The VeriFormer-conditioned output includes correct valid && ready gating on transfer, consistent module I/O wiring, synchronous reset, and signal widths consistent with the data path graph (Zhao et al., 27 Sep 2025).

The associated error analysis identifies common baseline failure modes as incorrect signal widths and literal sizes, missing or incorrect sensitivity lists, miswired module interfaces, and incomplete always blocks leading to latch inference. The paper states that structure-aware prompting mitigates these failures by biasing the LLM to obey graph-derived constraints involving bit-widths, control-flow gating, and module I/O alignment (Zhao et al., 27 Sep 2025).

6. Empirical results, related methods, and limitations

Evaluation is reported on VerilogEval (Human, Machine, v2) and RTLLM (v1.1, v2), with functional correctness measured by pass@k,

$\mathrm{sim}$7

using $\mathrm{sim}$8 and $\mathrm{sim}$9. Syntax correctness is also reported on subsets (Zhao et al., 27 Sep 2025).

On VerilogEval v1 (Machine), VeriGRAG-Qwen2.5-14B achieves pass@1 = 82.9 and pass@5 = 89.4, while VeriGRAG-Qwen2.5-7B achieves pass@1 = 80.2 and pass@5 = 88.2. Relative to strong baselines, improvements of 1–7% in pass@1 are reported depending on model size. On VerilogEval v2 (Function), VeriGRAG-Qwen2.5-14B achieves pass@1 = 62.2 and pass@5 = 69.5, outperforming VeriPrefer-Qwen2.5-14B on pass@1 by +3.6%; VeriGRAG-Qwen2.5-7B achieves pass@1 = 58.9 and pass@5 = 65.6, beating VeriPrefer-Qwen2.5-7B by +9.6% in pass@1. On RTLLM v1.1 (Function), VeriGRAG-Qwen2.5-7B reaches pass@1 = 57.6 and pass@5 = 68.0, and VeriGRAG-DeepSeek-7B reaches pass@1 = 56.9 and pass@5 = 67.6. On RTLLM v2 (Function), VeriGRAG-Qwen2.5-7B reaches pass@1 = 57.7 and pass@5 = 67.9, and VeriGRAG-DeepSeek-7B reaches pass@1 = 56.7 and pass@5 = 67.1. Syntax pass@5 reaches 99.9–100.0 for some model variants (Zhao et al., 27 Sep 2025).

Ablation results attribute a substantial share of these gains to structure-aware soft prompting. Three settings are compared: the base LLM without structural information, VeriGRAG with structure-aware soft prompts and a classic dual-encoder retriever, and VeriGRAG+Retriever(KD) with a student retriever distilled from a cross-attention teacher. On VerilogEval-Human, VeriGRAG-Qwen2.5-14B improves pass@1 by +13.3% over the pretrained model; on VerilogEval v2, it improves pass@1 by +17.9% over the pretrained model; distillation adds small gains (Zhao et al., 27 Sep 2025).

Inference overhead is characterized as modest. The reported runtime consists of a FAISS lookup over cached graph embeddings, a forward pass through the lightweight VeriFormer on the graph side only with 96 query tokens, and standard LLM decoding. The stated reasons are that graph embeddings are pre-indexed, LLM weights are frozen, soft prompts comprise only 96 continuous vectors, and no large cross-attention teacher is used at inference (Zhao et al., 27 Sep 2025).

VeriFormer is situated among several prior approaches to Verilog generation, including dataset curation in RTLCoder, discriminative guidance in BetterV, retrieval-augmented generation from text in AutoVCoder, code-to-code augmentation and self-reflection in OriGen, multi-agent decomposition in MAGE, hallucination mitigation in HaVen, hybrid reasoning in ReasoningV, and RL/DPO with testbench feedback in VeriPrefer. Its stated novelty is the two-stage cross-modal alignment of graph and code via VeriFormer together with structure-aware soft prompts conditioned on hardware data path graphs and enabled by a multimodal retriever distilled from a cross-attention teacher into a dual encoder (Zhao et al., 27 Sep 2025).

The paper also notes several limitations. Performance depends on the quality of graph extraction through the Yosys pipeline and on retrieval relevance; mismatched graphs can reduce gains. It also identifies possible domain overfitting to patterned circuits in the training corpora. Proposed future directions include end-to-end differentiable parsing, larger functionally validated corpora, dynamic retrieval during decoding, and joint fine-tuning of the LLM with structure-aware adapters. A plausible implication is that VeriFormer’s effectiveness is tightly coupled to the fidelity of the structural representation it receives, rather than to prompting alone (Zhao et al., 27 Sep 2025).