ChipMATE: Multi-Agent RTL Generation

• ChipMATE is a multi-agent framework that pairs independent Verilog and Python agents to generate and cross-verify RTL code from natural language specifications.
• It employs a backtracking-based iterative inference scheme within a two-stage training pipeline combining single-agent RL and joint multi-agent RL via X-GRPO.
• Empirical results show significant pass@1 improvements over single-agent baselines, highlighting its potential for industrial-grade, air-gapped RTL synthesis.

ChipMATE is a self-trained, fully offline, multi-agent framework for RTL code generation that replaces the usual “generate RTL once and judge it with a golden oracle” setup with a design-verification loop modeled on industrial practice. It pairs a Verilog RTL agent with a Python reference-model agent, has them generate independently from the same natural-language specification, and then cross-checks their outputs without treating either artifact as a golden oracle. The framework introduces backtrack-based iterative inference, a two-stage training pipeline with single-agent and joint multi-agent reinforcement learning, and a hybrid data-generation framework for Python reference models. Reported headline results include 75.0\% and 80.1\% pass@1 on VerilogEval V2 with 4B and 9B base models, respectively (Yu et al., 13 May 2026).

1. Industrial framing and problem definition

ChipMATE is positioned against three mismatches between prior academic RTL-generation systems and industrial chip-design practice. First, prior agentic systems such as MAGE and VerilogCoder assume that a golden testbench is available at generation time and can serve as an oracle for self-correction. Second, many high-performing systems are API-based, relying on closed-source frontier models that are incompatible with the air-gapped deployment requirements of chip vendors. Third, API-centric approaches cannot exploit proprietary in-house RTL codebases for fine-tuning, even though those corpora are often higher quality than public data. ChipMATE argues that recent self-trained RTL models address deployment constraints but remain single-turn generators, which is inadequate because correctness in industrial RTL development emerges through iterative verification rather than one-shot synthesis (Yu et al., 13 May 2026).

The framework’s central analogy is to the separation between design and verification in production hardware development. One agent plays the role of the RTL design engineer and generates Verilog; the other plays the role of the verification engineer and independently generates a cycle-accurate Python reference model. Python is chosen over SystemC because open-source LLMs are stronger in Python. Neither side sees the other’s code. The key claim is that disagreement between independently produced implementations provides useful debugging signal even when neither side can be assumed correct.

A plausible implication is that ChipMATE treats verification not as an external oracle but as a learned, collaborative process. That design choice distinguishes it from both oracle-dependent agentic systems and single-turn self-trained generators.

2. Multi-agent architecture and cross-verification workflow

At inference time, both agents receive the same natural-language specification and independently sample candidate implementations. A separate cross-language comparison tool then compiles the Verilog with Icarus Verilog, executes the Python model, drives both with the same stimuli, compares outputs cycle by cycle, and summarizes mismatches in natural language for subsequent refinement (Yu et al., 13 May 2026).

Component	Function	Output
Verilog agent	Generates the RTL module	Verilog
Python agent	Generates a cycle-accurate behavioral model	Python reference model
Cross-language verifier	Simulates both implementations on shared stimuli	Match ratio and mismatch diagnostics

The end-to-end inference workflow has three phases. First, the two agents independently sample $N$ candidate implementations. Second, the verifier compares them on 1000 randomly generated stimuli. Third, if the outputs disagree, both agents are prompted again with structured mismatch diagnostics and asked to refine their own code. The process can continue for multiple turns until agreement is reached or a turn limit is hit.

The feedback channel is deliberately symmetric. Because neither artifact is guaranteed correct, the mismatch prompt explicitly tells each agent that either it or its peer might be wrong. Raw waveform files are further processed by a waveform-to-natural-language converter that identifies the first divergent cycle and presents surrounding context in a form the agents can use. The examples shown in the paper include messages such as “Test 0, signal out: got=42, exp=37 (inputs: a=5, b=3, clk=1),” while still warning that the peer implementation is not a golden reference.

3. Backtracking, prompt structure, and inference control

A major design choice in ChipMATE is backtrack-based inference. The paper argues that, without a golden oracle, iterative refinement can easily propagate errors: an agent may change a correct or partially correct implementation in response to a peer’s bug, causing the pair to drift away from the right solution. To avoid this, ChipMATE only accepts a revised candidate if it strictly improves the match rate between Verilog and Python; otherwise, it rolls back to the last accepted version. This imposes a monotonicity constraint on cross-verification quality and is one of the framework’s most distinctive control mechanisms (Yu et al., 13 May 2026).

The prompts are correspondingly structured. For the Verilog agent, the prompt includes a code skeleton with the exact module name and parameterized port list, plus guidance on combinational logic, sequential logic, reset handling, latch avoidance, coding conventions, and a few-shot reasoning-then-code example. For the Python reference-model agent, the prompt provides a fixed class skeleton with exact class name, method signatures, signal names, and bit widths, and instructs the model to complete __init__ and eval without renaming anything. The Python prompt also encodes hardware-specific rules, including masking state and outputs to preserve bit widths and two’s-complement semantics, and treating each eval() call as one rising edge.

For turns $t>0$, the prompt additionally includes truncated previous code attempts, a structured verification-error fragment, and a short refinement instruction. The paper explicitly treats the interface skeleton as realistic because production specifications already define ports exactly.

The ablation evidence in the paper indicates that multi-agent interaction is not automatically beneficial. A naive multi-agent workflow without backtracking hurts performance substantially, whereas rollback-constrained refinement recovers and surpasses the single-agent baseline. This suggests that the framework’s benefit depends as much on inference control as on the presence of multiple agents.

4. Two-stage training pipeline and collaborative reinforcement learning

ChipMATE trains its two agents in two stages. Stage 1 trains the Verilog and Python agents independently so that each is individually competent before collaboration is attempted. Stage 2 then trains the pair jointly so they can use mismatch information productively and coordinate through iterative refinement (Yu et al., 13 May 2026).

In Stage 1, both agents undergo supervised fine-tuning (SFT) followed by single-agent RL using GRPO. The Verilog agent is trained on the 87K-sample QiMeng CodeV-R1 dataset. The Python agent is trained on a newly constructed 64.4K-sample reference-model dataset. To make GRPO informative, the training set is filtered to keep only problems whose post-SFT pass@10 falls in $[0.1, 0.9]$, excluding near-zero-variance cases. The paper states that full-parameter fine-tuning, not LoRA, is used because strong individual agents are a prerequisite for successful multi-agent RL.

The Stage 1 implementation details are explicit. The base models are Qwen3.5 at 4B and 9B scales. SFT is run for 6 epochs with learning rate $1\times 10^{-5}$, global batch size 64, and context length 16,384. RL uses verl with GRPO, global batch size and minibatch size 128, group size 16, learning rate $1\times 10^{-6}$, 300 steps, rollout temperature 1.0, instruction length 2,048, and response length 16,384.

Stage 2 introduces X-GRPO, a multi-turn, multi-agent extension inspired by Tree-of-Thought and AT-GRPO. At each turn, both agents independently sample $K$ candidate outputs. These are paired index-wise into $K \times K$ candidate pairs and all pairs are evaluated; the best-scoring pair becomes the shared prefix for the next turn. The objective is written as

$$\mathcal{L}(\theta^{(i)}) = -\,\mathbb{E}_{g\in\mathcal{B}_i}\!\left[\frac{1}{K}\sum_{c=1}^K \min\!\Big(r_g^{(c,i)}\,A_g^{(c)},\; \mathrm{clip} \big(r_g^{(c,i)},\,1{-}\varepsilon,\,1{+}\varepsilon\big)\,A_g^{(c)}\Big)\right].$$

The reward design is hierarchical. Each agent receives a local reward, a correct-fix bonus, and a team-match reward. The local reward uses the tiered score

$s \in \{0,\, 0.1,\, 0.2,\, 0.2 + 0.8c\},$

corresponding to compile failure, runtime error, I/O port mismatch, and partial pass rate $c$, respectively. The paper states that the aggregate reward combines local, fix, and team-match terms, and explicitly gives the weights as $t>0$0, $t>0$1, and $t>0$2. The extracted total-reward equation is visibly malformed in the supplied text, but the surrounding explanation makes clear that the intended structure is a weighted sum of those three components.

5. Hybrid generation of the Python reference-model corpus

A substantial part of ChipMATE’s contribution is the construction of the first large-scale public dataset for generating Python hardware reference models. The paper states that no public corpus of this kind existed, so the authors built one using a hybrid framework with three pipelines, ultimately yielding 64.4K high-quality Python reference-model training samples (Yu et al., 13 May 2026).

The first pipeline is LLM-API agentic distillation. A frontier model is asked to generate a Python reference model from a Verilog module, verify it against the golden Verilog, and iteratively self-correct using mismatch feedback. On the 87K-sample QiMeng-CodeV dataset, DeepSeek-R1 achieves only 17% single-turn success and 35% two-turn success. Processing the full dataset with four API keys costs over \$2,000 and more than 200 hours, producing only about 25K verified samples.

To scale beyond that, the paper introduces an IR-based conversion pipeline for the remaining data. It parses Verilog with PyVerilog, normalizes ports, signals, assignments, and blocks into a uniform IR, lowers combinational logic into pure Python functions and sequential logic into per-cycle update functions with explicit bit masks for two’s-complement semantics, and wraps the result into a TopModule Python class with matching interface. An LLM is then used only to append chain-of-thought to already-correct code. This pipeline completes in under two hours and yields 36K additional verified samples. The paper explicitly notes a limitation: the appended CoT is “unreal”, meaning it is not the actual reasoning process used to derive the code.

The third pipeline is category-specific augmentation. After training an early Python model and evaluating on VerilogEval, the authors identify weak performance on finite-state machines, multi-cycle protocol blocks such as counters and FIFOs, and bit-level arithmetic. They then collect more Verilog examples for these categories via web search, convert them with the IR pipeline, and apply a contamination filter to avoid overlap with benchmark test cases. This adds about 3.4K targeted samples.

The resulting supervision is intentionally asymmetric: the Verilog side uses an existing public corpus, while the Python side uses a new reference-model corpus generated by distillation, deterministic conversion, and targeted augmentation.

6. Empirical results and ablation evidence

The experiments evaluate ChipMATE on VerilogEval v2, RTLLM v2, ChipBench-SC (the self-contained subset of ChipBench, with 78 cases), and CVDP cid03. The base models are Qwen3.5-4B and Qwen3.5-9B, and evaluation reports pass@$t>0$3 for $t>0$4 with 10 responses per query (Yu et al., 13 May 2026).

Model / benchmark	VerilogEval v2 pass@1	RTLLM v2 pass@1	ChipBench-SC pass@1	CVDP cid03 pass@1
ChipMATE-Agents-9B	80.1%	75.8%	36.7%	40.4%
ChipMATE-Python-9B	82.4%	77.3%	43.3%	43.3%
DeepSeek V4	67.3%	—	—	—

The headline result is on VerilogEval v2: ChipMATE-Agents-4B reaches 75.0% pass@1, and ChipMATE-Agents-9B reaches 80.1%. The corresponding single-agent Verilog checkpoints, ChipMATE-Verilog-4B and ChipMATE-Verilog-9B, score 67.4% and 75.3%, respectively, showing that the full multi-agent workflow improves substantially over the single Verilog generator. Compared with previous self-trained state of the art, CodeV-R1 at 7B achieves 68.8% pass@1 on VerilogEval v2, and the paper summarizes gains over CodeV-R1 as 6.7%–13.6% pass@1 depending on benchmark. The paper also emphasizes that ChipMATE-Agents-9B outperforms DeepSeek V4 with 1600B parameters on all four RTL-generation benchmarks in the main table.

The separate Python-generation results are also central to the paper’s interpretation. ChipMATE-Python-9B attains 82.4% pass@1 on VerilogEval v2, 77.3% on RTLLM v2, 43.3% on ChipBench-SC, and 43.3% on CVDP cid03. In all four benchmarks, the Python agent outperforms the Verilog agent. The paper interprets this as evidence that Python reference-model generation is easier for LLMs after hardware-specific fine-tuning, and that Python-side accuracy effectively sets an upper bound for the multi-agent workflow.

The ablation studies identify four dominant effects. First, SFT provides the biggest jump: +19.2% on the 4B model and +22.0% on the 9B model on VerilogEval v2 pass@1. Second, single-agent RL and then multi-agent RL each add a further 1.4%–6.5%. Third, naive multi-agent interaction without backtracking reduces performance by 11.6%–14.3%. Fourth, adding backtracking rebounds performance by 16.0%–18.6%, surpassing the single-agent result by 4.3%–4.4%. The paper also explores inference hyperparameters and finds that Best-of-3 with 3 turns is optimal on the 4B checkpoint, reaching 75.6 pass@1 on VerilogEval.

7. Novelty, industrial significance, and limitations

The paper claims four main contributions: a cross-verification multi-agent workflow for RTL generation without a golden testbench or cloud API; a backtracking-based iterative inference scheme; a two-stage training pipeline with independent SFT+RL followed by joint multi-agent RL using X-GRPO and hierarchical rewards; and the first large-scale reference-model training corpus with 64.4K verified Python samples (Yu et al., 13 May 2026).

Relative to prior RTL-generation work, the novelty lies in combining an industrially motivated design-verification split with learnable mutual verification. Prior API-based systems depended on testbenches and external LLMs; prior self-trained systems remained single-turn Verilog generators and still relied on external oracles such as testbenches or reference implementations. ChipMATE instead trains two independent models, makes the verifier itself learnable, and coordinates the pair through rollout-time cross-comparison and reward sharing. The paper also distinguishes the approach from concurrent work such as SiliconMind-V1 by emphasizing the dedicated reference-model generator and its custom data pipeline.

Its industrial significance is tied to deployment conditions. Because ChipMATE is self-trained and offline, it is explicitly compatible with air-gapped environments and could, in principle, be applied to proprietary local RTL codebases. The paper does not report experiments on vendor-private data, but it frames local fine-tuning compatibility as a major advantage. It also notes that pass@1 matters disproportionately in industrial RTL flows because bug avoidance is valued more than best-of-many sampling.

The limitations are also explicit. The paper does not claim that Python reference models fully capture real industrial verification, where SystemC, C++, and more complex environments are common. The random-stimulus verifier is simpler than full industrial methodologies. The IR-based conversion pipeline produces correct Python efficiently, but the appended chain-of-thought is acknowledged to be artificial. The system’s ceiling also depends on a strong Python agent: the analysis suggests that if the reference-model side is weak, the multi-agent workflow’s performance drops correspondingly. Finally, although the framework is designed for proprietary deployment, all reported experiments are on public benchmarks rather than an actual industrial codebase.

Taken together, ChipMATE defines RTL generation as a collaborative, verification-centered process rather than a single-turn code-synthesis task. That redefinition—combined with rollback-constrained inference, staged RL, and dedicated reference-model data—is the basis for its empirical gains and its relevance to industrially realistic hardware-design automation.