---
title: Circuit-Aware Reward Training (CART)
url: https://www.emergentmind.com/topics/circuit-aware-reward-training-cart
type: topic
---

# Circuit-Aware Reward Training (CART)

Searching arXiv for the cited works and closely related context.
Circuit-Aware Reward Training (CART) denotes a class of reward-guided optimization procedures in which a generative model is trained against circuit-specific criteria rather than sequence likelihood alone. In the AutoCircuit-RL setting, “CART” is an *Editor’s term* for the paper’s “RL-refinement with AI feedback (RLAIF)”: an instruction-tuned LLM is further optimized with reward models that evaluate structural validity, efficiency, and output voltage so that generated topologies better satisfy design constraints [2506.03122]. AstRL realizes the same general idea in a graph-generation setting with simulator-embedded feedback, structural masks, symmetry-aware actions, behavioral cloning, and discriminator-based similarity rewards [2602.12402]. A separate paper uses the same acronym for a mechanistic interpretability framework for RLHF reward models; that usage concerns longtail robustness rather than electronic design [2509.24713].

## 1. Terminology, scope, and problem setting

In current usage, CART is not a single standardized algorithm. It refers, in the circuit-synthesis literature considered here, to reward training in which the reward signal is explicitly tied to circuit properties such as structural validity, efficiency, output voltage, or simulator-measured performance. This distinguishes CART from pure supervised netlist modeling and from heuristic search methods that do not learn reward surrogates or use RL-driven policy improvement [2506.03122].

| Usage | Reward source | Primary target |
|---|---|---|
| AutoCircuit-RL | Learned estimators for validity, efficiency, and output voltage | LLM-based topology generation |
| AstRL | Structural penalties, similarity reward, simulator-based domain reward | Graph-based AMS synthesis |
| CART in RLHF | Circuit-guided augmentation, regularization, curriculum, ensembling | Longtail robustness of reward models |

AutoCircuit-RL casts circuit synthesis as a one-step episodic RL problem. The state is the prompt \(X\), which may specify a component pool, \(V_{\text{in}}\), desired \(V_{\text{out}}\), or an efficiency threshold. The policy \(\pi_\theta\) is an autoregressive LLM that samples a sequence \(\hat{Y}=(y_1,\dots,y_T)\) encoding a full topology in incident encoding plus duty cycle. At the episode level, the entire generated netlist is treated as a single composite action and receives a scalar reward only after completion. The paper explicitly characterizes this as a bandit-style RL setup on sequences [2506.03122].

AstRL adopts a different RL formulation. Its state is a partially constructed circuit graph \(G=(V,E)\) with typed nodes and terminal-aware edges, and the agent performs structured graph edits at each timestep. This moves CART from sequence generation over text-encoded netlists to sequential topology construction at the level of individual transistors and nets [2602.12402].

## 2. Sequence-based CART in AutoCircuit-RL

AutoCircuit-RL has two training phases. The first is instruction tuning, or supervised fine-tuning, on pairs \((X_i,Y_i)\), where \(X_i\) is an instruction prompt encoding design constraints and \(Y_i\) is a valid circuit topology netlist in incident encoding plus duty cycle. Prompts are organized into three types: component constraint (C), component plus efficiency constraint (C+E), and component plus output-voltage constraint (C+V). The backbone is an autoregressive LLM \(\pi_\theta\), instantiated as Llama-3-8B or MPT-7B. The supervised objective is standard next-token log-likelihood over the target topology sequence [2506.03122].

The second phase is RL-refinement with AI feedback. Starting from the instruction-tuned base policy, the system adds reward models that score complete generations on structural validity, efficiency, and output voltage relative to the prompt. PPO with KL regularization is then used to update the policy toward a new RL-refined model while keeping it close to the supervised reference behavior. The paper writes the objective as
\[
\mathcal{L}_{RL} = r(X,\hat{Y}) - \eta \, KL\big(\pi_{\mathrm{RLAIF}}(\hat{Y}\mid X)\;\|\;\pi_\theta(\hat{Y}\mid X)\big).
\]
This is a KL-regularized RLHF-style formulation in which the circuit-aware reward substitutes for generic preference or heuristic signals [2506.03122].

A notable design choice is weighted data sampling in both SFT and RL. Simulated circuits are partitioned into four groups in \(V_{\text{out}}\)-efficiency space: low efficiency \((<0.05)\), medium efficiency \((0.05\text{–}0.7)\), high efficiency but \(V_{\text{out}}\approx V_{\text{in}}\), and high efficiency with desirable \(V_{\text{out}}\). The corresponding sampling weights are \(0.1\), \(0.25\), \(0.25\), and \(0.4\). This biases optimization toward high-quality regions of the design space [2506.03122].

Inference uses only the final RL-refined model. No RL loop and no simulation are invoked at inference time; the trained generator is used directly to produce circuit topologies from structured prompts [2506.03122].

## 3. Reward models, constraints, and optimization signals

The core of AutoCircuit-RL’s CART mechanism is a learned reward layer built from three RoBERTa-based estimators operating on netlist text in incident encoding plus duty cycle. The circuit validity estimator \(f_{\mathrm{valid}}\) is a binary classifier trained on valid and invalid netlists from an NGSpice-annotated dataset and achieves \(92\%\) F1. The circuit efficiency estimator \(f_{\mathrm{eff}}\) is a regression model that is evaluated by categorizing predicted efficiencies into predefined categories and achieves \(83\%\) macro F1. The output-voltage estimator \(f_{\mathrm{vout}}\) is a regression model with mean squared error approximately \(8\times 10^{-3}\) [2506.03122].

These estimators induce a piecewise reward function:
\[
r(X,\hat{Y})=
\begin{cases}
-1, & \text{if } s_{\mathrm{valid}} < 0.6 \\
1, & \text{if } s_{\mathrm{eff}} \text{ or } s_{\mathrm{vout}} \text{ meets constraints} \\
s_{\mathrm{eff}}, & \text{otherwise.}
\end{cases}
\]
The design is explicitly hierarchical. Structural validity is a hard gate: if the validity classifier is not sufficiently confident, the circuit receives maximum negative reward. Constraint satisfaction is prioritized next: if a valid circuit is predicted to meet the efficiency or \(V_{\text{out}}\) requirement, it receives full reward. For valid circuits that do not yet meet prompt-level thresholds, the reward is proportional to predicted efficiency, which shapes the search toward better candidates even when exact targets are missed [2506.03122].

NGSpice is essential but used offline. Random Search explores power-converter topologies with \(4\)–\(10\) components, each netlist is simulated at duty cycles \(0.1\), \(0.3\), \(0.5\), \(0.7\), and \(0.9\), and the resulting labels—validity, efficiency, and output voltage—are used both to train the reward estimators and to evaluate generated circuits with exact simulation at test time. The RL loop itself does not call NGSpice online; the learned estimators provide instant AI feedback during policy optimization [2506.03122].

AutoCircuit-RL also adds an Iterative Adaptation (IA) loop. Starting from an RL-tuned policy, the system generates many circuits by nucleus sampling, filters for predicted efficiency \(s_{\mathrm{eff}}>0.7\) and validity under \(f_{\mathrm{valid}}\), collects \(10{,}000\) such samples, and performs additional RL updates with the same objective. This loop is repeated for \(3\)–\(5\) IA iterations, each \(2\)–\(4\) epochs depending on backbone. The reported role of IA is to concentrate learning on high-reward regions after the broader RL phase has already improved policy behavior [2506.03122].

## 4. Graph-based and simulator-embedded CART in AstRL

AstRL formulates analog and mixed-signal synthesis directly as graph generation. Each partial circuit is represented as a graph \(G=(V,E)\) whose nodes are typed as active devices, passives, generic nets, and special nets, and whose edges encode terminal type such as gate, drain, source, bulk, or passive terminals. This representation is circuit-aware in a strong sense: nets are explicit graph objects, and edge attributes preserve terminal semantics that are behaviorally decisive in analog design [2602.12402].

Actions are structured and sampled autoregressively as
\[
a_t=[a_{\text{source node}}, a_{\text{target node}}, a_{\text{edge type}}, a_{\text{addition type}}, a_{\text{terminate}}].
\]
The action space is heavily masked. The source node must be an existing node; the target node may be an existing node or a new scaffold node, subject to legality constraints; edge types are limited by terminal availability; and termination is allowed only when structural completion constraints are satisfied. The environment also forbids direct component-to-component connection, net-to-net edges, and floating terminals. According to the paper, these masks ensure exploration remains within structurally feasible topology edits [2602.12402].

A distinctive inductive bias is symmetry-aware construction. Differential analog circuits are symmetric around the supply-ground axis, and AstRL introduces an addition-type component that chooses among five symmetry modifiers: single common-mode, symmetric pair, symmetric pair to common-mode component, symmetric pair to common-mode net, and common-mode to symmetric pair. These modifiers automatically mirror edits onto the symmetric branch and enforce consistent edge attributes. The reported purpose is to encode analog motifs such as differential pairs and symmetric loads directly into the action space [2602.12402].

The policy and discriminator use GINE with three layers and \(64\)-dimensional node embeddings. PPO with entropy regularization is combined with behavioral cloning on expert trajectories, with the BC term annealed over training iterations. Reward is decomposed as
\[
r(s_t,a_t)=r_{\text{validity}}+r_{\text{similarity}}+r_{\text{domain}}.
\]
Here \(r_{\text{validity}}\) penalizes structurally inconsistent actions with \(-2\) and leaves valid actions at \(0\); \(r_{\text{similarity}}\) is produced by a graph discriminator trained on expert subgraphs versus policy rollouts, yielding \(+1\) for expert-like partial graphs and \(-1\) otherwise; and \(r_{\text{domain}}\) combines simulator validity, multi-objective optimization over specifications, and a success bonus when all specs are satisfied. The simulator-validity term is reported as \(-2\) for structurally invalid final circuits, \(+3\) for structurally valid but simulation-invalid circuits, and \(+30\) for simulation-valid circuits, while a further \(+10\) is added if all specs are met [2602.12402].

This version of CART differs fundamentally from AutoCircuit-RL. AutoCircuit-RL uses learned surrogates to avoid online simulation and optimizes a text-generating LLM over full-netlist outputs. AstRL embeds the simulator in the training loop, works on graph-structured states, and relies on hard environment constraints and symmetry modifiers as part of the reward-design ecosystem [2506.03122].

## 5. Reported outcomes and relation to adjacent methods

AutoCircuit-RL reports gains that are explicitly attributed to RL with circuit-aware rewards. Relative to the best baselines, the framework generates approximately \(12\%\) more valid circuits, improves efficiency by approximately \(14\%\), and reduces duplicate generation rates by approximately \(38\%\). In detailed results for Llama-3, classifier-estimated validity reaches \(75.11\) versus \(67.60\) on \(4\)-component circuits and \(73.46\) versus \(65.75\) on \(5\)-component circuits. Simulator-estimated efficiency reaches \(71.65\) on \(4\)-component circuits versus best baseline values around \(63.87\)–\(63.19\). Duplicate Generation Rate drops from approximately \(\rho\approx 2.19\) for Gumbel Llama-3 to \(\rho\approx 1.29\) for Ac-RL. Success rates for Ac-RL (Llama-3) are \(99.08\%\) on C, \(80.90\%\) on C+E, \(71.30\%\) on C+V, and \(80.69\%\) overall, compared with \(96.04\%\), \(69.56\%\), \(60.36\%\), and \(71.27\%\) for non-RL Gumbel Llama-3. Under few-shot tuning with \(k=1000\) for \(6\)–\(10\)-component circuits, success rates are \(65.5\%\), \(63.8\%\), \(63.2\%\), \(60.4\%\), and \(58.5\%\), which the paper summarizes as over \(60\%\) success with limited training data [2506.03122].

AstRL reports \(100\%\) netlist validity across ring oscillator, comparator, and OTA tasks because of its action constraints. Simulation validity reaches up to approximately \(99\%\) for comparator and OTA when the full set of circuit-aware components is used. Spec satisfaction is highly task-dependent: \(13.6\%\) of generated ring-oscillator designs meet all specifications, \(99.2\%\) of comparator designs meet spec, and \(65.2\%\) of OTA designs meet spec with full AstRL. The OTA ablations are particularly diagnostic: removing structural masks collapses netlist validity to \(0\%\), and removing symmetry modifiers leaves many circuits functional and simulatable but drives spec satisfaction to \(0\%\) [2602.12402].

| System | Reported reward-aware outcomes | Context |
|---|---|---|
| AutoCircuit-RL | \(\sim 12\%\) more valid circuits, \(\sim 14\%\) higher efficiency, \(\sim 38\%\) lower DGR | LLM-based power-converter topology generation |
| AutoCircuit-RL | Over \(60\%\) success with limited data for \(6\)–\(10\) components | Few-shot generalization |
| AstRL | \(100\%\) netlist validity; up to \(\sim 99\%\) simulation validity | Simulator-in-the-loop AMS synthesis |
| AstRL | \(13.6\%\) RO, \(99.2\%\) comparator, \(65.2\%\) OTA meet spec | Task-specific spec satisfaction |

Relative to adjacent approaches, AutoCircuit-RL is presented as distinct from search-based EDA methods such as GA and tree search, which use explicit heuristics or hand-designed cost functions and may require hundreds of SPICE calls per design, including more than \(400\) simulations in Fan et al. 2021. It is also distinguished from GraphVAE and DAG-based generative models that focus on valid graph generation without directly optimizing efficiency or \(V_{\text{out}}\), and from LLM-based methods such as CircuitSynth, AnalogCoder, LaMAGIC, Auto-SPICE, AnalogXpert, and Atelier that rely on SFT or prompt engineering rather than learned reward models and RL refinement. Within the paper’s own ablations, Gumbel-based multi-objective training improves over plain SFT but remains approximately \(9\%\) behind Ac-RL in validity and efficiency, and removing IA reduces validity and efficiency—for Llama-3, from \(75.11/71.65\) with IA to \(71.08/69.50\) without it [2506.03122].

## 6. Homonymous usage, limitations, and extensions

The acronym “CART” also names an unrelated framework in RLHF: “Circuit-Aware Reward Training: A Mechanistic Framework for Longtail Robustness in RLHF” [2509.24713]. There, circuits are not electrical topologies but functionally coherent subnetworks inside a reward model. The framework proceeds through circuit discovery, vulnerability assessment, and targeted intervention via circuit-guided data augmentation, circuit regularization, progressive circuit strengthening, and circuit-aware ensembling. Its central loss is
\[
\mathcal{L}_{\text{CART}}(\theta)=
\mathcal{L}_{\text{head}}(\theta)+
\mathcal{L}_{\text{aug}}(\theta)+
\mathcal{L}_{\text{circuit}}(\theta)+
\mathcal{L}_{\text{prog}}(\theta),
\]
and the paper is explicit that it is primarily conceptual, sketching evaluation protocols rather than presenting full-scale benchmark evidence. The name collision is therefore substantive: one usage concerns circuit synthesis in EDA, while the other concerns mechanistic robustness of RLHF reward models [2509.24713].

Within circuit synthesis itself, the two EDA instantiations expose complementary tradeoffs. AutoCircuit-RL relies on offline NGSpice data to train surrogate reward models, making online RL tractable but introducing approximation error; the paper notes that reward-model misestimation can misguide RL, including near-misses around efficiency thresholds [2506.03122]. AstRL avoids surrogate mismatch by embedding the simulator in the loop, but the paper identifies sample efficiency as a challenge and notes that training runs take days [2602.12402].

The reported extension paths are consistent across the literature. AutoCircuit-RL discusses applying the same reward framework to more complex circuits and families, including filters, op-amps, and oscillators, and explicitly mentions additional metrics such as power consumption, component-level parameter estimation, and more complex design constraints [2506.03122]. AstRL highlights broader issues of multi-objective scalarization with fixed weights, scalability to larger hierarchical AMS blocks such as ADCs and PLLs, and robustness to process variation via Monte Carlo and PVT-corner-aware rewards [2602.12402]. Taken together, these works suggest that CART is best understood not as a fixed recipe but as a design pattern: combine circuit-aware representations, structurally meaningful action spaces, and rewards tied either to learned surrogates or to direct simulation so that policy optimization tracks electrical objectives rather than surface-form regularities.

Source: https://www.emergentmind.com/topics/circuit-aware-reward-training-cart