Circuit-Aware Reward Training (CART)

• Circuit-Aware Reward Training (CART) is a framework that integrates reward signals from circuit properties like efficiency and output voltage to guide generative model optimization.
• It utilizes either learned surrogate models from offline simulation data or embedded simulators to provide immediate feedback on structural validity and performance constraints.
• Implementations such as AutoCircuit-RL and AstRL demonstrate notable improvements in circuit validity, efficiency, and specification adherence over traditional EDA methods.

Searching arXiv for the cited works and closely related context. Circuit-Aware Reward Training (CART) denotes a class of reward-guided optimization procedures in which a generative model is trained against circuit-specific criteria rather than sequence likelihood alone. In the AutoCircuit-RL setting, “CART” is an Editor’s term for the paper’s “RL-refinement with AI feedback (RLAIF)”: an instruction-tuned LLM is further optimized with reward models that evaluate structural validity, efficiency, and output voltage so that generated topologies better satisfy design constraints (Vijayaraghavan et al., 3 Jun 2025). AstRL realizes the same general idea in a graph-generation setting with simulator-embedded feedback, structural masks, symmetry-aware actions, behavioral cloning, and discriminator-based similarity rewards (Guo et al., 12 Feb 2026). A separate paper uses the same acronym for a mechanistic interpretability framework for RLHF reward models; that usage concerns longtail robustness rather than electronic design (Liu, 29 Sep 2025).

1. Terminology, scope, and problem setting

In current usage, CART is not a single standardized algorithm. It refers, in the circuit-synthesis literature considered here, to reward training in which the reward signal is explicitly tied to circuit properties such as structural validity, efficiency, output voltage, or simulator-measured performance. This distinguishes CART from pure supervised netlist modeling and from heuristic search methods that do not learn reward surrogates or use RL-driven policy improvement (Vijayaraghavan et al., 3 Jun 2025).

Usage	Reward source	Primary target
AutoCircuit-RL	Learned estimators for validity, efficiency, and output voltage	LLM-based topology generation
AstRL	Structural penalties, similarity reward, simulator-based domain reward	Graph-based AMS synthesis
CART in RLHF	Circuit-guided augmentation, regularization, curriculum, ensembling	Longtail robustness of reward models

AutoCircuit-RL casts circuit synthesis as a one-step episodic RL problem. The state is the prompt $X$, which may specify a component pool, $V_{\text{in}}$, desired $V_{\text{out}}$, or an efficiency threshold. The policy $\pi_\theta$ is an autoregressive LLM that samples a sequence $\hat{Y}=(y_1,\dots,y_T)$ encoding a full topology in incident encoding plus duty cycle. At the episode level, the entire generated netlist is treated as a single composite action and receives a scalar reward only after completion. The paper explicitly characterizes this as a bandit-style RL setup on sequences (Vijayaraghavan et al., 3 Jun 2025).

AstRL adopts a different RL formulation. Its state is a partially constructed circuit graph $G=(V,E)$ with typed nodes and terminal-aware edges, and the agent performs structured graph edits at each timestep. This moves CART from sequence generation over text-encoded netlists to sequential topology construction at the level of individual transistors and nets (Guo et al., 12 Feb 2026).

2. Sequence-based CART in AutoCircuit-RL

AutoCircuit-RL has two training phases. The first is instruction tuning, or supervised fine-tuning, on pairs $(X_i,Y_i)$, where $X_i$ is an instruction prompt encoding design constraints and $Y_i$ is a valid circuit topology netlist in incident encoding plus duty cycle. Prompts are organized into three types: component constraint (C), component plus efficiency constraint (C+E), and component plus output-voltage constraint (C+V). The backbone is an autoregressive LLM $\pi_\theta$, instantiated as Llama-3-8B or MPT-7B. The supervised objective is standard next-token log-likelihood over the target topology sequence (Vijayaraghavan et al., 3 Jun 2025).

The second phase is RL-refinement with AI feedback. Starting from the instruction-tuned base policy, the system adds reward models that score complete generations on structural validity, efficiency, and output voltage relative to the prompt. PPO with KL regularization is then used to update the policy toward a new RL-refined model while keeping it close to the supervised reference behavior. The paper writes the objective as

$V_{\text{in}}$0

This is a KL-regularized RLHF-style formulation in which the circuit-aware reward substitutes for generic preference or heuristic signals (Vijayaraghavan et al., 3 Jun 2025).

A notable design choice is weighted data sampling in both SFT and RL. Simulated circuits are partitioned into four groups in $V_{\text{in}}$1-efficiency space: low efficiency $V_{\text{in}}$2, medium efficiency $V_{\text{in}}$3, high efficiency but $V_{\text{in}}$4, and high efficiency with desirable $V_{\text{in}}$5. The corresponding sampling weights are $V_{\text{in}}$6, $V_{\text{in}}$7, $V_{\text{in}}$8, and $V_{\text{in}}$9. This biases optimization toward high-quality regions of the design space (Vijayaraghavan et al., 3 Jun 2025).

Inference uses only the final RL-refined model. No RL loop and no simulation are invoked at inference time; the trained generator is used directly to produce circuit topologies from structured prompts (Vijayaraghavan et al., 3 Jun 2025).

3. Reward models, constraints, and optimization signals

The core of AutoCircuit-RL’s CART mechanism is a learned reward layer built from three RoBERTa-based estimators operating on netlist text in incident encoding plus duty cycle. The circuit validity estimator $V_{\text{out}}$0 is a binary classifier trained on valid and invalid netlists from an NGSpice-annotated dataset and achieves $V_{\text{out}}$1 F1. The circuit efficiency estimator $V_{\text{out}}$2 is a regression model that is evaluated by categorizing predicted efficiencies into predefined categories and achieves $V_{\text{out}}$3 macro F1. The output-voltage estimator $V_{\text{out}}$4 is a regression model with mean squared error approximately $V_{\text{out}}$5 (Vijayaraghavan et al., 3 Jun 2025).

These estimators induce a piecewise reward function: $V_{\text{out}}$6 The design is explicitly hierarchical. Structural validity is a hard gate: if the validity classifier is not sufficiently confident, the circuit receives maximum negative reward. Constraint satisfaction is prioritized next: if a valid circuit is predicted to meet the efficiency or $V_{\text{out}}$7 requirement, it receives full reward. For valid circuits that do not yet meet prompt-level thresholds, the reward is proportional to predicted efficiency, which shapes the search toward better candidates even when exact targets are missed (Vijayaraghavan et al., 3 Jun 2025).

NGSpice is essential but used offline. Random Search explores power-converter topologies with $V_{\text{out}}$8–$V_{\text{out}}$9 components, each netlist is simulated at duty cycles $\pi_\theta$0, $\pi_\theta$1, $\pi_\theta$2, $\pi_\theta$3, and $\pi_\theta$4, and the resulting labels—validity, efficiency, and output voltage—are used both to train the reward estimators and to evaluate generated circuits with exact simulation at test time. The RL loop itself does not call NGSpice online; the learned estimators provide instant AI feedback during policy optimization (Vijayaraghavan et al., 3 Jun 2025).

AutoCircuit-RL also adds an Iterative Adaptation (IA) loop. Starting from an RL-tuned policy, the system generates many circuits by nucleus sampling, filters for predicted efficiency $\pi_\theta$5 and validity under $\pi_\theta$6, collects $\pi_\theta$7 such samples, and performs additional RL updates with the same objective. This loop is repeated for $\pi_\theta$8–$\pi_\theta$9 IA iterations, each $\hat{Y}=(y_1,\dots,y_T)$0–$\hat{Y}=(y_1,\dots,y_T)$1 epochs depending on backbone. The reported role of IA is to concentrate learning on high-reward regions after the broader RL phase has already improved policy behavior (Vijayaraghavan et al., 3 Jun 2025).

4. Graph-based and simulator-embedded CART in AstRL

AstRL formulates analog and mixed-signal synthesis directly as graph generation. Each partial circuit is represented as a graph $\hat{Y}=(y_1,\dots,y_T)$2 whose nodes are typed as active devices, passives, generic nets, and special nets, and whose edges encode terminal type such as gate, drain, source, bulk, or passive terminals. This representation is circuit-aware in a strong sense: nets are explicit graph objects, and edge attributes preserve terminal semantics that are behaviorally decisive in analog design (Guo et al., 12 Feb 2026).

Actions are structured and sampled autoregressively as

$\hat{Y}=(y_1,\dots,y_T)$3

The action space is heavily masked. The source node must be an existing node; the target node may be an existing node or a new scaffold node, subject to legality constraints; edge types are limited by terminal availability; and termination is allowed only when structural completion constraints are satisfied. The environment also forbids direct component-to-component connection, net-to-net edges, and floating terminals. According to the paper, these masks ensure exploration remains within structurally feasible topology edits (Guo et al., 12 Feb 2026).

A distinctive inductive bias is symmetry-aware construction. Differential analog circuits are symmetric around the supply-ground axis, and AstRL introduces an addition-type component that chooses among five symmetry modifiers: single common-mode, symmetric pair, symmetric pair to common-mode component, symmetric pair to common-mode net, and common-mode to symmetric pair. These modifiers automatically mirror edits onto the symmetric branch and enforce consistent edge attributes. The reported purpose is to encode analog motifs such as differential pairs and symmetric loads directly into the action space (Guo et al., 12 Feb 2026).

The policy and discriminator use GINE with three layers and $\hat{Y}=(y_1,\dots,y_T)$4-dimensional node embeddings. PPO with entropy regularization is combined with behavioral cloning on expert trajectories, with the BC term annealed over training iterations. Reward is decomposed as

$\hat{Y}=(y_1,\dots,y_T)$5

Here $\hat{Y}=(y_1,\dots,y_T)$6 penalizes structurally inconsistent actions with $\hat{Y}=(y_1,\dots,y_T)$7 and leaves valid actions at $\hat{Y}=(y_1,\dots,y_T)$8; $\hat{Y}=(y_1,\dots,y_T)$9 is produced by a graph discriminator trained on expert subgraphs versus policy rollouts, yielding $G=(V,E)$0 for expert-like partial graphs and $G=(V,E)$1 otherwise; and $G=(V,E)$2 combines simulator validity, multi-objective optimization over specifications, and a success bonus when all specs are satisfied. The simulator-validity term is reported as $G=(V,E)$3 for structurally invalid final circuits, $G=(V,E)$4 for structurally valid but simulation-invalid circuits, and $G=(V,E)$5 for simulation-valid circuits, while a further $G=(V,E)$6 is added if all specs are met (Guo et al., 12 Feb 2026).

This version of CART differs fundamentally from AutoCircuit-RL. AutoCircuit-RL uses learned surrogates to avoid online simulation and optimizes a text-generating LLM over full-netlist outputs. AstRL embeds the simulator in the training loop, works on graph-structured states, and relies on hard environment constraints and symmetry modifiers as part of the reward-design ecosystem (Vijayaraghavan et al., 3 Jun 2025).

5. Reported outcomes and relation to adjacent methods

AutoCircuit-RL reports gains that are explicitly attributed to RL with circuit-aware rewards. Relative to the best baselines, the framework generates approximately $G=(V,E)$7 more valid circuits, improves efficiency by approximately $G=(V,E)$8, and reduces duplicate generation rates by approximately $G=(V,E)$9. In detailed results for Llama-3, classifier-estimated validity reaches $(X_i,Y_i)$0 versus $(X_i,Y_i)$1 on $(X_i,Y_i)$2-component circuits and $(X_i,Y_i)$3 versus $(X_i,Y_i)$4 on $(X_i,Y_i)$5-component circuits. Simulator-estimated efficiency reaches $(X_i,Y_i)$6 on $(X_i,Y_i)$7-component circuits versus best baseline values around $(X_i,Y_i)$8–$(X_i,Y_i)$9. Duplicate Generation Rate drops from approximately $X_i$0 for Gumbel Llama-3 to $X_i$1 for Ac-RL. Success rates for Ac-RL (Llama-3) are $X_i$2 on C, $X_i$3 on C+E, $X_i$4 on C+V, and $X_i$5 overall, compared with $X_i$6, $X_i$7, $X_i$8, and $X_i$9 for non-RL Gumbel Llama-3. Under few-shot tuning with $Y_i$0 for $Y_i$1–$Y_i$2-component circuits, success rates are $Y_i$3, $Y_i$4, $Y_i$5, $Y_i$6, and $Y_i$7, which the paper summarizes as over $Y_i$8 success with limited training data (Vijayaraghavan et al., 3 Jun 2025).

AstRL reports $Y_i$9 netlist validity across ring oscillator, comparator, and OTA tasks because of its action constraints. Simulation validity reaches up to approximately $\pi_\theta$0 for comparator and OTA when the full set of circuit-aware components is used. Spec satisfaction is highly task-dependent: $\pi_\theta$1 of generated ring-oscillator designs meet all specifications, $\pi_\theta$2 of comparator designs meet spec, and $\pi_\theta$3 of OTA designs meet spec with full AstRL. The OTA ablations are particularly diagnostic: removing structural masks collapses netlist validity to $\pi_\theta$4, and removing symmetry modifiers leaves many circuits functional and simulatable but drives spec satisfaction to $\pi_\theta$5 (Guo et al., 12 Feb 2026).

System	Reported reward-aware outcomes	Context
AutoCircuit-RL	$\pi_\theta$6 more valid circuits, $\pi_\theta$7 higher efficiency, $\pi_\theta$8 lower DGR	LLM-based power-converter topology generation
AutoCircuit-RL	Over $\pi_\theta$9 success with limited data for $V_{\text{in}}$00–$V_{\text{in}}$01 components	Few-shot generalization
AstRL	$V_{\text{in}}$02 netlist validity; up to $V_{\text{in}}$03 simulation validity	Simulator-in-the-loop AMS synthesis
AstRL	$V_{\text{in}}$04 RO, $V_{\text{in}}$05 comparator, $V_{\text{in}}$06 OTA meet spec	Task-specific spec satisfaction

Relative to adjacent approaches, AutoCircuit-RL is presented as distinct from search-based EDA methods such as GA and tree search, which use explicit heuristics or hand-designed cost functions and may require hundreds of SPICE calls per design, including more than $V_{\text{in}}$07 simulations in Fan et al. 2021. It is also distinguished from GraphVAE and DAG-based generative models that focus on valid graph generation without directly optimizing efficiency or $V_{\text{in}}$08, and from LLM-based methods such as CircuitSynth, AnalogCoder, LaMAGIC, Auto-SPICE, AnalogXpert, and Atelier that rely on SFT or prompt engineering rather than learned reward models and RL refinement. Within the paper’s own ablations, Gumbel-based multi-objective training improves over plain SFT but remains approximately $V_{\text{in}}$09 behind Ac-RL in validity and efficiency, and removing IA reduces validity and efficiency—for Llama-3, from $V_{\text{in}}$10 with IA to $V_{\text{in}}$11 without it (Vijayaraghavan et al., 3 Jun 2025).

6. Homonymous usage, limitations, and extensions

The acronym “CART” also names an unrelated framework in RLHF: “Circuit-Aware Reward Training: A Mechanistic Framework for Longtail Robustness in RLHF” (Liu, 29 Sep 2025). There, circuits are not electrical topologies but functionally coherent subnetworks inside a reward model. The framework proceeds through circuit discovery, vulnerability assessment, and targeted intervention via circuit-guided data augmentation, circuit regularization, progressive circuit strengthening, and circuit-aware ensembling. Its central loss is

$V_{\text{in}}$12

and the paper is explicit that it is primarily conceptual, sketching evaluation protocols rather than presenting full-scale benchmark evidence. The name collision is therefore substantive: one usage concerns circuit synthesis in EDA, while the other concerns mechanistic robustness of RLHF reward models (Liu, 29 Sep 2025).

Within circuit synthesis itself, the two EDA instantiations expose complementary tradeoffs. AutoCircuit-RL relies on offline NGSpice data to train surrogate reward models, making online RL tractable but introducing approximation error; the paper notes that reward-model misestimation can misguide RL, including near-misses around efficiency thresholds (Vijayaraghavan et al., 3 Jun 2025). AstRL avoids surrogate mismatch by embedding the simulator in the loop, but the paper identifies sample efficiency as a challenge and notes that training runs take days (Guo et al., 12 Feb 2026).

The reported extension paths are consistent across the literature. AutoCircuit-RL discusses applying the same reward framework to more complex circuits and families, including filters, op-amps, and oscillators, and explicitly mentions additional metrics such as power consumption, component-level parameter estimation, and more complex design constraints (Vijayaraghavan et al., 3 Jun 2025). AstRL highlights broader issues of multi-objective scalarization with fixed weights, scalability to larger hierarchical AMS blocks such as ADCs and PLLs, and robustness to process variation via Monte Carlo and PVT-corner-aware rewards (Guo et al., 12 Feb 2026). Taken together, these works suggest that CART is best understood not as a fixed recipe but as a design pattern: combine circuit-aware representations, structurally meaningful action spaces, and rewards tied either to learned surrogates or to direct simulation so that policy optimization tracks electrical objectives rather than surface-form regularities.