How Can Reinforcement Learning Achieve Expert-level Placement?

Abstract: Chip placement is a critical step in physical design. While reinforcement learning (RL)-based methods have recently emerged, their training primarily focuses on wirelength optimization, and therefore often fail to achieve expert-quality layouts. We identify the reward design as the primary cause for the performance gap with experts, and instead of formalizing intricate processes, we circumvent this by directly learning from expert layouts to derive a reward model. Our approach starts from the final expert layouts to infer step-by-step expert trajectories. Using these trajectories as demonstrations or preferences, we train a model that captures the latent implicit rewards in expert results. Experiments show that our framework can efficiently learn from even a single design and generalize well to unseen cases.

• The paper proposes a novel imitation-based reward model that learns from expert layout trajectories to overcome the limitations of hand-engineered rewards in chip placement.
• It demonstrates that demonstration-based reward learning (EIM-D) generalizes robustly on unseen designs with significant improvements in routed wirelength, congestion, and timing metrics.
• The approach reduces human effort in manual reward design and paves a scalable path for applying inverse reinforcement learning to complex VLSI physical design challenges.

Expert-Level Chip Placement via Reinforcement Learning Reward Model Imitation

Introduction

Achieving expert-level macro placement in VLSI physical design remains a formidable problem due to the reliance on intricate, often tacit, expert knowledge transcending simple heuristics like wirelength minimization. Despite advances in RL-based placement methods, such as AlphaChip and MaskPlace, existing systems are limited by their reward design—typically engineering handcrafted functions focused on wirelength (HPWL), which are insufficient to consistently reach expert-level power, performance, and area (PPA) metrics. The paper "How Can Reinforcement Learning Achieve Expert-level Placement?" (2604.25191) introduces an alternative paradigm: circumventing the need for explicit reward engineering by directly imitating final expert layouts to learn a robust, generalized reward model for RL-based chip placement.

Problem Analysis and Motivation

Traditional RL approaches to macro placement cast the process as an MDP, but the reward function is a critical bottleneck, as it typically optimizes surrogate metrics like HPWL. Human experts, in contrast, internalize complex domain knowledge regarding macro regularity, dead space, routing congestion, clock tree considerations, and other subtle, context-dependent factors. Attempts to inject human priors by encoding rules or using regularity masks (e.g., periphery biasing, macro grouping) have made partial progress but ultimately fail to capture unquantifiable aspects of expert intuition.

The empirical observation fueling this work is the abundance of expert-caliber layout instances produced in industry. Rather than converting implicit domain knowledge into explicit constraints or reward functions (an inherently lossy process), the approach leverages these expert placements as demonstration and preference data for reward learning. The hypothesis is that latent reward structures—otherwise inaccessible—can be effectively extracted via imitation learning, and then used to drive RL agents toward solutions rivaling human experts.

Methodology

The proposed framework, the Expert Imitation Model (EIM), proceeds via the following pipeline:

1. Expert Trajectory Inference: Each expert layout is decomposed into a sequential trajectory by assuming an incremental macro placement procedure. Each expert layout is reverse-engineered into a sequence of $(s, a, s')$ tuples, where $s$ is the partial placement state, $a$ the expert's next macro placement, and $s'$ the updated state. This yields dense, step-wise traces reflecting expert-level decision-making.
2. Preference and Demonstration Construction: Two types of expert signals are extracted:
   • Demonstrations: Direct action-state-next-state tuples reflecting the expert's choices.
   • Preferences: For each expert action in a given state, alternative (randomized, legal) actions are sampled. Explicit preference entries $(s, a_{\text{expert}}, a_{\text{random}})$ indicate that the expert's action is preferable.
3. Reward Model Learning: Two flavor of reward learning are investigated:
   • EIM-D (Demonstrations): Implements reward model extraction via max-entropy inverse RL, specifically the IQ-Learn algorithm, which sidesteps adversarial instability by unifying policy/reward parameters through soft Q-learning.
   • EIM-P (Preferences): Trains the reward model to score preferred actions higher than rejected actions, using a regularized RLHF-style preference loss.
4. Policy Optimization: A MaskPlace-style CNN encoder-decoder network is trained with PPO, using the learned reward model as the supervision signal instead of manually engineered rewards.

The full process is summarized in (Figure 1).

Figure 1: The proposed framework derives a reward model from expert placements, circumventing manual knowledge formalization by learning directly from final expert layouts.

Experimental Results

Benchmarks and Protocol

Experiments utilize two standard evaluation suites: ICCAD 2015 contest benchmarks and OpenROAD-flow-scripts, incorporating both seen (training) and unseen (held-out) chip designs. For each training case, a dataset of 50 expert layouts is curated. EIM-D and EIM-P are trained on single-design expert data, and then evaluated on both the training design and new, structurally diverse test cases. Comparisons include MaskPlace, EfficientPlace, and DREAMPlace 4.1.0, covering RL, hybrid, and analytical placement paradigms.

Reward Model Accuracy

The reward accuracy metric—reflecting the learned reward model's ability to rank the expert action highest among distractors—is near-perfect (0.99–1.00) for EIM-P and notably high (around 0.77–0.78) for EIM-D on training designs. Crucially, EIM-D generalizes better: On challenging unseen designs (e.g., superblue18), EIM-D retains substantially higher reward accuracy, while EIM-P exhibits pronounced overfitting.

Placement Quality and PPA Metrics

For both seen and unseen benchmarks, RL agents trained with EIM-derived rewards:

• Consistently outperform legacy MaskPlace on all PPA metrics, including routed wirelength (rWL), horizontal and vertical overflows (rOH/V), worst and total negative slack (WNS, TNS), and violated paths (NVP).
• Rival or surpass human expert layouts on several routing and timing metrics in design-specific evaluations.
• Demonstrate strong generalization: With reward models trained on only one design, EIM-D achieves up to 5–15% mean improvements in rWL, over 70% reductions in congestion, and 20–30% reduction in timing violations compared to RL and analytical baselines, even on structurally dissimilar chips.

EIM-P yields higher training accuracy but generalizes poorly; EIM-D, while slightly less accurate in-distribution, yields the best out-of-sample PPA performance. This dichotomy highlights the tradeoff between fitting expert traces and constructing a robust, transferable reward signal.

Theoretical and Practical Implications

The approach validates the hypothesis that reward design is the principal obstacle in expert-level RL chip placement, and that reward learning from expert outcomes bypasses the limitations of both manual reward engineering and policy cloning. It also demonstrates:

• Single-design reward model transferability, suggesting expert strategies can be generalized across chip architectures given sufficiently expressive policy and state representations.
• Reduction of human effort: The framework obviates the need for manually formalizing and maintaining complex reward logic as design heuristics evolve.
• The utility of inverse RL and preference-based reward modeling in large, high-dimensional industrial scheduling and planning scenarios, potentially extending to domains beyond VLSI physical design.

This work positions reward model imitation as a compelling avenue for bridging the gap between human and machine performance in intricate design automation problems.

Conclusion

This paper establishes that direct reward learning from expert layouts enables RL agents to achieve, and in some cases surpass, expert-level macro placement on both seen and unseen designs. It demonstrates that preference-based and demonstration-based reward inference methods, particularly IQ-Learn-based EIM-D, yield robust, generalizable reward models without recourse to heuristic engineering. The methodology is extensible and paves the way for future work in scaling expert dataset curation (e.g., via synthetic augmentation), integrating adversarial and diffusion-based imitation RL advances, and aligning physical design automation with broader trends in data-driven AI. Expert knowledge, distilled into reward models via imitation, offers a scalable path toward more autonomous and high-quality chip implementation frameworks.