Reinforcement Learning-Based Module Placement

• Reinforcement learning-based module placement is a sequential decision-making framework that casts chip design as a Markov Decision Process (MDP).
• The approach leverages advanced models like GATs, CNNs, and transformers to dynamically encode spatial and graph data for optimized placement decisions.
• It achieves significant improvements including up to 19.6% HPWL reduction and 3–6× speedup, demonstrating practical benefits in VLSI and system design.

Reinforcement learning–based module placement refers to the application of RL techniques to the combinatorial optimization problem of allocating discrete system modules—such as integrated circuit macros, chiplets, computation or communication nodes, or functional blocks—to physical or virtual regions, grids, or devices, subject to multifaceted constraints. This paradigm has emerged as a principal method in modern VLSI/PCB design, device placement for neural computation graphs, edge/fog and cloud application allocation, and general object assembly, aiming for jointly optimized performance, routability, and constraint satisfaction.

1. Markov Decision Process Formulation of Placement

In RL-based module placement, the problem is uniformly cast as a Markov decision process (MDP) $$\langle \mathcal{S}, \mathcal{A}, T, R, \gamma \rangle$$ in which an agent sequentially arrives at placement decisions for modules, guided by a reward signal encoding proxy or true measures of layout quality, resource usage, or physical response.

State and Action Representation

A canonical example is the macro-cell placer "SRLPlacer," where every state $s \in \mathcal{S}$ encodes the full partial placement: global metadata (e.g., netlist stats), an adjacency matrix $H$ reflecting the wiring graph, macro geometries $F$, the next module-to-place index $id$, and a binary placement mask for the current grid availability. Formally,

$s = (M, H, F, id, mask)$

with $mask \in \{0,1\}^{W \times W}$ denoting legal anchor sites on a discretized grid (Yu et al., 2024).

Actions $a \in \mathcal{A}$ typically correspond to the selection of a candidate grid cell for the current macro (or more generally, a tile-timeslot pair, a device assignment, etc.), and only legal actions under current constraints are allowed. In many frameworks, the action space is pruned dynamically to enforce hard constraints such as overlap prevention.

In device- and task-placement RL, the state can comprise computation graph embeddings, task/server feature matrices, and dependency encodings, while the action assigns current modules/tasks to compute devices or storage nodes, still respecting all physical or logical restrictions (Duan et al., 2024, Goudarzi et al., 2021).

2. RL Algorithms and Policy Network Architectures

The dominant suite of RL methods in module placement includes policy-gradient algorithms—Proximal Policy Optimization (PPO), actor-critic, or vanilla REINFORCE—with customized neural policy/value architectures to encode complex combinatorics, inter-module couplings, and high-dimensional spatial/topological context.

• Graph Attention/Convolutional Networks: SRLPlacer demonstrates the use of a multi-layer Graph Attention Network (GAT), processing netlist structure (adjacency matrix $H$, node feature matrix $F$) to yield permutation-invariant, relation-aware macro embeddings. The network selects the embedding for the currently active macro and outputs a $s \in \mathcal{S}$0 softmax over grid positions (Yu et al., 2024).
• Convolutional Neural Networks for Visual Mask Encoding: Visual mask–based approaches such as MaskPlace and ChiPFormer encode critical state information—including incremental wirelength costs, occupancy masks, and legal region maps—via ResNet-style CNNs for large-scale chip grids ($s \in \mathcal{S}$1 pixel input). Actions are sampled from a softmax over the spatial grid after masking illegal positions (Lai et al., 2022, Lai et al., 2023).
• Offline Decision Transformers: ChiPFormer trains a transformer-based policy on offline expert placement trajectories, conditioned on a learned circuit-token (via VGAE), yielding rapid and transferable placement policies that enable few-shot adaptation to unseen circuit topologies (Lai et al., 2023).
• RNNs, GNNs, and Global–Local Views: Multi-view embedding models fuse global grid/canvas information (via CNNs) and local netlist graph data (via GNNs) to capture both the spatial distribution and graph-theoretic constraints, as in DeepPlace/DeepPR (Cheng et al., 2021).

The RL loss generally combines the surrogate policy gradient objective (with policy-ratio clipping for PPO), a value-function loss, and an entropy regularizer to encourage exploration:

$s \in \mathcal{S}$2

with algorithm-specific hyperparameters chosen for stability and convergence (Yu et al., 2024, Lai et al., 2022).

3. Reward Shaping, Constraints, and Optimization Objectives

RL-based module placement requires carefully crafted reward functions to handle extreme sparsity and delayed task feedback.

Sparse and Immediate Rewards

• Wirelength and Congestion: Most module placement RL systems employ half-perimeter wirelength (HPWL) and estimated routing congestion as the main cost terms. For SRLPlacer, the immediate placement reward for each macro is

$s \in \mathcal{S}$3

and the overall trajectory reward after standard cell placement is

$s \in \mathcal{S}$4

with $s \in \mathcal{S}$5 denoting placement density (mean nearest-neighbor distance) (Yu et al., 2024).

• Overlap and Legality: Overlap is commonly enforced as a hard constraint by masking invalid actions or with large negative penalty rewards. SRLPlacer always outputs zero macro overlaps; MaskPlace hard-masks out overlaps via the position mask (Yu et al., 2024, Lai et al., 2022).
• Multi-Objective and Physical Constraints: In domains with coupled multiphysics constraints (e.g., combined thermal, mechanical, and geometric objectives), reward terms can comprise normalized inverses of maximum temperature, stress, and minimum centroid distance, with DNN surrogates employed to efficiently estimate physical field responses (Nair et al., 23 Apr 2025).
• Exploration and Intrinsic Bonus: Approaches suffering from reward sparsity may employ intrinsic bonuses, such as Random Network Distillation, providing a dense signal proportional to the novelty of each partial placement state (Cheng et al., 2021).

4. End-to-End Placement Pipelines and Integration

Most advanced RL-based placement systems use a hybrid pipeline, combining discrete sequential module placement with analytical or domain-specific refinement for subproblems unsuited to direct RL.

1. Sequential Macro Placement by RL: An RL agent sequentially places macros, chiplets, or blocks onto a discretized chip/interposer using the formulated MDP and RL algorithm (Yu et al., 2024, Deng et al., 2024).
2. Standard Cell/Secondary Placement: The output macro layout is handed off to an analytic placer (e.g., DREAMPlace) for standard cell legalization, further refinement, and congestion optimization (Yu et al., 2024, Cheng et al., 2021).
3. Joint/Hierarchical Learning: Some frameworks (e.g., DeepPR) pursue joint reinforcement learning for both placement and routing tasks, employing a multi-stage reward and curriculum (Cheng et al., 2021).
4. Cyclic RL and Metaheuristics: In certain pipelines, RL is used for initial placement followed by classical optimization such as simulated annealing, leveraging RL's fast convergence to “good enough” initial points and annealing's ability for local refinement (Vashisht et al., 2020).

The overall reward for each training episode is determined after both sequential RL-controlled macro placement and analytic standard cell routing, ensuring that the macro placements are guided by the downstream routability and wirelength consequences (Yu et al., 2024, Lai et al., 2023).

5. Applications Across Placement Domains and Benchmarks

RL-based placement has been applied across a spectrum of combinatorial optimization problems:

• VLSI and SoC Macro/Chiplet Placement: Macro placement for standard benchmarks (ISPD2005 adaptec1–4, bigblue1–4) demonstrates HPWL reductions up to 19.6% and 3–6 times speedup versus prior RL and analytic placers, always achieving zero macro overlaps and congestion-aware layouts (Yu et al., 2024, Lai et al., 2022).
• Chiplet Placement-Order Optimization: RL policies in chiplet-based systems are highly order-sensitive; ranking networks trained over rollout performance can select Pareto-efficient sequences to further reduce system wirelength (–10.05%) and improve peak temperature (–1.01%) (Deng et al., 2024).
• Device Placement for Neural Networks: RL guides node-to-device mapping in DAG computation graphs using structural GNN encoders and clustering, yielding up to 58.2% speedup in neural network inference relative to best CPU baselines (Duan et al., 2024, Addanki et al., 2019, Mirhoseini et al., 2017).
• Edge/Fog/Cloud Module Placement: Distributed RL (e.g., IMPALA-based actor-critic learners) for fog/IoT DAG placement reduces end-to-end application execution cost by up to 30% over state-of-the-art DRL and greedy approaches (Goudarzi et al., 2021).
• Mechanical Assembly and Multiphysics Placement: Physics-informed PPO with reward shaped by DNN surrogates of temperature/stress fields enables module placement under complex engineering constraints, outperforming global optimizers (Nair et al., 23 Apr 2025).

Benchmark Performance

Benchmark	SRLPlacer HPWL Reduction	SRLPlacer Speedup	MaskPlace HPWL Reduction	Reference
adaptec1 (ISPD2005)	–19.6% vs. fs50	3× vs. RePlAce	–60–90% vs. GraphPlace	(Yu et al., 2024, Lai et al., 2022)
bigblue4 (ISPD2005)	2–5% vs. DREAMPlace	6× vs. RePlAce

SRLPlacer is consistently among the top performers, and MaskPlace also achieves superior HPWL and density with zero overlaps.

6. Ablation Analyses, Generalization, and Limitations

Ablation studies across multiple works reinforce key architectural and reward components:

• Graph Attention vs. GCN: GAT-based backbones outperform GCNs by 6–7% HPWL reduction (Yu et al., 2024).
• Immediate Rewards: Sparse/delayed-only rewards impede convergence and increase final placement cost by 2–4% relative to schemes with well-designed immediate rewards (Yu et al., 2024, Lai et al., 2022).
• Visual vs. Graph Representations: CNN-based mask encodings in MaskPlace outperform hypergraph-based RL, particularly for representing spatial/timing pin offsets (Lai et al., 2022).

RL-based placement models exhibit strong generalization across circuit scales and domains (with proper architecture selection), and multi-task transformer-based policies (ChiPFormer) permit rapid adaptation to unseen circuits after minimal fine-tuning (Lai et al., 2023).

Key limitations include scaling to very large standard cell placements (current pixel/graph encodings are not feasible for millions of objects), reward shaping and constraint enforcement (especially for new logic/physics domains), and maintaining sample efficiency during training for industrial-scale designs. Integration with existing EDA and optimization tools for hybrid pipelines is required for practical adoption.

7. Outlook and Generalization

The RL-based module placement methodology unifies a spectrum of placement, mapping, and resource allocation problems under a sequential decision-making framework, enabled by advances in permutation-invariant state encoding, action-masking, reward shaping, and actor-critic training.

Notable extensions include:

• Hierarchical RL for multi-scale placement tasks,
• Unsupervised structure-aware embedding for arbitrary graphs and layouts,
• Generalized sequential/iterative improvement for non-grid, resource-constrained domains,
• Integration of physics surrogates for rapid “design-in-the-loop” optimization (Nair et al., 23 Apr 2025).

As new benchmarks and system integration challenges arise—including cloud/data-center workload placement, PCB/industrial assembly, and high-level system partitioning—the RL placement paradigm is expected to serve as a scalable, general-purpose foundation for sequential, constraint-rich combinatorial optimization.

---

Primary references:

• "Non-Overlapping Placement of Macro Cells based on Reinforcement Learning in Chip Design" (Yu et al., 2024)
• "Placement in Integrated Circuits using Cyclic Reinforcement Learning and Simulated Annealing" (Vashisht et al., 2020)
• "A Structure-Aware Framework for Learning Device Placements on Computation Graphs" (Duan et al., 2024)
• "Reinforcement learning framework for the mechanical design of microelectronic components under multiphysics constraints" (Nair et al., 23 Apr 2025)
• "MaskPlace: Fast Chip Placement via Reinforced Visual Representation Learning" (Lai et al., 2022)
• "ChiPFormer: Transferable Chip Placement via Offline Decision Transformer" (Lai et al., 2023)
• "On Joint Learning for Solving Placement and Routing in Chip Design" (Cheng et al., 2021)
• "Chiplet Placement Order Exploration Based on Learning to Rank with Graph Representation" (Deng et al., 2024)
• "Placeto: Learning Generalizable Device Placement Algorithms for Distributed Machine Learning" (Addanki et al., 2019)
• "Device Placement Optimization with Reinforcement Learning" (Mirhoseini et al., 2017)
• "A Distributed Deep Reinforcement Learning Technique for Application Placement in Edge and Fog Computing Environments" (Goudarzi et al., 2021)