---
title: Reinforcement Learning-Based Module Placement
url: https://www.emergentmind.com/topics/reinforcement-learning-based-module-placement
type: topic
---

# Reinforcement Learning-Based Module Placement

Reinforcement learning–based module placement refers to the application of RL techniques to the combinatorial optimization problem of allocating discrete system modules—such as integrated circuit macros, chiplets, computation or communication nodes, or functional blocks—to physical or virtual regions, grids, or devices, subject to multifaceted constraints. This paradigm has emerged as a principal method in modern VLSI/PCB design, device placement for neural computation graphs, edge/fog and cloud application allocation, and general object assembly, aiming for jointly optimized performance, routability, and constraint satisfaction.

## 1. Markov Decision Process Formulation of Placement

In RL-based module placement, the problem is uniformly cast as a Markov decision process (MDP) $\langle \mathcal{S}, \mathcal{A}, T, R, \gamma \rangle$ in which an agent sequentially arrives at placement decisions for modules, guided by a reward signal encoding proxy or true measures of layout quality, resource usage, or physical response.

### State and Action Representation

A canonical example is the macro-cell placer "SRLPlacer," where every state $s \in \mathcal{S}$ encodes the full partial placement: global metadata (e.g., netlist stats), an adjacency matrix $H$ reflecting the wiring graph, macro geometries $F$, the next module-to-place index $id$, and a binary placement mask for the current grid availability. Formally,
$$ s = (M, H, F, id, mask) $$
with $mask \in \{0,1\}^{W \times W}$ denoting legal anchor sites on a discretized grid [2407.18499].

Actions $a \in \mathcal{A}$ typically correspond to the selection of a candidate grid cell for the current macro (or more generally, a tile-timeslot pair, a device assignment, etc.), and only legal actions under current constraints are allowed. In many frameworks, the action space is pruned dynamically to enforce hard constraints such as overlap prevention.

In device- and task-placement RL, the state can comprise computation graph embeddings, task/server feature matrices, and dependency encodings, while the action assigns current modules/tasks to compute devices or storage nodes, still respecting all physical or logical restrictions [2405.14185, 2110.12415].

## 2. RL Algorithms and Policy Network Architectures

The dominant suite of RL methods in module placement includes policy-gradient algorithms—Proximal Policy Optimization (PPO), actor-critic, or vanilla REINFORCE—with customized neural policy/value architectures to encode complex combinatorics, inter-module couplings, and high-dimensional spatial/topological context.

- **Graph Attention/Convolutional Networks**: SRLPlacer demonstrates the use of a multi-layer Graph Attention Network (GAT), processing netlist structure (adjacency matrix $H$, node feature matrix $F$) to yield permutation-invariant, relation-aware macro embeddings. The network selects the embedding for the currently active macro and outputs a $W \times W$ softmax over grid positions [2407.18499].
- **Convolutional Neural Networks for Visual Mask Encoding**: Visual mask–based approaches such as MaskPlace and ChiPFormer encode critical state information—including incremental wirelength costs, occupancy masks, and legal region maps—via ResNet-style CNNs for large-scale chip grids ($N \times N$ pixel input). Actions are sampled from a softmax over the spatial grid after masking illegal positions [2211.13382, 2306.14744].
- **Offline Decision Transformers**: ChiPFormer trains a transformer-based policy on offline expert placement trajectories, conditioned on a learned circuit-token (via VGAE), yielding rapid and transferable placement policies that enable few-shot adaptation to unseen circuit topologies [2306.14744].
- **RNNs, GNNs, and Global–Local Views**: Multi-view embedding models fuse global grid/canvas information (via CNNs) and local netlist graph data (via GNNs) to capture both the spatial distribution and graph-theoretic constraints, as in DeepPlace/DeepPR [2111.00234].

The RL loss generally combines the surrogate policy gradient objective (with policy-ratio clipping for PPO), a value-function loss, and an entropy regularizer to encourage exploration:
$$
L(\theta) = L^{\rm CLIP}(\theta) - c_1 L_v(\theta) + c_2 S[\pi_\theta]
$$
with algorithm-specific hyperparameters chosen for stability and convergence [2407.18499, 2211.13382].

## 3. Reward Shaping, Constraints, and Optimization Objectives

RL-based module placement requires carefully crafted reward functions to handle extreme sparsity and delayed task feedback.

### Sparse and Immediate Rewards

- **Wirelength and Congestion**: Most module placement RL systems employ half-perimeter wirelength (HPWL) and estimated routing congestion as the main cost terms. For SRLPlacer, the immediate placement reward for each macro is
$$
R_i(s,a) = -\alpha_i\, WL(\tilde{P},H) - \beta_i\, D(\tilde{P})
$$
and the overall trajectory reward after standard cell placement is
$$
R_t = -\alpha_t\, WL(P,H) - \beta_t\, C(P,H)
$$
with $D(\tilde{P})$ denoting placement density (mean nearest-neighbor distance) [2407.18499].

- **Overlap and Legality**: Overlap is commonly enforced as a hard constraint by masking invalid actions or with large negative penalty rewards. SRLPlacer always outputs zero macro overlaps; MaskPlace hard-masks out overlaps via the position mask [2407.18499, 2211.13382].

- **Multi-Objective and Physical Constraints**: In domains with coupled multiphysics constraints (e.g., combined thermal, mechanical, and geometric objectives), reward terms can comprise normalized inverses of maximum temperature, stress, and minimum centroid distance, with DNN surrogates employed to efficiently estimate physical field responses [2504.17142].

- **Exploration and Intrinsic Bonus**: Approaches suffering from reward sparsity may employ intrinsic bonuses, such as Random Network Distillation, providing a dense signal proportional to the novelty of each partial placement state [2111.00234].

## 4. End-to-End Placement Pipelines and Integration

Most advanced RL-based placement systems use a hybrid pipeline, combining discrete sequential module placement with analytical or domain-specific refinement for subproblems unsuited to direct RL.

1. **Sequential Macro Placement by RL**: An RL agent sequentially places macros, chiplets, or blocks onto a discretized chip/interposer using the formulated MDP and RL algorithm [2407.18499, 2404.04943].
2. **Standard Cell/Secondary Placement**: The output macro layout is handed off to an analytic placer (e.g., DREAMPlace) for standard cell legalization, further refinement, and congestion optimization [2407.18499, 2111.00234].
3. **Joint/Hierarchical Learning**: Some frameworks (e.g., DeepPR) pursue joint reinforcement learning for both placement and routing tasks, employing a multi-stage reward and curriculum [2111.00234].
4. **Cyclic RL and Metaheuristics**: In certain pipelines, RL is used for initial placement followed by classical optimization such as simulated annealing, leveraging RL's fast convergence to “good enough” initial points and annealing's ability for local refinement [2011.07577].

The overall reward for each training episode is determined after both sequential RL-controlled macro placement and analytic standard cell routing, ensuring that the macro placements are guided by the downstream routability and wirelength consequences [2407.18499, 2306.14744].

## 5. Applications Across Placement Domains and Benchmarks

RL-based placement has been applied across a spectrum of combinatorial optimization problems:

- **VLSI and SoC Macro/Chiplet Placement**: Macro placement for standard benchmarks (ISPD2005 adaptec1–4, bigblue1–4) demonstrates HPWL reductions up to 19.6% and 3–6 times speedup versus prior RL and analytic placers, always achieving zero macro overlaps and congestion-aware layouts [2407.18499, 2211.13382].
- **Chiplet Placement-Order Optimization**: RL policies in chiplet-based systems are highly order-sensitive; ranking networks trained over rollout performance can select Pareto-efficient sequences to further reduce system wirelength (–10.05%) and improve peak temperature (–1.01%) [2404.04943].
- **Device Placement for Neural Networks**: RL guides node-to-device mapping in DAG computation graphs using structural GNN encoders and clustering, yielding up to 58.2% speedup in neural network inference relative to best CPU baselines [2405.14185, 1906.08879, 1706.04972].
- **Edge/Fog/Cloud Module Placement**: Distributed RL (e.g., IMPALA-based actor-critic learners) for fog/IoT DAG placement reduces end-to-end application execution cost by up to 30% over state-of-the-art DRL and greedy approaches [2110.12415].
- **Mechanical Assembly and Multiphysics Placement**: Physics-informed PPO with reward shaped by DNN surrogates of temperature/stress fields enables module placement under complex engineering constraints, outperforming global optimizers [2504.17142].

### Benchmark Performance

| Benchmark           | SRLPlacer HPWL Reduction | SRLPlacer Speedup | MaskPlace HPWL Reduction | Reference |
|---------------------|-------------------------|-------------------|-------------------------|-----------|
| adaptec1 (ISPD2005) | –19.6% vs. fs50         | 3× vs. RePlAce    | –60–90% vs. GraphPlace  | [2407.18499], [2211.13382] |
| bigblue4 (ISPD2005) | 2–5% vs. DREAMPlace     | 6× vs. RePlAce    |                         |           |

SRLPlacer is consistently among the top performers, and MaskPlace also achieves superior HPWL and density with zero overlaps.

## 6. Ablation Analyses, Generalization, and Limitations

Ablation studies across multiple works reinforce key architectural and reward components:

- **Graph Attention vs. GCN**: GAT-based backbones outperform GCNs by 6–7% HPWL reduction [2407.18499].
- **Immediate Rewards**: Sparse/delayed-only rewards impede convergence and increase final placement cost by 2–4% relative to schemes with well-designed immediate rewards [2407.18499, 2211.13382].
- **Visual vs. Graph Representations**: CNN-based mask encodings in MaskPlace outperform hypergraph-based RL, particularly for representing spatial/timing pin offsets [2211.13382].

RL-based placement models exhibit strong generalization across circuit scales and domains (with proper architecture selection), and multi-task transformer-based policies (ChiPFormer) permit rapid adaptation to unseen circuits after minimal fine-tuning [2306.14744].

Key limitations include scaling to very large standard cell placements (current pixel/graph encodings are not feasible for millions of objects), reward shaping and constraint enforcement (especially for new logic/physics domains), and maintaining sample efficiency during training for industrial-scale designs. Integration with existing EDA and optimization tools for hybrid pipelines is required for practical adoption.

## 7. Outlook and Generalization

The RL-based module placement methodology unifies a spectrum of placement, mapping, and resource allocation problems under a sequential decision-making framework, enabled by advances in permutation-invariant state encoding, action-masking, reward shaping, and actor-critic training.

Notable extensions include:
- **Hierarchical RL** for multi-scale placement tasks,
- **Unsupervised structure-aware embedding** for arbitrary graphs and layouts,
- **Generalized sequential/iterative improvement** for non-grid, resource-constrained domains,
- **Integration of physics surrogates for rapid “design-in-the-loop” optimization** [2504.17142].

As new benchmarks and system integration challenges arise—including cloud/data-center workload placement, PCB/industrial assembly, and high-level system partitioning—the RL placement paradigm is expected to serve as a scalable, general-purpose foundation for sequential, constraint-rich combinatorial optimization.

---

**Primary references:**
- "Non-Overlapping Placement of Macro Cells based on Reinforcement Learning in Chip Design" [2407.18499]
- "Placement in Integrated Circuits using Cyclic Reinforcement Learning and Simulated Annealing" [2011.07577]
- "A Structure-Aware Framework for Learning Device Placements on Computation Graphs" [2405.14185]
- "Reinforcement learning framework for the mechanical design of microelectronic components under multiphysics constraints" [2504.17142]
- "MaskPlace: Fast Chip Placement via Reinforced Visual Representation Learning" [2211.13382]
- "ChiPFormer: Transferable Chip Placement via Offline Decision Transformer" [2306.14744]
- "On Joint Learning for Solving Placement and Routing in Chip Design" [2111.00234]
- "Chiplet Placement Order Exploration Based on Learning to Rank with Graph Representation" [2404.04943]
- "Placeto: Learning Generalizable Device Placement Algorithms for Distributed Machine Learning" [1906.08879]
- "Device Placement Optimization with Reinforcement Learning" [1706.04972]
- "A Distributed Deep Reinforcement Learning Technique for Application Placement in Edge and Fog Computing Environments" [2110.12415]

Source: https://www.emergentmind.com/topics/reinforcement-learning-based-module-placement