Constrained Deep Reinforcement Learning (CDRL)

• Constrained Deep Reinforcement Learning (CDRL) is a class of methods that integrates explicit operational constraints into policy optimization to ensure feasibility and safety.
• It employs mechanisms like Lagrangian relaxation, action filtering, lexicographic prioritization, and external constraint guidance to manage resource budgets, safety, and performance requirements.
• CDRL has shown significant empirical gains across domains such as robotics, networking, and radar, outperforming standard reward shaping by achieving high-performance policies under strict constraints.

Constrained Deep Reinforcement Learning (CDRL) denotes a family of deep reinforcement learning methods in which policy optimization is carried out under explicit constraints rather than by reward maximization alone. In the literature represented here, those constraints include resource budgets, safety limits, feasibility conditions, deadline and reliability requirements, physical limits, and behavior specifications, and they are encoded through constrained Markov decision processes (CMDPs), Lagrangian saddle-point objectives, direct feasibility filtering, lexicographic action selection, or external constraint-guidance mechanisms (Bhatia et al., 2018). Across applications as diverse as multi-cloud edge computing, cognitive radar, robotics, wireless networks, fronthaul compression, legged locomotion, bin packing, and nonlinear control, the central technical problem is to optimize return while ensuring that selected actions or long-run trajectories satisfy operational requirements that are not naturally guaranteed by standard deep RL (Sun et al., 20 Jan 2025).

1. Formal problem classes and mathematical foundations

The dominant formalization of CDRL in these works is the CMDP. A typical formulation augments the standard MDP with one or more cost functions and thresholds, yielding objectives of the form

$$\max_{\pi} J_R(\pi)\quad \text{s.t.}\quad J_{C_k}(\pi)\le d_k,$$

or, in equivalent sign conventions, reward maximization subject to lower-bounded service or reliability metrics (Roy et al., 2021). In convex-constrained settings, the policy may be a mixed policy $p\in\Delta(\Pi)$, and feasibility is written as

$$\text{Find } p\in\Delta(\Pi)\quad\text{s.t.}\quad J(p)\in \mathcal C,$$

with $\mathcal C\subseteq \mathbb R^m$ a closed convex set (Cai et al., 2021).

Several papers instantiate this template with application-specific state, action, reward, and cost structures. In cognitive radar, the resource constraint is a normalized dwell-time budget,

$$\sum_{n=1}^N \frac{\tau_t^n}{T_0} \le \Theta_{\max},$$

and the discounted CMDP objective is

$$\max_{\pi} \sum_{m=0}^{\infty}\gamma^m U_{t+m}(\{\tau_{t+m}^n\}_{n=1}^N)$$

subject to a discounted cumulative time-budget constraint (Lu et al., 6 Jul 2025). In multi-UAV IoT networks, the CMDP tuple is written as

$$(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{P}_0, \mathcal{R}, \mathcal{C}, \zeta),$$

with per-UAV battery-depletion costs and long-term energy-sustainability constraints (Khairy et al., 2020). In latency-sensitive networking, the CMDP minimizes discounted cost

$$\widehat m_0(\pi)= \sum_{t=0}^{\infty}\gamma^t \mathbb{E}_s\left[m_0(\mathbf{s}(t),\mathbf{a}(t))\right]$$

subject to timely-throughput constraints $\widehat m^c(\pi)\ge 0$ for each commodity (Aygün et al., 4 Mar 2026).

A separate line of work uses constrained optimization without always centering the cost-threshold CMDP syntax. In secure edge computing, the objective is

$$\min_{\pi} \quad C(\pi)=\alpha_1 T(\pi)+\alpha_2 E(\pi)$$

subject to assignment, deadline, capacity, and binary-allocation constraints, with security overhead inserted directly into delay and energy terms (Sun et al., 20 Jan 2025). In online 3D bin packing, the problem is first written as an MDP and then extended with a cost function

$p\in\Delta(\Pi)$0

to express feasibility of placement actions, though the method ultimately enforces feasibility through masking rather than a standard primal-dual solver (Zhao et al., 2020).

These formulations clarify that CDRL is not a single algorithm. It is a modeling and optimization regime in which deep function approximation is combined with explicit admissibility structure. A recurrent misconception is that CDRL is synonymous with safe RL alone. The surveyed works contradict that view: constraints are used for energy sustainability, resource budgeting, direct behavior specification, semantic communication limits, reliable deadline delivery, and even physical stability in packing, not only for catastrophic safety avoidance (Roy et al., 2021).

2. Constraint-enforcement mechanisms

The surveyed literature uses several distinct enforcement strategies. One major family is Lagrangian relaxation. In cognitive radar, the constrained objective is relaxed as

$p\in\Delta(\Pi)$1

and the dual variable is updated online by projected gradient steps (Lu et al., 6 Jul 2025). In multi-UAV IoT networks, the penalized reward is

$p\in\Delta(\Pi)$2

with three time scales for value, policy, and multiplier updates (Khairy et al., 2020). In fronthaul compression, the dual variables weight a vector of reward and constraint heads, and the greedy policy is selected using $p\in\Delta(\Pi)$3 rather than a monolithic scalar critic (Grönland et al., 2023). In direct behavior specification, the Lagrangian is normalized through a softmax-like multiplier parameterization in order to prevent one multiplier from exploding under hard-to-satisfy constraints (Roy et al., 2021).

A second family enforces constraints through action filtering, masking, or projection. In secure resource allocation, infeasible actions are given a large negative bias through

$p\in\Delta(\Pi)$4

and this term is injected into both action selection and Bellman targets: $p\in\Delta(\Pi)$5 The method explicitly frames this as more than a reward penalty, because infeasible actions are structurally suppressed during policy improvement and bootstrapping (Sun et al., 20 Jan 2025). In online 3D bin packing, a predicted feasibility mask $p\in\Delta(\Pi)$6 modulates actor probabilities, infeasible loading positions are multiplied by a penalty factor of $p\in\Delta(\Pi)$7, and entropy is computed only over feasible actions (Zhao et al., 2020). In resource allocation with DDPG, one line of work compares projection-based feasibility repair, constrained softmax layers that ensure sum and local bounds by construction, and Approximate OptLayer, which enforces feasibility through differentiable clamping and redistribution (Bhatia et al., 2018).

A third family relies on lexicographic prioritization rather than weighted penalties. For chance-constrained control, separate DNNs approximate one primary objective and one value function per constraint. The feasible action set for the first $p\in\Delta(\Pi)$8 constraints is

$p\in\Delta(\Pi)$9

and action selection uses

$$\text{Find } p\in\Delta(\Pi)\quad\text{s.t.}\quad J(p)\in \mathcal C,$$0

This avoids tuning Lagrange multipliers or penalty weights when probability thresholds change (Giuseppi et al., 2020).

A fourth family externalizes constraints into side models. Scenario-based programming for robotics maps actions $$\text{Find } p\in\Delta(\Pi)\quad\text{s.t.}\quad J(p)\in \mathcal C,$$1, tracks an auxiliary scenario-model state $$\text{Find } p\in\Delta(\Pi)\quad\text{s.t.}\quad J(p)\in \mathcal C,$$2, and either penalizes blocked transitions or emits a separate cost

$$\text{Find } p\in\Delta(\Pi)\quad\text{s.t.}\quad J(p)\in \mathcal C,$$3

during training (Corsi et al., 2022). Constraint-guided RL generalizes this wrapper view by modifying observations $$\text{Find } p\in\Delta(\Pi)\quad\text{s.t.}\quad J(p)\in \mathcal C,$$4, repairing actions $$\text{Find } p\in\Delta(\Pi)\quad\text{s.t.}\quad J(p)\in \mathcal C,$$5, or injecting internal action masks $$\text{Find } p\in\Delta(\Pi)\quad\text{s.t.}\quad J(p)\in \mathcal C,$$6 before policy selection (Spieker, 2021).

These mechanisms support a useful conceptual distinction. Some CDRL methods enforce feasibility at the level of optimization geometry; others alter the admissible action set seen by the learner; still others treat constraints as a separate specification layer. The literature does not present a single consensus that one mechanism strictly dominates the others. Rather, each is tied to the structure of the action space, the type of constraint, and the acceptable tradeoff between exactness, scalability, and training stability.

3. Deep RL architectures and optimization patterns

The algorithmic substrate of CDRL in these papers is heterogeneous. Continuous-control problems frequently use actor-critic methods. Cognitive radar uses DDPG because the action is a continuous vector of dwell times $$\text{Find } p\in\Delta(\Pi)\quad\text{s.t.}\quad J(p)\in \mathcal C,$$7 (Lu et al., 6 Jul 2025). Doubly-dynamic ISAC precoding adopts a primal-dual DDPG architecture with separate reward and cost critics $$\text{Find } p\in\Delta(\Pi)\quad\text{s.t.}\quad J(p)\in \mathcal C,$$8 and $$\text{Find } p\in\Delta(\Pi)\quad\text{s.t.}\quad J(p)\in \mathcal C,$$9, and updates the actor by ascent on

$\mathcal C\subseteq \mathbb R^m$0

while the dual variable is updated from the estimated cost budget violation (Yang et al., 2024). In multi-UAV IoT networks, constrained PPO-Clip is used instead, with Gaussian policy, GAE, and a primal-dual loop over per-UAV energy constraints (Khairy et al., 2020). Latency-sensitive network control uses MADDPG with one centralized routing actor, multiple distributed scheduling actors, and a global critic (Aygün et al., 4 Mar 2026).

Value-based CDRL also appears repeatedly. Secure resource allocation uses an Action-Constrained DQN with experience replay, target network, adaptive $\mathcal C\subseteq \mathbb R^m$1-greedy exploration, prioritized experience replay, and dueling decomposition

$\mathcal C\subseteq \mathbb R^m$2

(Sun et al., 20 Jan 2025). Chance-constrained control uses a lexicographic Double DQN with one DNN per objective or constraint (Giuseppi et al., 2020). The fronthaul-compression work develops a DQN extension with $\mathcal C\subseteq \mathbb R^m$3 Q-networks and corresponding target networks, together with a separate-head Bellman operator (Grönland et al., 2023).

Some works expose a tension between value-based and policy-based CDRL. A policy-efficient reduction for convex constrained RL argues that value-based methods are rarely explored in constrained RL because they do not naturally randomize among multiple actions, then resolves this by reducing feasibility to distance minimization in measurement space and using any off-the-shelf RL algorithm as an oracle for

$\mathcal C\subseteq \mathbb R^m$4

where $\mathcal C\subseteq \mathbb R^m$5 (Cai et al., 2021). The same paper proposes a modified minimum norm point method that stores at most $\mathcal C\subseteq \mathbb R^m$6 policies while matching the $\mathcal C\subseteq \mathbb R^m$7 convergence rate of prior game-theoretic approaches (Cai et al., 2021).

The architecture itself can also be the locus of constraint. One paper uses “CDRL” in a broader sense to refer not to CMDP-style constraints but to structurally constrained representation learning: a DDQN backbone is replaced by a cerebellar-inspired module with large expansion, sparse connectivity, sparse activation, and fixed dendritic modulation. The resulting model uses about $\mathcal C\subseteq \mathbb R^m$8M parameters versus about $\mathcal C\subseteq \mathbb R^m$9M for the baseline (Zhang et al., 17 Feb 2026). This does not fit the standard safety- or cost-constrained interpretation; the paper explicitly states that its CDRL is “architecturally constrained / parameter-efficient / structurally biased” rather than a classical constrained optimization method (Zhang et al., 17 Feb 2026). This usage shows that the acronym is not semantically uniform across the literature.

4. Application domains and representative formulations

The breadth of CDRL applications is a defining feature of the field. The following table organizes representative problem classes and mechanisms already present in the literature.

Domain	Representative papers	Constraint mechanism
Resource allocation and networking	(Bhatia et al., 2018, Sun et al., 20 Jan 2025, Aygün et al., 4 Mar 2026, Grönland et al., 2023)	Projection, action-constraint terms, primal-dual CMDP
Radar and sensing	(Lu et al., 6 Jul 2025, Lu et al., 4 Jun 2026, Yang et al., 2024)	DDPG with dual update, time-budget or QoS constraints
Robotics and control	(Corsi et al., 2022, Roy et al., 2021, Lee et al., 2023, Wang et al., 2019)	Scenario costs, Lagrangian behavior constraints, CMDP physical limits
Combinatorial and feasibility-dominated tasks	(Zhao et al., 2020, Giuseppi et al., 2020)	Feasibility masking, lexicographic constraint selection

In secure serverless multi-cloud edge computing, the constrained decision includes task offloading destination, resource allocation, and security protection level. Security overhead is explicitly modeled through cryptographic operations

$$\sum_{n=1}^N \frac{\tau_t^n}{T_0} \le \Theta_{\max},$$0

with additional delay and energy terms $$\sum_{n=1}^N \frac{\tau_t^n}{T_0} \le \Theta_{\max},$$1 and $$\sum_{n=1}^N \frac{\tau_t^n}{T_0} \le \Theta_{\max},$$2 included in the optimization cost (Sun et al., 20 Jan 2025). This is notable because the constraint structure is not purely about physical capacity; it folds cryptographic overhead into resource-allocation decisions.

In radar resource management, the key tradeoff is between tracking existing targets and scanning for new ones under a revisit-interval budget $$\sum_{n=1}^N \frac{\tau_t^n}{T_0} \le \Theta_{\max},$$3. The tracking side uses EKF state estimation; scanning uses a UCA model and detection logic with GNN association and multi-detection confirmation (Lu et al., 6 Jul 2025). A later radar paper extends this formulation with weighted target priorities, explicit missed-target terms, DQL and DDPG variants, and scenario studies involving beam misalignment and clutter regions (Lu et al., 4 Jun 2026).

Networking applications reveal multiple notions of constraint. In multi-UAV IoT networks, the reward is normalized network capacity and the costs are battery-depletion terms $$\sum_{n=1}^N \frac{\tau_t^n}{T_0} \le \Theta_{\max},$$4, producing cooperative altitude-control behavior under energy sustainability constraints (Khairy et al., 2020). In latency-sensitive routing and scheduling, the state is a TTL-indexed queue system $$\sum_{n=1}^N \frac{\tau_t^n}{T_0} \le \Theta_{\max},$$5, and the core constraint is that timely throughput must exceed $$\sum_{n=1}^N \frac{\tau_t^n}{T_0} \le \Theta_{\max},$$6 for each commodity (Aygün et al., 4 Mar 2026). In fronthaul compression, the control action incrementally adjusts modulation order, weight quantization, and sub-band granularity to maximize fronthaul utilization while keeping latency and packet loss below specified thresholds (Grönland et al., 2023).

Robotics papers show that CDRL is not limited to hard safety budgets. Scenario-based programming encodes undesirable action patterns such as left-right-left oscillations or failure to move forward when the goal is ahead and the path is clear (Corsi et al., 2022). Direct behavior specification treats indicator costs as event-frequency constraints, so a threshold can be interpreted as the allowed rate of a behavior such as looking at a marker, not being on ground, not entering lava, or staying above minimum energy (Roy et al., 2021). In legged locomotion, physical constraints include command smoothness, joint speed, joint torque, joint position limits, and undesirable body contact, all separated from the task reward in a CMDP (Lee et al., 2023). In nonlinear control of the Duffing oscillator, the constraint is a bound on actuation magnitude $$\sum_{n=1}^N \frac{\tau_t^n}{T_0} \le \Theta_{\max},$$7, and the policy learns to push the state into the basin of attraction of the desired attractor rather than force it directly onto the target orbit (Wang et al., 2019).

The bin-packing case adds another axis: feasibility can be geometrical and physical rather than temporal or energetic. Feasible placement requires collision avoidance and one of three support conditions, including over $$\sum_{n=1}^N \frac{\tau_t^n}{T_0} \le \Theta_{\max},$$8 of bottom area plus all four corners, over $$\sum_{n=1}^N \frac{\tau_t^n}{T_0} \le \Theta_{\max},$$9 plus three corners, or over $$\max_{\pi} \sum_{m=0}^{\infty}\gamma^m U_{t+m}(\{\tau_{t+m}^n\}_{n=1}^N)$$0 support (Zhao et al., 2020). This makes CDRL applicable to tasks in which the main challenge is legal action generation in a structured combinatorial space.

5. Empirical findings and comparative patterns

Across the surveyed works, several recurrent empirical patterns appear. First, explicit constraint handling often outperforms reward shaping alone. In secure resource allocation, the action-constrained DQN reduces system cost at $$\max_{\pi} \sum_{m=0}^{\infty}\gamma^m U_{t+m}(\{\tau_{t+m}^n\}_{n=1}^N)$$1 tasks by $$\max_{\pi} \sum_{m=0}^{\infty}\gamma^m U_{t+m}(\{\tau_{t+m}^n\}_{n=1}^N)$$2 versus DQN, $$\max_{\pi} \sum_{m=0}^{\infty}\gamma^m U_{t+m}(\{\tau_{t+m}^n\}_{n=1}^N)$$3 versus DQN+NN, $$\max_{\pi} \sum_{m=0}^{\infty}\gamma^m U_{t+m}(\{\tau_{t+m}^n\}_{n=1}^N)$$4 versus LC, $$\max_{\pi} \sum_{m=0}^{\infty}\gamma^m U_{t+m}(\{\tau_{t+m}^n\}_{n=1}^N)$$5 versus CO, and $$\max_{\pi} \sum_{m=0}^{\infty}\gamma^m U_{t+m}(\{\tau_{t+m}^n\}_{n=1}^N)$$6 versus RC; at $$\max_{\pi} \sum_{m=0}^{\infty}\gamma^m U_{t+m}(\{\tau_{t+m}^n\}_{n=1}^N)$$7 GB data size, energy consumption is reduced by $$\max_{\pi} \sum_{m=0}^{\infty}\gamma^m U_{t+m}(\{\tau_{t+m}^n\}_{n=1}^N)$$8 versus DQN and $$\max_{\pi} \sum_{m=0}^{\infty}\gamma^m U_{t+m}(\{\tau_{t+m}^n\}_{n=1}^N)$$9 versus CO (Sun et al., 20 Jan 2025). The paper attributes this to feasibility being enforced inside action selection and Bellman targets rather than only in the reward (Sun et al., 20 Jan 2025).

Second, primal-dual methods repeatedly achieve feasible high-performance policies where unconstrained or fixed-penalty baselines do not. In multi-UAV IoT networks, the proposed algorithm achieves a temporal average capacity $$(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{P}_0, \mathcal{R}, \mathcal{C}, \zeta),$$0 higher than a feasible reward-shaped DRL solution and only $$(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{P}_0, \mathcal{R}, \mathcal{C}, \zeta),$$1 lower than the energy-constraint-free system (Khairy et al., 2020). In latency-sensitive networking, the CDRL-based solution satisfies reliability constraints more consistently than Backpressure and Universal Max-Weight as load increases, while also achieving lower resource allocation cost per episode (Aygün et al., 4 Mar 2026). In fronthaul compression, DQN and SAC improve average fronthaul utilization by $$(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{P}_0, \mathcal{R}, \mathcal{C}, \zeta),$$2 on average compared to a conservative reference scheme while keeping average latency below $$(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{P}_0, \mathcal{R}, \mathcal{C}, \zeta),$$3 (Grönland et al., 2023).

Third, explicit physical constraints improve real-world transfer in robotics. In legged locomotion, unconstrained PPO achieves reward $$(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{P}_0, \mathcal{R}, \mathcal{C}, \zeta),$$4 but has $$(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{P}_0, \mathcal{R}, \mathcal{C}, \zeta),$$5 violations per episode, whereas N-P3O attains reward $$(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{P}_0, \mathcal{R}, \mathcal{C}, \zeta),$$6 with $$(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{P}_0, \mathcal{R}, \mathcal{C}, \zeta),$$7 violations (Lee et al., 2023). The same study reports successful sim-to-real transfer on a wheeled-legged robot under joint-speed and torque limits (Lee et al., 2023). Scenario-based programming for mapless navigation similarly reports that constrained training dramatically improves safety and performance while preserving the baseline’s high success rate around $$(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{P}_0, \mathcal{R}, \mathcal{C}, \zeta),$$8 (Corsi et al., 2022).

Fourth, feasibility-aware action pruning is especially powerful in structured combinatorial problems. In online 3D bin packing, the constrained actor-critic with feasibility prediction and projection outperforms online and offline baselines on three benchmarks, with reported space utilizations of $$(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{P}_0, \mathcal{R}, \mathcal{C}, \zeta),$$9 on RS, $$\widehat m_0(\pi)= \sum_{t=0}^{\infty}\gamma^t \mathbb{E}_s\left[m_0(\mathbf{s}(t),\mathbf{a}(t))\right]$$0 on CUT-1, and $$\widehat m_0(\pi)= \sum_{t=0}^{\infty}\gamma^t \mathbb{E}_s\left[m_0(\mathbf{s}(t),\mathbf{a}(t))\right]$$1 on CUT-2 (Zhao et al., 2020). The same paper reports AI average space utilization $$\widehat m_0(\pi)= \sum_{t=0}^{\infty}\gamma^t \mathbb{E}_s\left[m_0(\mathbf{s}(t),\mathbf{a}(t))\right]$$2 versus $$\widehat m_0(\pi)= \sum_{t=0}^{\infty}\gamma^t \mathbb{E}_s\left[m_0(\mathbf{s}(t),\mathbf{a}(t))\right]$$3 for human participants in a user study (Zhao et al., 2020). This suggests that when infeasibility is a dominant source of sample inefficiency, explicit masks may be more effective than retrospective penalties.

Fifth, fixed heuristics are repeatedly beaten by adaptive constrained policies. In cognitive radar, fixed time-allocation baselines are always worse than CDRL in tracking performance, while CDRL consistently achieves higher utility (Lu et al., 6 Jul 2025). In the more detailed radar study, DDPG-based CDRL attains about $$\widehat m_0(\pi)= \sum_{t=0}^{\infty}\gamma^t \mathbb{E}_s\left[m_0(\mathbf{s}(t),\mathbf{a}(t))\right]$$4 s average track initiation time versus about $$\widehat m_0(\pi)= \sum_{t=0}^{\infty}\gamma^t \mathbb{E}_s\left[m_0(\mathbf{s}(t),\mathbf{a}(t))\right]$$5 s for Equal Allocation and about $$\widehat m_0(\pi)= \sum_{t=0}^{\infty}\gamma^t \mathbb{E}_s\left[m_0(\mathbf{s}(t),\mathbf{a}(t))\right]$$6 s for Distance-based Allocation (Lu et al., 4 Jun 2026). In wireless collaborative DRL, semantic-aware HFDRL yields up to $$\widehat m_0(\pi)= \sum_{t=0}^{\infty}\gamma^t \mathbb{E}_s\left[m_0(\mathbf{s}(t),\mathbf{a}(t))\right]$$7 higher final return than baselines and up to $$\widehat m_0(\pi)= \sum_{t=0}^{\infty}\gamma^t \mathbb{E}_s\left[m_0(\mathbf{s}(t),\mathbf{a}(t))\right]$$8 gain over uniform RB allocation (Lotfi et al., 2021).

A plausible implication is that the value of CDRL is highest when the admissible set is both narrow and state-dependent. That interpretation is consistent with the strongest gains appearing in domains with dynamic resource budgets, strict deadline or physical feasibility requirements, or large structured action spaces.

6. Debates, limitations, and scope of the term

Several objective controversies or ambiguities emerge from the literature. The first concerns terminology. Some papers use CDRL in the standard CMDP sense of reward optimization under cost or feasibility constraints (Khairy et al., 2020), whereas another explicitly uses “constrained” to mean architectural sparsity and parameter-efficient inductive bias rather than explicit safety or resource constraints (Zhang et al., 17 Feb 2026). This suggests that “CDRL” is not semantically stable unless the paper specifies what is being constrained: policy behavior, expected costs, admissible actions, or model structure.

The second concerns penalties versus direct feasibility enforcement. Multiple papers argue that pure reward shaping is inadequate. Secure resource allocation states that penalty-only methods can still allow the agent to explore and reinforce infeasible actions during learning (Sun et al., 20 Jan 2025). Online 3D bin packing likewise emphasizes that negative reward shaping is weaker than feasibility masking and still allows invalid placements at test time (Zhao et al., 2020). Yet penalty and primal-dual methods remain central in continuous control and networked settings because direct masking is often unavailable or intractable. The literature therefore presents no universal replacement of one paradigm by the other.

The third concerns scalability and memory. Convex constrained RL via mixed policies may require storing many policies in game-theoretic approaches, motivating the reduction method with worst-case optimal policy efficiency $$\widehat m_0(\pi)= \sum_{t=0}^{\infty}\gamma^t \mathbb{E}_s\left[m_0(\mathbf{s}(t),\mathbf{a}(t))\right]$$9 (Cai et al., 2021). Joint discrete action spaces also scale poorly: centralized DQL in radar faces action-space growth $\widehat m^c(\pi)\ge 0$0, and fronthaul compression notes exponential growth in joint action space with the number of cells, leading to autoregressive SAC factorization (Lu et al., 4 Jun 2026). Resource-constrained allocation with DDPG similarly develops approximate differentiable projection layers because exact OptLayer is too slow at scale (Bhatia et al., 2018).

The fourth concerns training stability. Dual variables can oscillate or destabilize learning. Fronthaul compression reports that SAC multipliers oscillate more due to target-network delay (Grönland et al., 2023). Direct behavior specification introduces multiplier normalization precisely because unnormalized multipliers can diverge and collapse learning (Roy et al., 2021). Legged locomotion finds that normalized P3O is more stable than its unnormalized form, while FOCOPS is less robust in that experimental setting (Lee et al., 2023).

The fifth concerns dependence on auxiliary models. Some CDRL methods remain partly model-dependent. Attractor selection learns the policy in a model-free way, but reward computation depends on a basin-of-attraction classifier trained from long-horizon simulations on $\widehat m^c(\pi)\ge 0$1 labeled grid samples (Wang et al., 2019). Scenario-based robotics depends on expert-authored rule sets (Corsi et al., 2022). Constraint-guided RL assumes the correctness of the constraint model and notes runtime overhead from solving or applying it at each step (Spieker, 2021).

Taken together, these limitations indicate that CDRL is best understood as a design space rather than a finished recipe. Its common principle is explicit admissibility-aware learning. Its unresolved engineering questions concern how to specify constraints, how to enforce them without sacrificing sample efficiency or stability, and how to scale them in high-dimensional or multi-agent systems.

7. Relationship to adjacent areas and future directions

CDRL overlaps with safe RL, resource-aware RL, constrained control, and behavior specification, but the surveyed papers show that it is broader than any one of these categories. Safe RL is one major use case, especially in robotics and physical systems (Lee et al., 2023). However, direct behavior specification for non-player characters, semantic-aware collaborative training over wireless links, and secure cryptographic task offloading all use constraints to express requirements that are not reducible to immediate hazard avoidance (Roy et al., 2021).

The literature also shows multiple bridges to neighboring optimization traditions. Lexicographic DeepRL connects constrained RL to chance-constrained control and priority-based decision rules (Giuseppi et al., 2020). The policy-efficient reduction approach connects constrained RL to conditional gradient and minimum norm point methods in convex geometry (Cai et al., 2021). Approximate OptLayer and constrained projection methods connect actor-critic RL to differentiable optimization layers (Bhatia et al., 2018). Semantic-aware collaborative DRL couples policy learning with OFDMA resource allocation and knowledge-graph style similarity selection (Lotfi et al., 2021).

Several recurring research directions are already explicit in these works. One is stronger feasibility handling in continuous spaces without large action quantization overhead, motivating DDPG-based primal-dual methods, Wolpertinger architectures, and approximate differentiable projections (Yang et al., 2024). Another is more interpretable constrained learning, motivating multi-head value decomposition and explicit multiplier tracking in fronthaul compression (Grönland et al., 2023). A third is richer specification languages, as seen in scenario-based programming and indicator-cost behavior specification (Corsi et al., 2022). A fourth is improved policy efficiency and memory efficiency for mixed or randomized policies in convex constrained RL (Cai et al., 2021).

A plausible implication is that future CDRL systems will increasingly combine these strands: structured specification of admissible behavior, online dual adaptation, feasibility-aware action generation, and domain-specific inductive biases. The surveyed corpus already points toward that synthesis. It treats constraints not as an afterthought attached to reward engineering, but as a primary object of model design, optimization, and evaluation.