---
title: Action-Constrained Imitation Learning (ACIL)
url: https://www.emergentmind.com/topics/action-constrained-imitation-learning-acil
type: topic
---

# Action-Constrained Imitation Learning (ACIL)

to=arxiv_search.query ￣奇米json
{"query":"Action-Constrained Imitation Learning arXiv 2508.14379 constraint-aware imitation learning autonomous racing 2503.07737 SCOPIL 2507.06780", "max_results": 10}
to=arxiv_search.query ോഗjson
{"query":"ti:\"Action-Constrained Imitation Learning\" OR ti:\"A Simple Approach to Constraint-Aware Imitation Learning with Application to Autonomous Racing\" OR ti:\"Learning safe, constrained policies via imitation learning: Connection to Probabilistic Inference and a Naive Algorithm\"", "max_results": 10}
Action-Constrained Imitation Learning (ACIL) denotes an imitation-learning setting in which the imitator must learn from demonstrations generated by an expert with a larger effective action capability, while the imitator itself is restricted to a feasible action set. In the formalization introduced in 2025, the central difficulty is not merely that some expert actions are infeasible, but that infeasible actions induce a mismatch in future state visitation and therefore a mismatch in occupancy measure. ACIL accordingly shifts the learning target away from exact state-action reproduction and toward state trajectories that remain as similar as possible to the expert’s under the imitator’s action constraints [2508.14379].

## 1. Formal definition and problem structure

The strict ACIL formulation models the environment as an action-constrained MDP,
$$
\mathcal{M} := (\mathcal{S}, \mathcal{A}, P, \mu, \gamma, \mathcal{C}),
$$
where $\mathcal{C}(s)\subseteq \mathcal{A}$ is the feasible action set at state $s$, and the imitator must satisfy $\pi\in \Pi_{\mathcal C}$. The expert policy $\pi_e$ acts in the ambient action space $\mathcal A$ and may select actions outside $\mathcal C(s)$ for some states. The resulting objective is not exact state-action imitation but minimization of discrepancy between the learner’s and the expert’s state sequences:
$$
\pi^*:=\underset{\pi\in \Pi_{\mathcal{C}}}{\arg\min}\; \mathbb{E}_{\tau_e\sim \pi_e,\;\tau\sim \pi} \big[d(\sigma(\tau), \sigma(\tau_e)) \mid s_0\sim \mu\big].
$$
Here $\sigma(\tau)$ denotes the state sequence of trajectory $\tau$, and $d(\cdot,\cdot)$ is allowed to compare sequences of different lengths [2508.14379].

This formulation makes the action-capability gap explicit. The expert and imitator share the same state and action semantics, but the imitator is the weaker agent because it must remain in $\mathcal C(s)$. A constrained learner may therefore fail to reproduce not only expert actions but also the expert’s timing and downstream state distribution. In this sense, ACIL differs from ordinary learning from demonstrations, where realizability of the demonstrator’s behavior is usually assumed.

Related 2025 work broadens the practical scope of the topic without adopting exactly the same formal definition. One line treats constraint-aware imitation as behavioral cloning augmented with a learned safety penalty based on predicted next-state feasibility [2503.07737]. Another line treats constraints as implicit in demonstrations from a constraint-abiding expert and learns a maximum-entropy policy with a KL-based imitation term and dual gradient descent [2507.06780]. This suggests a useful distinction between ACIL in the narrow sense of explicit feasible action sets and ACIL in a broader sense of imitation under feasibility or safety restrictions.

## 2. Occupancy measure distortion and why standard imitation learning fails

The defining conceptual claim of ACIL is that action constraints create an unavoidable occupancy-measure mismatch. If
$$
\operatorname{supp}(\pi_e(\cdot\mid s)) \not\subseteq \mathcal{C}(s),
$$
then any feasible learner must satisfy $\rho_\pi(s,a)=0$ for $a\notin \mathcal C(s)$, while the expert may have $\rho_{\pi_e}(s,a)>0$ on those same state-action pairs. Exact occupancy matching is therefore infeasible in general. The 2025 ACIL paper terms the resulting phenomenon **occupancy measure distortion** [2508.14379].

A common constrained-RL baseline is to compute an unconstrained action $\tilde a$ and project it to feasibility with $\Gamma_{\mathcal C(s)}(\tilde a)$. ACIL argues that such projection does not resolve the imitation problem, because projected actions alter the next state, and the induced state deviation compounds over time. The mismatch is therefore trajectory-level rather than timestep-local. The Maze2D example described in the paper makes this concrete: a learner that takes the “closest” feasible turn still turns too late, collides, and departs from the intended route, whereas a temporally warped but feasible path can remain behaviorally similar [2508.14379].

This diagnosis explains why several standard imitation-learning families become ill-posed under action constraints. Behavior cloning regresses toward infeasible expert actions and suffers amplified covariate shift once executed actions are projected or clipped. Adversarial imitation methods such as GAIL attempt to match an unreachable expert occupancy measure. Inverse RL can recover a reward, but if the learner’s feasible set differs from the expert’s, the same reward need not produce the same occupancy measure. State-only methods avoid direct action mismatch, yet they still try to imitate trajectories generated by unconstrained actions and therefore do not eliminate the capability-gap problem. The common failure mode is not merely lack of safety, but mismatch between the target behavior class and the learner’s feasible behavior class.

## 3. DTWIL and surrogate action-constrained demonstrations

The principal algorithm proposed for strict ACIL is **DTWIL (Dynamic Time Warping Imitation Learning)**. Its strategy is to replace infeasible expert demonstrations with **surrogate, action-constrained demonstrations** before running a standard imitation learner. The method decomposes the problem into two stages: first, generate a surrogate dataset $\mathcal D_{\text{sur}}$ consisting of feasible trajectories that start from the same initial states and remain close to the expert’s state trajectories; second, train any off-the-shelf IL method on those feasible surrogates [2508.14379].

For an expert trajectory $\tau_e$, surrogate construction is posed as
$$
\tau_{\text{sur}} = \arg\min_{\tau\in \mathbb{T}_{\mathcal C}} d(\sigma(\tau),\sigma(\tau_e)),
$$
where $\mathbb T_{\mathcal C}$ is the set of feasible trajectories. The discrepancy is instantiated with **Dynamic Time Warping (DTW)** rather than pointwise $\ell_2$ matching. DTW is critical because constrained learners often require more steps than the expert to realize a comparable path. The DTW distance is defined through a monotone alignment path over the pairwise distance matrix and is computed by the standard dynamic-programming recursion [2508.14379].

Direct optimization over variable-length feasible trajectories is handled through **Model Predictive Control (MPC)**. At each surrogate timestep, MPC plans over horizon $H$, rolls out candidate feasible action sequences under a learned forward model, and scores them by DTW against the current segment of the expert trajectory. Only the first action is executed, after which planning is repeated. Candidate sequences are optimized with the **Cross-Entropy Method (CEM)**, and feasibility is enforced during sampling through rejection sampling. A learned ensemble of probabilistic neural dynamics models, with ensemble size $5$, supplies the predictive model used during planning [2508.14379].

Two implementation mechanisms are especially central. The first is the **progression parameter** $t_{\text{pg}}$, which tracks how far the surrogate has progressed along the expert trajectory. It advances asynchronously according to the DTW warping path rather than synchronizing one learner step to one expert step. The second is **Expert Regularized Control (ERC)**, used in delicate environments such as Hopper, where early planned actions are blended with projected expert actions. The paper reports that replanning every step is better than executing several open-loop MPC steps, and that asynchronous progression is materially better than synchronous progression on Hopper [2508.14379].

The downstream significance of DTWIL is architectural rather than merely procedural. It decouples constraint satisfaction from policy fitting. Once the surrogate trajectories are feasible, the imitation learner no longer targets an impossible expert occupancy measure but a realizable surrogate occupancy that preserves the expert’s state-space behavior up to temporal warping.

## 4. Constraint-aware and safety-aware extensions

Two closely related 2025 methods address similar concerns from different angles. **Constraint-aware Behavior Cloning** augments behavioral cloning with a learned safety penalty derived from a differentiable approximation of a one-step predictive safety filter. The setup considers a nonlinear deterministic discrete-time system with partial observation,
$$
x_{k+1} = f(x_k, u_k), \qquad y_k = h(x_k) + n_k,
$$
subject to $x_k\in\mathcal X$ and $u_k\in\mathcal U$. The ideal imitation objective explicitly requires
$$
f(x,\pi_\theta(y)) \in \mathcal{R}_\infty^B(\mathcal X_f),
$$
where $\mathcal{R}_\infty^B(\mathcal X_f)$ denotes the backward reachable tube of states from which admissible control can reach the target set while satisfying constraints. Because the dynamics and safe set are assumed unavailable, the method learns a surrogate dynamics model $\hat f_{\phi_f}$ and a safety classifier $\hat p_{\phi_p}(x)\approx p(x\in \mathcal{R}_\infty^B(\mathcal X_f))$, leading to the final objective
$$
\mathcal{L}_{\text{clone}} + \mathcal{L}_{\text{safety}}
=
\Vert \pi_\theta(y)-\pi_\beta(x)\Vert_2^2
-\lambda \log p\!\left(f(x,\pi_\theta(y))\in \mathcal{R}_\infty^B(\mathcal X_f)\right).
$$
The method uses DAgger-style data aggregation, a safety auto-labeling heuristic based on successful and failed rollouts, local convex-hull filtering for ambiguous failed states, and updates the dynamics model every $k_f=5$ iterations and the safety classifier every $k_p=10$ iterations. Crucially, “During test time, only the policy network $\pi_\theta$ is used, while all other modules participate solely in training” [2503.07737].

**SCOPIL** addresses constrained behavior through a different formal route. It assumes a discounted MDP and a dataset of demonstrations from a constraint-abiding expert, but does not assume explicit access to the constraint costs. Instead, it formulates imitation as maximum-entropy RL with a KL-based adherence term and dual optimization. The main objective is
$$
\max_{\pi} J(\pi)+\mathcal H(\pi)
\quad \text{s.t.} \quad Dist_\rho(\pi_E,\pi)\le \delta,
$$
and its practical Lagrangian, implemented atop SAC, combines reward, entropy, and a demonstration-based term derived from $D_{KL}(\pi_E\|\pi)$. The method is model-free, off-policy, and hybrid offline–online: it uses a fixed demonstration set $\mathcal D$ alongside environment interaction through a replay buffer $\mathcal B$. It does not project actions, does not construct an explicit safe action set, and does not guarantee per-step feasibility; its constraint signal is transferred from demonstration occupancy patterns rather than hard action masks [2507.06780].

| Formulation | Core mechanism | Test-time handling |
|---|---|---|
| DTWIL | Feasible surrogate demonstrations via MPC + DTW | Standard IL on surrogate data |
| Constraint-aware Behavior Cloning | BC plus learned one-step safety penalty | Only policy network used |
| SCOPIL | Max-ent RL plus KL imitation and dual descent | No mask or projection layer |

This comparison clarifies an important terminological issue. In the strict sense, ACIL refers to the explicit action-constrained setting formalized through $\mathcal C(s)$ and addressed by DTWIL. In a broader and practically common sense, the term also overlaps with safety-aware or constraint-aware imitation methods that shape policies through predicted-state feasibility or demonstration-induced constraint information rather than online action projection.

## 5. Empirical findings

The strict ACIL paper evaluates DTWIL on four continuous-control domains: Maze2d-Medium-v1, HalfCheetah-v3, Hopper-v2, and Robosuite Table-Wiping. The comparisons include both learning-from-demonstration and learning-from-observation baselines: GAIL, BCO, GAIfO, OPOLO, CFIL-s, CFIL-sa, SAIL, and DIFO. All online methods are limited to **50K environment steps** during training. Across nearly all tasks, DTWIL is reported to perform best or among the best in return while also achieving the smallest DTW distance to expert trajectories. Representative results include Maze2D M+B return $0.77 \pm 0.04$ with DTW distance $4.0 \pm 0.1$; HalfCheetah HC+B return $2669 \pm 4$, compared with $1222 \pm 260$ for SAIL, and DTW distance $7.4$ versus $26.5$ for SAIL; Hopper H+B return $2844 \pm 57$; and Table-Wiping W+L2 return $70 \pm 4$. Ablations show that DTW-based alignment outperforms $\ell_2$ alignment on HC+B ($2669.4$ versus $2157.5$) and H+B ($2844.7$ versus $1054.5$), that asynchronous progression dominates synchronous progression on Hopper ($2844.68\pm57.77$ versus $153.52\pm1.20$), that ERC markedly improves Hopper performance (Return-BC $2844.7$ with ERC versus $889.7$ without), and that replanning every step is superior to executing 3 or 5 open-loop steps [2508.14379].

Constraint-aware Behavior Cloning is validated in CARLA-based autonomous racing with customized vehicle dynamics under three settings: image-feedback path following at conservative speed, full-state autonomous racing, and image-feedback autonomous racing. The reported metrics include successful consecutive laps completed without constraint violation, training epochs required to reach sustained success, lap time, lap-time variability, and qualitative distance from walls. In the most difficult image-feedback racing setting, the learned policy completed **50 consecutive laps within 80 epochs**, while the baseline **failed to exceed 10 laps**. Reported early-stopped rollout statistics were average lap time $= 5.665$ s, max lap time $= 5.8$ s, min lap time $= 5.5$ s, and expert average lap time $= 4.805$ s. In full-state racing, the method achieved higher return with less training effort, similar average lap time to the baseline and expert, and lower lap-time variability, which the paper interprets as greater consistency [2503.07737].

SCOPIL is evaluated in the Marble Maze environment with 9 discrete board-rotation actions under three settings: Simple, Multiple-constraints, and Two-modes. In the Simple setting, SCOPIL reports horizontal violations $0.045$, cyclic violations $0.042$, and total violations $0.087$, versus total violations $1.54$ for ICRL and $1.99$ for SAC. In the Two-modes setting, it reports horizontal violations $0.0$, cyclic violations $0.02$, and total violations $0.025$. The paper emphasizes that safety is not encoded in the reward, which remains task-related, and attributes its multimodal robustness to the demonstration-based constraint signal and the use of $D_{KL}(\pi_E\|\pi)$ rather than adversarial occupancy matching [2507.06780].

Taken together, these results support three distinct empirical claims. First, strict ACIL benefits from replacing infeasible demonstrations with realizable surrogates rather than attempting direct expert matching. Second, training-time safety shaping can materially improve consistency and constraint satisfaction even when only the policy is deployed at test time. Third, demonstration-induced constraint information can reduce violations and preserve multimodal safe behavior without explicit constraint costs.

## 6. Limitations, misconceptions, and conceptual boundaries

A recurrent misconception is that ACIL is equivalent to clipping or projecting actions into a feasible set. The ACIL formulation rejects that reduction: projection changes the transition and thereby alters future occupancy, so it does not solve the imitation problem by itself [2508.14379]. A second misconception is that state-only imitation automatically resolves action infeasibility. The ACIL analysis argues otherwise, because state-only learners may still target trajectories that were generated by unconstrained actions and remain physically unrealizable for the constrained imitator [2508.14379].

The main limitations differ across formulations. DTWIL relies on a learned forward model, incurs substantial offline computation from MPC, CEM, and DTW scoring, and assumes the expert and imitator share the same state and action dimensions. The paper does not provide strong formal optimality guarantees and notes possible failure modes when constraints are extremely tight, when the expert path remains infeasible even after temporal warping, when instability amplifies model error, or when the planning horizon is too short [2508.14379]. Constraint-aware Behavior Cloning explicitly provides **no formal guarantees of constraint satisfaction for the deployed policy**, because safety is enforced only as a soft penalty via learned dynamics, learned safe-set probabilities, heuristic safety labels, and no runtime filter [2503.07737]. SCOPIL likewise offers no hard guarantees of zero violations at execution time, since it does not learn an explicit admissible action set and enforces constraints only through a dual-weighted objective shaped by demonstrations [2507.06780].

The conceptual boundary of ACIL is therefore best drawn along the mechanism of feasibility. In the narrow sense, ACIL studies imitation when the learner’s feasible action set is smaller than the expert’s and standard occupancy matching is structurally unattainable. In the broader neighboring literature, feasibility is often encoded indirectly—through predicted successor-state safety, backward-reachability surrogates, or demonstration-induced occupancy constraints. This suggests that contemporary work on ACIL is organized around a common question—how to imitate when expert behavior is not directly executable—while differing sharply in whether feasibility is enforced by surrogate demonstration generation, by training-time safety shaping, or by soft distributional adherence.

Source: https://www.emergentmind.com/topics/action-constrained-imitation-learning-acil