RARLAP: Reward-Augmented Learning for AP

• The paper introduces a reward-augmented framework for continuous autonomous parking, comparing three reward strategies to guide policy optimization.
• It shows that the milestone-augmented reward (MAR) with on-policy optimization achieves a 91% success rate and smoother, potential-field-like trajectories.
• The study establishes a reproducible benchmark in a Unity-based simulation, highlighting structured reward design to enhance safety and sim-to-real transfer.

Searching arXiv for the specified paper to ground the article and verify bibliographic metadata. Reward-Augmented Learning Framework for AP (RARLAP) is a reward-augmented reinforcement-learning framework for continuous-control autonomous parking (AP), introduced in "Reward-Augmented Reinforcement Learning for Continuous Control in Precision Autonomous Parking via Policy Optimization Methods" (Suleman et al., 25 Jul 2025). It targets a regime characterized by tight spatial constraints, frequent close-range obstacle interactions, and stringent safety margins, and is trained entirely within a high-fidelity Unity-based custom 3D simulation environment. The framework systematically compares three structured reward strategies—goal-only reward (GOR), dense proximity reward (DPR), and milestone-augmented reward (MAR)—under both on-policy and off-policy policy optimization. In the reported experiments, on-policy MAR achieves a 91\% success rate and yields smoother trajectories and more robust behavior, whereas GOR and DPR fail to guide effective learning (Suleman et al., 25 Jul 2025).

1. Problem setting and conceptual scope

RARLAP is formulated for precision autonomous parking as a continuous-domain control problem rather than as a discrete planning task. The motivating premise is that conventional rule-based and model-predictive methods often lack the adaptability and generalization needed to handle the nonlinear and environment-dependent complexities of AP. The proposed response is not a change in vehicle dynamics or sensing, but a change in how the learning signal is structured: reward augmentation is used to induce smooth and adaptable policy behavior during policy optimization (Suleman et al., 25 Jul 2025).

Within that framing, the paper’s central comparison is between reward sparsity, dense but unstructured proximity feedback, and reward shaping that introduces an explicit intermediate milestone. This comparison is operationalized across both on-policy and off-policy optimization paradigms. A key implication of the reported results is that AP performance is highly sensitive not merely to whether the reward is sparse or dense, but to whether the reward embeds trajectory structure that is compatible with the parking geometry.

2. Markov decision process formulation

The parking task is cast as a model-free Markov decision process defined by the tuple $$(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R})$$ (Suleman et al., 25 Jul 2025). The state space is

$$s_t = [\,\mathbf{p}^{\rm agent}_t,\;\mathbf{p}^{\rm target}_t,\;\mathbf{R}^t,\;d_t\,],$$

where $$\mathbf{p}^{\rm agent}_t,\mathbf{p}^{\rm target}_t\in\mathbb{R}^3$$ are the global 3D positions of the vehicle and parking slot, $\mathbf{R}^t=[r^t_1,\dots,r^t_N]\in\mathbb{R}^N$ is the raycast-distance vector with $N=8$, and

$$d_t=\|\mathbf{p}^{\rm target}_t-\mathbf{p}^{\rm agent}_t\|_2$$

is the Euclidean distance to the goal.

The action space is a scalar control $a_t\in[-1,1]$ that governs the incremental change in steering angle,

$$S_t^\angle = S_{t-1}^\angle + \Delta S_{t-1}^\angle(a_t).$$

This action design makes the control problem explicitly continuous and ties policy output directly to steering evolution rather than to a discrete maneuver library. State transitions are model-free and realized through Unity physics, so the transition kernel is sampled as $s_{t+1}\sim\mathcal{P}(\cdot|s_t,a_t)$.

The optimization target is the discounted return

$$J(\theta)=\mathbb{E}_{\tau\sim\pi_\theta}\Bigl[\sum_{t=0}^\infty\Gamma\,r(s_t,a_t)\Bigr],$$

with discount factor $$s_t = [\,\mathbf{p}^{\rm agent}_t,\;\mathbf{p}^{\rm target}_t,\;\mathbf{R}^t,\;d_t\,],$$0. In this formulation, reward design becomes the main instrument for shaping policy search, because neither an explicit vehicle model nor a handcrafted planner is imposed on the learned controller.

3. Structured reward design

RARLAP evaluates three reward functions that differ in sparsity, granularity, and safety guidance (Suleman et al., 25 Jul 2025). Let $$s_t = [\,\mathbf{p}^{\rm agent}_t,\;\mathbf{p}^{\rm target}_t,\;\mathbf{R}^t,\;d_t\,],$$1, let $$s_t = [\,\mathbf{p}^{\rm agent}_t,\;\mathbf{p}^{\rm target}_t,\;\mathbf{R}^t,\;d_t\,],$$2 denote the collision threshold, let $$s_t = [\,\mathbf{p}^{\rm agent}_t,\;\mathbf{p}^{\rm target}_t,\;\mathbf{R}^t,\;d_t\,],$$3 denote the goal region, and let $$s_t = [\,\mathbf{p}^{\rm agent}_t,\;\mathbf{p}^{\rm target}_t,\;\mathbf{R}^t,\;d_t\,],$$4 denote the agent, goal, and milestone positions.

The goal-only reward is a sparse terminal structure:

$$s_t = [\,\mathbf{p}^{\rm agent}_t,\;\mathbf{p}^{\rm target}_t,\;\mathbf{R}^t,\;d_t\,],$$5

GOR supplies success and collision signals plus a living penalty, but no intermediate shaping. The reported behavior is very slow learning, a low reward plateau, and unstable off-policy value evolution.

The dense proximity reward adds continuous distance-based feedback:

$$s_t = [\,\mathbf{p}^{\rm agent}_t,\;\mathbf{p}^{\rm target}_t,\;\mathbf{R}^t,\;d_t\,],$$6

with $$s_t = [\,\mathbf{p}^{\rm agent}_t,\;\mathbf{p}^{\rm target}_t,\;\mathbf{R}^t,\;d_t\,],$$7. DPR provides a denser gradient than GOR, and the paper reports a faster initial rise. However, it also reports early plateauing at suboptimal reward and oscillatory back-and-forth trajectories, which the authors attribute to the absence of structural guidance despite the dense signal.

The milestone-augmented reward introduces an explicit intermediate region:

$$s_t = [\,\mathbf{p}^{\rm agent}_t,\;\mathbf{p}^{\rm target}_t,\;\mathbf{R}^t,\;d_t\,],$$8

and

$$s_t = [\,\mathbf{p}^{\rm agent}_t,\;\mathbf{p}^{\rm target}_t,\;\mathbf{R}^t,\;d_t\,],$$9

Here $$\mathbf{p}^{\rm agent}_t,\mathbf{p}^{\rm target}_t\in\mathbb{R}^3$$0 ensures nonnegativity once past the milestone region. The paper describes this reward as injecting a mid-trajectory milestone signal that acts as an intermediate shaping reward and combines safety guidance with structured proximity to goal once in the milestone region. The resulting trajectories are reported as clean, direct, and “potential-field”-like.

A plausible implication is that MAR does not merely densify feedback; it restructures the optimization landscape by separating approach behavior into at least two phases, before and after milestone entry. That interpretation is consistent with the reported contrast between DPR and MAR.

4. Policy optimization under on-policy and off-policy paradigms

RARLAP is evaluated with both PPO-based on-policy optimization and SAC-based off-policy optimization (Suleman et al., 25 Jul 2025). The on-policy variant is denoted ON-POM, and the off-policy variant is denoted OFF-POM.

For PPO, the clipped surrogate objective is

$$\mathbf{p}^{\rm agent}_t,\mathbf{p}^{\rm target}_t\in\mathbb{R}^3$$1

with

$$\mathbf{p}^{\rm agent}_t,\mathbf{p}^{\rm target}_t\in\mathbb{R}^3$$2

and $$\mathbf{p}^{\rm agent}_t,\mathbf{p}^{\rm target}_t\in\mathbb{R}^3$$3 computed via GAE. The value-function loss is

$$\mathbf{p}^{\rm agent}_t,\mathbf{p}^{\rm target}_t\in\mathbb{R}^3$$4

and the total loss is

$$\mathbf{p}^{\rm agent}_t,\mathbf{p}^{\rm target}_t\in\mathbb{R}^3$$5

The parameters are updated jointly by gradient descent on $$\mathbf{p}^{\rm agent}_t,\mathbf{p}^{\rm target}_t\in\mathbb{R}^3$$6.

For SAC, the Q-function loss is

$$\mathbf{p}^{\rm agent}_t,\mathbf{p}^{\rm target}_t\in\mathbb{R}^3$$7

with soft value

$$\mathbf{p}^{\rm agent}_t,\mathbf{p}^{\rm target}_t\in\mathbb{R}^3$$8

and policy loss

$$\mathbf{p}^{\rm agent}_t,\mathbf{p}^{\rm target}_t\in\mathbb{R}^3$$9

The reported implementation updates $\mathbf{R}^t=[r^t_1,\dots,r^t_N]\in\mathbb{R}^N$0 and $\mathbf{R}^t=[r^t_1,\dots,r^t_N]\in\mathbb{R}^N$1 via their respective gradients, with replay-buffer sampling in the off-policy case and an optional entropy-temperature auto-tuning step “if implemented.”

The high-level training loop follows the same reward-selection interface for both paradigms: initialize the policy network; if ON-POM, also initialize the value network; if OFF-POM, initialize Q-network and replay buffer; collect trajectories or transitions in Unity; compute the reward according to GOR, DPR, or MAR; and perform the corresponding PPO or SAC updates. This common interface allows the reward design to be compared while keeping the control setting fixed.

5. Unity-based simulation environment

The framework is trained and evaluated in a custom Unity-based 3D simulator (Suleman et al., 25 Jul 2025). The agent vehicle uses a realistic mesh with dimensions $\mathbf{R}^t=[r^t_1,\dots,r^t_N]\in\mathbb{R}^N$2 m, Ackermann steering physics, and collider meshes. Perception is represented through 8 raycasts—4 front and 4 rear, with $\mathbf{R}^t=[r^t_1,\dots,r^t_N]\in\mathbb{R}^N$3 lateral placement and a range of 0–8 m.

The environment models walls and obstacles as simple box colliders and parked cars as rectangular colliders that are slightly bloated for safety. Parking scenarios include configurable empty and occupied slots, a mid-mission milestone collider, and an arrival-trigger collider. These components are directly aligned with the reward structure, especially MAR, for which milestone entry is a key event.

The training setup includes 12 concurrent Unity instances sharing a central training process. Domain randomization is supported through Unity ML-Agents configuration, with slot positions, obstacle arrangements, and friction parameters varied across runs. The code is publicly available, and the paper presents the simulator, multi-instance rollout configuration, and reward study as a reproducible benchmark for continuous-steering AP. This suggests that reproducibility in RARLAP is treated as part of the framework rather than as an auxiliary implementation detail.

6. Empirical performance and reported significance

The empirical comparison distinguishes sharply among the three reward strategies and between on-policy and off-policy optimization under MAR (Suleman et al., 25 Jul 2025). In the convergence curves, GOR exhibits very slow learning, a low reward plateau, and unstable OFF-POM value behavior. DPR rises faster initially but plateaus early at suboptimal reward and shows high variance. MAR rises rapidly to approximately 3000 for ON-POM and approximately 2000 for OFF-POM, with stable low loss.

Distance-to-goal analysis shows that MAR produces tight clustering close to 0 m. Trajectory analysis further differentiates the methods: GOR yields erratic spiral behavior with wide oscillations; DPR produces oscillatory back-and-forth motion; and MAR yields clean, direct “potential-field”-like paths. The paper’s comparative discussion characterizes GOR as sparse terminal reward without shaping, DPR as dense proximity reward without structural guidance, and MAR as structured reward shaping with safety guidance and milestone-based progression.

The reported deployment metrics under MAR are as follows:

Setting	Success / Collision	Final distance / Steps / Train time
ON-POM (MAR)	91\% / 9\%	2.191 m / 201 / 26.8 min
OFF-POM (MAR)	50\% / 50\%	7.287 m / 143 / 268.7 min

These results underpin the paper’s principal takeaways. Structured reward augmentation is presented as critical for complex, spatially constrained AP tasks; combining MAR with on-policy optimization yields superior sample- and time-efficiency, summarized in the paper as training in 27 min versus 269 min; and the synergy of reward structuring with synchronous on-policy updates is reported to accelerate convergence, reduce policy variance, and enhance safety margins. The authors further state that this paves the way for real-world sim-to-real transfer in precision autonomous parking. A cautious interpretation is that the paper establishes the efficacy of structured reward design within its simulator and optimization setup, while positioning sim-to-real transfer as a forward-looking consequence rather than a completed empirical validation.