Curriculum Reinforcement Learning Framework

• Curriculum reinforcement learning frameworks are student–teacher systems that sequence tasks with adaptive difficulty to boost autonomous driving agent robustness.
• The framework leverages a graph-based MARL teacher to generate realistic NPC behaviors, balancing cooperative and adversarial traffic scenarios.
• Adaptive task sequencing based on student success metrics ensures continuous exposure to both common and safety-critical driving situations.

Curriculum reinforcement learning frameworks are designed to improve learning efficiency and robustness by sequencing tasks, data, or environments in a way that progressively increases complexity or challenge. In the context of autonomous driving, this approach is leveraged to expose agents to a broad spectrum of routine and safety-critical scenarios, thereby enhancing policy robustness and generalization in real-world conditions (Abouelazm et al., 25 Jul 2025).

1. Student–Teacher Curriculum Architecture

The framework is structured as a student–teacher system, where the “student” is a deep reinforcement learning (RL) agent representing the ego vehicle, and the “teacher” is a multi-agent RL (MARL) system that controls non-player character (NPC) traffic participants. The teacher orchestrates NPC behaviors by conditioning on an auxiliary difficulty parameter, denoted as λ, as well as on continuous feedback derived from student performance metrics.

The teacher’s policy is graph-based, incorporating full observability over traffic topology, while the student agent operates under partial observability, reflecting realistic on-vehicle sensor constraints (RGB camera, LiDAR, and kinematic measurements).

2. Adaptive Curriculum Learning Mechanism

Curriculum learning is realized through an adaptive progression of task difficulty, mediated by the teacher. Initially, the teacher is trained to produce a wide range of traffic behaviors across a discrete set of difficulty levels (λ-set, e.g., λ ∈ {–1, –0.75, ..., 1}). During student training, the current difficulty λ is adaptively selected based on the student's observed success rate over recent episodes.

A recalibration phase ensures synchronization between student skill and the initial task level at each curriculum round. If the student’s success rate surpasses a threshold T_success, λ is incremented; if it falls below T_fail, the environment is maintained or simplified. To maintain exposure to previously mastered tasks and prevent forgetting, the framework implements episodic sampling of prior (easier) levels with probability P_old.

3. Graph-Based Multi-Agent RL Teacher

The teacher leverages a graph-based MARL architecture to generate dynamically coupled NPC behaviors. Each NPC agent independently receives local rewards but operates with access to a shared state embedding that encodes all agents’ spatial-temporal trajectories and topological lane graph features.

This joint representation incorporates:

• 1D convolutional encodings + GRU for individual agent histories,
• Heterogeneous message passing over a vectorized lane graph describing road geometry and inter-vehicle relations.

The teacher’s reward function is defined as: $$\mathbb{R}_{\text{NPC}} = \mathbb{R}_{\text{NPC}}^{\text{intrinsic}} + \mathbb{R}_{\text{NPC}}^{\text{extrinsic}}$$ where

$$\mathbb{R}_{\text{NPC}}^{\text{intrinsic}} = (1 - K(d)) \cdot \max(\epsilon, 1 - |\lambda|) \cdot \mathbb{R}_{\text{NPC}}^{\text{driving}}$$

$$\mathbb{R}_{\text{NPC}}^{\text{extrinsic}} = K(d) \cdot \mathbb{R}_{\text{student}}^{\text{driving}} \cdot \begin{cases} \lambda, & \text{if } |\lambda| > \epsilon \ \operatorname{sgn}(\lambda) \cdot \epsilon, & \text{otherwise} \end{cases}$$

with $K(d) = \exp(-d^2 / (2\sigma^2))$ as a radial basis function dependent on agent-student distance d and scaling parameter σ.

This design enables the teacher to adapt NPC behavior realism and challenge according to specified λ, capturing a continuum from cooperative (altruistic) to adversarial conduct.

4. Adaptive Task Sequencing and Diversity

The teacher adapts the curriculum trajectory on-line using the student's measured success rate, ensuring that the student is consistently exposed to a mix of common scenarios (to encourage generalizable, assertive behavior) and rare, safety-critical situations (to ensure robust hazard mitigation).

Whenever the student achieves a performance above T_success, the curriculum advances to more adversarial, high-difficulty NPC strategies (lower λ). If performance falls below T_fail, the task is either held constant or simplified. The episodic reintroduction of earlier levels via P_old sampling counteracts catastrophic forgetting and maintains student robustness.

This adaptive, performance-driven sequencing enables emergent behaviors arising jointly from student agent learning dynamics and adaptive scenario generation by the teacher.

5. Realistic Perception and Partial Observability

A key architectural constraint is the partial observability imposed on the student agent. Unlike the teacher, which has privileged access to the full simulation state (agent, NPC, and topology), the student’s observation model is restricted to:

• An RGB camera feed,
• LiDAR grid-mapped point cloud,
• Vehicle state vector (velocities, accelerations).

This addresses the sim-to-real gap, ensuring that the policy learned under curriculum transfer is feasible for deployment on real autonomous vehicles with limited perceptual coverage.

6. Performance Metrics and Empirical Results

Student policy performance is rigorously evaluated by:

• Cumulative episode reward ($\mathbb{R}$),
• Success rate (percentage reaching the goal without collision or leaving the road),
• Terminal statistics (collision rate, off-road rate, timeout rate),
• Route progress (RP),
• Mean velocity along the reference route.

Empirical experiments demonstrate that curriculum-trained agents:

• Achieve higher $\mathbb{R}$ and success rates than rule-based traffic-trained agents,
• Exhibit more realistic and assertive driving (e.g., higher route progress and velocity),
• Avoid degenerate behaviors such as passivity (waiting excessively for NPCs to clear the intersection).

A plausible implication is improved transferability and safety in more diverse and adversarial real-world driving conditions.

7. Implications, Limitations, and Extensibility

This student-teacher curriculum framework demonstrates that explicit, behaviorally diverse, and adaptively sequenced training scenarios can substantially elevate the sample efficiency, robustness, and behavioral realism of RL-based autonomous driving agents compared to rule-based scenario design.

The framework’s reliance on full observability for the teacher is well-suited to current simulation environments but may be constrained in partially observable or real-world multi-agent deployments.

Potential limitations include:

• The assumption that the curriculum can be accurately calibrated by student success thresholds may not generalize across all task distributions or agent architectures.
• Dependence on sophisticated graph-based MARL modeling may increase implementation complexity relative to simpler curriculum generation approaches.

Future work may include extending this curriculum paradigm to multi-agent competitive driving, leveraging its automatic behavior generation and diversity adaptation, or integrating real-world perception uncertainty and physical system constraints into teacher curriculum strategy, to further bridge the sim-to-real gap (Abouelazm et al., 25 Jul 2025).