Highway-env Simulator for Autonomous Driving

• Highway-env Simulator is a Python-based modular platform that simulates multi-lane highways and complex traffic scenarios for autonomous driving research.
• It provides extensible state and action definitions, customizable reward functions, and strict OpenAI Gym-style API compatibility for various DRL methods.
• Researchers leverage its diverse scenario modeling and LLM-driven reward evolution to achieve improved policy efficiency and robust simulation metrics.

The highway-env simulator is a Python-based, modular platform for research in autonomous driving, deep reinforcement learning (DRL), and closed-loop driving benchmarks. It provides canonical multi-lane highway, intersection, merge, and roundabout scenarios, with extensible state/action definitions, reward functions, and programmatic customizability. highway-env underpins contemporary algorithmic studies ranging from classic DRL to LLM-driven reward evolution and personalized multi-modal driving evaluation. Its design philosophy is strict OpenAI Gym-style API compatibility, high-throughput training, and fine-grained scenario/traffic control, which has made it a central testbed in the field (Dong et al., 2023, Han et al., 2024, Kou et al., 8 May 2025).

1. Environment Architecture and Scenario Modeling

highway-env is structured around the OpenAI Gym/Gymnasium reinforcement learning interface. The environment loop exposes standardized entrypoints: env.reset() for initial state sampling and env.step(action) for transition dynamics and reward, returning tuples of (observation, reward, done, info). Rendering is supported via 2D bird’s-eye and forward-facing camera projections or can be run in headless mode for accelerated, batch RL (Dong et al., 2023).

Scenario management comprises six principal built-in modes: Highway, Merge, Roundabout, Intersection, Racetrack, and Parking. Scenarios are defined by composable parameters—such as lane count, curvature, and stochastic traffic density—and researchers may supply custom road geometries or traffic profiles by inheriting from core Environment and RoadNetwork classes. This modularity supports arbitrary scenario extension and integration of novel behavioral models (Dong et al., 2023).

Observation space may consist of a vector encoding ego vehicle kinematics, nearby vehicle states (relative position, velocity, heading), lane centerline geometries, and, optionally, a rendered image (BEV or camera). Actions are either continuous (TRPO: steering angle $\delta$, acceleration $\alpha$) or discrete (DQN and RL/LLM workflows: left/right lane change, acceleration, braking, idle) to support diverse RL algorithm integration (Dong et al., 2023, Han et al., 2024, Kou et al., 8 May 2025).

2. State, Action, and Sensor Modeling

The system state at time $t$ is a vector $s_t$ comprising:

• Ego vehicle: position $(x, y)$, velocity $(v_x, v_y)$, heading $\varphi$,
• For $N$ surrounding vehicles: relative position, velocity, heading,
• Road attributes: lane centerline, width, curvature (absolute or ego-centric).

The action space $\mathcal{A}$ supports:

• Continuous: $\mathcal{A}_\text{cont} = \{\delta,\alpha\}$, $\alpha$0, $\alpha$1 (for policy-gradient methods),
• Discrete: $\alpha$2 (for value-based methods) (Dong et al., 2023, Han et al., 2024).

Sensor channels include:

• Numeric state vectors,
• 2D or 3D pixel images (BEV, semantic segmentation, perspective, via wrappers or extensions),
• In personalized frameworks, BEV images of configurable resolution—for example, 336×336 in PAD-Highway, with per-frame vehicle and lane overlays (Kou et al., 8 May 2025).

3. Reward Formulation and Evolution

Reward design in highway-env is highly configurable. The standard baseline is a linear sum of features (progress, lane-keeping, collision penalty), e.g.,

$\alpha$3

with empirically tuned weights (Han et al., 2024).

Advanced modifications use geometric quantities:

• Signed lane distance, $\alpha$4,
• Lane heading difference, LHD,

$\alpha$5

plus severe penalties for collisions ($\alpha$6) (Dong et al., 2023).

Reward function generation and optimization are increasingly automated via LLM-driven frameworks, where candidate Python reward code is iteratively generated, scored by RL training, and refined using closed-loop text-based feedback (“RL–LLM reflection loop”). This leads to empirically higher success rates and improved policy efficiency, notably yielding a reported 22% improvement over expert-designed rewards on the same highway-env configurations (Han et al., 2024).

4. Scenario Extensions and Custom Modules

The platform supports integration of custom scenarios and logic:

• ComplexRoads: An extended composite scenario embedding two merges, two four-way intersections, two roundabouts, and multi-lane segments into a single map, with randomized spawn/destination sampling for both ego and background vehicles. Traffic follows deterministic "perfect driver" kinematic bicycle models with configurable density (Dong et al., 2023).
• Lane-tracking and on-lane logic are improved using signed distance and heading criteria, increasing reward function robustness when vehicles depart from designated lanes (Dong et al., 2023).
• For personalized driving research, the PAD-Highway benchmark wraps highway-env to emit BEV images and state vectors. Personalization is further enabled via MLLM-driven prompts (e.g., “fast,” “normal,” “slow”) and per-action danger-level prediction (Kou et al., 8 May 2025).

Custom traffic generation and logic modules can be incorporated by subclassing the Vehicle or BehaviorModel objects, and the reward function or logger can be extended for domain-specific evaluation metrics.

5. Integration with Deep Reinforcement Learning Algorithms

highway-env is designed for efficient RL training with state-of-the-art algorithms:

• DQN: Standard value-based agent with three hidden layers (256 ReLU units each), outputting Q-values for all discrete actions, $\alpha$7-greedy exploration schedule, large replay buffer, and default $\alpha$8 (Dong et al., 2023, Han et al., 2024).
• TRPO: Policy-gradient agent with actor (2 layers × 64 units, $\alpha$9 activation) and a Gaussian policy parameterization, trust-region updates ($t$0), and generalized advantage estimation ($t$1). Critic may share the backbone or be a separate MLP (Dong et al., 2023).

Training is conducted over 10,000–100,000 gradient steps, typically with independent seeds per algorithm and scenario to ensure robust evaluation. Metrics are logged at validation intervals and include speed, comfort (jerk), steering activity, on-lane percentage, collision rate, and simulation runtime.

6. Evaluation Benchmarks and Performance Metrics

Mainline evaluation scenarios include racetrack (curved lane following), roundabout entry/exit, intersection negotiation, merge maneuvers, and the composite ComplexRoads map (Dong et al., 2023). PAD-Highway additionally provides 250 hours of video data (32,000 × 30 s driving episodes) with extensive annotation (scene fact extraction, per‐action danger levels, ground-truth actions) for evaluation under personalized prompts (Kou et al., 8 May 2025).

Key metrics for benchmarking include:

• Average speed (m/s),
• Peak/total jerk (comfort index),
• Total steering angle executed (efficiency measure),
• On-lane rate (percent within lane bounds),
• Collision rate,
• Safe following distance rate,
• Lane-keeping rate,
• Comfort (average/jerk acceleration on axes),
• Total distance covered and completion success rate.

For instance, in ComplexRoads with TRPO and customized reward, reported results are speed 16.2 m/s, total jerk 13.2, on-lane rate 99.9%, and collision rate 9% (Dong et al., 2023).

Success rate improvements via LLM-evolved rewards (LanERD) are empirically observed as 22% higher than baseline on test suites of three-lane and four-lane scenarios of varying traffic density (Han et al., 2024). PAD-Highway assesses models via survival rate under collision constraints, driving distance, average speed, density, safe distance-keeping, and comfort statistics, under both expert and human driving policies (Kou et al., 8 May 2025).

7. Extension, Customization, and Pseudocode Workflows

The highway-env platform is intended for rapid prototyping and experiment extension:

• New road geometries are defined by subclassing AbstractEnv or RoadNetwork, with user-supplied lane graph and connection logic.
• Alternative traffic or vehicle behaviors are implemented by subclassing and injecting into the traffic manager.
• Evaluating new RL algorithms requires only the implementation of standard Gym agent interfaces: select_action(), store_transition(), and update() (Dong et al., 2023).

A canonical agent–environment training loop, as employed by both DRL and LLM-driven workflows, follows:

$t$2

For LLM-guided reward design workflows (LanERD), the process additionally involves iteratively generating Python reward functions, evaluating by batch DQN training, and feeding back statistics for automated prompt refinement (Han et al., 2024).

In summary, highway-env is a reference RL driving platform enabling reproducible, extensible research across core DRL, LLM-induced policy shaping, and personalized, multi-modal autonomous driving analysis (Dong et al., 2023, Han et al., 2024, Kou et al., 8 May 2025).