---
title: Highway-env Simulator for Autonomous Driving
url: https://www.emergentmind.com/topics/highway-env-simulator
type: topic
---

# Highway-env Simulator for Autonomous Driving

The highway-env simulator is a Python-based, modular platform for research in autonomous driving, deep reinforcement learning (DRL), and closed-loop driving benchmarks. It provides canonical multi-lane highway, intersection, merge, and roundabout scenarios, with extensible state/action definitions, reward functions, and programmatic customizability. highway-env underpins contemporary algorithmic studies ranging from classic DRL to LLM-driven reward evolution and personalized multi-modal driving evaluation. Its design philosophy is strict OpenAI Gym-style API compatibility, high-throughput training, and fine-grained scenario/traffic control, which has made it a central testbed in the field [2306.11466, 2406.10540, 2505.05240].

## 1. Environment Architecture and Scenario Modeling

highway-env is structured around the OpenAI Gym/Gymnasium reinforcement learning interface. The environment loop exposes standardized entrypoints: `env.reset()` for initial state sampling and `env.step(action)` for transition dynamics and reward, returning tuples of (observation, reward, done, info). Rendering is supported via 2D bird’s-eye and forward-facing camera projections or can be run in headless mode for accelerated, batch RL [2306.11466].

Scenario management comprises six principal built-in modes: Highway, Merge, Roundabout, Intersection, Racetrack, and Parking. Scenarios are defined by composable parameters—such as lane count, curvature, and stochastic traffic density—and researchers may supply custom road geometries or traffic profiles by inheriting from core `Environment` and `RoadNetwork` classes. This modularity supports arbitrary scenario extension and integration of novel behavioral models [2306.11466].

Observation space may consist of a vector encoding ego vehicle kinematics, nearby vehicle states (relative position, velocity, heading), lane centerline geometries, and, optionally, a rendered image (BEV or camera). Actions are either continuous (TRPO: steering angle $\delta$, acceleration $\alpha$) or discrete (DQN and RL/LLM workflows: left/right lane change, acceleration, braking, idle) to support diverse RL algorithm integration [2306.11466, 2406.10540, 2505.05240].

## 2. State, Action, and Sensor Modeling

The system state at time $t$ is a vector $s_t$ comprising:
- Ego vehicle: position $(x, y)$, velocity $(v_x, v_y)$, heading $\varphi$,
- For $N$ surrounding vehicles: relative position, velocity, heading,
- Road attributes: lane centerline, width, curvature (absolute or ego-centric).

The action space $\mathcal{A}$ supports:
- **Continuous**: $\mathcal{A}_\text{cont} = \{\delta,\alpha\}$, $\delta \in [-\pi/2, \pi/2]$, $\alpha \in [-5, 5]$ (for policy-gradient methods),
- **Discrete**: $\mathcal{A}_\text{disc} = \{\text{LANE\_LEFT}, \text{LANE\_RIGHT}, \text{ACCELERATE}, \text{BRAKE}, \text{IDLE}\}$ (for value-based methods) [2306.11466, 2406.10540].

Sensor channels include:
- Numeric state vectors,
- 2D or 3D pixel images (BEV, semantic segmentation, perspective, via wrappers or extensions),
- In personalized frameworks, BEV images of configurable resolution—for example, 336×336 in PAD-Highway, with per-frame vehicle and lane overlays [2505.05240].

## 3. Reward Formulation and Evolution

Reward design in highway-env is highly configurable. The standard baseline is a linear sum of features (progress, lane-keeping, collision penalty), e.g.,
\[
R(s,a) = \alpha_1 \frac{v_\text{ego}}{v_\text{max}} + \alpha_2 \mathbf{1}\{\ell_\text{ego} = \ell_\text{rightmost}\} - \alpha_3 \mathbf{1}\{\text{collision}\} - \alpha_4 \mathbf{1}\{a \in \{\text{LANE\_LEFT}, \text{LANE\_RIGHT}\}\}
\]
with empirically tuned weights [2406.10540].

Advanced modifications use geometric quantities:
- Signed lane distance, $d(c, \mathcal{L})$,
- Lane heading difference, LHD,
\[
R_S(c, \mathcal{L}) = 
\begin{cases}
\displaystyle
\frac{\cos|\mathrm{LHD}| \cdot v_c}{20 \max(1, |d(c, \mathcal{L})|)} & v_c \geq 0 \\
0 & v_c < 0
\end{cases}
\]
plus severe penalties for collisions ($R = -10$) [2306.11466].

Reward function generation and optimization are increasingly automated via LLM-driven frameworks, where candidate Python reward code is iteratively generated, scored by RL training, and refined using closed-loop text-based feedback (“RL–LLM reflection loop”). This leads to empirically higher success rates and improved policy efficiency, notably yielding a reported 22% improvement over expert-designed rewards on the same highway-env configurations [2406.10540].

## 4. Scenario Extensions and Custom Modules

The platform supports integration of custom scenarios and logic:
- **ComplexRoads**: An extended composite scenario embedding two merges, two four-way intersections, two roundabouts, and multi-lane segments into a single map, with randomized spawn/destination sampling for both ego and background vehicles. Traffic follows deterministic "perfect driver" kinematic bicycle models with configurable density [2306.11466].
- Lane-tracking and on-lane logic are improved using signed distance and heading criteria, increasing reward function robustness when vehicles depart from designated lanes [2306.11466].
- For personalized driving research, the PAD-Highway benchmark wraps highway-env to emit BEV images and state vectors. Personalization is further enabled via MLLM-driven prompts (e.g., “fast,” “normal,” “slow”) and per-action danger-level prediction [2505.05240].

Custom traffic generation and logic modules can be incorporated by subclassing the Vehicle or BehaviorModel objects, and the reward function or logger can be extended for domain-specific evaluation metrics.

## 5. Integration with Deep Reinforcement Learning Algorithms

highway-env is designed for efficient RL training with state-of-the-art algorithms:
- **DQN**: Standard value-based agent with three hidden layers (256 ReLU units each), outputting Q-values for all discrete actions, $\epsilon$-greedy exploration schedule, large replay buffer, and default $\gamma=0.99$ [2306.11466, 2406.10540].
- **TRPO**: Policy-gradient agent with actor (2 layers × 64 units, $\tanh$ activation) and a Gaussian policy parameterization, trust-region updates ($\delta=0.01$), and generalized advantage estimation ($\lambda=0.97$). Critic may share the backbone or be a separate MLP [2306.11466].

Training is conducted over 10,000–100,000 gradient steps, typically with independent seeds per algorithm and scenario to ensure robust evaluation. Metrics are logged at validation intervals and include speed, comfort (jerk), steering activity, on-lane percentage, collision rate, and simulation runtime.

## 6. Evaluation Benchmarks and Performance Metrics

Mainline evaluation scenarios include racetrack (curved lane following), roundabout entry/exit, intersection negotiation, merge maneuvers, and the composite ComplexRoads map [2306.11466]. PAD-Highway additionally provides 250 hours of video data (32,000 × 30 s driving episodes) with extensive annotation (scene fact extraction, per‐action danger levels, ground-truth actions) for evaluation under personalized prompts [2505.05240].

Key metrics for benchmarking include:
- Average speed (m/s),
- Peak/total jerk (comfort index),
- Total steering angle executed (efficiency measure),
- On-lane rate (percent within lane bounds),
- Collision rate,
- Safe following distance rate,
- Lane-keeping rate,
- Comfort (average/jerk acceleration on axes),
- Total distance covered and completion success rate.

For instance, in ComplexRoads with TRPO and customized reward, reported results are speed 16.2 m/s, total jerk 13.2, on-lane rate 99.9%, and collision rate 9% [2306.11466].

Success rate improvements via LLM-evolved rewards (LanERD) are empirically observed as 22% higher than baseline on test suites of three-lane and four-lane scenarios of varying traffic density [2406.10540]. PAD-Highway assesses models via survival rate under collision constraints, driving distance, average speed, density, safe distance-keeping, and comfort statistics, under both expert and human driving policies [2505.05240].

## 7. Extension, Customization, and Pseudocode Workflows

The highway-env platform is intended for rapid prototyping and experiment extension:
- New road geometries are defined by subclassing `AbstractEnv` or `RoadNetwork`, with user-supplied lane graph and connection logic.
- Alternative traffic or vehicle behaviors are implemented by subclassing and injecting into the traffic manager.
- Evaluating new RL algorithms requires only the implementation of standard Gym agent interfaces: `select_action()`, `store_transition()`, and `update()` [2306.11466].

A canonical agent–environment training loop, as employed by both DRL and LLM-driven workflows, follows:

```python
initialize ENV (e.g. ComplexRoads) with parameters
initialize AGENT policy πθ (DQN or TRPO)
for iteration = 1 to N_training_steps do
    s0 ← ENV.reset()
    for t = 0 to T_max do
        a_t ← AGENT.select_action(s_t)   # ε-greedy (DQN) or sample πθ(.|s_t) (TRPO)
        s_{t+1}, r_t, done, info ← ENV.step(a_t)
        AGENT.store_transition(s_t, a_t, r_t, s_{t+1}, done)
        if AGENT.ready_to_update() then
            AGENT.update()
        if done then break
        s_t ← s_{t+1}
    end for
    if iteration mod EVAL_INTERVAL == 0 then
        evaluate AGENT on validation scenarios
        log metrics (speed, jerk, on_lane_rate, collisions)
end for
save best AGENT checkpoint
```

For LLM-guided reward design workflows (LanERD), the process additionally involves iteratively generating Python reward functions, evaluating by batch DQN training, and feeding back statistics for automated prompt refinement [2406.10540].

In summary, highway-env is a reference RL driving platform enabling reproducible, extensible research across core DRL, LLM-induced policy shaping, and personalized, multi-modal autonomous driving analysis [2306.11466, 2406.10540, 2505.05240].

Source: https://www.emergentmind.com/topics/highway-env-simulator