PlanT 2.0: Object-Centric Planning Transformer

• PlanT 2.0 is a planning-only, object-centric transformer that predicts future ego trajectories in CARLA using ground-truth objects, route, SD-map, and speed-limit tokens.
• It improves on its predecessor by expanding input classes and decoupling lateral and longitudinal planning through a multi-token linear decoder, achieving strong performance on benchmarks.
• Controlled perturbation studies with PlanT 2.0 expose shortcut learning and biases in closed-loop driving pipelines, underscoring key data-centric challenges and future research opportunities.

PlanT 2.0 is a lightweight, fully open, object-centric planning transformer for autonomous driving research in CARLA. It is a planning-only, privileged-information model: given ground-truth objects, route information, a small SD-map token, and a discrete speed-limit token, it predicts a future ego trajectory in bird’s-eye view that is later tracked by low-level controllers. Conceived as an upgrade to the original PlanT for the substantially harder CARLA Leaderboard 2.0 regime, it is notable not only for strong closed-loop performance on Longest6 v2, Bench2Drive, and CARLA validation routes, but also for enabling controlled perturbation studies that expose shortcut learning, benchmark biases, and structural flaws in closed-loop driving pipelines (Gerstenecker et al., 10 Nov 2025).

1. Historical setting and research objective

PlanT 2.0 emerged from two converging pressures. The first was scientific: sensor-based end-to-end driving models are difficult to manipulate systematically, because controlled edits to camera images or point clouds are cumbersome and expensive. The second was benchmark-driven: CARLA Leaderboard 2.0 introduced harder, higher-speed, multi-actor scenarios—such as construction sites, cut-ins, emergency vehicles, and explicit stop-sign compliance—under which the original PlanT and Leaderboard 1.0-oriented systems perform poorly (Gerstenecker et al., 10 Nov 2025).

Within the CARLA ecosystem, PlanT 2.0 occupies the privileged, object-centric end of the design space. It contrasts with sensor-based end-to-end systems such as UniAD, TF++, HiP-AD, Raw2Drive, and SimLingo, as well as with rule-based privileged experts such as PDM-Lite and RL-based privileged planners such as Think2Drive and CaRL. Its purpose is deliberately diagnostic: by operating on object-level scene descriptions rather than raw sensory tensors, it allows direct intervention on the input scene and makes the learned planning policy easier to probe (Gerstenecker et al., 10 Nov 2025).

Relative to the original PlanT, the system extends both representation and control. Its input classes are expanded from a comparatively narrow set to vehicles, pedestrians, static objects, emergency vehicles, traffic lights encoded as stop-line boxes, stop signs encoded as stop-line boxes, an SD-map token, a discrete speed-limit token, and an improved route representation using 20 BEV points instead of 2 boxes. Its spatial support is enlarged from a 30 m radius to a combination of a 50 m radius behind the ego vehicle and an ellipse in front with 100 m extent along $x$ and 50 m along $y$. Its output representation is also revised: the best variant decouples lateral and longitudinal planning, using 20 spatially equidistant path points for steering and 8 temporal waypoints for speed, rather than a single waypoint sequence used for both axes in the original model (Gerstenecker et al., 10 Nov 2025).

2. Representation, transformer design, and control interface

The core input is a set of object tokens in ego-centric BEV. Each object is represented by center position $(x_i, y_i)$, orientation $\theta_i$, extents $(w_i, l_i)$, a scalar speed or velocity $v_i$, and a semantic type drawn from vehicle, pedestrian, static object, emergency vehicle, stop sign, or traffic-light stop line. This raw descriptor is mapped through a class-specific linear projection,

$$\mathbf{z}_i = W_{\text{class}(i)} \mathbf{o}_i + b_{\text{class}(i)},$$

producing a token sequence that can be manipulated directly at the object level (Gerstenecker et al., 10 Nov 2025).

Three additional inputs are central. The route token is derived from 20 route points in BEV coordinates sampled at 1 m spacing along the global route and encoded through a linear layer into a single token. The speed-limit token is a learned embedding drawn from a 4-way discrete set for CARLA speed limits. The SD-map token is produced from an RGB BEV map covering a $64 \times 64$ m area centered on the ego vehicle at 0.5 m resolution, yielding a $128 \times 128$ image whose channels encode background, drivable road, solid lane markings, and broken lane markings. A ResNet-18 encoder followed by a linear layer converts this map into a single token, allowing the transformer to distinguish lane topology, including same-direction versus opposite-direction lanes, albeit at coarse spatial fidelity (Gerstenecker et al., 10 Nov 2025).

These tokens are processed by a standard transformer with multi-head self-attention,

$$\mathrm{Attn}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$

together with standard feed-forward blocks and layer norms. The paper emphasizes continuity with PlanT v1: the major intellectual changes lie not in transformer depth or hidden size, which are inherited, but in the semantics of the input and output interface (Gerstenecker et al., 10 Nov 2025).

The trajectory decoder is where PlanT 2.0 departs most strongly from the original model. Three representations are considered: WPS, with 8 temporal waypoints at 4 Hz over 2 seconds; PATH, with 20 spatially equidistant path points plus a discrete speed classifier; and P+WP, the final configuration, combining 20 path points for lateral control with 8 temporal waypoints for longitudinal control. Three generation schemes are studied: a single-token GRU, a multi-token GRU, and a multi-token linear head. The best-performing configuration is P+WP with the multi-token linear decoder. For the path points, learned point tokens are appended to the transformer sequence, and the decoder predicts 2D increments,

$y$0

An analogous cumulative-sum construction is used for the temporal waypoints. The cumulative sum imposes ordering and smoothness, while the simple linear decoder outperforms more complex GRU-based alternatives in the reported experiments (Gerstenecker et al., 10 Nov 2025).

Low-level actuation is delegated to PDM-Lite controllers. Longitudinal control uses linear regression; lateral control uses PID with variable lookahead. The design therefore separates high-level planning from actuation, while still exposing a key limitation later emphasized in the analysis: waypoint-based outputs entangle geometric intent and assumed vehicle dynamics (Gerstenecker et al., 10 Nov 2025).

3. Training data, imitation-learning setup, and evaluation protocol

The training set is based on the TF++ dataset for Leaderboard 2.0 and comprises approximately 500k samples generated entirely by PDM-Lite, a rule-based expert capable of solving all Leaderboard 2.0 scenarios. The training corpus includes Town13, which also appears in validation; ablations are additionally trained without Town13. Several dataset fixes are applied: invalid extent values for some static objects are corrected, samples in which a stationary ego vehicle is hit by another car are filtered out, and more cautious expert behavior is enforced around emergency vehicles in OppositeVehicleRunningRedLight and OppositeVehicleTakingPriority scenarios (Gerstenecker et al., 10 Nov 2025).

Training uses supervised imitation learning. The model regresses path points and waypoints, or path points plus a speed classifier depending on the representation under study. The exposition characterizes the trajectory supervision as effectively pointwise regression over the predicted 2D path and waypoint sequences. No reinforcement learning loss and no auxiliary perception objectives are introduced. To improve recovery from off-expert states, the same augmentation used in TF++ is adopted: the ego vehicle is randomly translated and rotated by small amounts, and all object positions are re-expressed in the new ego frame, thereby creating off-trajectory examples (Gerstenecker et al., 10 Nov 2025).

The reported training procedure uses 30 epochs, batch size 128, a learning rate of $y$1, and a factor-10 reduction in learning rate at the final epoch. Experiments run on 2 $y$2 RTX 2080 Ti GPUs and take about 6 hours per training run. For reproducibility, three independent training seeds are used per configuration, together with three evaluation seeds per model, and results are reported as mean $y$3 standard deviation (Gerstenecker et al., 10 Nov 2025).

Evaluation spans three CARLA 2.0 benchmarks. CARLA validation routes contain 20 long routes of approximately 12.4 km average length with 36 different scenarios. Reported metrics are Route Completion (RC), Infraction Score (IS), Driving Score (DS), and Normalized Driving Score (NDS), with

$y$4

Bench2Drive contains 44 scenario types over 220 very short routes and reports Driving Score and Success Rate (SR). Longest6 v2 contains 36 medium-length routes and reports RC, DS, and collisions per kilometer (CP) (Gerstenecker et al., 10 Nov 2025).

4. Reported performance and benchmark position

The empirical results establish PlanT 2.0 as a strong privileged planner in CARLA 2.0, while simultaneously showing a substantial remaining gap to the rule-based expert on the hardest benchmark (Gerstenecker et al., 10 Nov 2025).

Benchmark	PlanT 2.0	Comparator context
CARLA validation routes	NDS $y$5, RC $y$6	Expert: NDS 61.55, RC 92.35
Bench2Drive	DS $y$7, SR $y$8	Think2Drive: DS 91.85, SR 85.41; Expert: DS 97.02, SR 92.27
Longest6 v2	DS $y$9, RC $(x_i, y_i)$0, CP 0.03	CaRL: DS $(x_i, y_i)$1, RC $(x_i, y_i)$2, CP 0.01; Expert: DS 73, RC 100, CP 0.00

On CARLA validation routes, PlanT 2.0 substantially outperforms sensor-based baselines reported in the same study. UniAD records NDS 0.00 and RC 1.42, TF++ records NDS 4.94 and RC 68.53, and TF++ without Town13 records NDS 2.12 and RC 50.20. Yet the same benchmark also reveals pronounced town sensitivity: removing Town13 from training drops PlanT 2.0 to NDS $(x_i, y_i)$3 and RC $(x_i, y_i)$4, indicating strong domain dependence on evaluation-town geometry (Gerstenecker et al., 10 Nov 2025).

On Bench2Drive, PlanT 2.0 reaches a new state of the art among learning-based privileged planners in Driving Score, slightly exceeding Think2Drive in DS while remaining close in SR. On Longest6 v2, it surpasses both CaRL and the expert in DS, while nearly saturating Route Completion. A plausible implication is that the simpler scenario distribution of Longest6 v2 is more compatible with imitation of deterministic expert behavior than the heterogeneous long-route setting of the validation benchmark. That interpretation is consistent with the later failure analysis, which emphasizes brittleness under distribution shift and scenario variation (Gerstenecker et al., 10 Nov 2025).

5. Failure analysis, shortcut learning, and structural critique

The central intellectual contribution of PlanT 2.0 lies in what its perturbation studies reveal. Because the scene is encoded as a set of object vectors, experiments can directly translate obstacles, add or remove actors, rotate the ego frame, or modify the speed-limit token. This makes it possible to study causal response patterns rather than only aggregate benchmark scores (Gerstenecker et al., 10 Nov 2025).

In construction scenarios, perturbations show a lack of genuine spatial reasoning. When a construction obstacle is moved from the ego lane to the opposite lane, or to the right shoulder, the model still reacts as if its own lane were blocked. Conversely, when cones behind the construction sign are removed but the warning sign is retained, the model ignores the sign and behaves as though no obstacle exists. The paper interprets this as evidence that the model has learned a pattern tied to the canonical cone formation rather than a concept of “obstacle blocking ego lane.” The deeper claim is data-centric: low obstacle diversity in training permits shortcut strategies that suffice for benchmark success (Gerstenecker et al., 10 Nov 2025).

A related defect appears in trajectory generalization. When the ego vehicle is initialized progressively closer to a construction obstacle, the model fails to steepen its lane-change trajectory and instead repeats a familiar geometric template, producing collisions. Inspection of the expert data indicates that PDM-Lite generates highly stereotyped avoidance trajectories. This suggests memorization of a small family of canonical motions rather than continuous adaptation of the maneuver to distance and speed (Gerstenecker et al., 10 Nov 2025).

Safety margins also emerge as a hidden weakness. In obstacle-avoidance scenarios, PlanT 2.0 sometimes passes obstacles with too little clearance, especially when returning to the lane, causing side-swipes. Because the controller is identical to the expert’s controller, the study attributes this not to actuation but to brittle expert trajectories that run very close to obstacles. Small prediction errors are then enough to convert nominal imitation into collision (Gerstenecker et al., 10 Nov 2025).

The perturbation analysis further exposes positional shortcuts. In scenarios with obstructions and oncoming traffic, rotating the ego vehicle by roughly 10–15 degrees causes predicted speed to jump suddenly to normal driving values even though the oncoming lane remains unsafe. The proposed explanation is that, in the expert dataset, significant ego yaw reliably coincides with already-safe overtaking; the model thus learns “ego rotation implies road clear” instead of checking current object positions. The same structural logic appears in ParkingCutIn, where the model anticipates a parked car’s cut-in correctly at larger distances but, when moved too close to the vehicle, abruptly predicts acceleration. Because CARLA triggers the parked vehicle’s behavior partly as a function of ego velocity, early braking by the model can delay the trigger and create a state absent from training. The model’s anticipation thus becomes brittle under altered timing (Gerstenecker et al., 10 Nov 2025).

Waypoint-based planning introduces an additional representational issue. The spacing of waypoints encodes both intended geometry and assumed dynamics, so the model’s output implicitly bundles “where to go” with “how fast the vehicle can get there.” In a pedestrian crossing scenario, increasing the speed-limit token from 50 to 100 to 120 km/h shifts the predicted stopping point farther downstream. If the actual ego speed is low but the speed limit is high, this can yield counterintuitive or unsafe braking behavior. The paper notes that simply adding ego velocity as an input risks causal confusion, where the model uses action-derived signals improperly, rather than resolving the interface problem cleanly (Gerstenecker et al., 10 Nov 2025).

Two further limitations concern under-specified states and sparse environment encoding. Because the training data contains only successful expert drives, the model sees almost no near-crash or post-crash states; after collisions it may accelerate and attempt to continue, sometimes causing additional collisions. Meanwhile, in scenarios such as InvadingTurn, fences on the road edge are not represented as explicit objects and are only weakly visible in the coarse SD-map. PlanT 2.0 can therefore misjudge right-side free space and collide with fences while avoiding invading traffic. The paper treats this as a representational trade-off: sparse object tokens improve controllability, but fine-grained occupancy features remain hard to encode without exploding token count and quadratic attention cost (Gerstenecker et al., 10 Nov 2025).

These observations underpin a broader critique of closed-loop driving benchmarks. High Driving Score or Success Rate can coexist with shallow scene understanding, because deterministic experts, repetitive scenario templates, and metrics that reward route completion more than realism create incentives for exploitation of structural quirks rather than robust causal reasoning. PlanT 2.0’s diagnostic value lies precisely in making those shortcuts visible (Gerstenecker et al., 10 Nov 2025).

6. Data-centric implications, future directions, and name ambiguity

The paper’s response to these findings is explicitly data-centric. It argues for greater obstacle diversity; more varied expert behavior with multiple valid trajectories; scenario timing randomization, especially for trigger-based interactions such as ParkingCutIn; inclusion of near-crash and post-crash states with explicitly defined desired behavior; richer occupancy representations for fences, curbs, and barriers; and planning interfaces that separate intent from dynamics more cleanly. It also suggests that PlanT 2.0 can serve as a privileged reference system for interpreting failures in sensor-based driving models, even though direct transfer of all conclusions beyond CARLA remains to be established (Gerstenecker et al., 10 Nov 2025).

The name “PlanT 2.0” is not unique to autonomous driving in recent arXiv usage. In one unrelated line of work, it denotes a proposed extension of an AI-plus-IoT plant communication platform in which sensors, ThingSpeak, Firebase, Flutter, and a Gemini-based chatbot allow plants to “talk” to humans; the proposed PlanT 2.0 there adds richer sensing, multimodal AI, predictive analytics, and broader deployment scale (Agarwal et al., 2024). In another unrelated usage, the phrase appears as a natural label for a next-generation vegetation-recognition pipeline based on DINOv2 ViT-B/14, multi-head taxonomy-aware classification, multi-scale tiling, and dynamic thresholding for PlantCLEF-style quadrat analysis (Herasimchyk et al., 14 Aug 2025).

This suggests that “PlanT 2.0” currently functions less as a stable domain-unique term than as a reused label for second-generation plant- or planning-related systems. In the autonomous-driving literature, however, its established meaning is the object-centric CARLA planner that couples competitive benchmark performance with a systematic critique of dataset bias, expert rigidity, and evaluation-induced shortcut learning (Gerstenecker et al., 10 Nov 2025).