PilotBench: A Benchmark for General Aviation Agents with Safety Constraints

Abstract: As LLMs advance toward embodied AI agents operating in physical environments, a fundamental question emerges: can models trained on text corpora reliably reason about complex physics while adhering to safety constraints? We address this through PilotBench, a benchmark evaluating LLMs on safety-critical flight trajectory and attitude prediction. Built from 708 real-world general aviation trajectories spanning nine operationally distinct flight phases with synchronized 34-channel telemetry, PilotBench systematically probes the intersection of semantic understanding and physics-governed prediction through comparative analysis of LLMs and traditional forecasters. We introduce Pilot-Score, a composite metric balancing 60% regression accuracy with 40% instruction adherence and safety compliance. Comparative evaluation across 41 models uncovers a Precision-Controllability Dichotomy: traditional forecasters achieve superior MAE of 7.01 but lack semantic reasoning capabilities, while LLMs gain controllability with 86--89% instruction-following at the cost of 11--14 MAE precision. Phase-stratified analysis further exposes a Dynamic Complexity Gap-LLM performance degrades sharply in high-workload phases such as Climb and Approach, suggesting brittle implicit physics models. These empirical discoveries motivate hybrid architectures combining LLMs' symbolic reasoning with specialized forecasters' numerical precision. PilotBench provides a rigorous foundation for advancing embodied AI in safety-constrained domains.

• The paper introduces PilotBench, a phase-annotated benchmark of 708 flight segments that jointly evaluates numerical prediction, instruction following, and certified safety compliance.
• Qwen3-32B achieves the strongest overall result with a 9.54 MAE and 90.23 Pilot-Score, while traditional FlightPatchNet records the best MAE at 7.01 but cannot interpret natural-language instructions.
• LLM errors rise sharply during high-dynamic Climb and Approach phases, and Physics-CoT reduces safety violations by about 25%, supporting hybrid systems that combine language models with specialized flight forecasters.

Motivation and problem statement

PilotBench addresses a specific gap in the evaluation of LLMs as embodied agents: whether models trained on text corpora can perform physics-governed prediction while respecting certified safety constraints. Existing aviation benchmarks emphasize regression accuracy alone, and general reasoning benchmarks such as AGIEval lack physical grounding. The authors position general aviation as a testbed where trajectory prediction requires simultaneous numerical precision, phase-aware reasoning, and compliance with flight envelopes, and they construct a benchmark that measures all three jointly.

Benchmark construction

The corpus comprises 708 trajectory segments collected over a ten-day campaign: 38 hours 18 minutes of synchronized sensor data from 31 VFR circuits on a DA40 and 22 dual-instruction sorties on a C172N. Each aircraft carried dual-frequency RTK-GNSS, an air-data computer, and an inertial reference unit at 20 Hz; channels were aligned to GNSS 1-PPS, resampled to 10 Hz, and consolidated into 34 standardized variables. Quality control discarded frames with HDOP above 2 or inertial residuals beyond $2.5\sigma$ (removing 1.7% of the corpus), and mutual-information leakage was kept below 0.7%.

Flights are segmented into nine phases derived from the standard traffic pattern—five straight segments (P1–P5) and four transitions (T1–T4)—with boundaries defined by heading change rate, roll angle, and vertical speed. Phase annotation is notable given that 45% of labels were missing; rule-based seeding with contrastive boundary refinement and expert vetting achieved Cohen's kappa of 0.93, indicating strong inter-annotator agreement. An avionics engineer and former LOSA auditor reviewed all segments for safety compliance.

Pilot-Score

The composite metric $\mathcal{F}$ weights regression accuracy at 60% (itself split 60/40 between MAE- and RMSE-based sub-scores via piecewise linear scoring) against instruction adherence at 40%, decomposed into Field Completeness (0.5), Field Validity (0.3), and Format (0.2). Scores are bounded in $[5,100]$ with a floor of 5 to prevent total-failure degeneracy. A distinctive design choice is alignment with Required Navigation Performance standards: $\mathcal{F} \geq 90$ maps to RNP 0.1, $\geq 70$ to RNP 0.3–1.0, and $\geq 50$ to RNP 2–4. This gives the score operational meaning rather than treating it as an arbitrary weighted average, though the mapping itself is a calibration assumption inherited from FAA guidance rather than an empirically validated equivalence.

Main results across 41 models

Evaluation covered 41 LLMs (Qwen, DeepSeek, GLM, InternLM, GPT, Doubao families) under deterministic decoding on eight A100 GPUs. Representative results:

Model	MAE	Instruction Following	Pilot-Score
Qwen3-32B	9.54	97.43	90.23
Qwen2.5-72B-Instruct	11.91	96.64	88.58
DeepSeek-V3	11.94	96.00	88.31
QwQ-32B	9.72	83.95	84.90
Qwen2.5-7B-Instruct	35.53	95.32	71.11
GLM-4-9B-Chat	483.79	70.21	31.08

Qwen3-32B achieves the best LLM MAE (9.54) and the top Pilot-Score (90.23). Two findings stand out. First, scaling helps non-linearly: architecture and training matter more than parameter count, since Qwen2.5-72B does not dominate smaller reasoning-tuned models. Second, syntactic and semantic competence are decoupled in instructive ways—GLM-4-9B-Chat maintains near-perfect format compliance but attains only 0.7% field validity, while QVQ-72B-Preview is numerically accurate yet frequently refuses instructions (instruction-following of 2.7%), rendering it operationally unusable despite its regression quality.

The Precision-Controllability Dichotomy

A nine-configuration ablation comparing three traditional forecasters (FlightPatchNet, DLinear, PatchTST) against six LLM variants quantifies the paper's central trade-off. FlightPatchNet achieves MAE of 7.01—the best in the study—but cannot process natural-language instructions, making it inert in human-on-the-loop settings. LLMs occupy MAE 11.28–13.59 while achieving 86–89% instruction adherence. Qwen3-32B with Physics-CoT prompting offers the best compromise (Pilot-Score 69.21, violation rate below 0.8%, IF = 88.9).

The prompting ablation further shows that accuracy and safety improve through distinct mechanisms: in-context learning acts primarily as a syntax stabilizer (e.g., −1.19 MAE for GPT-4o), whereas Physics-CoT functions as a semantic constraint injector, reducing violation rates by roughly 25%—a safety gain ICL alone cannot deliver. This implies that reliable safety-critical behavior requires explicit physical-feasibility reasoning before output generation, not merely exemplar conditioning.

The Dynamic Complexity Gap

Phase-stratified analysis reveals that all models perform robustly in low-dynamic phases such as Cruise and Descent (MAE 7–10), but LLM error spikes above MAE 13 in Climb and Approach—phases with coupled changes in altitude, airspeed, and configuration—while FlightPatchNet remains stable below MAE 8. The authors interpret this as evidence that LLMs' implicit physics models are brittle under coupled nonlinear dynamics, adequate for steady-state extrapolation but fragile during aggressive maneuvering. This is arguably the most consequential empirical result, because it identifies precisely where LLM-based aviation agents would fail in deployment.

Limitations and open questions

Several caveats bear directly on the generality of these conclusions. The corpus derives exclusively from VFR traffic-pattern circuits on two aircraft types over ten days, so phase coverage is narrow and conditions are benign; performance under IMC, crosswind extremes, or emergency procedures is untested. The one-step prediction task sidesteps multi-horizon error accumulation, which is where autoregressive drift would likely compound the Dynamic Complexity Gap. The RNP mapping of Pilot-Score is asserted rather than validated against operational certification data, and the 60/40 weighting, while motivated by prior aviation safety assessments, remains a design choice. Finally, the hybrid architecture the results motivate—LLMs for intent understanding, specialized forecasters for high-frequency control—is proposed but not implemented or evaluated here, leaving open how such a system would arbitrate between semantic and numerical modules online.

Conclusion

PilotBench provides a rigorously curated, phase-annotated benchmark with a safety-aligned composite metric, and its evaluation of 41 models yields two durable empirical findings: a Precision-Controllability Dichotomy separating traditional forecasters from instruction-following LLMs, and a Dynamic Complexity Gap showing that LLM physics reasoning degrades sharply in high-workload flight phases. The evidence supports hybrid designs in which LLMs handle intent and constraint interpretation while specialized forecasters supply numerical fidelity, though validating such architectures—and extending coverage beyond benign VFR circuits—remains open work.