- The paper introduces an open-source, 300-question benchmark covering airport ground operations, ICAO and FAA regulations, aviation knowledge, and operational scenarios, validated by aviation practitioners.
- The strongest evaluated model, GPT-5.5, achieved 82.7% accuracy versus an informal expert reference of approximately 95%, while open-weight Qwen3.5 reached 77.3%, showing progress but persistent reliability gaps.
- The benchmark exposes uneven performance and evaluation risks, including higher failure rates on FAA regulations, reasoning-heavy tasks, possible data contamination, and answer-key errors that models helped identify.
Pre-Flight is an open source benchmark of 300 multiple choice questions designed to measure LLM competence in aviation operational knowledge, a domain the authors argue is poorly served by general purpose evaluation. The benchmark covers international airport ground operations, ICAO and US FAA regulations, aviation general knowledge, and complex operational scenarios, and was authored and reviewed by practitioners with backgrounds in air traffic management, ground operations, and commercial flying (2607.01829). The central empirical finding is that even the strongest model evaluated reaches 82.7% accuracy against an informal expert reference of roughly 95%, leaving a persistent gap below expert level reliability. The authors position domain specific evaluation of this kind as a necessary precondition for responsible deployment of generative AI in non safety critical aviation operations.
Motivation and positioning
Aviation business operations span network planning, airport operations, asset management, and airspace design, and are characterized by heterogeneous data: legacy formats such as NOTAMs and Type B messages over AFTN coexist with modern geospatial XML schemas, and much operationally relevant knowledge remains unstructured text. Despite two decades of rationalization efforts such as ICAO's System Wide Information Management (SWIM), migration is slow, and the combination of structured and unstructured information is precisely where generative AI is proposed. The authors restrict scope to non safety critical business operations, consistent with the FAA's Roadmap for Artificial Intelligence Safety Assurance.
The paper situates Pre-Flight against general benchmarks such as MMLU, which contain little aviation operational content, and against established domain benchmarks in medicine (MedQA) and law (LegalBench). Two recent aerospace efforts are identified as complementary: ALUE, from the FAA and MITRE, targets systemic air traffic management language understanding with a roadmap toward multimodal and retrieval grounded tasks, while PilotBench evaluates LLMs as agents on physics governed flight trajectory and attitude prediction. Pre-Flight occupies a distinct layer: declarative operational and regulatory knowledge, delivered as a fixed, openly released dataset with a rolling leaderboard distributed in the inspect_evals package.
A recurring concern motivating the design is that strong aggregate scores can mask shallow competence; the calibration gap between expressed confidence and accuracy, documented in factuality work, is a specific liability in a domain whose safety culture depends on precise understanding of system limitations.
Benchmark construction and composition
Each item presents a question with four or five answer choices and exactly one correct answer, stored in JSONL with a unique identifier, stem, options, and target. Sources include airport Ground Operations Safety Manuals (GOSM), 14 CFR, ICAO Annexes, and general aviation documentation. The composition is deliberately weighted toward ground operations:
| Category |
Items |
Share |
| International airport ground operations |
152 |
50.7% |
| ICAO rules and regulations |
85 |
28.3% |
| FAA rules and regulations |
51 |
17.0% |
| Aviation trivia |
8 |
2.7% |
| Complex ground scenarios |
4 |
1.3% |
The two smallest categories rest on very few items and are explicitly described as indicative. The current public release is an "easy" tier under the MIT licence on Hugging Face; a harder tier is deliberately withheld to preserve discriminative power as models improve. Validation was expert based and source grounded but partial: not every item received independent second review, and no formal interannotator agreement statistic was computed.
Experimental setup
Evaluation uses the Inspect framework with the standard multiple choice solver and exact match scoring; no LLM judge is used. Prompting is zero shot with default sampling parameters, one epoch over the 300 item set, giving a binomial standard error of approximately 0.025. Models are evaluated on a rolling basis as released, and results are reported as a snapshot of 29 June 2026 spanning 44 models from early 2024 to mid 2026. Thirteen earlier models are shown at their original scores and have not been rerun on the current dataset revision, so the frontier line mixes dataset revisions before mid 2025 and is indicative only in that region.
Results
Three findings stand out. First, the gap to expert performance is large and persistent: GPT-5.5 leads at 82.7%, roughly twelve points below the informal expert reference. Second, progress has been gradual: the frontier rose from the mid 60s to the low 70s through 2024, then added only about eight points across 2025 and into 2026. Third, open weight models are competitive at the frontier, with a locally run quantised Qwen3.5 122B variant at 77.3%, ahead of GPT-5.1 and behind only the four leading API systems. A striking outlier is ALLaM 2 7B at 25.3%, at or near chance, demonstrating that a model strong in other settings can collapse entirely on this domain.
Selected results from the snapshot:
| Model |
Organisation |
Accuracy |
| GPT-5.5 |
OpenAI |
82.7% |
| GPT-5 |
OpenAI |
80.3% |
| Claude Opus 4.8 |
Anthropic |
79.0% |
| Gemini 2.5 Pro |
Google |
79.0% |
| Qwen3.5 122B-A10B int4 (local) |
Alibaba |
77.3% |
| GPT-OSS 20B |
OpenAI |
63.7% |
| Llama 3.1 8B |
Meta AI |
58.0% |
| ALLaM 2 7B |
SDAIA |
25.3% |
Item level analysis over the 31 model cohort scored on the current revision shows uneven difficulty: 9 items (3.0%) were failed by every model, and the micro averaged failure rate is 28.4%. US (FAA) regulations are by far the hardest category at 53.0% mean failure, with eight of the nine all-fail items being US regulation items. The authors attribute the relative ease of international material to the sector's open documentation culture placing it in training data, which also helps explain why even small, older models score above 70%. Separating recall from reasoning, 60.3% of items require reasoning, and models fare somewhat worse on these (30.1% versus 25.9% failure), supporting the claim that aggregate knowledge scores overstate operational competence.
An important caveat qualifies the FAA result: inspection of the most failed items revealed answer key errors rather than model failures. On items such as the definition of "operational control" and the aircraft dispatcher's duty to brief the pilot in command, models converged almost unanimously on options matching the governing regulation more closely than the recorded key. Two keys were corrected; several remaining items show the same signature and are flagged for expert revalidation. The FAA failure rate should therefore be read as a modest upper bound. The authors note a useful methodological byproduct: strong, consistent disagreement between many independent models and a key is an efficient way to surface benchmark errors.
Qualitative failure modes
Three illustrative probes, not part of the scored dataset, characterize failure patterns beyond multiple choice accuracy. On a simplified river crossing puzzle, several models produced elaborate but incorrect solutions by pattern matching against the familiar training data version rather than reasoning from the stated problem; reasoning oriented models that review intermediate steps largely avoided this. On a snow clearance scheduling task with physical and temporal constraints, models managed basic scheduling but sometimes selected orderings that violated stand availability. On a 1970s Aeroflot route map, models reliably identified the hub but struggled with day of week schedule interpretation and route optimization, suggesting vision LLMs can extract structure from dense diagrams but fall short of reliable multistep interpretation.
Limitations
The benchmark measures recognition rather than open ended generation or operational judgement, and high accuracy here should not be read as fitness for deployment. Training data contamination cannot be excluded for the public tier; the strong performance of small, older models on widely published material is consistent with some contamination, and the withheld harder tier is intended to mitigate this. The expert reference is informal, deriving from a small, self selected sample of conference attendees under uncontrolled conditions, and should be read as indicative rather than a validated human baseline. Category sizes are markedly uneven, making per category rates for the smallest categories noisy. Answer key quality is a further limitation, with the FAA failure rate an upper bound pending revalidation. A small number of items reference an external figure not reproduced in the text only prompt; these are retained deliberately as extreme retrieval items requiring the model to reconstruct the figure's content from context. The qualitative probes are small in number and should not be generalized quantitatively.
Conclusion
Pre-Flight provides an open, reproducible measure of aviation operational knowledge and demonstrates that even the strongest current models leave a clear gap below the reliability expected of domain experts. The rolling leaderboard design, distribution through inspect_evals, and planned harder tier give the benchmark a mechanism for remaining discriminative as models improve. Open questions the work leaves include whether the persistent gap reflects genuine gaps in regulatory knowledge or in reasoning over it, how the harder tier resolves the contamination question, and whether model surfaced answer key errors can be systematically incorporated into benchmark maintenance.