- The paper introduces FRAMe, an end-to-end eVTOL flight-planning system that combines an LLM planner with geometry-filtered RAG memory and a multimodal coach agent for validation and preference alignment.
- The full system achieved validity rates of 88.9%–93.8% for three planners, while model-specific gains included 12% fewer waypoints for o3-mini and 9% greater clearance for o4-mini.
- The results show that FRAMe outperforms A* on clearance and waypoint efficiency with a small distance tradeoff, but retrieval and coaching benefits depend on the model, metric headroom, and reliable human validation.
- What design choices make geometry-constrained retrieval safer than unrestricted semantic RAG for flight planning?
- How do LLM planner capabilities and prompting strategies affect validity, clearance, waypoint count, and distance?
- How could human feedback, dynamic obstacles, weather, energy constraints, and air-traffic rules be integrated into the FRAMe architecture?
- What evaluation methods can reduce bias when a multimodal LLM coach judges operator preference alignment?
- Find recent papers about LLM-based autonomous flight planning.
Motivation and problem setting
Classical path planners such as A*, RRT, and RRT* guarantee obstacle-free trajectories but require explicitly specified mathematical objectives, which makes them poorly suited to capturing subjective, context-dependent operator intent in Advanced Air Mobility (AAM) operations. The paper addresses this gap with FRAMe, an end-to-end flight planning system in which a planner LLM converts natural-language mission prompts into eVTOL waypoint sequences, while two auxiliary modules—a retrieval-augmented generation (RAG) memory and a multi-modal coach agent—improve validity and preference alignment. The work extends the authors' prior line of research on LLM-based route selection and chain-of-thought flight planning, and positions itself against related efforts including SayCan-style grounding, LLM-A*/LLM+P hybrid planners, retrieval-augmented embodied planning (P-RAG, STRAP), and UAV-specific systems such as TypeFly and GSCE.
System architecture
FRAMe takes as input a KML-encoded scenario (origin, destination, flyable airspace, polygonal no-fly zones) plus an operator preference expressed in natural language. The planner agent converts scenario placemarks into natural-language messages, applies one of several prompting strategies (the authors report that a customized zero-shot chain-of-thought variant performs best), optionally augments the prompt with retrieved plans, and emits waypoints together with natural-language reasoning for explainability.
The multi-modal coach agent, implemented with o4-mini, performs a three-stage review: (1) rule-based geometric checks of validity—waypoint containment in the flyzone, correct origin/destination connectivity, and no segment intersecting a no-fly zone; (2) a vision-based assessment of preference alignment on a rendered image of the plan; and (3) optional free-form operator feedback. The resulting record is inserted into the vector database.
The RAG module embeds the operator's preference text and retrieves the K=2 most similar past plans by cosine distance over preference embeddings. A notable design decision is that retrieval candidates are restricted to records with identical scenario geometry (same flyzone, polygons, origin, destination); the embedding similarity only ranks within that set. Geometric relevance is therefore guaranteed by exact matching rather than inferred from embedding distance, which avoids a common failure mode of semantic retrieval over spatial problems.
Evaluation protocol
Experiments use Dallas–Fort Worth scenarios at three difficulty levels (2, 4, and 7 no-fly polygons), four planner LLMs (o3-mini, o4-mini, DeepSeek-R1, GPT-5.4), and four conditions: A*, Baseline (LLM alone), +RAG, and +RAG+Coach. Preference alignment is evaluated without human annotators via three geometrically computable objectives: minimize flight distance, minimize waypoint count, and maximize polygon clearance. A warmup phase (150 runs with neutral preferences) seeds the database; the ablation phase then fixes it read-only, isolating the marginal contribution of each component.
Results
Validity. The full FRAMe configuration achieves the highest validity for every planner, reaching 88.9% (o3-mini), 90.7% (o4-mini), 93.8% (DeepSeek-R1), and 55.6% (GPT-5.4). Two findings stand out. First, for o3-mini, +RAG alone slightly regresses validity (79.6% → 79.1%), implying that unvalidated retrieval can surface misleading neighbors; the coach is what recovers and extends the gain, so coach review is load-bearing rather than redundant. Second, GPT-5.4 lags substantially across all conditions despite augmentation, which the authors attribute to weaker instruction-following under geometric constraints.
Preference capture. Preference effects are model-specific and headroom-bounded. o3-mini reduces mean waypoint count from 4.06 to 3.58 (−12%) under the full system; o4-mini increases minimum clearance from 5.58 km to 6.07 km (+9%); DeepSeek-R1 peaks on clearance at +RAG (7.50 km) but retreats slightly with the coach added, suggesting retrieval alone is the stronger signal for its spatial reasoning. Notably, o3-mini's clearance declines monotonically with augmentation (5.88 → 5.35 → 4.52 km), because coach-driven revisions prioritize fixing validity violations over maximizing separation. The distance metric barely moves for any model, since all planners already operate within roughly 6 km of the ~190 km geodesic at Baseline.
A* comparison. A* achieves 100% validity by construction but is preference-blind: its mean clearance (1.91 km) is well below every LLM condition (3.9–5.0 km), and its waypoint count (18.2) is 3–5× higher because it tracks grid discretization rather than operator intent. FRAMe outperforms A* on clearance and waypoint preferences while accepting a small distance overhead (roughly +5–7 km).
Per-difficulty analysis shows the difficulty–validity relationship is model-specific: DeepSeek-R1 peaks on Easy/Medium (98.7%/97.3%) but drops to 85.3% on Hard, whereas o4-mini peaks on Hard (96.0%), indicating richer geometric structure anchors its decisions rather than overwhelming them.
Limitations
The paper is explicit about several constraints. The preference-capture evidence is strongest for clearance (and waypoints for planners starting above the floor) and weakest for distance, a consequence of evaluation geometry rather than framework design. Aggregate metrics averaged across preferences can move non-target metrics unfavorably—for example, aggregate clearance declines for o3-mini and GPT-5.4 under +RAG+Coach when clearance was not requested—so aggregate columns should be read as side-effect summaries. The coach's alignment verdict inherits known LLM-as-judge biases (position, verbosity, self-preference); validity is grounded in rule-based checks, but no human-rater study has been conducted. All experiments are simulation-only with synthetic static polygons; dynamic traffic, energy limits, vehicle dynamics, weather, and live flight tests remain unaddressed, though a subscale-UAV web prototype integrated with Mission Planner is presented as an intermediate validation step.
Conclusion
FRAMe demonstrates that coupling an LLM planner with a geometry-guaranteed RAG memory and a multi-modal coach yields the highest plan validity across four state-of-the-art models, with the coach providing a non-redundant lift over retrieval alone—including recovering validity where retrieval regresses it. Preference capture is real but conditional on metric headroom and model identity, and the system trades a small distance overhead relative to A* for substantially better clearance and waypoint economy. The open questions left by the paper concern human validation of the coach's alignment judgments, extension to dynamic constraints and real airspace traffic, and confirmation through live flight tests.