AppEvalPilot: AI in Flight Training & GUI Testing

• AppEvalPilot is an ambiguous term denoting two distinct AI systems: one for flight training and one for automated GUI testing.
• The flight training framework uses behavioral cloning to learn an expert-like model that provides error-aware, formative feedback to student pilots.
• The GUI testing system generates adaptive test cases and evaluates software using structured judgment metrics, improving interactive software quality assessment.

AppEvalPilot is an ambiguous term in the arXiv literature. In one usage, it denotes an AI-based tutoring framework for simulator-based flight training that learns a “teacher” agent from expert pilot demonstrations and then compares student control actions against that learned reference to provide formative feedback during straight-and-level flight (Guevarra et al., 2022). In a later and unrelated usage, it denotes an automated, agent-as-a-judge GUI testing system within the RealDevWorld software-evaluation framework, where it derives test cases from requirements, interacts with generated applications through their interfaces, and judges functional correctness, visual fidelity, and runtime behavior (Bian et al., 17 Aug 2025). The two systems share a name but address different technical problems, use different methodological stacks, and are evaluated under different criteria.

1. Terminological scope and disambiguation

The name “AppEvalPilot” refers to two distinct research artifacts rather than a single continuously developed framework. One belongs to AI-assisted aviation training; the other belongs to automated evaluation of production-ready software.

Usage	Domain	Core function
AppEvalPilot (Guevarra et al., 2022)	Flight training	Learn an expert-like “teacher” agent and provide formative feedback to student pilots
AppEvalPilot (Bian et al., 17 Aug 2025)	GUI software evaluation	Generate tests, interact with applications through the GUI, and judge whether requirements are satisfied

This dual usage matters bibliographically and conceptually. In the aviation paper, the central object is a learned simulator-resident tutor intended to reduce reliance on scarce human flight instructors by supporting pilot-training throughput. In the RealDevWorld paper, the central object is an automated judge for open-ended repository-level software tasks, introduced to address failures of static checks and brittle pass/fail scripts in evaluating real applications (Guevarra et al., 2022).

2. Flight-training AppEvalPilot: problem setting and learned teacher model

In “Augmenting Flight Training with AI to Efficiently Train Pilots,” AppEvalPilot is presented as an AI-based tutoring framework for flight training inside a simulator environment using X-Plane (Guevarra et al., 2022). The motivating problem is framed explicitly as one of pilot-training throughput: commercial pilot demand is high, but qualified instructors are limited, so an automated assistant that can observe student behavior, detect mistakes, and provide formative feedback could allow instructors to supervise more trainees efficiently.

The prototype is intentionally narrow and operates as a proof of concept. It focuses on the canonical “straight and level” maneuver, described as foundational for other maneuvers because it requires maintaining constant heading and altitude. Both expert instructors and students perform the maneuver in the simulator under visual flight rules and clear weather. For demonstration collection, expert pilots flew straight and level for about 12.5 minutes total, producing 25 trials of 30 seconds each. To increase diversity, the target heading was randomized between approximately $\pm 30^\circ$ from the starting heading across trials.

The learned “teacher” is trained as a supervised imitation-learning policy by behavioral cloning. Given simulator state observations $s_i$, the policy $\pi_\theta$ is optimized to imitate expert actions $\pi_e(s_i)$ by minimizing

$$L(\theta)=\sum_{i=1}^{N}||\pi_{\theta}(s_i)-\pi_e(s_i)||^2,$$

where $N$ is the number of state-action pairs in the demonstration set. In the instantiated system, the policy outputs the aircraft yoke controls corresponding to pitch and roll, and the implementation was trained using Stable Baselines. Evaluation is performed on 10 held-out trials with randomized headings, measuring average heading error over time; training is stopped when this error stops improving. The reported evaluation plot indicates that the learned controller becomes good enough to generalize to unseen starts and targets for this simple maneuver (Guevarra et al., 2022).

3. Flight-training AppEvalPilot: error detection and formative feedback

Once trained, the teacher policy is used not merely to mimic expert flying but to serve as a model of correct control behavior against which student actions can be compared (Guevarra et al., 2022). The tutoring logic is explicitly based on policy disagreement: if a student’s control choices diverge significantly from what the expert-like agent would do in the same situation, the system flags an error and provides feedback.

To construct this layer, the authors recorded poorly performed student flights and asked pilots to annotate critiques of the observed errors. From these critiques, two main error categories were identified for straight-and-level flight: failure to keep altitude and airspeed constant, and overshooting the target. Comparison between student and teacher is performed directly on control outputs. Let $p_a(t), r_a(t)$ denote the agent’s pitch and roll at time $t$, and $p_s(t), r_s(t)$ the student’s pitch and roll. The paper defines threshold-based agreement tests:

$|p_a(t)-p_s(t)| \ge D_1$

for the first error type, and

$s_i$0

for the second, where $s_i$1 and $s_i$2 are user-defined nonnegative thresholds. If the student’s pitch choice differs too much from the teacher’s, the system interprets this as a problem with maintaining altitude or airspeed; if the roll choice differs too much, it interprets this as a likely heading-control or overshoot issue. These are simple distance-based error detectors and do not explicitly infer latent intent; disagreement with the expert policy is used as a proxy for a likely mistake.

The feedback mechanism is visual and formative. The paper describes the design as an instance of “informative tutoring,” meaning it provides verification feedback, error flagging, and strategic hints (Guevarra et al., 2022). When a threshold is exceeded, the system displays a black square containing two lines representing the student and agent statuses. The position $s_i$3 of each line corresponds to $s_i$4 and $s_i$5, and the line slope encodes the roll angle. The intent is to expose the degree of mismatch between the student’s maneuver and the expert-like reference trajectory so that the learner can visually infer that the objective is to make the two line representations overlap.

4. Flight-training AppEvalPilot: empirical status, claimed benefits, and limitations

The aviation instantiation is explicitly small-scale and exploratory (Guevarra et al., 2022). The reported evidence is that a behavioral-cloning teacher can learn a usable straight-and-level policy from 25 demonstration trials and that the resulting teacher can be used to compare against student control actions. The paper does not report detailed learning curves beyond the average heading-error plot, does not present a controlled user study quantifying learning gains, retention, or transfer, and does not include ablations comparing alternative policies, thresholds, or feedback styles. The authors explicitly note that evaluating the effectiveness of the feedback is future work.

The main benefits claimed are reduced instructor workload and the potential to scale pilot training. The main limitations are equally explicit: the system is demonstrated only on a single basic maneuver, relies on relatively simple threshold-based comparison rather than richer reasoning, and has not yet been validated in a full student-learning study (Guevarra et al., 2022). This suggests that the contribution is primarily architectural and proof-of-concept oriented: it shows how an expert-like control policy can be repurposed as a reference model for error-aware tutoring, but it does not yet establish broad pedagogical efficacy.

5. RealDevWorld AppEvalPilot: agent-as-a-judge GUI testing

In “You Don’t Know Until You Click: Automated GUI Testing for Production-Ready Software Evaluation,” AppEvalPilot is an automated, agent-as-a-judge GUI testing system inside the RealDevWorld framework (Bian et al., 17 Aug 2025). RealDevBench supplies the tasks, and AppEvalPilot serves as the judge: it generates tests from the task requirements, interacts with the built application through its GUI like a human tester, and then decides whether the software satisfies the intended features.

The motivation is that prior evaluation methods are too shallow for production-ready applications. The paper argues that function-level benchmarks and repository-level benchmarks often rely on static checks, code similarity, unit tests, or scripted integration tests, and that these methods miss interactive behavior, UI flows, runtime state changes, multimodal inputs, and failures that only appear after clicking, scrolling, typing, or navigating (Bian et al., 17 Aug 2025). AppEvalPilot is introduced to close this “you don’t know until you click” gap.

Mechanistically, the system follows a three-stage pipeline. First, it generates task-specific test cases from the requirement description and feature list, using few-shot learning and domain knowledge to infer requirement-to-test mappings, with a professional test-engineer style prompt and a cap of about 15–20 cases. Second, it executes those test cases by simulating realistic GUI-based user behavior. It interacts directly with the running application through the interface, using both visual information and accessibility-tree text. The paper describes a hierarchical action space with four atomic actions: Open(app), Run(code), Tell(answer), and Stop. Run performs scripted GUI manipulation through PyAutoGUI to simulate mouse and keyboard input, while execution is adaptive rather than purely scripted through a Plan-and-Solve style loop using historical reasoning and model-based planning.

Third, AppEvalPilot evaluates the observed outcome against the intended behavior. For each test case it produces a structured report of actions and outcomes, then classifies the case as Pass, Fail, or Uncertain. The paper explicitly frames the assessment as covering functional correctness, visual fidelity, and runtime behavior (Bian et al., 17 Aug 2025). RealDevBench, the benchmark AppEvalPilot is designed to evaluate, contains 194 open-ended repository-level tasks across four domains—Display, Analysis, Data, and Game—with multimodal materials such as images, audio, documents, and structured datasets.

The paper formalizes the evaluation workflow as a pipeline:

$s_i$6

and notes that the equation appears typeset imperfectly in the source, while the conceptual structure remains test identification, execution, and judgment.

6. RealDevWorld AppEvalPilot: judgment formalization, results, and failure modes

The judgment logic in the RealDevWorld paper converts true/false/uncertain labels into binary scores:

$s_i$7

and defines feature-level human ground truth quality as

$s_i$8

where $s_i$9 and $\pi_\theta$0 is the number of features (Bian et al., 17 Aug 2025). The paper uses accuracy for judgment correctness and Pearson correlation to measure alignment with human labels at both test-case and feature levels.

On the 49-task validation subset, the paper reports AppEvalPilot(Claude) reaching feature-level quality 0.73, feature-level alignment 0.85, test-case-level quality 0.74, test-case-level alignment 0.81, and accuracy 0.92 (Bian et al., 17 Aug 2025). It is reported to outperform GUI model baselines such as Claude-3.5-Sonnet and UI-Tars, and GUI-agent frameworks such as WebVoyager and Browser-Use. The comparison given in the paper includes Browser-Use with Claude at feature-level alignment 0.58 and test-case-level alignment 0.61, versus 0.85 and 0.81 for AppEvalPilot. The paper also reports 9.0 minutes per app and cost 0.26, compared with Browser-Use’s 13.50 minutes and 1.13 cost units, and a distribution overlap rate of 0.96 with human scores, versus 0.75 for code-quality scoring and 0.55 for visual-quality scoring. It further states that AppEvalPilot-generated test cases improve downstream GUI testing performance for baseline systems by an average of 0.17.

The paper’s failure analysis identifies several limitations: the agent can hallucinate when evidence is incomplete, struggle with low-quality or misaligned test cases, require advanced reasoning and real-time responsiveness, and misinterpret what “pass” should mean when the true requirement is more than mere element presence (Bian et al., 17 Aug 2025). The system is therefore not presented as a perfect replacement for human testing, but as a scalable, production-oriented approximation that captures the interaction-dependent nature of real software more faithfully than static evaluation.

A plausible implication is that the two AppEvalPilot systems are best understood as homonymous frameworks rather than variants of one research program. One uses behavioral cloning and threshold-based disagreement to support simulator-based flight instruction; the other uses adaptive GUI interaction and structured judgment to evaluate production-ready software. The shared label can therefore obscure substantial differences in objective, ontology, and evaluation protocol.