End-to-end physical-accuracy evaluation of downstream remote-sensing products

Evaluate the end-to-end physical accuracy of downstream remote-sensing science products, such as change-detection maps measured by Kappa, by replacing the Mock Execution Engine’s schema, dependency, and sensor-compatibility checks with execution on real imagery through GDAL, SNAP, or Google Earth Engine.

Background

The SimCRAFT framework uses a Mock Execution Engine to validate tool-call schemas, inter-tool dependencies, and sensor compatibility during synthetic trajectory generation. This validation occurs at the workflow-planning level and does not execute the generated workflows against real satellite imagery or assess the scientific validity of their outputs.

Consequently, the paper leaves unresolved whether trajectories accepted by the Mock Execution Engine produce physically accurate downstream products, including change-detection results. The authors identify real-software execution and product-level evaluation, such as Kappa-based assessment of change-detection maps, as future work.