Checked Program Recovery from Execution Video: A Sound Oracle for Untrusted Generators

Abstract: A growing class of tools recovers a program from observations of its behavior using an untrusted generator, a neural model or a search, that proposes candidates with no correctness guarantee. We study how to make such recovery trustworthy, in the concrete setting of recovering a runnable Scratch program from a recording of its execution. The recording shows what the program does but never its code; many programs produce the same video, so the source cannot be recovered, and the right target is a program that behaves the same as far as the camera can tell, made precise with a lens. The core is a two-tier validation oracle with a deliberate verdict asymmetry. A static checker proves lens-equivalence to a reference and issues a certificate that, granting the partial-order independence quotient adequate, never accepts a wrong program; a renderer can only refute or witness finite agreement, never certify. Around it, Vid2Prog reads each sprite's motion, visibility, and timing from the video and a known-asset manifest and synthesizes a candidate source-free; a closed loop renders and runs recovery again for ground truth. Under the exact lens the oracle makes no false accept on 246 labeled differing pairs, including an adversarial battery built to trap its concurrency quotient; on inputs outside the vocabulary and on real projects it abstains or refutes, accepting none we test. In-vocabulary recoveries reproduce their source frame for frame and 80% earn a static certificate, while whole real projects, mostly outside the vocabulary, recover at 14%, a vocabulary-bound rate the system never inflates with a wrong answer. A frontier vision-LLM recovers none of the matched programs single-shot, which oracle-in-the-loop repair lifts only to a few while the structured pipeline recovers all, the gap a sound checker makes for an untrusted generator.

• The paper introduces a formal framework that recovers lens-equivalent Scratch programs from execution videos using a two-tier validation oracle and description-length prior.
• It integrates a neuro-symbolic pipeline, Vid2Prog, that combines pixel-level perception with program synthesis to extract executable representations from noisy video inputs.
• The evaluation demonstrates robust performance with zero false positives and high recovery rates across diverse execution scenarios and adversarial tests.

Checked Program Recovery from Execution Video: A Sound Oracle for Untrusted Generators

Problem Statement and Formalization

This paper formulates the problem of extracting executable Scratch programs from videos of their execution, where source code is not directly observable. The challenge arises from two core issues: (a) the generator that proposes candidate programs—be it neural or search-based—has no correctness guarantee, and (b) the observation (the execution video) is a lossy, partial function of the program's semantics. Multiple programs can be observationally indistinguishable under the "camera lens," which only captures rendered, user-visible behavior.

The authors introduce a formalization rooted in the concept of observation lenses, parameterizing which aspects of execution (sprite position, visibility, etc.) are visible. They rigorously characterize the non-injective nature of the observation map and provide constructive families of distinct programs collapsed under the camera lens. This formalization underpins their treatment of program equivalence: rather than targeting recovery of the original source, the objective becomes the identification (and, for evaluation, certification) of a program that is lens-equivalent to the lost original, across all schedules, inputs, and random seeds.

They further introduce a description-length prior, selecting the class representative with minimal program length, penalizing camera-invisible structure, establishing uniqueness and the lower bound on recoverability for each lens.

The Two-Tier Validation Oracle

Central to the trustworthiness of recovered programs is the two-tier validation oracle, enforcing a soundness discipline. The first (static) tier operates by static checking of lens equivalence between candidate and reference, based on a canonical intermediate representation, feature multiset abstraction, and partial-order normalization for concurrency. Only this tier can positively certify equivalence. The authors prove a static soundness guarantee relative to the adequacy of the partial-order quotient for the compiled semantics, ensuring that a positive verdict never admits a false equivalent.

The second (render) tier performs deterministic re-execution and direct comparison with the video, identifying divergences by schedule and input. It can only refute or witness finite agreements, never certify equivalence, thereby enforcing an asymmetric verdict discipline. This renders overfit answers (e.g., pose tables mimicking observations) non-certifiable and guards against acceptance on the basis of sampled (finite) execution only.

Orthogonally, the oracle can abstain, which, together with the soundness of certification and the inability of the render witness to certify, provides a robust, compositional structure for downstream trust.

Vid2Prog: Neuro-Symbolic Recovery Pipeline

The implemented pipeline (Vid2Prog) integrates pixel-level perception, neuro-symbolic program synthesis, and oracle validation. Given an execution video and an asset manifest (sprites, costumes, backdrop), perception produces denoised sprite tracks using foreground segmentation and least-squares denoising, supported by meticulous calibration of costume centroids to achieve sub-pixel accuracy. Synthesis proceeds by extracting parametric motion/visibility/costume motifs, employing verification gates to guarantee exact agreement with observable tracks, and emitting compact, canonical programs.

Where motion is input-gated and invisible under passive observation, an active inference extension renders the original as a black box under synthesized interaction sequences (e.g., key presses), recovers the gating logic, and validates against both passive and active timelines. This module embodies an active learning approach for dynamic program behaviors not manifest in passive traces.

The pipeline refrains from emitting unjustified programs: abstention rather than optimistic guessing aligns with the system’s soundness discipline.

Evaluation: Soundness, Discrimination, and Robustness

Systematic closed-loop evaluation is carried out on procedurally generated benchmarks and real-world Scratch programs, using a gold-standard renderer and varying the camera lens, noise, and sample rate. The key empirical results are:

• No observed false acceptance: Across 246 adversarial program pairs and 604 labeled pairs, the static tier never accepts an incorrect candidate, validating the verdict asymmetry.
• Strong recovery within-vocabulary: 100% render-witnessed recovery on in-vocabulary programs, with 80% obtaining static certificates of lens-equivalence.
• Discrimination against near-miss perturbations: All structurally close but gold-different programs are refuted at pixel-level tolerance, with the only failures tied to the perception noise threshold.
• Robustness to degraded input: Recovery performance is invariant under reduced frame rate (down to 5fps), compressed H.264 video, and alternate renderer outputs.
• Active inference: All input-gated behaviors are recovered when the pipeline is allowed black-box interaction, matching the prevalence of input-gated programs in the wild (31%).
• Comparison to vision-LLMs: Commercial VLMs (e.g., Claude Sonnet 4.6, MiniMax-M3) fail to recover matched programs (0-2.5%), demonstrating the necessity of precise structural fitting and sound checking; even with oracle-in-the-loop repair, their performance remains limited.

In end-to-end trials over real Scratch projects, the system achieves 14% recovery—tracking exactly the synthesis vocabulary’s reach—without a single wrong certification.

Theoretical and Practical Implications

The results establish a trustworthy recovery framework for source-unavailable, user-facing program artifacts, under bounded but formally characterized observability. The discipline of separating trust (the checker) from inference (the generator) not only ensures practical no-false-accept deployment but also sets a template for future work in multimodal, program-level data integration.

The methodology generalizes to domains where lenses and sound equivalence checkers are available, potentially influencing future design of program analysis and repair systems across both educational and industrial settings. The work paves the way toward formalizing limits of observation-based recovery, integrating richer perception, and automating recovery for a broader class of interactive and event-driven software artifacts.

Conclusion

The presented approach rigorously addresses the problem of certified program recovery from execution video, embedding a formal soundness discipline in the validation oracle and neuro-symbolic pipeline. Empirical results confirm both the discrimination and coverage properties of the system, exposing the gap in commercial VLMs and establishing Vid2Prog as a trustworthy front end to program-based analysis from video. Future development trajectories include expanding synthesis vocabulary, relaxing asset assumptions, and further mechanizing semantic quotient adequacy. The discipline formalized herein is broadly transferable to other untrusted generative settings, provided a sound equivalence oracle is available.