SchedCheck: Schedule-Robustness Analysis for Event-Driven Block Programs

Abstract: Block-based languages such as Scratch let beginners assemble interactive programs from sprites and scripts. These programs are concurrent in practice: green-flag scripts, broadcasts, and clones run as cooperatively scheduled threads over shared sprite and stage state, and their authors never write a thread. We show that such programs contain schedule-sensitive behaviors whose observable result depends on an execution order the language leaves open. Editing, saving, or remixing a project can produce a copy with the same blocks but a different layer order, changing the order the virtual machine starts scripts. We formalize the schedule space a Scratch virtual machine can realize as the permutations of the initial executable-target order, and define schedule-robustness against a lattice of observation lenses over a fixed horizon. A partial-order exploration runs one schedule per dependence-equivalence class, and on projects small enough to enumerate, an independent oracle confirms it recovers every realizable outcome. On larger projects, representatives stand in for the factorial under the validated dependence model. SchedCheck implements this on the production Scratch VM. Across 224 real student projects, at least 21% of the concurrent ones are schedule-sensitive at the grading lens, and a uniform random sample of public projects replicates the rate at 17.6%, with two real remixes of a deployed animation arranging its letters differently. On hand-built fault pairs and a generated benchmark of 32 spec-defined faults across four classes, the tool detects and localizes every schedule fault, with a logic-fault control reporting clean. The oracle exposed four unsoundness gaps in the dependence model, all repaired. The method is parametric in the execution model, instantiating unchanged on a second cooperative event loop.

• The paper introduces schedule-robustness in event-driven block programs by formalizing how observable outcomes remain invariant across different sprite execution orders.
• It employs partial-order reduction through static footprint analysis to efficiently collapse numerous sprite orderings into manageable equivalence classes, offering substantial computational savings.
• Empirical evaluation on student and public Scratch projects reveals that a significant portion exhibit schedule-sensitive behavior undetected by standard testing or static analysis tools.

Schedule-Robustness Analysis for Event-Driven Block Programs: A Technical Perspective on SchedCheck

Motivation and Problem Statement

Event-driven block-based environments like Scratch are the de facto computational model for early programming education, with substantial deployment in classrooms and online communities. Although Scratch abstracts concurrency details from the user, its underlying cooperative event-loop execution model induces implicit parallelism: green-flag scripts, broadcasts, and clones all run as threads that interact over shared global and sprite-local state. The virtual machine’s execution semantics leave certain schedule choices—specifically, the order in which sprites’ event handler scripts fire—underdetermined. The result is that observational program behaviors can depend on execution orderings which are neither visible in nor controlled by the Scratch “surface language.”

SchedCheck directly targets this axis of uncertainty, formalizing and analyzing schedule-robustness: the property that the observable outcome of a program remains invariant across all real, user-reproducible thread orderings. Critically, SchedCheck connects these semantically meaningful schedule changes with concrete user actions (e.g., sprite layer reordering and remixing), validating that all fault incidences detected are “reproducible,” and not merely artifacts of theoretical interleaving.

Formalization of the Schedule Space and Observation Lenses

SchedCheck’s foundational insight is that, on the production Scratch VM, the only significant indeterminacy at the schedule level is the relative order of sprites with executable scripts upon initialization. Specifically, execution differs solely in the permutation of $k$ contributing sprite targets, formalized as the symmetric group $S_k$. All additional events (e.g., broadcast, clone creation) are deterministic functions of this initial ordering; the stage target is always rendered as a fixed suffix and does not contribute additional schedule freedom.

Program behaviors are classified through a lattice of observation lenses, ranging from the coarse (final-state of variables and sprite poses) to the fine (full per-tick execution traces). The robustness verdict is parameterized over this lattice; monotonicity holds, so robustness at a finer lens implies robustness at all coarser ones.

Algorithmic Schedule Exploration: Partial-Order Reduction

Exhaustively enumerating all $k!$ sprite orderings is computationally intractable for all but the smallest projects. SchedCheck exploits partial-order reduction techniques, leveraging static footprint analysis to cluster schedule permutations into dependence-equivalence classes. Here, two schedules are equivalent if they order every pair of dependent sprites identically (i.e., induce the same acyclic orientation on the conflict graph derived from inter-script data dependencies). The exploration algorithm outputs one canonical representative per equivalence class, reducing the search effort from $k!$ runs to the combinatorial number of acyclic orientations of the conflict graph—which scales far better for sparse dependencies.

A soundness/completeness guarantee is formalized: provided the footprint model is a conservative over-approximation, the reduced exploration characterizes all and only possible schedule-sensitive behaviors that real users might observe.

Empirical Characterization, Detection, and Localization

SchedCheck was evaluated on two major corpora: 224 student submissions from a course and 250 randomly sampled public Scratch projects. The findings are robust:

• Prevalence of schedule sensitivity: Among concurrent projects (multiple sprites), 21% (course) and 17.6% (public) showed outcome variance under realizable schedule changes at the grading lens.
• Reduction savings: For projects where the dependence graph is sparse, the reduction yields substantial cost savings, with some 40-sprite projects collapsing $40!$ schedules into a single equivalence class.
• Detection power: On both hand-constructed mutants and generated benchmarks, SchedCheck detected every schedule-sensitivity instance seeded. The method not only detects, but localizes faults to precise (script pair, contended resource) tuples. Attribution utilizes set-valued differentials, overcoming limitations of binary divergence detection.
• Orthogonality and necessity: Many schedule-sensitive faults are invisible to standard test generation and static analysis, falling outside what input perturbation or pattern-based “linter” tools discover.

Notably, two “real-world” remix case studies confirm that schedule-divergent executions are not just theoretical: observed remixes of popular projects do realize distinct schedule-induced outcomes with non-cosmetic differences.

Theoretical Implications

SchedCheck provides a formally justified and empirically validated methodology for sound schedule-fault analysis, sidestepping the state space explosion plaguing concurrency verification. The central theoretical contributions include:

• A precise characterization of the executable schedule space of block-based event-driven VMs
• A connection between lens-parametric behavioral equivalence and partial-order reduction
• Sound compositional reduction from schedule permutations to equivalence classes, empowered by static resource footprints

This work neither overshoots into intractable general concurrency interleaving nor underapproximates schedule hazards realizable by users, which positions it with strong claims of exhaustiveness and reproducibility.

Practical Impact and Future Directions

SchedCheck’s findings have direct implications for the reliability of education-facing block-based programming environments:

• Upwards of one in five Scratch projects in authentic classroom and “in-the-wild” contexts are schedule-sensitive in user-reproducible ways. These behaviors are not flagged by current test generators or static checkers, confirming the need for robust schedule-analysis tooling in educational settings.
• The reduction mechanism is parametric in the execution model and does not require any changes to the runtime; implementation operates on the production VM in a headless harness.

In principle, the methodology generalizes to other event-driven, block-based or visual programming environments with user-facing concurrency abstractions. Extensions may target event models with richer forms of timing nondeterminism, or integrate dynamic symbolic execution to automatically extract more precise resource footprints without losing soundness. Integrating SchedCheck into online Scratch editors/graders would enable real-time feedback for millions of learners and instructors, improving the pedagogical robustness of early computer science education.

Conclusion

SchedCheck systematically explores and diagnoses schedule-sensitive behaviors in event-driven block-based programs, specifically targeting the concurrency hazards arising in standard Scratch scenarios. By formalizing the realizable schedule space, introducing lens-parametric robustness definitions, and employing a scalable reduction that is empirical and oracle-validated, the method achieves strong detection, precise localization, and practical scalability. The tool reveals substantive schedule fragility in fielded code and should inform best practices for the design and assessment of block-based educational environments.

---

Reference:

"SchedCheck: Schedule-Robustness Analysis for Event-Driven Block Programs" (2607.00623)