EcoScratch: Cost-Effective Multimodal Repair for Scratch Using Execution Feedback

Abstract: Scratch is the most popular programming environment for novices, with over 1.15 billion projects created worldwide. Unlike traditional languages, correctness in Scratch is defined by visible behavior on the stage rather than by code structure alone, so programs that appear correct in the workspace can still fail at runtime due to timing, event ordering, or cross-sprite interactions. Visual execution evidence such as gameplay videos can therefore be essential for diagnosis and repair. However, capturing and processing this evidence inside an automated repair loop introduces substantial overhead. Probing execution, recording stage behavior, rebuilding executable .sb3 projects, and verifying candidate fixes consume time, monetary cost, and resources across an entire repair trajectory rather than a single model call. We present EcoScratch, a repair pipeline that uses lightweight runtime signals to decide whether the next attempt stays text-only or escalates to multimodal prompting. The controller also sets the JSON Patch budget and verification effort, so evidence choice and repair budget are coupled inside the same decision. EcoScratch rebuilds candidate fixes into executable .sb3 projects and records per-trajectory traces, monetary cost, local-runtime energy. We evaluate 12 models on 100 executable Scratch repair projects under four controller settings, yielding 4800 repair trajectories. In this matrix, a selective multimodal policy gives the strongest observed success-cost-energy tradeoff. It reaches the highest generation success (30.3%) while using less average cost and local-runtime energy than the two non-adaptive multimodal baselines under the same bounded trajectory budget; text-only remains the lowest-cost floor. Across the evaluated matrix, multimodal evidence helps most when it is used to control escalation within a bounded trajectory budget rather than applied uniformly.

• The paper presents EcoScratch, which uses a runtime feedback loop to selectively choose between text-only and multimodal debugging evidence.
• It reports a 30.3% generation success and 8.0% strict verification rate while reducing cost and energy by up to 50% compared to non-adaptive strategies.
• The adaptive controller design optimizes resource allocation by dynamically escalating evidence collection based on lightweight execution probes.

Cost-Efficient Multimodal Automated Repair in Scratch via Execution Feedback

Introduction and Motivation

The paper "EcoScratch: Cost-Effective Multimodal Repair for Scratch Using Execution Feedback" (2603.29624) addresses the central challenge of enabling scalable, automated debugging and program repair in Scratch, the dominant block-based programming environment for novice programmers. Scratch program correctness is inherently determined by animated runtime behavior ("the stage") rather than static code organization. This introduces unique difficulties: code edits may superficially improve scripts without rectifying behavioral bugs manifesting only during execution. Furthermore, fully automated repair loops leveraging multimodal evidence (especially screenshots or gameplay videos) introduce significant computational, monetary, and energy overhead, greatly limiting system scalability.

The paradigm shift introduced by EcoScratch is to frame multimodality in automated repair as a deployment-time allocation problem, not a model capability issue. The pipeline uses lightweight runtime probes to decide selectively whether to use text-only evidence or to escalate to multimodal prompting. The controller jointly manages patch budgets and verification effort, tightly coupling the cost of richer evidence to the likelihood that it will yield improved repair. This design aims to balance generation success with minimal end-to-end trajectory costs.

Figure 1: The complexity of Scratch debugging stems from the mismatch between plausible code-local edits and their actual impact on the animated stage, motivating execution-centered repair approaches.

System Architecture

EcoScratch accepts a buggy Scratch project, gold reference, and test harness. The core runtime is structured as a closed execution loop: a lightweight probe summarizes observed behavior, generating a scheduling signal for policy selection. The key architectural contribution is the separation and interaction among four controller modes:

• Text-only (minimal evidence, lowest cost)
• Always-on multimodal (maximum evidence, highest cost)
• Fixed multimodal (uniform, predetermined multimodal evidence for each case)
• Heuristic (runtime-signal-guided; the paper’s primary proposal)

Patch candidates are expressed as bounded RFC 6902-style JSON edits, enabling precise auditing of model actions and robust staged verification. The system records all artifacts, allowing trajectory-level accounting of cost, runtime energy, and repair outcomes.

Figure 2: EcoScratch's pipeline integrates runtime probing, adaptive evidence selection, patch generation, and staged checker loops, outputting cost and energy records for each trajectory.

Experimental Framework

The evaluation matrix comprises 4,800 complete repair trajectories—12 concrete LLMs from the OpenAI and Gemini families, × four controller modes, × 100 projects per model-mode. Central metrics include:

• Generation success (applied patch candidate)
• Strict verification success (patch passes customized verifier)
• Trajectory-level cost in USD
• Host-side local-runtime energy (Wh)

The experimental setup ensures that model comparisons are paired within each project, controlling for project- and provider-specific confounds. A significant emphasis is placed on empirical resource tradeoffs as measured over complete probe–repair–verify trajectories, in the spirit of Green AI [schwartz2020green].

Results and Analysis

Success–Cost–Energy Tradeoff

The Heuristic controller consistently achieves the best tradeoff for multimodal repair, with 30.3% generation success and 8.0% strict verification success. Notably, Heuristic outperforms Always-on and Fixed multimodal modes by 3–7 percentage points in generation success while reducing monetary cost and energy by 34–50%, with no increase in average attempts per project. Text-only is the minimal expenditure baseline but achieves inadequate bug repair rates (16.0% generation success, 1.7% strict).

Figure 3: Across all models and controller settings, Heuristic achieves higher success rates at lower cost and energy compared to non-adaptive multimodal policies; each point represents a model aggregated over 100 project trajectories.

Consistency Across Models and Providers

Per-model and per-provider analyses confirm these findings: for every model in both OpenAI and Gemini families, Heuristic strictly dominates or ties non-adaptive baselines in success and efficiency. The selective escalation policy's advantage is not restricted to a single LLM lineage or implementation.

Figure 4: Heuristic’s gains in generation success relative to always-on and fixed multimodal strategies are always accompanied by meaningful reductions in cost for all tested models, confirming robust efficiency improvements.

Selective Multimodality and Quality Assurance

Heuristic is not merely drifting toward text-only repair. It uses images in 31.3% of cases—down sharply from the 78% rate of always-on policies—yet maintains or improves strict-pass ratios. This demonstrates that adaptive escalation delivers substantial evidence cost savings without sacrificing downstream verification quality or yielding spurious candidates.

Practical and Theoretical Implications

EcoScratch advances the state of automated block-based programming repair by precisely separating policy control, evidence selection, and resource-effort budgeting. The results challenge the assumption that automatically escalating to multimodal prompting is necessary or cost-effective in all cases. The pipeline suggests that lightweight execution-based triage (e.g., comparing runtime divergence and early stage symptoms) is a critical system primitive for controlled deployment at scale.

From an operational perspective, the implication is that LLM-driven repair assistants for block-based and visual programming (Scratch and analogous environments) should reify artifact collection, including image and video, as an explicit scheduling decision. Selective multimodality, guided by cheap runtime signals, is more cost-effective than a naïve always-on strategy, even with high-performing LLMs. This finding is robust across current-generation foundation models.

Theoretically, this work extends the generate-validate paradigm to block-based, event-driven, and asset-rich domains, where visual evidence and behavioral verification are first-class constraints. The controller architecture developed by EcoScratch can be adapted to related multimodal repair and feedback frameworks.

Limitations and Threats to Validity

There are several limitations that may circumscribe generalization. The heuristic controller is fixed without policy learning; further gains could possibly be realized by tuning thresholds or leveraging RL-based adaptation, especially as runtime context or evidence channels diversify. The 100-project dataset, while designed for controlled comparison, does not capture the full breadth of classroom-scale or open-ended learner tasks. All energy and cost metrics are measured host-side, which may not reflect full cloud deployment or provider compute costs.

Future Directions

Further advances could include controller adaptation via policy learning, integration of fine-grained behavioral traces or temporal video, and application to more diverse program genres. Extending the EcoScratch paradigm to interactive, dialog-driven repair assistants may enable practical incorporation into classroom and MOOC-scale platforms. Broader adoption would also benefit from research into full-system environmental impact, as modeled in recent LLM sustainability studies [ren2024reconciling, jegham2025hungry].

Conclusion

EcoScratch introduces a principled, execution-feedback-centered approach for cost-effective, multimodal automated repair in Scratch. By decoupling evidence escalation from default model capability and enabling selective, runtime-guided multimodal prompt construction, the system achieves a dominated frontier in the success–cost–energy space for all tested models. This work underscores the deployment importance of adaptive policy design in multimodal LLM-based program repair and provides compelling empirical justification for runtime-aware artifact selection. Outcomes from EcoScratch are directly relevant for scalable deployment of automated repair, feedback, and tutoring infrastructure in block-based programming ecosystems.

(2603.29624)