Resolve the Phaser comparison between RSIGame and Play2Code for Qwen3.8-27B

Determine whether RSIGame significantly outperforms Play2Code on Phaser game-development tasks when both use Qwen3.8-27B as the generator under the reported matched-budget evaluation protocol.

Background

The main experiments report that RSIGame has a higher mean Overall score than Play2Code for the Qwen3.8-27B Phaser setting. However, the paired statistical test does not establish a significant difference: the mean improvement is 1.55 points, with a 95% confidence interval spanning zero and p = 0.394.

The authors therefore explicitly characterize this comparison as unresolved. The open issue is whether the observed numerical advantage reflects a reliable performance difference or sampling variability under the 140-task benchmark.

References

Against Play2Code, the improvement is significant for both Godot generators and for GPT-5.5 on Phaser; the Qwen3.8-27B Phaser comparison remains unresolved (+1.55, p=0.394).

— RSIGame: Autonomous Agentic Game Development with Recursive Self-improvement  (2609.39045 - Wu et al., 30 Sep 2026) in Appendix, Section ‘Statistical Reliability of Main Results’, paragraph ‘Main comparisons’