Native symbolic reasoning in video-based neural computers

Determine how to achieve reliable native symbolic reasoning in video-based neural computer models operating on command-line interfaces, and identify the architectural or training advances necessary to close current performance gaps on arithmetic and related symbolic probes.

Background

The authors evaluate arithmetic probes in terminal settings using their CLIGen model and find low accuracy compared to a reported outlier (Sora2).

They show that reprompting can boost scores dramatically but argue this reflects conditioning rather than native computation, concluding that dependable symbolic reasoning within the model’s runtime remains unresolved.