Explain the lower performance of programming-competition benchmarks

Determine whether the comparatively poor performance of large language models on programming-competition program-synthesis benchmarks is caused by the way programming-competition problems are written, particularly their use of a style that obscures the underlying algorithms needed to solve them.

Background

The paper contrasts its Mostly Basic Programming Problems (MBPP) dataset with programming-competition datasets such as APPS. The authors argue that MBPP uses simpler and more literal descriptions of programming tasks, whereas competition problems often embed the algorithmic requirements in a story-like presentation. They propose, as a conjecture, that this linguistic difference accounts for the generally poor synthesis performance reported on competition benchmarks.

The conjecture remains unresolved in the paper: the authors do not empirically isolate the effect of problem-description style from other factors such as task difficulty, algorithmic complexity, dataset composition, or model training. Establishing whether description style is the principal cause would inform the design of future program-synthesis benchmarks and methods.