Is Next Token Prediction Sufficient for GPT? Exploration on Code Logic Comprehension (2404.08885v1)

Abstract: LLMs has experienced exponential growth, they demonstrate remarkable performance across various tasks. Notwithstanding, contemporary research primarily centers on enhancing the size and quality of pretraining data, still utilizing the next token prediction task on autoregressive transformer model structure. The efficacy of this task in truly facilitating the model's comprehension of code logic remains questionable, we speculate that it still interprets code as mere text, while human emphasizes the underlying logical knowledge. In order to prove it, we introduce a new task, "Logically Equivalent Code Selection," which necessitates the selection of logically equivalent code from a candidate set, given a query code. Our experimental findings indicate that current LLMs underperform in this task, since they understand code by unordered bag of keywords. To ameliorate their performance, we propose an advanced pretraining task, "Next Token Prediction+". This task aims to modify the sentence embedding distribution of the LLM without sacrificing its generative capabilities. Our experimental results reveal that following this pretraining, both Code Llama and StarCoder, the prevalent code domain pretraining models, display significant improvements on our logically equivalent code selection task and the code completion task.

• The paper shows that standard next token prediction fails to capture the logical structure of code as it treats code merely as text.
• The paper introduces the ‘Logically Equivalent Code Selection’ task, revealing that models like Code Llama and StarCoder underperform by up to 23.99%.
• The paper proposes a novel next token prediction+ method that marries structural awareness with generative abilities to improve code logic comprehension.

Evaluation of "Is Next Token Prediction Sufficient for GPT? Exploration on Code Logic Comprehension"

In recent developments within the domain of LLMs, such as GPT variants, the prevailing strategy has been to enhance model performance through the augmentation of training data, refining the model architectures, and scaling up parameter sizes. These improvements have propelled LLMs to excel across diverse tasks, most notably in code-related tasks where models have demonstrated substantial utility. However, the effectiveness of next token prediction tasks in imparting a true understanding of programming code's logic remains underexplored. The paper by Qi et al. critically examines this gap, positing that current models may interpret code primarily as text devoid of inherent logical structures, which is divergent from how humans comprehend code by emphasizing its logical underpinnings.

The authors introduce a novel task named "Logically Equivalent Code Selection," demanding the selection of functionally identical code from a set of candidates when given a query code. This addition aims to evaluate the pure logic comprehension capabilities of LLMs beyond mere syntactic prediction. Empirical findings reveal that prominent code-domain models, namely Code Llama and StarCoder, underperform on this task, suggesting that the models predominantly treat code as an unordered collection of keywords.

To address these limitations, Qi et al. propose an advanced pretraining task, "Next Token Prediction+," aimed at refining sentence embedding distributions while preserving generative prowess. This strategy enhances the model's capacity to discern logical equivalency in code, as evidenced by marked improvements in the logically equivalent code-selection task and in code completion tasks. This improvement is quantified with an enhancement in accuracy of up to 23.99% across different model configurations, signifying a meaningful shift towards more human-like understanding of code logic.

The critical insights from this paper are manifold. Firstly, it accentuates the distinction between understanding textual sequences and genuine code comprehension. Secondly, it underscores the potential of targeted training strategies in bridging the cognitive gap between LLMs and human users in code logic tasks. The token prediction+ method sets a precedent for future model training methodologies by introducing a balanced approach that marries structural awareness with generative function.

Potential developments inferred from this research could lead to more nuanced LLMs capable of executing complex reasoning tasks, particularly those necessitating deep logical comprehension. Further speculations may involve adapting similar logic-oriented training to other domains characterized by structured logic akin to code, including mathematical proofs or scientific reasoning.

In conclusion, Qi et al.'s exploration of the sufficiency of next token prediction in fostering true code comprehension by LLMs offers a fresh paradigm in model training strategies. By proposing a practicable solution that enhances both comprehension and generative capabilities, this paper contributes significantly to ongoing efforts to refine AI's interaction with structured logic inherent in programming and beyond. The insights gleaned from this paper may pave the way for the next generation of LLMs, potentially heralding advancements in both their utility and reliability in professional and academic settings.