Steering Large Language Models between Code Execution and Textual Reasoning (2410.03524v1)

Abstract: While a lot of recent research focuses on enhancing the textual reasoning capabilities of LLMs by optimizing the multi-agent framework or reasoning chains, several benchmark tasks can be solved with 100% success through direct coding, which is more scalable and avoids the computational overhead associated with textual iterating and searching. Textual reasoning has inherent limitations in solving tasks with challenges in math, logics, optimization, and searching, which is unlikely to be solved by simply scaling up the model and data size. The recently released OpenAI GPT Code Interpreter and multi-agent frameworks such as AutoGen have demonstrated remarkable proficiency of integrating code generation and execution to solve complex tasks using LLMs. However, based on our experiments on 7 existing popular methods for steering code/text generation in both single- and multi-turn settings with 14 tasks and 6 types of LLMs (including the new O1-preview), currently there is no optimal method to correctly steer LLMs to write code when needed. We discover some interesting patterns on when models use code vs. textual reasoning with the evolution to task complexity and model sizes, which even result in an astonishingly inverse scaling law. We also discover that results from LLM written code are not always better than using textual reasoning, even if the task could be solved through code. To mitigate the above issues, we propose three methods to better steer LLM code/text generation and achieve a notable improvement. The costs of token lengths and runtime are thoroughly discussed for all the methods. We believe the problem of steering LLM code/text generation is critical for future research and has much space for further improvement. Project Page, Datasets, and Codes are available at https://yongchao98.github.io/CodeSteer/.

Steering LLMs between Code Execution and Textual Reasoning

The paper, "Steering LLMs between Code Execution and Textual Reasoning," addresses the limitations of current LLMs in choosing between textual reasoning and code execution for problem-solving. Traditionally, LLMs have demonstrated distinct proficiencies in generating text and code. The challenge lies in identifying the optimal modality for various tasks—particularly those involving mathematical, logical, and planning elements—where coding can offer substantial procedural advantages, yet, textual reasoning remains prevalent.

Key Observations and Challenges

1. Inefficacy of Traditional Methods: The paper assessed seven existing strategies for steering code/text generation across six LLMs and 14 task categories. No single method proved consistently superior, highlighting the need for adaptive steering based on task complexity and model capability.
2. Inverse Scaling Law: One intriguing finding was the inverse scaling effect—smaller models, like GPT-3.5, occasionally outperformed larger counterparts such as GPT-4o when augmented with code interpreters. This reversals main contributors included larger models’ overconfidence in textual reasoning and underutilization of coding where beneficial.
3. Collider of Task Complexity: The complexity of tasks meditated the choice between code and text. For example, GPT-4o's CI toggled heuristically; it managed simple tasks textually but complex ones through code—though struggled with medium complexity.
4. Pitfalls of Code-Centric Approaches: Contrary to expectations, merely forcing models to generate code did not ensure higher accuracy. The quality of code matters: symbolic, non-functional code is frequently produced if models are inappropriately cued to switch modalities.

Proposed Solutions and Findings

The authors proposed improved methodologies to address these limitations:

1. Multi-Agent Frameworks: Inspired by multi-agent paradigms, an integration method (Code + Text + Summary) where models first generate solutions in both code and text, followed by an evaluation to yield a refined response, showed enhanced performance across most models.
2. Iterative Refinement: Providing multi-turn reflection and self-correction embedded within LLM processing allowed models to refine solutions iteratively, proving especially beneficial for code-based reasoning cases.
3. Self-Esteem Scores: Implementing a self-evaluation mechanism before choosing the output modality helped guide LLMs in adopting a more nuanced approach. Better-informed decisions are made between code and text, leveraging situational appropriateness.

Implications and Future Directions

The paper highlights the criticality of dynamically steering LLMs' reasoning modalities. Accurate steering can amplify model utility in diverse domains—automation, robotics, complex data analysis, and AI-driven logic tasks. Yet, steering mechanisms require refinement which can benefit from:

• Adaptive System Prompts: Fine-tuning system prompts towards more context-aware ones may better guide modality selection.
• Specialized Training: Tailoring LLM training paradigms to incorporate task-specific reasoning strategies, fostering a deeper integration of code and text outputs.
• Enhanced Reflective Procedures: Building upon iterative feedback loops that might tighten error margins in sequential decision-making.

The prospects for LLM applications expand as these steering techniques evolve, ultimately aiming for a seamlessly blended paradigm of code-generation and textual reasoning proficient models. Accentuating the efficacy and efficiency, future research might concentrate on multi-modal LLMs tailored for generalized as well as domain-specific problem-solving.