What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspective

Abstract: What makes a difference in the post-training of LLMs? We investigate the training patterns of different layers in LLMs through the lens of the gradient. We are specifically interested in how fast vs. slow thinking affects the layer-wise gradients, given the recent popularity of training LLMs on reasoning paths such as chain-of-thoughts (CoT) and process rewards. In our study, fast thinking without CoT leads to larger gradients and larger differences of gradients across layers than slow thinking (Detailed CoT), indicating the learning stability brought by the latter. Additionally, we study whether the gradient patterns can reflect the correctness of responses when training different LLMs using slow vs. fast thinking paths. The results show that the gradients of slow thinking can distinguish correct and irrelevant reasoning paths. As a comparison, we conduct similar gradient analyses on non-reasoning knowledge learning tasks, on which, however, trivially increasing the response length does not lead to similar behaviors of slow thinking. Our study strengthens fundamental understandings of LLM training and sheds novel insights on its efficiency and stability, which pave the way towards building a generalizable System-2 agent. Our code, data, and gradient statistics can be found in: https://github.com/MingLiiii/Layer_Gradient.

• The paper demonstrates that slow thinking using detailed chain-of-thought minimizes gradient fluctuations, leading to more stable training across LLM layers.
• It reveals through SVD analysis that slow thinking gradients better discern correct responses compared to fast thinking approaches.
• The study highlights that instruction-tuned models may not outperform pre-trained ones in reasoning tasks, suggesting a need for hybrid training strategies.

Analysis of Gradient Dynamics in LLMs: Fast vs. Slow Thinking

The investigation into the inner dynamics of LLMs as presented in "What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspective" provides a noteworthy contribution to understanding the training behaviors of these models. The study explores how LLMs respond to training paradigms characterized by fast and slow thinking processes, particularly in relation to gradient dynamics across different layers. This analysis, leveraging Singular Value Decomposition (SVD) of layer-wise gradients, sheds light on the stability, efficiency, and correctness of LLM outputs under varied cognitive processing simulations.

Key Findings

1. Gradient Stability Across Layers: The paper demonstrates that training LLMs with slow thinking practices, which incorporate detailed chain-of-thought (CoT) reasoning paths, results in more uniform gradient norms across layers compared to fast thinking strategies. This finding underscores a reduction in gradient fluctuations and potentially enhances training stability. Specifically, the nuclear norm measurements revealed smaller gradients on detailed CoT tasks, suggesting that slow thinking minimizes learning misalignments with pre-trained model weights.
2. Response Correctness Identification: Through an analysis of gradients, the study establishes that slow thinking gradients are discerning in identifying correct versus irrelevant responses. In contrast, fast thinking models without CoT pathways exhibit similar gradient behaviors irrespective of response correctness, suggesting an insufficient alignment mechanism when rationalization paths are omitted.
3. Pre-training vs. Instruction-Tuning: It is shown that instruction-tuned LLMs are not inherently better at recognizing incorrect reasoning paths compared to general pre-trained models. However, when evaluated on simplified CoT paths, instruction-tuned models exhibited significantly different gradient characteristics, denoting potential discrepancies with their training data.
4. Inapplicability to Knowledge Tasks: The examination further reveals that the observed gradient properties in reasoning tasks do not extend to knowledge-based tasks, such as Wikipedia content processing. Simply extending response length in these tasks does not replicate the gradient behavioral patterns observed in slow thinking, indicating a unique interaction between reasoning processes and gradient stabilizations.

Implications and Future Directions

Practical Considerations: The insights from this research could be pivotal in refining LLM training processes, particularly in designing training regimes that leverage slow cognitive simulations to enhance response accuracy and reduce harmful content generation. The identification of stable gradient norms associated with detailed CoT suggests a training path that could be focused on improving model robustness and interpretability.

Theoretical Insights: This study advances the theoretical understanding of how cognitive paradigms, mirrored in training strategies, affect the internal gradient dynamics of LLMs. By adopting a gradient-centric analysis, it reveals layer-specific sensitivities that could inform architectural adjustments in future model designs.

Speculative Future Research: One promising direction for future exploration involves the development of hybrid training methodologies that dynamically adjust between fast and slow cognitive simulations based on task requirements or detected gradient instabilities. Furthermore, the extrapolation of these findings to other LLM architectures or more domain-specific tasks could unravel more generalized principles guiding efficient model training.

In conclusion, this paper's exploration into the gradient dynamics of fast versus slow thinking within LLMs enriches our understanding of the subtleties involved in training these complex models. The findings encourage a detailed consideration of thought process simulations in model optimizations, paving the way toward more stable and interpretable LLMs.