Generative AI Act II: Test Time Scaling Drives Cognition Engineering (2504.13828v3)

Abstract: The first generation of LLMs - what might be called "Act I" of generative AI (2020-2023) - achieved remarkable success through massive parameter and data scaling, yet exhibited fundamental limitations such as knowledge latency, shallow reasoning, and constrained cognitive processes. During this era, prompt engineering emerged as our primary interface with AI, enabling dialogue-level communication through natural language. We now witness the emergence of "Act II" (2024-present), where models are transitioning from knowledge-retrieval systems (in latent space) to thought-construction engines through test-time scaling techniques. This new paradigm establishes a mind-level connection with AI through language-based thoughts. In this paper, we clarify the conceptual foundations of cognition engineering and explain why this moment is critical for its development. We systematically break down these advanced approaches through comprehensive tutorials and optimized implementations, democratizing access to cognition engineering and enabling every practitioner to participate in AI's second act. We provide a regularly updated collection of papers on test-time scaling in the GitHub Repository: https://github.com/GAIR-NLP/cognition-engineering

• The paper introduces test-time scaling as a core mechanism to evolve AI from static knowledge retrieval to dynamic, multi-step cognition engineering.
• It proposes a three-phase scaling hypothesis—pre-training, post-training, and test-time—that builds interlinked cognitive bridges for complex inference.
• It details advanced methods such as parallel sampling, tree search, and multi-turn correction to enhance AI reasoning and decision-making.

This paper, "Generative AI Act II: Test Time Scaling Drives Cognition Engineering" (2504.13828), proposes a framework for understanding the evolution of LLMs, dividing it into two distinct "acts."

Act I (approx. 2020-2023): This era focused on scaling model parameters and training data. LLMs became powerful knowledge retrieval and management systems, capable of engaging in dialogue and generating content. The primary interaction method developed during this time was prompt engineering. However, Act I models exhibited limitations:

• Knowledge Latency: Difficulty incorporating emerging information not present in static training data.
• Shallow Reasoning: Struggles with multi-step, complex logical problems.
• Limited Thought Processes: Inability to demonstrate deep, human-like thinking, especially for novel questions.

Act II (approx. 2024-present): This act marks a shift from knowledge retrieval to thought construction, driven by test-time scaling (TTS) techniques. The goal is Cognition Engineering, defined as the systematic development of AI thinking capabilities using TTS paradigms and targeted training (like reinforcement learning). This approach aims to create a "mind-level" connection with AI through language-based thoughts, moving beyond simple dialogue.

Core Concepts:

1. Cognition Engineering:
   • Focuses on developing deep cognition (complex reasoning, creativity, metacognition) rather than just knowledge acquisition. It aligns with moving from the "Knowledge" to the "Wisdom" level in the DIKW pyramid.
   • Emphasizes intentional construction of cognitive abilities over purely emergent capabilities from scaling.
   • Shifts focus from imitating human outputs (behavior) to imitating human thought processes.
   • Aims for dynamic thinking at inference time and knowledge creation.
2. Three Scaling Phases Hypothesis:
   • Pre-training Scaling: Forms foundational "knowledge islands."
   • Post-training Scaling: Densifies these islands with more connections between related concepts.
   • Test-time Scaling: Builds dynamic reasoning pathways ("cognitive bridges") between distant knowledge islands, enabling complex, multi-hop inference.
3. Technical Pillars for Act II:
   • Knowledge Foundation: Modern LLMs are trained on richer, more curated data (code, math, scientific literature), providing the necessary base knowledge.
   • Test-Time Scaling Foundation: Techniques like Chain-of-Thought (CoT), tree search, and self-correction provide a "cognitive workspace" for extended reasoning during inference.
   • Self-Training Foundation: Methods like Reinforcement Learning (RL) allow models to learn complex cognitive behaviors (reflection, backtracking) and discover novel reasoning strategies, potentially surpassing human capabilities.

Test-Time Scaling (TTS) Methods: The paper abstracts TTS as a search strategy $M$ guiding a generator $g$ for a query $q$: $y \sim M(.|q,g,\phi)$. Key goals are maximizing performance for a given compute budget (scaling efficiency). Major TTS methods include:

• Parallel Sampling: Generating multiple candidate responses ($N$) and selecting the best via:
  • Best-of-N (BoN): Using a scoring function/verifier.
  • Majority Voting (Self-Consistency): Choosing the most frequent answer.
  • Combined Strategies: Weighting votes by scores.
  • Efficiency: Improved via query-aware sampling, early stopping, model size/sample trade-offs, better verifiers, inference-aware fine-tuning. Scaling Pass@N often outpaces practical BoN/Maj@1 performance due to imperfect verifiers.
• Tree Search: Framing problems as search over trees (token, step, or solution level) using algorithms like BFS, DFS, or MCTS, guided by value functions (self-eval, trained models like PRMs, likelihoods, consistency scores, rollouts).
  • Efficiency: Improved by algorithm choice, reducing value function cost, adaptive breadth, pruning redundant nodes.
• Multi-turn Correction: Iteratively refining responses using feedback (self-generated or external tools/models) and a refinement model.
  • Efficiency: Depends heavily on feedback quality and the model's refinement ability, which can be improved via training (e.g., RISE, SCoRe).
• Long CoT: Extended reasoning chains incorporating cognitive behaviors like reflection, backtracking, verification, and divergent thinking, often elicited via specialized training.
  • Efficiency: Improved by prompting conciseness, finetuning on compressed traces, query-aware compression (learning optimal length, RL with length penalties, routing), model merging, compressing intermediate states, or reasoning in latent space.

Comparison & Ensemble: Long CoT is presented as having high potential (adaptivity, complex behaviors) but requires training. Other methods are often training-free but less adaptive. Ensemble methods combine strengths, e.g., using parallel sampling with tree search or self-correction.

Training Strategies for TTS:

• Scaling Reinforcement Learning (RL): Training models (often using PPO or GRPO algorithms) with verifiable (rule-based) rewards (e.g., for math/code correctness) can elicit long CoT and complex cognitive behaviors ("RL Scaling phenomenon"). Success depends on algorithm choice, reward design (outcome vs. process, rule vs. model), base model selection (pre-existing cognitive patterns help), data quality/quantity, and multi-stage training (cold start SFT, iterative lengthening, curriculum learning). The paper includes a table of recipes for RL challenges.
• Supervised Fine-tuning (SFT): Directly fine-tuning models on long CoT data (distilled from capable models like DeepSeek-R1 or synthesized). Less computationally intensive than RL but potentially limited by teacher model quality and memorization concerns. Data quality appears more critical than quantity beyond a certain point.
• Iterative Self-reinforced Learning (ISRL): Using TTS outputs (e.g., from tree search or parallel sampling) to generate training data for offline SFT or DPO updates, creating a self-improvement loop. Less effective than online RL due to off-policy nature and lack of negative gradients (in SFT versions).

Applications & Progress: TTS methods are improving performance across various domains:

• Mathematics: Significant gains in problem-solving (e.g., DeepSeek-R1 on AIME) and formal theorem proving.
• Code: Enhanced code generation, debugging, and performance on benchmarks like SWE-bench and competitive programming (e.g., o1 models).
• Multimodality: Improving reasoning in VLMs (image/video understanding) and emerging applications in multimodal generation.
• Agents: Enabling long-horizon planning and complex task execution (e.g., DeepResearch, CUA).
• Embodied AI: Improving high-level planning and low-level control policies for robotics.
• Safety: Enhancing factuality, robustness, and alignment via methods like self-checking, structured debate, and deliberative reasoning.
• RAG: Tackling complex queries requiring multi-hop reasoning over retrieved documents.
• Evaluation: Improving LLM-as-a-Judge capabilities through deeper reasoning or multi-agent approaches.

Implications: Cognition engineering via TTS leads to:

• Cognition Data Engineering: A shift towards collecting/generating data representing thought processes, not just outputs.
• Reward & Environment Engineering: Need for nuanced rewards for complex/subjective tasks and specialized "cognitive environments" for training.
• Human-AI Cognitive Partnership: Moving towards bidirectional cognitive exchange and amplification.
• Research Acceleration: AI systems acting as partners in scientific discovery.

Infrastructure & Future Directions:

• Infrastructure for RL and MCTS needs optimization for scale and efficiency (e.g., vLLM, OpenRLHF, veRL).
• Key future directions include exploring new architectures beyond Transformers, pretraining on cognition data, advancing RL scaling (reproducibility, broader domains), developing better evaluation methods for cognitive processes, and furthering AI for scientific discovery.

Conclusion: The paper frames cognition engineering, powered by test-time scaling and advanced training, as the defining characteristic of generative AI's "Act II." This paradigm shift moves beyond knowledge retrieval towards building AI systems capable of deep thought, reasoning, and collaborative problem-solving with humans.