DynaAct: Large Language Model Reasoning with Dynamic Action Spaces (2511.08043v1)

Abstract: In modern sequential decision-making systems, the construction of an optimal candidate action space is critical to efficient inference. However, existing approaches either rely on manually defined action spaces that lack scalability or utilize unstructured spaces that render exhaustive search computationally prohibitive. In this paper, we propose a novel framework named \textsc{DynaAct} for automatically constructing a compact action space to enhance sequential reasoning in complex problem-solving scenarios. Our method first estimates a proxy for the complete action space by extracting general sketches observed in a corpus covering diverse complex reasoning problems using LLMs. We then formulate a submodular function that jointly evaluates candidate actions based on their utility to the current state and their diversity, and employ a greedy algorithm to select an optimal candidate set. Extensive experiments on six diverse standard benchmarks demonstrate that our approach significantly improves overall performance, while maintaining efficient inference without introducing substantial latency. The implementation is available at https://github.com/zhaoxlpku/DynaAct.

• The paper introduces a dynamic action space framework that leverages submodular maximization to select compact, diverse actions for efficient LLM reasoning.
• It utilizes a proxy action space from extensive demonstration corpora and a learned embedding function to balance utility and diversity.
• Experimental results across benchmarks, including a 6.8% gain on MATH-500, validate the method’s scalability, robustness, and improved accuracy.

Dynamic Action Space Construction for LLM Sequential Reasoning

Motivation and Problem Formulation

The paper "DynaAct: LLM Reasoning with Dynamic Action Spaces" addresses a central limitation in current approaches to LLM-based reasoning framed as sequential decision-making: the inadequacy of static or heuristically constructed action spaces for efficient, scalable reasoning across diverse problem domains. Previous work in MDP-based reasoning either manually defines granular action spaces—sacrificing generalization and domain transfer—or leverages unstructured, open-ended spaces—leading to intractable search and computational inefficiency.

DynaAct introduces a principled framework for dynamic action space construction, targeting two critical properties: scalability (automatic extraction and learning of action spaces from demonstration corpora) and compactness (selection of information-dense candidates at each reasoning step). The framework formulates the candidate action space selection as a subset selection problem over a proxy space, optimizing both utility and diversity using submodular functions.

Figure 1: Overview of the proposed method. Given the proxy action space $\mathcal{A}$, a submodular function selects the optimal candidate subset $\mathcal{A}_t$ for reasoning steps, balancing utility and diversity.

Methodology

Proxy Action Space Extraction

The action space $\mathcal{A}$ is approximated by clustering a large corpus (Open-Platypus, 24,652 problems) into $k$ groups, extracting universal observation sketches via LLM queries. Each observation encodes generalized subgoals or actions, designed for broad applicability and efficient reuse. The resulting proxy space is deduplicated and scaled as needed for domain-specific or cross-disciplinary agents.

Submodular Function Design

To select the compact, optimal subset $\mathcal{A}_t$ at each reasoning step $t$, a submodular function $F(\mathcal{A}_t; s_t)$ is defined:

$$F(\mathcal{A}_t; s_t) = \alpha\, f_{\text{util}}(\mathcal{A}_t; s_t) + \beta\, f_{\text{div}}(\mathcal{A}_t)$$

• $f_{\text{util}}$: Expected utility, measuring relevance to the current state via a log-sum-exp of dot products between learned embeddings.
• $f_{\text{div}}$: Diversity, promoting minimal pairwise similarity within the candidate set.

The embedding function $\mathbf{e}(\cdot)$ is learned using a Q-learning objective, so that $\mathbf{e}(s_t)^\top \mathbf{e}(a)$ estimates the expected value for choosing action $a$ in state $s_t$. Parameters $\alpha$ and $\beta$ control the utility-diversity trade-off, empirically shown to favor utility for best performance.

Greedy Submodular Maximization

Given the proven submodular structure (see Appendix for proof), the combinatorial optimization of candidate selection ($|\mathcal{A}_t| = m$) is efficiently solved using a greedy algorithm with linear complexity in $|\mathcal{A}|$. Embedding caching further reduces runtime for batch inference.

Sequential Reasoning Pipeline

Action selection at each step is governed by Monte Carlo Tree Search (MCTS), estimating $Q(s_t, a)$ via simulated rollouts, balancing exploration and exploitation. Only the compact candidate set is explored, dramatically reducing computational cost relative to open-ended autoregressive generation. The base LLM remains frozen; only the embedding model for submodular selection is fine-tuned.

Experimental Evaluation

Benchmarks and Results

Experiments span six benchmarks in general, reasoning, and math domains: MMLU, MMLU-Pro, GPQA, ARC-Challenge, GSM8K, and MATH-500. DynaAct consistently outperforms strong baselines (Zero-shot CoT, self-consistency, RAP, and rStar) across all tasks, with a notable gain of 6.8% accuracy over rStar on MATH-500—indicating significant advancement in complex mathematical reasoning.

• The method's gains are most pronounced in tasks with deep reasoning requirements, confirming the impact of dynamic, state-optimized action spaces.

Ablations and Analysis

Component ablations demonstrate that both the utility and diversity terms, as well as explicit Q-learning, are indispensable for optimal performance. Removing any component leads to accuracy degradation, especially on high-difficulty math problems.

Compactness studies confirm that DynaAct achieves higher accuracy with smaller action spaces; increasing candidate set size in suboptimal methods (e.g., RAP) yields only marginal improvement, indicating redundancy.

Figure 2: Comparison of RAP and DynaAct on different candidate set sizes and rollouts for MATH-500, showing compactness and superior scaling efficiency for DynaAct.

Utility analysis demonstrates increased "critical step coverage"—the fraction of solutions containing essential insights in problem-solving—relative to manually designed broad action spaces (rStar).

Latency benchmarks show that DynaAct introduces only minor runtime overhead compared to baselines, with the main computational cost attributed to embedding forward passes, which are efficiently cached and reused.

Robustness and Scalability

Performance remains stable as the proxy action space is scaled from $40,000$ to $1,000,000$ actions, confirming the framework's suitability for large corpora. The greedy algorithm and embedding caching contribute to manageable resource demands.

Sensitivity analysis on utility-diversity weighting indicates that utility must be strongly prioritized; diversity substantially enhances compactness and robustness, but overemphasizing it leads to suboptimal accuracy.

Case studies across general and math domains illustrate enhanced stepwise reasoning, selection of problem-relevant actions, and transparent traceability in generated solutions.

Figure 3: Case paper: solution to inscribed hexagon angle problem, showing reasoning progression and explicit action selection.

Theoretical and Practical Implications

The proposed dynamic action space construction framework provides a scalable, data-driven alternative to manual action engineering or costly brute-force reasoning. By leveraging submodular optimization, action selection is both tractable and effective, supporting generalization across tasks and domains.

DynaAct's robust mathematical justification and empirical evidence suggest the method is orthogonal to test-time search strategies—future work can extend compact action space selection to alternative search algorithms beyond MCTS.

Practical applications include interpretable reasoning agents, educational tutors, scientific assistants, and automated theorem solvers—domains where efficiency and transparency in decision-making are essential. The explicit structure in reasoning traces facilitates further auditing and controllability.

Limitations and Future Directions

DynaAct is dependent on the quality of extracted observation sketches for its proxy action space, and on the chosen backbone LLM for transition modeling. Computational efficiency in extremely large or specialized domains may require additional optimization or integration with more advanced search and planning strategies. Addressing potential bias and safe deployment in data-driven action extraction is a critical direction for real-world impact.

Future research can integrate domain-adaptive backbone models, investigate joint learning of transition and value functions, or extend the submodular framework for multi-agent or multimodal reasoning settings. Exploring synergy with code-form planning and hybrid symbolic-LLM frameworks may broaden the range of tractable sequential reasoning tasks.

Conclusion

DynaAct formalizes dynamic action space construction for LLM reasoning as a submodular optimization problem, achieving strong empirical results, especially in combinatorial and mathematical tasks. The framework’s automatic, scalable action selection strategy offers concrete improvements over manual or static heuristics, and underpins efficient, interpretable LLM reasoning for complex problem-solving scenarios. This work sets a new direction for flexible and principled reasoning architectures in AI systems.