Submodular Function Design

• Submodular Function Design is the construction of set functions exhibiting a diminishing returns property, ensuring efficient subset selection.
• It underpins dynamic action space construction by balancing state-conditional utility with diverse candidate selection, as demonstrated in DynaAct.
• The approach employs greedy algorithms that achieve near-optimal approximation guarantees, offering practical scalability and enhanced inference accuracy.

Submodular function design concerns the construction of set functions $F: 2^V \to \mathbb{R}$ that exhibit the diminishing returns property—specifically, submodularity—for the purpose of optimizing selection in combinatorial environments. In contemporary decision-making systems, especially those involving large candidate action spaces, submodular function design governs the greedy selection of compact, informative subsets that balance utility and diversity according to rigorous mathematical criteria. This principle underpins recent frameworks for dynamically constructing action spaces in sequential reasoning with LLMs, most notably embodied in DynaAct (Zhao et al., 11 Nov 2025), where an explicit submodular objective enables efficient, state-conditional extraction of action sets.

1. Submodular Functions: Definition and Key Properties

A set function $F: 2^V \to \mathbb{R}$, with ground set $V$, is submodular if, for all $A \subseteq B \subseteq V$ and $x \in V\setminus B$,

$F(A \cup \{x\}) - F(A) \ge F(B \cup \{x\}) - F(B).$

This encodes the principle of diminishing returns: adding $x$ to a smaller set provides a greater or equal incremental gain compared to adding it to a larger set. Submodular functions are monotonic if $F(A) \le F(B)$ whenever $A \subseteq B$, but monotonicity is not required for all results. Submodularity ensures that greedy maximization algorithms can achieve strong approximation guarantees for subset selection problems.

The standard result is that, for maximizing a monotone submodular function $F$ under a cardinality constraint $|C| \leq k$, the greedy algorithm achieves at least a $(1-1/e)$-approximation to the optimal value.

2. Motivations for Submodular Function Design in Action Selection

In the context of dynamic action space construction for sequential decision-making (e.g., LLM-based reasoning), exhaustive search over the full candidate set $\mathcal{A}$ is infeasible. The principle is to construct, per current state $s_t$, a small action subset $\mathcal{A}_t \subseteq \mathcal{A}$ of fixed size $m$ that is maximally effective for inference. Submodular functions are used to formalize the desiderata of (i) state-conditional utility (actions relevant to $s_t$) and (ii) diversity (coverage of distinct strategies or perspectives), enabling a tractable greedy selection that leverages their well-established approximation guarantees.

3. Concrete Formulation: Joint Utility and Diversity

In DynaAct (Zhao et al., 11 Nov 2025), the submodular function is defined as: $$F(C; s_t) = \alpha\, f_{\text{util}}(C; s_t) + \beta\, f_{\text{div}}(C)$$ with $\alpha, \beta \ge 0$. For a candidate set $C \subseteq \mathcal{A}$:

• The utility term: $$f_{\text{util}}(C; s_t) = \log\left(\sum_{a\in C} \exp\bigl(\mathbf{e}(s_t)^{\top}\mathbf{e}(a)\bigr)\right)$$ where $\mathbf{e}(\cdot)$ denotes learned embeddings that approximately align with $Q(s_t, a)$. This promotes selection of actions that are individually relevant to the current context.
• The diversity term: $$f_{\text{div}}(C) = \sum_{a_i\in C} \min_{a_j\in C,\,j \ne i} \left[1 - \mathbf{e}(a_i)^{\top}\mathbf{e}(a_j)\right]$$ encourages a set of actions whose embeddings are as far apart as possible, thus discouraging redundancy.

Lemma 1 (Zhao et al., 11 Nov 2025) shows that such a function, being a non-negative linear combination of submodular terms, retains submodularity—crucial for enabling the efficient selection of high-quality action sets.

4. Greedy Maximization and Approximation Guarantee

Selecting the best subset of size $m$ at each decision point entails solving: $\max_{C \subseteq \mathcal{A},\,|C| = m} F(C; s_t)$ The standard greedy algorithm initializes $X = \emptyset$ and iteratively augments it by adding the candidate $a^*$ which maximizes the marginal gain $F(X \cup \{a\}; s_t) - F(X; s_t)$, until $|X| = m$. For monotonic submodular $F$, the greedy set $X$ satisfies $F(X) \geq (1 - 1/e) F(C^*)$ where $C^*$ is the optimal subset.

Computational complexity is $O(m^2 |\mathcal{A}|)$, as each of the $m$ selection rounds involves scanning all remaining candidates and evaluating $F$ (which is $O(m)$ per evaluation if embedding dot products are cached).

5. Application: Dynamic Action Space Construction in Sequential Reasoning

In LLM reasoning systems that perform Markov Decision Process (MDP)-based multi-step inference, the full action set $\mathcal{A}$ may be automatically mined from corpora via LLM-based "sketch extraction". For each state $s_t$, a submodular objective as above is used to select $\mathcal{A}_t \subseteq \mathcal{A}$ of cardinality $m$. The selected set then serves as the search space for the subsequent inference phase (e.g., via Monte Carlo Tree Search) (Zhao et al., 11 Nov 2025).

The submodular design is critical for scalability: compared to prior manual or unstructured action spaces, the greedy submodular selection of DynaAct achieves denser, more informative candidate sets, empirically improving both efficiency and downstream task accuracy on a range of benchmarks (e.g., +6.8 accuracy on MATH-500 compared to rStar and RAP baselines). Ablation studies confirm substantial drops in performance when either the utility or diversity terms are omitted or if submodular maximization is replaced by a non-submodular scheme.

6. Empirical Results and Theoretical Analysis

Empirically, DynaAct demonstrates performance gains across classification, math, and reasoning datasets against zero-shot chain-of-thought, self-consistency, and fixed-action MCTS baselines. Main results include:

Model	MMLU	MMLU-Pro	GPQA	ARC-C	GSM8K	MATH-500
DynaAct	70.22	51.40	39.39	88.31	89.16	61.00
rStar	68.61	48.81	36.87	86.43	87.11	54.20
RAP	69.46	48.70	38.89	85.41	87.79	51.60

Latency overhead per example remains moderate (e.g., 57.60s for DynaAct on MATH-500 vs 54.72s for rStar) due to the compactness of the selected action set ($m=5$).

The approximation guarantee from submodular greedy maximization ensures that this efficiency/accuracy result is not accidental but rooted in the inherent structure of the designed objective function.

7. Extensions, Limitations, and Open Challenges

Limitations of submodular function design in this context include dependency on embedding quality (which may be limited by the underlying LLM), reliance on the availability and representativeness of the consolidated action corpus, and the compute costs of downstream search (e.g., MCTS).

Extensions under consideration include domain-specific or multilingual proxy construction for $\mathcal{A}$, feedback-driven online refinement of the candidate action pool during inference, and generalization of the objective to incorporate mutual information or end-to-end learned policy/value networks.

Alternative approaches could involve beam search or more efficient UCT variants as replacements for MCTS, as well as further research into submodular function classes better suited for structured domains.

---

Submodular function design serves as the theoretical and practical foundation for efficient subset selection in modern dynamic action space construction, balancing relevance and diversity in a principled manner that is provably near-optimal. This approach is particularly impactful in scenarios—such as LLM-based reasoning—where candidate action sets are combinatorially large but standard exhaustive or manual approaches are impractical (Zhao et al., 11 Nov 2025).