---
title: 'ParaSDM: Parameterized Sequential Decision-Making'
url: https://www.emergentmind.com/topics/parametrized-sequential-decision-making-parasdm
type: topic
---

# ParaSDM: Parameterized Sequential Decision-Making

Parametrized Sequential Decision-Making (ParaSDM) denotes a class of sequential decision problems in which the policy, objective, model, action space, or environment depends on parameters that vary across instances or over time. In the cited literature, those parameters range from environment and risk variables in stochastic-reward MDPs, to the size and topology of parameterized MDPs, to state and action parameters in maximum-entropy control, to shared continuous design variables such as facility locations, and to continuous arguments attached to discrete actions [1907.04269], [2410.18293], [2006.09646], [2507.22338], [2512.20831]. The literature suggests that ParaSDM is best understood as a family of related formalisms rather than a single universally fixed definition: what unifies them is the requirement to map parameter variation to policies, values, risks, or design choices quickly enough to remain effective when tasks, constraints, or system scales change.

## 1. Conceptual scope and parameterization

A central distinction in ParaSDM is the object being parameterized. In the risk-sensitive scheme of “A Scheme for Dynamic Risk-Sensitive Sequential Decision Making,” the parameter space $\Theta$ is factored into environment/process parameters $q_e$ and risk parameters $q_p$, and a neural function $f:\Theta \to (\text{policy representation}, \text{risk measures})$ is trained to approximate an ideal decision-maker $\nu(\theta)$ [1907.04269]. In probabilistic verification, a parameterized MDP is a family $M(\theta)=(S_\theta,A_\theta,P_\theta,s_\theta,G_\theta)$ in which $\theta$ changes the state space, action space, transition probabilities, and possibly rewards; the synthesis objective is then to obtain a single parameter-agnostic policy that remains effective for large instances [2410.18293].

A different axis of parameterization appears in “Towards a Unified Framework for Sequential Decision Making,” where an SDM task is defined as $T=(M_{\text{train}},M_{\text{test}})$ and context-aware policies receive an MDP context $\mu$, drawn from subsets of $\{S,A,T,C,s_i,s_g,D,V\}$, to support generalization across tasks [2310.02167]. In “Parameterized MDPs and Reinforcement Learning Problems -- A Maximum Entropy Principle Based Framework,” the parameter vector is $\theta=(\zeta,\eta)$, with state parameters $\zeta$ and action parameters $\eta$, and the goal is to determine both the optimal policy and the optimal parameters [2006.09646]. In “Contextual Preference Distribution Learning,” the learned object is a context-conditioned distribution $q_\theta(c\mid X)$ over latent preference coefficients, which is then used to generate scenarios for downstream risk-averse optimization [2603.17139].

ParaSDM also encompasses formulations in which the sequential choice itself is hybrid. In “Context-Sensitive Abstractions for Reinforcement Learning with Parameterized Actions,” the grounded action space is
\[
\tilde{A}=\bigcup_{a_d\in A_d}\{a_d\}\times X_{a_d},
\]
so the agent must choose both a discrete action label and a continuous parameter vector governing its execution [2512.20831]. A related but structurally distinct variant appears in “Parameterized Exploration,” where only the exploration component of a policy is parameterized, through schedules such as $\eta(T,t,\theta)$ that depend on the horizon and the current state of knowledge of the dynamics [1907.06090]. In parameterized games, the parameter vector $\lambda$ may affect payoffs, action spaces, or information states, and the operational objective becomes fast online selection of a good strategy from an offline-constructed library or parametric decision list [2104.14744].

This variety implies that ParaSDM is not restricted to one modeling tradition. It touches risk-sensitive RL, verification, planning, game theory, hybrid-action RL, structured optimization, and graph-based decision systems [1907.04269], [2410.18293], [2310.02167], [2104.14744], [2603.19501].

## 2. Formal models, returns, and utility criteria

One canonical ParaSDM formalization is the parameterized MDP family. For reachability objectives, a finite MDP is written as $M=(S,A,P,s_0,G)$, and a ParaMDP is a family $M(\theta)=(S_\theta,A_\theta,P_\theta,s_\theta,G_\theta)$ indexed by $\theta\in\Theta$ [2410.18293]. For fixed $\theta$, the optimal Bellman equation for discounted reward takes the standard form
\[
V_\theta^*(s)=\max_{a\in A(s)}\sum_{s'}P_\theta(s' \mid s,a)\,[R_\theta(s,a)+\gamma V_\theta^*(s')],
\]
whereas reachability probabilities satisfy corresponding linear optimality equations [2410.18293].

Risk-sensitive ParaSDM augments this picture with explicit return moments. In the stochastic-reward MDP of [1907.04269], the discounted return is
\[
\Phi=\sum_{t=0}^{T-1}\gamma^tR_t,
\]
with
\[
\mu_R(\pi,\theta)=\mathbb{E}_{\pi,\theta}[\Phi],\qquad
\sigma_R^2(\pi,\theta)=\mathrm{Var}_{\pi,\theta}[\Phi].
\]
The paper studies objectives and constraints expressed through these moments, including the mean–variance criterion
\[
J(\pi;\theta)=\mu_R(\pi,\theta)-\lambda\,\sigma_R^2(\pi,\theta),
\]
and constrained forms such as $\mu_R(\pi,\theta)/\sigma_R^2(\pi,\theta)\ge q$ [1907.04269]. Under Normal returns, Value-at-Risk and Conditional VaR reduce to explicit functions of $\mu_R$ and $\sigma_R$,
\[
\mathrm{VaR}_{\alpha}=\mu_R-z_\alpha\sigma_R,\qquad
\mathrm{CVaR}_{\alpha}=\mu_R-\frac{\sigma_R\phi(z_\alpha)}{1-\alpha},
\]
which is the basis for the paper’s claim that most law-invariant risk measures used in RL practice can be evaluated or estimated from mean and variance of return under mild assumptions [1907.04269].

A second major formal line is maximum-entropy control. In [2006.09646], the parameterized cost objective is
\[
J^{\pi}_{\theta}(s)=
\mathbb{E}_{\tau\sim p_{\pi,\theta}(\cdot\mid s_0=s)}
\Bigg[
\sum_{t=0}^{\infty}\gamma^t\,c_{\theta}\big(x_t(\zeta),u_t(\eta),x_{t+1}(\zeta)\big)
\Bigg],
\]
and the optimization is regularized by trajectory Shannon entropy, yielding a free-energy objective of the form $H(\tau)-\beta\,\mathbb{E}[\sum_t\gamma^t c_\theta]$ [2006.09646]. The induced optimal policy has Gibbs/Boltzmann form, and the associated Bellman operator is a contraction. “Time-Varying Parameters in Sequential Decision Making Problems” adopts the same MEP foundation but treats subsets of the parameters as time-varying and manipulable, using the smooth free-energy as a control Lyapunov function [2201.10273].

An axiomatic generalization of return structure is given in “Utility Theory for Sequential Decision Making.” Under the VNM axioms plus memorylessness, there exist $r:S\times A\times S\to\mathbb{R}$ and $m:S\times A\times S\to\mathbb{R}_{>0}$ such that
\[
u(t\cdot \tau)=r(t)+m(t)u(\tau),
\]
which motivates Affine-Reward MDPs and the Bellman recursion
\[
V(s)=\max_{a\in A}\mathbb{E}_{s'\sim T(\cdot\mid s,a)}[\,r(s,a,s')+\beta(s,a,s')V(s')\,].
\]
Stronger additivity forces $\beta\equiv 1$, recovering cumulative scalar rewards, while path-obliviousness yields potential-difference utilities of the form $u(\tau)=\Phi(s')-\Phi(s)$ [2206.13637].

A further extension arises in hybrid action spaces. In [2512.20831], the optimal action-value function for parameterized actions is
\[
Q^*(s,a_d,a_c)=r(s,a_d,a_c)+\gamma\,\mathbb{E}_{s'}\Big[\max_{a'_d\in A_d,\;a'_c\in X_{a'_d}}Q^*(s',a'_d,a'_c)\Big].
\]
This formalization makes explicit that ParaSDM can concern simultaneous optimization over discrete choices and continuous execution parameters rather than only task-level environment parameters.

## 3. Learning and synthesis mechanisms

One class of ParaSDM methods learns an explicit parameter-to-policy map. In [1907.04269], the workflow is: sample $\theta$ from prespecified intervals; build the stochastic-reward MDP $\mathcal{M}(\theta)$; enumerate all deterministic policies $\pi\in\Pi$; apply the state-augmentation transformation (SAT) to obtain deterministic state-based rewards; compute mean and variance using Sobel’s variance formula; evaluate the chosen risk objective; and train a feed-forward neural network with MSE loss and Adam so that $f(\theta)$ outputs both a compact policy representation and risk measures. SAT is critical because it preserves the probability measure on trajectories while converting reward randomness into a form where
\[
v=(I-\gamma P)^{-1}r,\qquad
\psi=(I-\gamma^2P)^{-1}\theta
\]
can be computed exactly [1907.04269].

A different mechanism is symbolic generalization from small solved instances. The “1–2–3–Go!” pipeline first selects a small set of feasible base instances, solves them with exact model checking, collects reachable non-goal state–action labels, and then trains an axis-aligned decision tree using Gini impurity until leaves are pure [2410.18293]. The learned policy $\pi_{\mathrm{DT}}$ can then be applied to arbitrarily large parameter instances without explicit construction of their full state spaces. The method emphasizes “generalizability by explainability”: predicates over structured state variables are reused as a compact policy description across scales [2410.18293].

Maximum-entropy approaches constitute a third family. In [2006.09646], the inner loop solves a soft Bellman fixed point and the outer loop updates state and action parameters using gradients of the free energy. “Parametrized Multi-Agent Routing via Deep Attention Models” specializes this perspective to Facility-Location and Path Optimization (FLPO): the Maximum Entropy Principle defines a Gibbs distribution over paths, and a Shortest Path Network (SPN), described as a permutation-invariant encoder-decoder, amortizes the stage-wise Gibbs policy so that shared continuous parameters such as facility locations can be optimized by backpropagation through sampled path costs [2507.22338].

Other methods place the learned parameterization on uncertainty models rather than directly on policies. In [2603.17139], the model is an amortized, context-conditioned distribution $q_\theta(c\mid X)$ trained by squared moment matching and a bounded-variance score-function estimator. The learned preference distribution is not the terminal output; it feeds a second-stage scenario-based CVaR optimization. In [2512.20831], by contrast, the main learned object is an abstraction. State–Parameterized-Action Conditional Abstraction Trees (SPA-CATs) and Action Parameter Trees progressively refine both state partitions and action-parameter partitions online, and a tabular abstract $Q$ function is updated with TD($\lambda$). The heterogeneity score
\[
H(\bar{s},\bar{a})=\beta\cdot SD[\delta(\bar{s},\bar{a})]+(1-\beta)\cdot SD[\hat V(s)\mid s\in\bar s]
\]
drives refinement toward regions where greater representational resolution improves performance [2512.20831].

The breadth of these mechanisms suggests that ParaSDM is as much about amortization and structure exploitation as about parameterization itself. Some methods amortize exact planning or model checking; some amortize risk-sensitive optimization; some amortize scenario generation; and some amortize representation learning. That interpretation is inferential, but it aligns with the recurring pattern that offline structure is exploited to reduce online decision latency.

## 4. Time variation, generalization, and scaling

Several ParaSDM papers are explicitly motivated by time variation. In [1907.04269], deployment assumes $\theta_t=(q_{e,t},q_{p,t})$ changes over epochs as prices, demand, reliability, and risk appetite evolve, so training samples $\theta$ over specified intervals to cover these dynamics. The same paper recommends periodic resampling and retraining when new regimes emerge, and it notes that heavy tails or seasonal spikes may require richer labels than Normal-based VaR/CVaR mappings [1907.04269].

A stronger treatment of time variation is given in [2201.10273]. There, the parameter vector is partitioned into pre-specified parameters with known dynamics and manipulable parameters with control-designed dynamics. The sum of entropy-regularized optimal values,
\[
\mathcal{V}(\Upsilon)=\sum_s V_{\beta\Upsilon}^*(s),
\]
is used as a control Lyapunov function. The resulting feedback law guarantees $\dot{\mathcal V}\le 0$, asymptotic convergence of the manipulable-parameter gradient to zero, and a control that is Lipschitz continuous and bounded under the stated assumptions [2201.10273]. This moves ParaSDM from static amortization toward online tracking of local optima.

Another route to time variation is reformulation. “Towards Enabling Learning for Time-Varying finite horizon Sequential Decision-Making Problems” proposes topography lifting, which reinterprets a finite-horizon, time-varying Para-SDM as an equivalent time-invariant problem by stage-indexed duplication of entities and strict action-topology constraints [2504.02129]. Under that construction, every feasible finite-horizon path corresponds to a path in the lifted problem with the same cumulative cost, and the policy correspondence is one-to-one. The practical aim is to make time-varying problems accessible to learning methods that rely on stationarity [2504.02129].

Scaling across instance size rather than time is central in verification and games. In [2410.18293], large $\theta$ values cause state-space explosion, and decision-tree policies are used to bypass explicit exploration of huge ParaMDPs. In “Human strategic decision making in parametrized games,” the same challenge is addressed through sample-based strategy libraries and parametric decision lists $L=(F,O,S)$, with an online rule of the form “If $f_i(\lambda)\,o_i\,0$ then play $s_i$; else …” [2104.14744]. For one-dimensional $\lambda$ and continuous payoffs, the paper proves
\[
\lim_{t\to\infty}\mathbb{E}[\epsilon_t]=0,
\]
where $\epsilon_t$ is the exploitability of the nearest-neighbor strategy selected from $t$ sampled parameter points [2104.14744].

A related but narrower scaling issue appears in exploration control. In [1907.06090], only the exploration schedule is parameterized, but the tuning is explicitly horizon-aware and knowledge-aware. Schedules such as
\[
\eta(T,t,\theta)=\frac{\theta_0}{1+\exp[-\theta_2(T-t-\theta_1)]}
\]
or $\varepsilon_t=\sigma(\theta_0+\theta_1 t/T+\theta_2U_t)$ encode dependence on both the remaining horizon and uncertainty features. The outer optimization over $\theta$ is done by Gaussian process Bayesian optimization using a model of the environment or a confidence distribution over models [1907.06090].

Finally, time-varying graph expansion produces another form of ParaSDM. In [2603.19501], the state is $(G_t,x_t,\theta_t)$, the action is a tap-update vector for a graph filter, and the transition law includes stochastic graph growth. The filter taps are treated as agents, and a context-aware graph neural network parameterizes a multi-agent policy that updates the filter to optimize discounted cumulative reward over the expansion horizon [2603.19501].

## 5. Applications and reported empirical behavior

ParaSDM has been instantiated in inventory control, quantitative verification, 5G small-cell placement, ridesharing, mobile health, expanding-graph filtering, hybrid-action control, and multi-agent routing. The reported outcomes vary by formulation, but they consistently evaluate whether parameter-aware structure improves either decision quality, computational cost, or both.

| Instantiation | Setting | Reported outcome |
|---|---|---|
| Risk-sensitive ParaSDM [1907.04269] | Inventory control with two suppliers, stochastic rewards, and VaR objective | Dataset size $\approx 10^5$; convergence within $\approx 10$ epochs; empirical “hit rate” on both training and validation $\approx 95\%$ |
| Decision-tree ParaMDP synthesis [2410.18293] | 21 model+property instances from QVBS-style benchmarks | Near-optimal values in 13/21 cases; in 2 additional cases, it outperforms Smart LSS |
| Maximum-entropy parameterized MDP [2006.09646] | 5G small cell network design | Joint design yields costs “as low as 65% of the former”; model-based vs. model-free solutions differ by $\sim 1.9\%$ and $\sim 0.3\%$; entropy-over-paths gives a $\sim 5\%$ lower cost than the policy-entropy-only variant; no annealing gives $\approx 11\%$ higher cost |
| Parameterized exploration [1907.06090] | mHealth glucose-control MDP | At $T=25$, tuned AR(2) linear MCRew is $-13.08$ vs. $-17.57$ for $\varepsilon=0.05$; at $T=50$, tuned AR(2) linear MCRew is $-16.92$ vs. $-33.83$ for $\varepsilon=0.5\,t^{-1}$ |
| Time-varying Para-SDM control [2201.10273] | Dynamic multi-UAV communication network | Frame-by-frame re-optimization is reported to be approximately $10^4$ times more computationally expensive |
| Topography lifting [2504.02129] | Small cell network / FLPO-style finite-horizon Para-SDM | Near-identical optimal costs to the time-varying solution with significantly better runtimes across 10 scenarios |
| Contextual preference distribution learning [2603.17139] | Synthetic ridesharing environment with CVaR assignment | Average post-decision surprise reduced by up to $114\times$ vs. a risk-neutral approach with perfect predictions and up to $25\times$ vs. leading risk-averse baselines |
| Expanding-network graph filtering [2603.19501] | Synthetic graphs, MovieLens-100K cold-start, COVID prediction | G-MARL achieves the lowest RMSE across the reported tasks |
| Deep attention FLPO [2507.22338] | Parametrized multi-agent routing with shared facility locations | Up to $100\times$ speedup in policy inference and gradient computation; average optimality gap of approximately $6\%$; over $10\times$ lower cost than metaheuristic baselines; matches Gurobi’s optimal cost with annealing at a $1500\times$ speedup |
| Context-sensitive abstractions [2512.20831] | OfficeWorld, Pinball, Multi-City Transport, Robot Soccer Goal | Across 50 seeds, PEARL-flexible and PEARL-uniform outperform MP-DQN and HyAR in cumulative return during training and success probability of the greedy policy; HyAR learns only in Soccer; MP-DQN fails across tasks |

Taken together, these results indicate that ParaSDM methods are especially effective when parameter variation has exploitable regularity: reusable predicates in verification, reusable path structure in routing, reusable risk structure in stochastic control, or reusable context-to-distribution mappings in preference-driven optimization. That conclusion is interpretive, but it is consistent with the empirical pattern across the surveyed papers.

## 6. Limitations, contested boundaries, and open directions

The surveyed literature does not support a single canonical boundary for ParaSDM. Some papers treat it as policy generalization across parameterized task families [2410.18293], some as joint policy–parameter optimization [2006.09646], some as dynamic parameter tracking [2201.10273], some as contextual uncertainty learning for downstream optimization [2603.17139], and some as hybrid discrete–continuous control [2512.20831]. The literature therefore suggests that reducing ParaSDM to parameterized actions alone, or to parameterized MDP size alone, is too narrow.

Method-specific limitations are explicit. In the risk-sensitive framework of [1907.04269], VaR/CVaR estimates based only on $\mu_R$ and $\sigma_R$ are biased for heavy-tailed or skewed returns, the deterministic policy space grows combinatorially, and generalization to out-of-distribution $\theta$ is not guaranteed. In [2410.18293], there is no sample-complexity analysis or PAC-style generalization bound, and the paper identifies failure modes when policies do not generalize across horizons or when changing transition probabilities alter optimal decisions in ways not captured by state-variable predicates. In [1907.06090], performance degrades with model misspecification, and the paper explicitly notes the absence of formal regret bounds.

The scaling papers also leave open theoretical questions. In [2603.19501], no convergence or regret guarantees are provided for the growing-graph MARL setting, and stability of the time-varying filters is not theoretically analyzed. In [2512.20831], SPA-CATs are motivated by grouping behaviorally similar states and actions, but the paper states that it does not provide formal bisimulation-type conditions or performance-loss bounds; abstractions only refine and do not merge. In [2201.10273], convergence is to stationary points of the entropy-regularized objective, not to guaranteed global optima. In [2603.17139], misspecification of the learned distribution family and covariate drift are identified as sources of degradation, and richer families such as normalizing flows or copulas are proposed as extensions.

Open directions recur across subfields. The verification literature proposes active learning over $\theta$, feature learning, hybrid verification-learning loops, and symbolic integration [2410.18293]. Risk-sensitive ParaSDM points toward richer labels such as empirical quantiles via simulation when Normal or elliptical approximations fail [1907.04269]. Preference-based ParaSDM points toward online or continual learning and richer distributional models [2603.17139]. Expanding-network ParaSDM raises questions about safe policy updates, Lyapunov critics, and stability-aware parameterizations [2603.19501]. Hybrid-action abstraction methods raise the complementary question of whether state and action abstractions can be endowed with formal value-loss guarantees while retaining the empirical sample-efficiency gains already observed [2512.20831].

These open problems are not peripheral. They define the current frontier of ParaSDM: how to obtain reusable parameter-aware policies and decision rules without losing calibration, robustness, interpretability, or theoretical control as parameters drift, action spaces hybridize, and system scales expand.

Source: https://www.emergentmind.com/topics/parametrized-sequential-decision-making-parasdm