ParaSDM: Parameterized Sequential Decision-Making

• ParaSDM is a framework for sequential decision-making where policies, objectives, models, and actions adapt based on varying parameters.
• It encompasses diverse paradigms including risk-sensitive MDPs, maximum-entropy control, and hybrid-action formulations to tackle practical challenges.
• Methods in ParaSDM leverage amortization and symbolic generalization to reduce online computation and enhance scalability in applications.

Parametrized Sequential Decision-Making (ParaSDM) denotes a class of sequential decision problems in which the policy, objective, model, action space, or environment depends on parameters that vary across instances or over time. In the cited literature, those parameters range from environment and risk variables in stochastic-reward MDPs, to the size and topology of parameterized MDPs, to state and action parameters in maximum-entropy control, to shared continuous design variables such as facility locations, and to continuous arguments attached to discrete actions (Ma et al., 2019, Azeem et al., 2024, Srivastava et al., 2020, Basiri et al., 30 Jul 2025, Nayyar et al., 23 Dec 2025). The literature suggests that ParaSDM is best understood as a family of related formalisms rather than a single universally fixed definition: what unifies them is the requirement to map parameter variation to policies, values, risks, or design choices quickly enough to remain effective when tasks, constraints, or system scales change.

1. Conceptual scope and parameterization

A central distinction in ParaSDM is the object being parameterized. In the risk-sensitive scheme of “A Scheme for Dynamic Risk-Sensitive Sequential Decision Making,” the parameter space $\Theta$ is factored into environment/process parameters $q_e$ and risk parameters $q_p$, and a neural function $$f:\Theta \to (\text{policy representation}, \text{risk measures})$$ is trained to approximate an ideal decision-maker $\nu(\theta)$ (Ma et al., 2019). In probabilistic verification, a parameterized MDP is a family $$M(\theta)=(S_\theta,A_\theta,P_\theta,s_\theta,G_\theta)$$ in which $\theta$ changes the state space, action space, transition probabilities, and possibly rewards; the synthesis objective is then to obtain a single parameter-agnostic policy that remains effective for large instances (Azeem et al., 2024).

A different axis of parameterization appears in “Towards a Unified Framework for Sequential Decision Making,” where an SDM task is defined as $T=(M_{\text{train}},M_{\text{test}})$ and context-aware policies receive an MDP context $\mu$, drawn from subsets of $\{S,A,T,C,s_i,s_g,D,V\}$, to support generalization across tasks (Núñez-Molina et al., 2023). In “Parameterized MDPs and Reinforcement Learning Problems -- A Maximum Entropy Principle Based Framework,” the parameter vector is $q_e$0, with state parameters $q_e$1 and action parameters $q_e$2, and the goal is to determine both the optimal policy and the optimal parameters (Srivastava et al., 2020). In “Contextual Preference Distribution Learning,” the learned object is a context-conditioned distribution $q_e$3 over latent preference coefficients, which is then used to generate scenarios for downstream risk-averse optimization (Hudson et al., 17 Mar 2026).

ParaSDM also encompasses formulations in which the sequential choice itself is hybrid. In “Context-Sensitive Abstractions for Reinforcement Learning with Parameterized Actions,” the grounded action space is

$q_e$4

so the agent must choose both a discrete action label and a continuous parameter vector governing its execution (Nayyar et al., 23 Dec 2025). A related but structurally distinct variant appears in “Parameterized Exploration,” where only the exploration component of a policy is parameterized, through schedules such as $q_e$5 that depend on the horizon and the current state of knowledge of the dynamics (Clifton et al., 2019). In parameterized games, the parameter vector $q_e$6 may affect payoffs, action spaces, or information states, and the operational objective becomes fast online selection of a good strategy from an offline-constructed library or parametric decision list (Ganzfried, 2021).

This variety implies that ParaSDM is not restricted to one modeling tradition. It touches risk-sensitive RL, verification, planning, game theory, hybrid-action RL, structured optimization, and graph-based decision systems (Ma et al., 2019, Azeem et al., 2024, Núñez-Molina et al., 2023, Ganzfried, 2021, Gao et al., 19 Mar 2026).

2. Formal models, returns, and utility criteria

One canonical ParaSDM formalization is the parameterized MDP family. For reachability objectives, a finite MDP is written as $q_e$7, and a ParaMDP is a family $q_e$8 indexed by $q_e$9 (Azeem et al., 2024). For fixed $q_p$0, the optimal Bellman equation for discounted reward takes the standard form

$q_p$1

whereas reachability probabilities satisfy corresponding linear optimality equations (Azeem et al., 2024).

Risk-sensitive ParaSDM augments this picture with explicit return moments. In the stochastic-reward MDP of (Ma et al., 2019), the discounted return is

$q_p$2

with

$q_p$3

The paper studies objectives and constraints expressed through these moments, including the mean–variance criterion

$q_p$4

and constrained forms such as $q_p$5 (Ma et al., 2019). Under Normal returns, Value-at-Risk and Conditional VaR reduce to explicit functions of $q_p$6 and $q_p$7,

$q_p$8

which is the basis for the paper’s claim that most law-invariant risk measures used in RL practice can be evaluated or estimated from mean and variance of return under mild assumptions (Ma et al., 2019).

A second major formal line is maximum-entropy control. In (Srivastava et al., 2020), the parameterized cost objective is

$q_p$9

and the optimization is regularized by trajectory Shannon entropy, yielding a free-energy objective of the form $$f:\Theta \to (\text{policy representation}, \text{risk measures})$$0 (Srivastava et al., 2020). The induced optimal policy has Gibbs/Boltzmann form, and the associated Bellman operator is a contraction. “Time-Varying Parameters in Sequential Decision Making Problems” adopts the same MEP foundation but treats subsets of the parameters as time-varying and manipulable, using the smooth free-energy as a control Lyapunov function (Srivastava et al., 2022).

An axiomatic generalization of return structure is given in “Utility Theory for Sequential Decision Making.” Under the VNM axioms plus memorylessness, there exist $$f:\Theta \to (\text{policy representation}, \text{risk measures})$$1 and $$f:\Theta \to (\text{policy representation}, \text{risk measures})$$2 such that

$$f:\Theta \to (\text{policy representation}, \text{risk measures})$$3

which motivates Affine-Reward MDPs and the Bellman recursion

$$f:\Theta \to (\text{policy representation}, \text{risk measures})$$4

Stronger additivity forces $$f:\Theta \to (\text{policy representation}, \text{risk measures})$$5, recovering cumulative scalar rewards, while path-obliviousness yields potential-difference utilities of the form $$f:\Theta \to (\text{policy representation}, \text{risk measures})$$6 (Shakerinava et al., 2022).

A further extension arises in hybrid action spaces. In (Nayyar et al., 23 Dec 2025), the optimal action-value function for parameterized actions is

$$f:\Theta \to (\text{policy representation}, \text{risk measures})$$7

This formalization makes explicit that ParaSDM can concern simultaneous optimization over discrete choices and continuous execution parameters rather than only task-level environment parameters.

3. Learning and synthesis mechanisms

One class of ParaSDM methods learns an explicit parameter-to-policy map. In (Ma et al., 2019), the workflow is: sample $$f:\Theta \to (\text{policy representation}, \text{risk measures})$$8 from prespecified intervals; build the stochastic-reward MDP $$f:\Theta \to (\text{policy representation}, \text{risk measures})$$9; enumerate all deterministic policies $\nu(\theta)$0; apply the state-augmentation transformation (SAT) to obtain deterministic state-based rewards; compute mean and variance using Sobel’s variance formula; evaluate the chosen risk objective; and train a feed-forward neural network with MSE loss and Adam so that $\nu(\theta)$1 outputs both a compact policy representation and risk measures. SAT is critical because it preserves the probability measure on trajectories while converting reward randomness into a form where

$\nu(\theta)$2

can be computed exactly (Ma et al., 2019).

A different mechanism is symbolic generalization from small solved instances. The “1–2–3–Go!” pipeline first selects a small set of feasible base instances, solves them with exact model checking, collects reachable non-goal state–action labels, and then trains an axis-aligned decision tree using Gini impurity until leaves are pure (Azeem et al., 2024). The learned policy $\nu(\theta)$3 can then be applied to arbitrarily large parameter instances without explicit construction of their full state spaces. The method emphasizes “generalizability by explainability”: predicates over structured state variables are reused as a compact policy description across scales (Azeem et al., 2024).

Maximum-entropy approaches constitute a third family. In (Srivastava et al., 2020), the inner loop solves a soft Bellman fixed point and the outer loop updates state and action parameters using gradients of the free energy. “Parametrized Multi-Agent Routing via Deep Attention Models” specializes this perspective to Facility-Location and Path Optimization (FLPO): the Maximum Entropy Principle defines a Gibbs distribution over paths, and a Shortest Path Network (SPN), described as a permutation-invariant encoder-decoder, amortizes the stage-wise Gibbs policy so that shared continuous parameters such as facility locations can be optimized by backpropagation through sampled path costs (Basiri et al., 30 Jul 2025).

Other methods place the learned parameterization on uncertainty models rather than directly on policies. In (Hudson et al., 17 Mar 2026), the model is an amortized, context-conditioned distribution $\nu(\theta)$4 trained by squared moment matching and a bounded-variance score-function estimator. The learned preference distribution is not the terminal output; it feeds a second-stage scenario-based CVaR optimization. In (Nayyar et al., 23 Dec 2025), by contrast, the main learned object is an abstraction. State–Parameterized-Action Conditional Abstraction Trees (SPA-CATs) and Action Parameter Trees progressively refine both state partitions and action-parameter partitions online, and a tabular abstract $\nu(\theta)$5 function is updated with TD($\nu(\theta)$6). The heterogeneity score

$\nu(\theta)$7

drives refinement toward regions where greater representational resolution improves performance (Nayyar et al., 23 Dec 2025).

The breadth of these mechanisms suggests that ParaSDM is as much about amortization and structure exploitation as about parameterization itself. Some methods amortize exact planning or model checking; some amortize risk-sensitive optimization; some amortize scenario generation; and some amortize representation learning. That interpretation is inferential, but it aligns with the recurring pattern that offline structure is exploited to reduce online decision latency.

4. Time variation, generalization, and scaling

Several ParaSDM papers are explicitly motivated by time variation. In (Ma et al., 2019), deployment assumes $\nu(\theta)$8 changes over epochs as prices, demand, reliability, and risk appetite evolve, so training samples $\nu(\theta)$9 over specified intervals to cover these dynamics. The same paper recommends periodic resampling and retraining when new regimes emerge, and it notes that heavy tails or seasonal spikes may require richer labels than Normal-based VaR/CVaR mappings (Ma et al., 2019).

A stronger treatment of time variation is given in (Srivastava et al., 2022). There, the parameter vector is partitioned into pre-specified parameters with known dynamics and manipulable parameters with control-designed dynamics. The sum of entropy-regularized optimal values,

$$M(\theta)=(S_\theta,A_\theta,P_\theta,s_\theta,G_\theta)$$0

is used as a control Lyapunov function. The resulting feedback law guarantees $$M(\theta)=(S_\theta,A_\theta,P_\theta,s_\theta,G_\theta)$$1, asymptotic convergence of the manipulable-parameter gradient to zero, and a control that is Lipschitz continuous and bounded under the stated assumptions (Srivastava et al., 2022). This moves ParaSDM from static amortization toward online tracking of local optima.

Another route to time variation is reformulation. “Towards Enabling Learning for Time-Varying finite horizon Sequential Decision-Making Problems” proposes topography lifting, which reinterprets a finite-horizon, time-varying Para-SDM as an equivalent time-invariant problem by stage-indexed duplication of entities and strict action-topology constraints (Tiwari et al., 2 Apr 2025). Under that construction, every feasible finite-horizon path corresponds to a path in the lifted problem with the same cumulative cost, and the policy correspondence is one-to-one. The practical aim is to make time-varying problems accessible to learning methods that rely on stationarity (Tiwari et al., 2 Apr 2025).

Scaling across instance size rather than time is central in verification and games. In (Azeem et al., 2024), large $$M(\theta)=(S_\theta,A_\theta,P_\theta,s_\theta,G_\theta)$$2 values cause state-space explosion, and decision-tree policies are used to bypass explicit exploration of huge ParaMDPs. In “Human strategic decision making in parametrized games,” the same challenge is addressed through sample-based strategy libraries and parametric decision lists $$M(\theta)=(S_\theta,A_\theta,P_\theta,s_\theta,G_\theta)$$3, with an online rule of the form “If $$M(\theta)=(S_\theta,A_\theta,P_\theta,s_\theta,G_\theta)$$4 then play $$M(\theta)=(S_\theta,A_\theta,P_\theta,s_\theta,G_\theta)$$5; else …” (Ganzfried, 2021). For one-dimensional $$M(\theta)=(S_\theta,A_\theta,P_\theta,s_\theta,G_\theta)$$6 and continuous payoffs, the paper proves

$$M(\theta)=(S_\theta,A_\theta,P_\theta,s_\theta,G_\theta)$$7

where $$M(\theta)=(S_\theta,A_\theta,P_\theta,s_\theta,G_\theta)$$8 is the exploitability of the nearest-neighbor strategy selected from $$M(\theta)=(S_\theta,A_\theta,P_\theta,s_\theta,G_\theta)$$9 sampled parameter points (Ganzfried, 2021).

A related but narrower scaling issue appears in exploration control. In (Clifton et al., 2019), only the exploration schedule is parameterized, but the tuning is explicitly horizon-aware and knowledge-aware. Schedules such as

$\theta$0

or $\theta$1 encode dependence on both the remaining horizon and uncertainty features. The outer optimization over $\theta$2 is done by Gaussian process Bayesian optimization using a model of the environment or a confidence distribution over models (Clifton et al., 2019).

Finally, time-varying graph expansion produces another form of ParaSDM. In (Gao et al., 19 Mar 2026), the state is $\theta$3, the action is a tap-update vector for a graph filter, and the transition law includes stochastic graph growth. The filter taps are treated as agents, and a context-aware graph neural network parameterizes a multi-agent policy that updates the filter to optimize discounted cumulative reward over the expansion horizon (Gao et al., 19 Mar 2026).

5. Applications and reported empirical behavior

ParaSDM has been instantiated in inventory control, quantitative verification, 5G small-cell placement, ridesharing, mobile health, expanding-graph filtering, hybrid-action control, and multi-agent routing. The reported outcomes vary by formulation, but they consistently evaluate whether parameter-aware structure improves either decision quality, computational cost, or both.

Instantiation	Setting	Reported outcome
Risk-sensitive ParaSDM (Ma et al., 2019)	Inventory control with two suppliers, stochastic rewards, and VaR objective	Dataset size $\theta$4; convergence within $\theta$5 epochs; empirical “hit rate” on both training and validation $\theta$6
Decision-tree ParaMDP synthesis (Azeem et al., 2024)	21 model+property instances from QVBS-style benchmarks	Near-optimal values in 13/21 cases; in 2 additional cases, it outperforms Smart LSS
Maximum-entropy parameterized MDP (Srivastava et al., 2020)	5G small cell network design	Joint design yields costs “as low as 65% of the former”; model-based vs. model-free solutions differ by $\theta$7 and $\theta$8; entropy-over-paths gives a $\theta$9 lower cost than the policy-entropy-only variant; no annealing gives $T=(M_{\text{train}},M_{\text{test}})$0 higher cost
Parameterized exploration (Clifton et al., 2019)	mHealth glucose-control MDP	At $T=(M_{\text{train}},M_{\text{test}})$1, tuned AR(2) linear MCRew is $T=(M_{\text{train}},M_{\text{test}})$2 vs. $T=(M_{\text{train}},M_{\text{test}})$3 for $T=(M_{\text{train}},M_{\text{test}})$4; at $T=(M_{\text{train}},M_{\text{test}})$5, tuned AR(2) linear MCRew is $T=(M_{\text{train}},M_{\text{test}})$6 vs. $T=(M_{\text{train}},M_{\text{test}})$7 for $T=(M_{\text{train}},M_{\text{test}})$8
Time-varying Para-SDM control (Srivastava et al., 2022)	Dynamic multi-UAV communication network	Frame-by-frame re-optimization is reported to be approximately $T=(M_{\text{train}},M_{\text{test}})$9 times more computationally expensive
Topography lifting (Tiwari et al., 2 Apr 2025)	Small cell network / FLPO-style finite-horizon Para-SDM	Near-identical optimal costs to the time-varying solution with significantly better runtimes across 10 scenarios
Contextual preference distribution learning (Hudson et al., 17 Mar 2026)	Synthetic ridesharing environment with CVaR assignment	Average post-decision surprise reduced by up to $\mu$0 vs. a risk-neutral approach with perfect predictions and up to $\mu$1 vs. leading risk-averse baselines
Expanding-network graph filtering (Gao et al., 19 Mar 2026)	Synthetic graphs, MovieLens-100K cold-start, COVID prediction	G-MARL achieves the lowest RMSE across the reported tasks
Deep attention FLPO (Basiri et al., 30 Jul 2025)	Parametrized multi-agent routing with shared facility locations	Up to $\mu$2 speedup in policy inference and gradient computation; average optimality gap of approximately $\mu$3; over $\mu$4 lower cost than metaheuristic baselines; matches Gurobi’s optimal cost with annealing at a $\mu$5 speedup
Context-sensitive abstractions (Nayyar et al., 23 Dec 2025)	OfficeWorld, Pinball, Multi-City Transport, Robot Soccer Goal	Across 50 seeds, PEARL-flexible and PEARL-uniform outperform MP-DQN and HyAR in cumulative return during training and success probability of the greedy policy; HyAR learns only in Soccer; MP-DQN fails across tasks

Taken together, these results indicate that ParaSDM methods are especially effective when parameter variation has exploitable regularity: reusable predicates in verification, reusable path structure in routing, reusable risk structure in stochastic control, or reusable context-to-distribution mappings in preference-driven optimization. That conclusion is interpretive, but it is consistent with the empirical pattern across the surveyed papers.

6. Limitations, contested boundaries, and open directions

The surveyed literature does not support a single canonical boundary for ParaSDM. Some papers treat it as policy generalization across parameterized task families (Azeem et al., 2024), some as joint policy–parameter optimization (Srivastava et al., 2020), some as dynamic parameter tracking (Srivastava et al., 2022), some as contextual uncertainty learning for downstream optimization (Hudson et al., 17 Mar 2026), and some as hybrid discrete–continuous control (Nayyar et al., 23 Dec 2025). The literature therefore suggests that reducing ParaSDM to parameterized actions alone, or to parameterized MDP size alone, is too narrow.

Method-specific limitations are explicit. In the risk-sensitive framework of (Ma et al., 2019), VaR/CVaR estimates based only on $\mu$6 and $\mu$7 are biased for heavy-tailed or skewed returns, the deterministic policy space grows combinatorially, and generalization to out-of-distribution $\mu$8 is not guaranteed. In (Azeem et al., 2024), there is no sample-complexity analysis or PAC-style generalization bound, and the paper identifies failure modes when policies do not generalize across horizons or when changing transition probabilities alter optimal decisions in ways not captured by state-variable predicates. In (Clifton et al., 2019), performance degrades with model misspecification, and the paper explicitly notes the absence of formal regret bounds.

The scaling papers also leave open theoretical questions. In (Gao et al., 19 Mar 2026), no convergence or regret guarantees are provided for the growing-graph MARL setting, and stability of the time-varying filters is not theoretically analyzed. In (Nayyar et al., 23 Dec 2025), SPA-CATs are motivated by grouping behaviorally similar states and actions, but the paper states that it does not provide formal bisimulation-type conditions or performance-loss bounds; abstractions only refine and do not merge. In (Srivastava et al., 2022), convergence is to stationary points of the entropy-regularized objective, not to guaranteed global optima. In (Hudson et al., 17 Mar 2026), misspecification of the learned distribution family and covariate drift are identified as sources of degradation, and richer families such as normalizing flows or copulas are proposed as extensions.

Open directions recur across subfields. The verification literature proposes active learning over $\mu$9, feature learning, hybrid verification-learning loops, and symbolic integration (Azeem et al., 2024). Risk-sensitive ParaSDM points toward richer labels such as empirical quantiles via simulation when Normal or elliptical approximations fail (Ma et al., 2019). Preference-based ParaSDM points toward online or continual learning and richer distributional models (Hudson et al., 17 Mar 2026). Expanding-network ParaSDM raises questions about safe policy updates, Lyapunov critics, and stability-aware parameterizations (Gao et al., 19 Mar 2026). Hybrid-action abstraction methods raise the complementary question of whether state and action abstractions can be endowed with formal value-loss guarantees while retaining the empirical sample-efficiency gains already observed (Nayyar et al., 23 Dec 2025).

These open problems are not peripheral. They define the current frontier of ParaSDM: how to obtain reusable parameter-aware policies and decision rules without losing calibration, robustness, interpretability, or theoretical control as parameters drift, action spaces hybridize, and system scales expand.