Policy-Augmented Graphical Hybrid Models

• The paper introduces pKG as dynamic probabilistic models that integrate physics-based functions with data-driven policy components for interpretable process control.
• It details a discrete-time framework using state and action vectors alongside Shapley value sensitivity analysis to assess input significance and uncertainty.
• Methodological enhancements like linear–Gaussian approximations and TFWW-VRT permutation sampling yield significant computational acceleration and robust sensitivity estimation.

Policy-Augmented Graphical Hybrid Models (pKG) constitute a class of dynamic, probabilistic models designed to capture the stochastic and causal structure of controlled processes, with a particular focus on domains exhibiting high complexity and uncertainty, such as biomanufacturing. These models explicitly integrate mechanistic (knowledge-based) and data-driven (policy-based) components—encoding both the physics/chemistry of the system and parametric control policies—thus supporting interpretable and optimal process control in real-world settings (Zhao et al., 2024).

1. Mathematical Structure of pKG Models

pKG models operate on a discrete-time horizon $t = 1, ..., H$ and represent the system evolution via two principal sequences:

• State vectors $\pmb s_t \in \mathbb{R}^n$ (critical quality attributes, CQAs)
• Action vectors $\pmb a_t \in \mathbb{R}^m$ (controlled process parameters, CPPs)

The state transition is governed by a knowledge-based function $f_t$ reflecting first-principles or kinetic equations, parameterized by unknowns $\pmb w_t$, and perturbed by noise $\pmb e_{t+1}$: $$\pmb s_{t+1} = f_t(\pmb s_t, \pmb a_t; \pmb w_t) + \pmb e_{t+1}, \quad \pmb e_{t+1} \sim \text{“noise”}$$ The control policy is modeled as a (potentially nonlinear) parametric map: $\pmb a_t = h_t(\pmb s_t; \pmb\theta_t)$ where $\pmb\theta_t$ are policy parameters optimized for process outcomes.

The generative (graphical) model entails the joint distribution: $$p(\tau, \pmb e_{1:H}, \pmb w, \pmb \theta) = p(\pmb w)\,p(\pmb\theta)\,p(\pmb s_1) \prod_{t=1}^{H-1} p(\pmb e_{t+1})\,p(\pmb s_{t+1}|\pmb s_t,\pmb a_t, \pmb w_t)\, \delta[\pmb a_t - h_t(\pmb s_t;\pmb\theta_t)]$$ where the Dirac delta encodes the deterministic policy constraint.

A cumulative reward functional is defined as: $$J(\pmb\theta, \tau; \pmb w) = \sum_{t=1}^H r_t(\pmb s_t, \pmb a_t)$$ allowing for reward/cost design tailored to domain-specific criteria.

2. Shapley-Value Sensitivity Analysis Framework (SV-pKG)

Criticality of process inputs under uncertainty is quantified via Shapley value (SV) sensitivity analysis, conceptualizing each uncertain random factor ($e_t^k$), policy parameter ($\theta_t^i$), and model parameter ($w_t^i$) as a “player” in a cooperative game. The performance measure associated with a subset $U \subseteq \O$ (“active” inputs) is denoted $g(U)$.

Shapley value for input $o \in \O$ is: $\mathrm{Sh}(Y \mid o) = \sum_{U \subseteq \O\setminus\{o\}} \frac{(|\O| - |U| - 1)!\,|U|!}{|\O|!} [g(U \cup \{o\}) - g(U)]$ or, equivalently, as an expectation over random input permutations: $$\mathrm{Sh}(Y \mid o) = \mathbb{E}_{\pi}\left[ g(P_o(\pi) \cup \{o\}) - g(P_o(\pi)) \right]$$ where $P_o(\pi)$ is the set of inputs preceding $o$ in permutation $\pi$.

Types of value functions $g(U)$ employed:

• Random factors $e$:
  • Predictive: $$g(U \mid \pmb w) = \mathbb{E}[Y \mid \{e \in U\}, \pmb w]$$
  • Variance-based: $$g(U \mid \pmb w) = \mathbb{E}[\operatorname{Var}(Y \mid \{e \notin U\}, \pmb w)]$$
• Policy parameters $\theta$:
  • Predictive: $g(U \mid \pmb w) = \mathbb{E}[Y \mid \theta_{\O\setminus U} = 0, \pmb w]$
  • Variance-based: $g(U \mid \pmb w) = \operatorname{Var}[Y \mid \theta_{\O\setminus U} = 0, \pmb w]$
• Model parameters $w$:
  • Predictive: $g(U) = \mathbb{E}[Y \mid w_{\O\setminus U} = 0]$
  • Variance-based: $g(U) = \mathbb{E}[\operatorname{Var}(Y \mid w_{\O\setminus U})]$

SVs are estimated using a sampling-based pipeline:

1. Draw $Q$ model parameter samples $\pmb w^{(q)} \sim p(\pmb w \mid \D)$.
2. For each, draw $D$ permutations $\pi^{(d)}$.
3. Compute marginal SV contributions, by simulation for nonlinear pKG or in closed form for linear-Gaussian cases.
4. Average over all $(q, d)$ pairs.

The estimator satisfies unbiasedness, and concentration inequalities (Chebyshev, Hoeffding) provide sample size bounds to achieve prescribed estimation accuracy.

3. Linear–Gaussian Approximation of pKG

For high-frequency, heavily instrumented processes, the pKG system is often approximated by a linear-Gaussian model: $$\pmb s_{t+1} = \pmb\mu^s_{t+1} + (\pmb\beta_t^s)^\top(\pmb s_t - \pmb\mu^s_t) + (\pmb\beta_t^a)^\top(\pmb a_t - \pmb\mu^a_t) + \pmb e_{t+1}$$ where $\pmb e_{t+1} \sim N(0, \Sigma_e)$, linear policy $$\pmb a_t = \pmb\mu^a_t + \pmb\theta_t^\top(\pmb s_t - \pmb\mu^s_t)$$, and linear reward.

Explicit recursive forms allow significant computational acceleration:

• Predictive SVs for random factors are given by:

$$\mathrm{Sh}(\pmb s_{t+1} \mid e_h^k) = \mathbb{E}_{\pmb w}\left[ \mathbf R_{h,t} e_h^k \pmb e_k \right]$$

$$\mathrm{Sh}(J \mid e_h^k) = \mathbb{E}_{\pmb w}\left[\sum_{t=0}^{H-1} \alpha_{t+1} \mathbf R_{h,t} e_h^k \pmb e_k \right]$$

• Variance-based SVs involve covariance-propagation terms:

$$\mathrm{Sh}(\pmb s_{t+1} \mid e_h^k) = \mathbb{E}_{\pmb w}\left[\mathbf R_{t+1}\left(\mathbf V_{t+1} \odot \frac{1}{2}\left(\pmb 1^{\,l} + \pmb 1^{\,l\top}\right)\right)\mathbf R_{t+1}^\top \right]$$

Closed forms are generally not available for policy and model parameter SVs, but updates can still be performed in $O(1)$ time per step, producing overall costs of $O(QD H^2 n^4 m)$ (predictive SV) and $O(QD H^3 n^4 m)$ (variance SV), a substantial reduction compared to naïve brute-force methods.

4. Permutation Sampling and Variance Reduction: TFWW-VRT

Accurate SV estimation requires low-discrepancy permutation sampling. The TFWW-VRT approach (TFWW: number-theoretic hypersphere transformation, VRT: variance reduction techniques) applies:

• Quasi-Monte Carlo sampling in the unit cube, mapped to the sphere using the TFWW recipe.
• Embedding via fixed orthonormal projection, and sorting to generate $D$ permutations.
• Antithetic sampling: for each permutation $\pi$, also process $\operatorname{Reverse}(\pi)$.
• TFWW achieves $O(Ds)$ complexity (s = number of players), surpassing traditional Balanced/Marginal Techniques (BMT, SCT) in permutation quality and estimator MSE.

Final SV algorithm:

1. Sample $Q$ parameter sets; generate $D$ TFWW-VRT permutations.
2. For each combination, compute value-function increments via simulation (nonlinear) or closed form (Gaussian); reuse subcomputations where feasible.
3. Accumulate/average for SVs.

Empirical results confirm 20–50% lower SV mean squared error compared with BMT and SCT at equal computation, with dramatic runtime improvements (from hours to minutes for process horizons $H \approx 30$).

5. Computational Efficiency and Practical Implications

Efficiency enhancements in SV-pKG stem from algorithmic re-use and the structure of linear–Gaussian models. For nonlinear pKG models, pathway reuse reduces cost of value-function evaluation from $O(H)$ times the number of runs to $O(1)$ per run, yielding an $O(H)$-fold acceleration. In linear–Gaussian pKG, the recursive structure enables predictive SV computation for random factors in $O(QD H^2 n^4 m)$ and variance SV in $O(QD H^3 n^4 m)$, much faster than traditional brute-force.

Permutation sampling improvements via TFWW-VRT further decrease estimator variance and time complexity. The cumulative impact is efficient sensitivity analysis at scale, supporting robust interpretation, model validation, and stable optimal control in complex, uncertain process environments.

6. Application Domains and Significance

pKG models and the SV-pKG analysis pipeline were developed to address complexity and uncertainty in biomanufacturing, where both mechanistic knowledge and data-driven control are critical for risk-informed optimization. The framework generalizes to any domain requiring interpretable, causality-respecting analysis of stochastic decision processes with hybrid model components. The integration of linear–Gaussian approximations, optimal sampling strategies, and scalable SV estimation allows for tractable sensitivity quantification in high-dimensional, multistage systems, thereby advancing the state-of-the-art in both theoretical and applied process analytics (Zhao et al., 2024).