Policy-Augmented Graphical Hybrid Models

• The paper introduces pKG as dynamic probabilistic models that integrate physics-based functions with data-driven policy components for interpretable process control.
• It details a discrete-time framework using state and action vectors alongside Shapley value sensitivity analysis to assess input significance and uncertainty.
• Methodological enhancements like linear–Gaussian approximations and TFWW-VRT permutation sampling yield significant computational acceleration and robust sensitivity estimation.

Policy-Augmented Graphical Hybrid Models (pKG) constitute a class of dynamic, probabilistic models designed to capture the stochastic and causal structure of controlled processes, with a particular focus on domains exhibiting high complexity and uncertainty, such as biomanufacturing. These models explicitly integrate mechanistic (knowledge-based) and data-driven (policy-based) components—encoding both the physics/chemistry of the system and parametric control policies—thus supporting interpretable and optimal process control in real-world settings (Zhao et al., 2024).

1. Mathematical Structure of pKG Models

pKG models operate on a discrete-time horizon $t = 1, ..., H$ and represent the system evolution via two principal sequences:

• State vectors $\pmb s_t \in \mathbb{R}^n$ (critical quality attributes, CQAs)
• Action vectors $\pmb a_t \in \mathbb{R}^m$ (controlled process parameters, CPPs)

The state transition is governed by a knowledge-based function $f_t$ reflecting first-principles or kinetic equations, parameterized by unknowns $\pmb w_t$, and perturbed by noise $\pmb e_{t+1}$: $$\pmb s_{t+1} = f_t(\pmb s_t, \pmb a_t; \pmb w_t) + \pmb e_{t+1}, \quad \pmb e_{t+1} \sim \text{“noise”}$$ The control policy is modeled as a (potentially nonlinear) parametric map: $\pmb a_t = h_t(\pmb s_t; \pmb\theta_t)$ where $\pmb\theta_t$ are policy parameters optimized for process outcomes.

The generative (graphical) model entails the joint distribution: $$p(\tau, \pmb e_{1:H}, \pmb w, \pmb \theta) = p(\pmb w)\,p(\pmb\theta)\,p(\pmb s_1) \prod_{t=1}^{H-1} p(\pmb e_{t+1})\,p(\pmb s_{t+1}|\pmb s_t,\pmb a_t, \pmb w_t)\, \delta[\pmb a_t - h_t(\pmb s_t;\pmb\theta_t)]$$ where the Dirac delta encodes the deterministic policy constraint.

A cumulative reward functional is defined as: $\pmb s_t \in \mathbb{R}^n$0 allowing for reward/cost design tailored to domain-specific criteria.

2. Shapley-Value Sensitivity Analysis Framework (SV-pKG)

Criticality of process inputs under uncertainty is quantified via Shapley value (SV) sensitivity analysis, conceptualizing each uncertain random factor ($\pmb s_t \in \mathbb{R}^n$1), policy parameter ($\pmb s_t \in \mathbb{R}^n$2), and model parameter ($\pmb s_t \in \mathbb{R}^n$3) as a “player” in a cooperative game. The performance measure associated with a subset $\pmb s_t \in \mathbb{R}^n$4 (“active” inputs) is denoted $\pmb s_t \in \mathbb{R}^n$5.

Shapley value for input $\pmb s_t \in \mathbb{R}^n$6 is: $\pmb s_t \in \mathbb{R}^n$7 or, equivalently, as an expectation over random input permutations: $\pmb s_t \in \mathbb{R}^n$8 where $\pmb s_t \in \mathbb{R}^n$9 is the set of inputs preceding $\pmb a_t \in \mathbb{R}^m$0 in permutation $\pmb a_t \in \mathbb{R}^m$1.

Types of value functions $\pmb a_t \in \mathbb{R}^m$2 employed:

• Random factors $\pmb a_t \in \mathbb{R}^m$3:
  • Predictive: $\pmb a_t \in \mathbb{R}^m$4
  • Variance-based: $\pmb a_t \in \mathbb{R}^m$5
• Policy parameters $\pmb a_t \in \mathbb{R}^m$6:
  • Predictive: $\pmb a_t \in \mathbb{R}^m$7
  • Variance-based: $\pmb a_t \in \mathbb{R}^m$8
• Model parameters $\pmb a_t \in \mathbb{R}^m$9:
  • Predictive: $f_t$0
  • Variance-based: $f_t$1

SVs are estimated using a sampling-based pipeline:

1. Draw $f_t$2 model parameter samples $f_t$3.
2. For each, draw $f_t$4 permutations $f_t$5.
3. Compute marginal SV contributions, by simulation for nonlinear pKG or in closed form for linear-Gaussian cases.
4. Average over all $f_t$6 pairs.

The estimator satisfies unbiasedness, and concentration inequalities (Chebyshev, Hoeffding) provide sample size bounds to achieve prescribed estimation accuracy.

3. Linear–Gaussian Approximation of pKG

For high-frequency, heavily instrumented processes, the pKG system is often approximated by a linear-Gaussian model: $f_t$7 where $f_t$8, linear policy $f_t$9, and linear reward.

Explicit recursive forms allow significant computational acceleration:

• Predictive SVs for random factors are given by:

$\pmb w_t$0

$\pmb w_t$1

• Variance-based SVs involve covariance-propagation terms:

$\pmb w_t$2

Closed forms are generally not available for policy and model parameter SVs, but updates can still be performed in $\pmb w_t$3 time per step, producing overall costs of $\pmb w_t$4 (predictive SV) and $\pmb w_t$5 (variance SV), a substantial reduction compared to naïve brute-force methods.

4. Permutation Sampling and Variance Reduction: TFWW-VRT

Accurate SV estimation requires low-discrepancy permutation sampling. The TFWW-VRT approach (TFWW: number-theoretic hypersphere transformation, VRT: variance reduction techniques) applies:

• Quasi-Monte Carlo sampling in the unit cube, mapped to the sphere using the TFWW recipe.
• Embedding via fixed orthonormal projection, and sorting to generate $\pmb w_t$6 permutations.
• Antithetic sampling: for each permutation $\pmb w_t$7, also process $\pmb w_t$8.
• TFWW achieves $\pmb w_t$9 complexity (s = number of players), surpassing traditional Balanced/Marginal Techniques (BMT, SCT) in permutation quality and estimator MSE.

Final SV algorithm:

1. Sample $\pmb e_{t+1}$0 parameter sets; generate $\pmb e_{t+1}$1 TFWW-VRT permutations.
2. For each combination, compute value-function increments via simulation (nonlinear) or closed form (Gaussian); reuse subcomputations where feasible.
3. Accumulate/average for SVs.

Empirical results confirm 20–50% lower SV mean squared error compared with BMT and SCT at equal computation, with dramatic runtime improvements (from hours to minutes for process horizons $\pmb e_{t+1}$2).

5. Computational Efficiency and Practical Implications

Efficiency enhancements in SV-pKG stem from algorithmic re-use and the structure of linear–Gaussian models. For nonlinear pKG models, pathway reuse reduces cost of value-function evaluation from $\pmb e_{t+1}$3 times the number of runs to $\pmb e_{t+1}$4 per run, yielding an $\pmb e_{t+1}$5-fold acceleration. In linear–Gaussian pKG, the recursive structure enables predictive SV computation for random factors in $\pmb e_{t+1}$6 and variance SV in $\pmb e_{t+1}$7, much faster than traditional brute-force.

Permutation sampling improvements via TFWW-VRT further decrease estimator variance and time complexity. The cumulative impact is efficient sensitivity analysis at scale, supporting robust interpretation, model validation, and stable optimal control in complex, uncertain process environments.

6. Application Domains and Significance

pKG models and the SV-pKG analysis pipeline were developed to address complexity and uncertainty in biomanufacturing, where both mechanistic knowledge and data-driven control are critical for risk-informed optimization. The framework generalizes to any domain requiring interpretable, causality-respecting analysis of stochastic decision processes with hybrid model components. The integration of linear–Gaussian approximations, optimal sampling strategies, and scalable SV estimation allows for tractable sensitivity quantification in high-dimensional, multistage systems, thereby advancing the state-of-the-art in both theoretical and applied process analytics (Zhao et al., 2024).