Bayesian HRM-SDT Model in AI Clinical Assessments

• The model is a hierarchical framework integrating IRT with SDT to account for multi-rater biases using a probabilistic, ordered-logit approach.
• It distinctly separates latent constructs including learner competency, case difficulty, and rater decision behavior within AI-assisted clinical assessment pipelines.
• Empirical validations using HMC and NUTS confirm robust performance, effective rater bias calibration, and reliable estimation of latent abilities.

The Bayesian HRM-SDT (Hierarchical Rater-Mediated Signal Detection Theory) model is a probabilistic framework for psychometric analysis, explicitly formulated to separate the latent constructs of learner ability, case difficulty, and rater decision behavior in settings where observable ratings derive from complex, noisy, and potentially multi-rater evaluation pipelines. It provides a unified model for analyzing ratings generated from AI-assisted clinical assessments, particularly in contexts involving virtual standardized patients, AI-simulated learners, and item-based scoring rubrics. The HRM-SDT model extends classic item response theory (IRT) by embedding a hierarchical SDT framework for rater mediation, fully specifying both the generative process for scores and the structure of uncertainties at each analytic level (Gin et al., 26 Jan 2026).

1. Model Structure and Notation

The Bayesian HRM-SDT model dimensions are organized as follows:

• $i = 1,...,N$: virtual learner (examinee)
• $j = 1,...,J$: rater
• $l = 1,...,L$: rubric item
• $p = 1,...,P$: competency dimension (here, %%%%4%%%%, corresponding to ACGME domains)
• $k = 1,...,K-1$ for $K=5$ rating categories

Observed Data:

• $Y_{i j l} \in \{1,2,3,4,5\}$: rubric score assigned by rater $j$ to learner $i$ for item $l$ (when meaningful)
• $A_{i j l} \in \{0,1\}$: applicability gate; 1 if the item is scorable, 0 otherwise

Latent Variables:

• $\theta_i \in \mathbb{R}^P$: $P$-dimensional competency vector for learner $i$
• $n_{i l} \in \{1,...,5\}$: latent true performance state on item $l$
• $$\tilde{n}_{i l} = (n_{i l} - 1)/(K-1) \in \{0, \frac{1}{4}, \frac{1}{2}, \frac{3}{4}, 1\}$$: normalized performance
• $\psi_{q,p}$: case-by-competency shift, for scenario $q = q(l)$ and competency $p = g(l)$
• $d_j > 0$: rater $j$'s baseline detection (sensitivity)
• $\Delta^d_{j,g}$: multiplicative detection shift for rater $j$ on competency group $g$
• $C_{j,k}$: rater $j$'s baseline category thresholds
• $\Delta^c_{j,g}$: additive threshold shift for rater $j$ on group $g$

Hierarchical Levels:

• Level 1: Learner and case effects jointly determine latent performance $n_{i l}$
• Level 2: Rater SDT: noisy latent evidence $W_{i j l}$ is formed and discretized into rubric score $Y_{i j l}$ via rater-specific thresholds
• Applicability: Ratings only incorporated if $A_{i j l}=1$ (modeled via a logistic Bernoulli gating layer)

The table below summarizes the primary entities:

Symbol	Description	Domain
$\theta_i$	Learner $i$'s competency vector	$\mathbb{R}^P$
$n_{i l}$	Latent "true" performance on item $l$	$\{1,\ldots,5\}$
$d_j$	Rater $j$'s baseline detection	$\mathbb{R}^{+}$
$C_{j,k}$	Rater $j$'s category thresholds	$\mathbb{R}$ with $C_{j,1}<...<C_{j,K-1}$

This formalization enables explicit modeling of the dependencies between examinee competency profiles, variable case/item demands, and rater-specific behavioral characteristics (Gin et al., 26 Jan 2026).

2. Likelihood Specification and SDT Layer

The likelihood factorizes in two stages, leveraging both IRT and SDT conventions:

• Stage 1: IRT-Like Latent Performance
  • The item-specific linear predictor $S_{i l} = \alpha_l^T \theta_i + \psi_{q(l),g(l)}$ (where $\alpha_l$ is a unit-normalized loading vector on competencies) determines the distribution over latent performance states $n_{i l}$ via an ordered-logit model:

  $$\Pr(n_{i l} \leq k | \theta, \psi, b) = o(b_{l,k} - S_{i l})$$

  with $b_{l,k}$ as the item step parameters and $o(x)$ the logistic CDF.

• Stage 2: Rater SDT Generative Model
  • Performance $\tilde{n}_{i l}$ (normalized) is mapped to rater evidence $$W_{i j l} = d_{j,l} \cdot \tilde{n}_{i l} + \epsilon$$, where $d_{j,l} = d_j \cdot \exp(\Delta^d_{j,g(l)})$ and $\epsilon \sim \text{Logistic}(0,1)$.
  • Discretization is applied:

  $$Y_{i j l} = m \quad \text{iff} \quad C_{j,m-1,l} < W_{i j l} \leq C_{j,m,l}$$

  with $C_{j,k,l} = C_{j,k} + \Delta^c_{j,g(l)}$.

• Applicability Gate: $$A_{i j l} \sim \text{Bernoulli}(\text{logit}^{-1}(w_{q(l),g(l)}))$$ with non-applicable ratings omitted.

The full probability of observing $Y_{i j l}$ is expressed as a marginalized ordered logit in terms of the underlying model parameters. Each rating is included in the likelihood only if gated by $A_{i j l} = 1$ (Gin et al., 26 Jan 2026).

3. Prior Hierarchy and Regularization Schemes

The Bayesian HRM-SDT approach specifies a hierarchical prior structure:

• $$\theta_i \sim \mathcal{N}_P(\mu_\theta, \Sigma_\theta)$$
• $$\psi_{q,p} \sim \mathcal{N}(0, \sigma^2_{\text{case},p})$$, centered for identifiability ($\sum_q \psi_{q,p}=0$, each $p$)
• $b_{l,k} \sim \mathcal{N}(\mu_{b,k}, \sigma_b^2)$, subject to $b_{l,1}<...<b_{l,K-1}$
• $\log d_j \sim \mathcal{N}(\mu_d, \sigma_d^2)$; $\Delta^d_{j,g} \sim \mathcal{N}(0, \sigma^2_{d,g})$
• $C_{j,k} \sim \mathcal{N}(\mu_{c,k}, \sigma_c^2)$ with order constraints; $\Delta^c_{j,g} \sim \mathcal{N}(0, \sigma^2_{c,g})$
• $w_{q,g} \sim \mathcal{N}(\mu_w, \sigma_w^2)$

Hyperparameters ($\mu$'s) receive broad normal priors and all $\sigma$'s employ half-normal or half-Cauchy distributions. This approach promotes stable regularization while supporting the model’s flexibility in capturing heterogeneous rating patterns and latent abilities (Gin et al., 26 Jan 2026).

4. Posterior Computation and MCMC Implementation

Inference is performed using Hamiltonian Monte Carlo (HMC) with the No-U-Turn Sampler (NUTS), implemented in a JAX-backed environment. Key estimation parameters:

• 4 independent chains
• 1000 warmup iterations per chain
• 750 posterior draws per chain (total 3000)
• Target acceptance probability: 0.99

Discrete latent states $n_{i l}$ are analytically marginalized to maintain posterior differentiability. Convergence diagnostics include:

• $\hat{R}$ (Gelman-Rubin statistic) $<$ 1.01 for all key parameters
• Effective sample size (ESS) $>$ 500
• Visual trace inspection confirming absence of divergences and robust chain mixing

This configuration ensures rigorous uncertainty quantification and effective separation of latent constructs central to the psychometric framework (Gin et al., 26 Jan 2026).

5. Model Interpretation: SDT Parameters and Psychological Structure

Detection parameters $d_j$ (and $d_{j,l}$) constitute analogues of SDT sensitivity ($d'$): higher values correspond to a rater's evidence variable $W$ being more tightly coupled to underlying true performance $\tilde{n}$. Category thresholds $C_{j,k,l}$ (combining $C_{j,k}$ and $\Delta^c_{j,g}$) correspond to severity or leniency criteria; elevating all thresholds reflects a globally more severe rater.

The relationship to standard SDT is explicit:

• For two-category SDT: $$\Pr(\text{response} = \text{"signal"}) = \Phi(d'/2 - c)$$
• In the HRM-SDT model:

$$\Pr(Y_{i j l}=m) = \Phi(C_{j,m,l} - d_{j,l} \tilde{n}_{i l}) - \Phi(C_{j,m-1,l} - d_{j,l} \tilde{n}_{i l})$$

under a normal rather than logistic link.

Effective sensitivity and criterion parameters can be computed: $d'_{j,g} = d_j \exp(\Delta^d_{j,g})$, $\beta_{j,g,k} = \exp(C_{j,k} + \Delta^c_{j,g})$ in log-odds form (Gin et al., 26 Jan 2026).

6. Empirical Validation and Model Fit Assessment

Posterior predictive checking is central to evaluating model adequacy:

• Simulated replicated $Y$’s are compared to observed distributions (category counts, item-mean vs. ability curves, rating variances).
• Learner recovery: Posterior means for $\theta_i$ significantly correlate with generating competencies across dimensions (observed Pearson $r \in [0.38, 0.76]$).
• Dimensional separability: Comparison of estimated and true correlation matrices for $\theta$ reveals some induced cross-dimension correlation (max $|r| \approx 0.54$), indicating potential item coverage refinement needs.
• Cross-case consistency: Between-learner correlations of case-conditional $\theta_i$ for case pairs are as high as $r \approx 0.65$; within-learner profile correlations across cases average $r \approx 0.57$.
• Rater-effect summaries: Posterior intervals for $\Delta^d_{j,g}$ are small ($\approx \pm 0.1$ log-units). In contrast, $\Delta^c_{j,g}$ spans $\approx [-1.3, +1.2]$, indicating severity shifts dominate detection shifts across competencies.

Item-Characteristic Curves allow identification of items with near-floor, near-ceiling, or ideal discrimination properties. All convergence diagnostics confirm acceptable inference quality: $\hat{R} < 1.01$, ESS $> 200$, and no divergent transitions (Gin et al., 26 Jan 2026).

7. Significance for AI-Enabled Clinical Assessment

By integrating virtual patient scenarios, AI-simulated learners, and individualized rater models, the Bayesian HRM-SDT framework enables disentanglement of ability, case, and rater effects in complex, multi-axis competency assessments. Its probabilistic, hierarchical structure supports robust, interpretable, and generalizable estimates of learner competency, and provides a principled basis for stress-testing and validating AI-assisted evaluation pipelines prior to adoption in human-facing educational settings (Gin et al., 26 Jan 2026).

A plausible implication is that applying the HRM-SDT model can reveal subtle interactions between rating conditions, case-based difficulty, and rater response strategies that would be conflated under less granular analytic frameworks. This is foundational for staged, safety-centric deployment of AI-driven assessment systems in high-stakes domains.