A Comparison of Joint and Stepwise Dynamic Cognitive Diagnostic Models

Abstract: To extend cognitive diagnostic models (CDMs) to longitudinal settings, stepwise approaches that integrate a CDM model with a latent transition model and covariates are widely used due to their flexibility. Previous research has shown that stepwise estimation can yield biased results, motivating classification-error correction as a means of improving inference over uncorrected stepwise procedures. In this study, we evaluate a unified Bayesian dynamic cognitive diagnostic model that jointly estimates measurement (item parameters, latent attribute profiles) and transition components (transition parameters) in longitudinal settings with covariates. We compare this joint approach with the bias-corrected stepwise latent transition CDM through a Monte Carlo study. Results demonstrate that joint modeling provides more accurate recovery of transition parameters, particularly under limited test length and sample size, underscoring its advantages for longitudinal diagnostic analysis and offering practical guidance for applied researchers.

• The paper demonstrates that the joint Bayesian dynamic CDM yields significantly more accurate transition parameter estimates than the bias-corrected stepwise method.
• It employs Monte Carlo simulations across varying sample sizes and test lengths to assess classification accuracy and item parameter recovery.
• The study highlights practical implications for educational assessments by advocating joint estimation to avoid error propagation inherent in sequential approaches.

Comparative Evaluation of Joint and Stepwise Dynamic Cognitive Diagnostic Models

Introduction

This study rigorously evaluates two frameworks for longitudinal cognitive diagnostic modeling (CDMs) with covariates: (1) a joint Bayesian dynamic CDM that utilizes simultaneous estimation of measurement and transition submodels, and (2) a bias-corrected stepwise (three-step) latent transition CDM, in which measurement and transition components are estimated sequentially with explicit correction for misclassification error. The research addresses the critical problem of parameter estimation bias and uncertainty propagation in stepwise approaches, quantifying comparative performance with respect to classification accuracy and parameter recovery—particularly transition parameters in attribute acquisition processes—across varying sample size and test length regimes.

Methodological Frameworks

Bias-Corrected Stepwise Latent Transition CDM

The bias-corrected stepwise approach follows the classic three-step protocol:

1. Measurement Model Fitting: At each time point, a DINA model is independently estimated for item response data (assuming known, invariant Q-matrix), yielding posterior probabilities over latent attribute profiles.
2. Latent State Assignment and Classification Error Estimation: Participants are assigned discrete attribute profiles using a MAP rule; classification error probabilities (CEPs) are estimated via the misclassification matrix, capturing the uncertainty inherent in hard assignments.
3. Structural Model Estimation with Error Correction: The transition (latent Markov) model with covariates is fit by marginalizing over possible latent state sequences, with CEPs integrated into the likelihood function. Covariate effects are introduced on both initial attribute mastery and transition probabilities via logistic regression models.

Joint Bayesian Dynamic CDM

The joint modeling framework specifies the same item response (DINA) and latent transition structures but estimates all parameters—item, initial mastery, and transition (with covariates)—jointly within a unified Bayesian framework using Markov Chain Monte Carlo (MCMC) inference. Weakly to moderately informative priors are specified for all parameters, with attribute mastery modeled as an absorbing state (monotonic acquisition). This approach allows uncertainty in measurement and transition components to be modeled coherently via the joint posterior.

Simulation Design

Comprehensive Monte Carlo simulations were conducted with two time points, two latent attributes, and varying combinations of sample size ($N \in \{200, 400, 600\}$) and test length ($J \in \{6, 18, 30\}$). Covariates were generated from a trivariate normal with moderate correlation ($\rho = 0.4$), and DINA item parameters were drawn from moderate-quality uniform distributions. Recovery was evaluated via mean absolute error (MAE), root mean squared error (RMSE), and attribute-wise agreement rates (AAR) across 100 replications per setting.

Empirical Results

Classification and Item Parameter Recovery

Both joint and stepwise approaches achieved high classification accuracy, with AAR increasing in larger-$N$ and longer-test scenarios. Item parameter ($g, s$) estimation was robust for both methods, with small RMSE and MAE.

Covariate and Transition Parameter Estimation

Key finding: The joint Bayesian model demonstrated consistently superior accuracy in estimating covariate effects on attribute acquisition (transition parameters), particularly in low-sample and short-test conditions, with MAE/RMSE reductions often an order of magnitude relative to the stepwise approach.

For example, under ($N = 200, J = 6$), the joint method yielded MAEs of 0.10–0.14 for transition slopes versus 0.80–1.0 for stepwise. In large-sample conditions, estimation for both approaches converged, but stepwise estimates of transition covariate effects remained substantially more variable (sometimes by factors of 8–10).

Slope estimates for covariate effects on the initial attribute states were moderately more accurate for the joint model, but differences were attenuated.

Robustness and Sensitivity

Sensitivity analyses with varying covariate correlation ($0 \leq \rho \leq 0.8$) demonstrated stability for both item parameter and latent class recovery. However, the variance and bias for transition parameter estimates in the stepwise model were sensitive to both $\rho$ and sample size, with the stepwise method always performing worse (in both bias and variance) compared to the joint method.

Computationally, the joint MCMC framework exhibited tractable runtimes in the scenarios considered (single-core CPU, 0.8–6 minutes per replicate).

Theoretical and Practical Implications

The results reinforce established concerns regarding error propagation in sequential estimation protocols: stepwise assignment amplifies uncertainty in subsequent transition estimation, and even with bias correction (via CEPs), the sequential framework fails to match the inferential stability of joint estimation, particularly in modest data scenarios. This has direct implications for the use of stepwise procedures in field studies, where short forms and moderate $N$ are common—joint Bayesian modeling provides more trustworthy inference about learning/transition dynamics and covariate effects.

Practically, these findings advocate for adoption of joint estimation in applied longitudinal CDM analyses where parameter stability is critical. Furthermore, the simulation infrastructure and code provided facilitate reproducibility and continued experimentation by the research community.

Methodologically, the work points to further extensions: application to alternative CDMs beyond DINA (e.g., DINO, G-DINA, higher-order DINA), incorporation of time-varying covariates, and joint estimation when the Q-matrix is unknown. The latter is particularly relevant for large-scale assessment consortia data, with implications for PISA, PIAAC, and TIMSS.

Conclusion

This study provides conclusive evidence that joint Bayesian dynamic cognitive diagnostic modeling yields more accurate and reliable attribution of learning transitions and covariate effects than the bias-corrected stepwise approach—especially under constraints common to educational and psychological assessment. The joint framework’s superior transition parameter recovery, computational tractability, and robustness to covariate correlation and test characteristics mark it as the method of choice for longitudinal CDM with covariates. Future research should continue to develop unified frameworks for modeling structural and measurement components in longitudinal diagnostic inference, especially in the presence of Q-matrix uncertainty and complex item/covariate structures.