Design Effect Ratios for Bayesian Survey Models: A Diagnostic Framework for Identifying Survey-Sensitive Parameters

Abstract: Bayesian hierarchical models fit to complex survey data require variance correction for the sampling design, yet applying this correction uniformly harms parameters already protected by the hierarchical structure. We propose the Design Effect Ratio -- the ratio of design-corrected to model-based posterior variance -- as a per-parameter diagnostic identifying which quantities are survey-sensitive. Closed-form decompositions show that fixed-effect sensitivity depends on whether identifying variation lies between or within clusters, while random-effect sensitivity is governed by hierarchical shrinkage. These results yield a compute-classify-correct workflow adding negligible overhead to Bayesian estimation. In simulations spanning 54 scenarios and 10,800 replications of hierarchical logistic regression, selective correction achieves 87-88% coverage for survey-sensitive parameters -- matching blanket correction -- while preserving near-nominal coverage for protected parameters that blanket correction collapses to 20-21%. A threshold of 1.2 produces zero false positives, with a separation ratio of approximately 4:1. Applied to the 2019 National Survey of Early Care and Education (6,785 providers, 51 states), the diagnostic flags exactly 1 of 54 parameters for correction; blanket correction would have narrowed the worst remaining interval to 4.3% of its original width. The entire pipeline completes in under 0.03 seconds, bridging design-based and model-based survey inference.