Sensitivity and Early Detection of Bayesian Causal Impact Models for Marketing Interventions

Abstract: Marketing systems frequently undergo operational changes that may affect performance, making timely detection of adverse effects essential for decision making. While Bayesian Causal Impact models are widely used to estimate the causal effects of interventions, their ability to support early operational monitoring remains less explored. This paper proposes a simulation based framework to evaluate the sensitivity and detection capabilities of Bayesian causal impact analysis under controlled performance degradations. Using daily traffic data from an abandoned cart marketing journey, we repeatedly perturb the observed outcomes and assess alarm activation probabilities across different effect magnitudes, confidence levels, and detection horizons. Two alarm criteria are analyzed: one based on the proportion of observations falling below the predictive lower bound and another based on consecutive days of negative cumulative impact. Results show that detection performance depends strongly on the interaction between effect size, confidence level, and evaluation horizon. In particular, proportion based criteria become less effective as the monitoring horizon increases, whereas persistence based criteria provide more stable and operationally meaningful detection behavior. The proposed framework extends causal impact analysis beyond retrospective effect estimation, offering a practical methodology for quantifying detection sensitivity and supporting monitoring decisions in dynamic marketing environments.