Hierarchical modeling and statistical calibration for photometric redshifts

Abstract: The cosmological exploitation of modern photometric galaxy surveys requires both accurate (unbiased) and precise (narrow) redshift probability distributions derived from broadband photometry. Existing methodologies do not meet those requirements. Standard template fitting delivers interpretable models and errors, but lacks flexibility to learn inaccuracies in the observed photometry or the spectral templates. Machine learning addresses those issues, but requires representative training data, and the resulting models and uncertainties cannot be interpreted in the context of a physical model or outside of the training data. We present a hierarchical modeling approach simultaneously addressing the issues of flexibility, interpretability, and generalization. It combines template fitting with flexible (machine learning-like) models to correct the spectral templates, model their redshift distributions, and recalibrate the photometric observations. By optimizing the full posterior distribution of the model and solving for its (thousands of) parameters, one can perform a global statistical calibration of the data and the SED model. We apply this approach to the public Dark Energy Survey Science Verification data, and show that it provides more accurate and compact redshift posterior distributions than existing methods, as well as insights into residual photometric and SED systematics. The model is causal, makes predictions for future data (e.g., additional photometric bandpasses), and its internal parameters and components are interpretable. This approach does not formally require the training data to be complete or representative; in principle it can even work in regimes in which few or no spectroscopic redshifts are available.