Systematic Spectral Distortion from Digital Whitening in Radio Telescopes and Implications for 21 cm Cosmology

Abstract: We identify a systematic distortion of the gain-vs.-frequency function of radio telescopes caused by digital flattening ("whitening") of the signal's spectrum followed by re-quantization, a common pair of processes in the signal processing of modern telescopes. Wide-bandwidth telescopes often have a large variation of signal power over frequency. Flattening of the spectrum allows samples of the channelized signal to be represented in a small number of bits, allowing efficient downstream processing. However, we show that this produces subtle systematic error in the measured spectra. We explore this effect in data from the Owens Valley Radio Observatory's Long Wavelength Array (OVRO-LWA) and through detailed semi-analytic simulations. Although the effect can be small so that it has heretofore been unrecognized, we demonstrate that it produces distortion of the spectrum at a level that is problematic for some science, in particular 21 cm cosmology. Finally, we explore mitigation strategies, showing that the effect can be substantially reduced by careful choice of the gain distribution along the signal path or by incorporating dithering in the re-quantization step.