Detection of Malicious DNS-over-HTTPS Traffic: An Anomaly Detection Approach using Autoencoders

Abstract: To maintain the privacy of users' web browsing history, popular browsers encrypt their DNS traffic using the DNS-over-HTTPS (DoH) protocol. Unfortunately, encrypting DNS packets prevents many existing intrusion detection systems from using plaintext domain names to detect malicious traffic. In this paper, we design an autoencoder that is capable of detecting malicious DNS traffic by only observing the encrypted DoH traffic. Compared to previous works, the proposed autoencoder looks for anomalies in DoH traffic, and thus can detect malicious traffic that has not been previously observed, i.e., zero-day attacks. We run extensive experiments to evaluate the performance of our proposed autoencoder and compare it to that of other anomaly detection algorithms, namely, local outlier factor, one-class support vector machine, isolation forest, and variational autoencoders. We find that our proposed autoencoder achieves the highest detection performance, with a median F-1 score of 99\% over several types of malicious traffic.