VATE: a trade-off between memory and preserving time for high accuracy cardinalities estimation under sliding time window

Abstract: Host cardinality is one of the important attributes in the field of network research. The cardinality estimation under sliding time window has become a research hotspot in recent years because of its high accuracy and small delay. This kind of algorithms preserve the time information of sliding time window by introducing more powerful counters. The more counters used in these algorithms, the higher the estimation accuracy of these algorithms. However, the available number of sliding counters is limited due to their large memory footprint or long state-maintenance time. To solve this problem, a new sliding counter, asynchronous timestamp (AT), is designed in this paper which has the advantages of less memory consumption and low state-maintenance time. AT can replace counters in existing algorithms. On the same device, more AT can be used to achieve higher accuracy. Based on AT, this paper designs a new multi-hosts cardinalities estimation algorithm VATE. VATE is also a parallel algorithm that can be deployed on GPU. With the parallel processing capability of GPU, VATE can estimate cardinalities of hosts in a 40 Gb/s high-speed network in real time at the time granularity of 1 second.