PKA:An Extension of Sheldon M. Ross's Method for Fast Large-Scale Variance Computation

Abstract: The paper introduces Prior Knowledge Acceleration (PKA), a method to speed up variance calculations by leveraging prior knowledge of the variance in the original dataset. PKA enables the efficient updating of variance when adding new data, reducing computational costs by avoiding full recalculation. We derive expressions for both population and sample variance using PKA and compare them to Sheldon M. Ross's method. Unlike Sheldon M. Ross's method, the PKA method is designed for processing large data streams online like online machine learning. Simulated results show that PKA can reduce calculation time in most conditions, especially when the original dataset or added one is relatively large. While this method shows promise in accelerating variance computations, its effectiveness is contingent on the assumption of constant computational time.