Exponentially Weighted Moving Models (2404.08136v2)

Abstract: An exponentially weighted moving model (EWMM) for a vector time series fits a new data model each time period, based on an exponentially fading loss function on past observed data. The well known and widely used exponentially weighted moving average (EWMA) is a special case that estimates the mean using a square loss function. For quadratic loss functions EWMMs can be fit using a simple recursion that updates the parameters of a quadratic function. For other loss functions, the entire past history must be stored, and the fitting problem grows in size as time increases. We propose a general method for computing an approximation of EWMM, which requires storing only a window of a fixed number of past samples, and uses an additional quadratic term to approximate the loss associated with the data before the window. This approximate EWMM relies on convex optimization, and solves problems that do not grow with time. We compare the estimates produced by our approximation with the estimates from the exact EWMM method.

• The paper introduces an approximate recursive method for EWMMs that uses a fixed-size window to manage non-quadratic loss functions.
• It significantly reduces computational overhead and storage by approximating tail losses using Taylor expansion or a direct fitting method.
• Extensive numerical experiments validate the approach in settings like quantile estimation, logistic regression, and sparse inverse covariance estimation.

Efficient Approximation Methods for Exponentially Weighted Moving Models

Introduction

Exponentially Weighted Moving Models (EWMMs) offer a flexible framework for fitting time-varying models to vector time series data. By assigning exponentially decaying weights to past observations, EWMMs prioritize more recent data without disregarding historical information. While EWMMs generalize the concept of Exponentially Weighted Moving Averages (EWMAs), their utility extends to a broad array of models, including quantile estimation, covariance estimation, and regression models amongst others.

The fundamental challenge in employing EWMMs lies in their computation. For models with quadratic loss functions, an exact recursive computation akin to EWMAs is possible. However, for non-quadratic loss functions, direct computation necessitates storing the entire history of data, rendering the process increasingly impractical over time. This paper introduces an approximation method for EWMMs that circumvents this issue by maintaining only a fixed-size window of recent observations, thereby ensuring constant computation and storage requirements over time.

Exponentially Weighted Moving Models Overview

The EWMM framework integrates both historical and recent data in model fitting, with the significance of each data point diminishing exponentially into the past. This setup offers a more reactive and adaptable model compared to traditional rolling window models (RWMs), which equally weigh all observations within a fixed window. The flexibility of EWMMs is illustrated through their capability to estimate various statistics and parameters, such as quantiles and covariance matrices, effectively capturing the underlying dynamics of time series data.

Recursive and Approximate Methods

For quadratic loss functions, EWMMs benefit from a straightforward recursive computation methodology, significantly reducing the computational overhead by avoiding the need to revisit the entire data history. This approach is directly applicable to a range of models, including exponentially weighted least squares and sparse inverse covariance estimation.

Conversely, for non-quadratic loss functions, an exact recursive solution is absent. To address this, the paper innovates with an approximate recursion technique that relies on a fixed-size data window alongside a quadratic approximation of the "tail" loss – the loss attributable to data outside this window. This approximation is crafted through either a Taylor expansion method, suitable for twice-differentiable loss functions, or a direct fitting method that utilizes an additional subset of past data to refine the tail loss approximation.

Numerical Examples and Implications

The paper presents extensive numerical experiments to validate the effectiveness of the proposed approximate EWMM method in various settings, including quantile estimation, logistic regression, and sparse inverse covariance estimation. The results affirm that the approximations closely mirror the exact EWMMs while requiring significantly less computational resources. Particularly, the examples emphasize the method's adaptability to both smoothly varying datasets and scenarios requiring parameter regularization.

Conclusion and Future Work

By introducing a practical and computationally efficient method for approximating EWMMs, this paper addresses a critical gap in the application of time-varying models to large datasets. The approximation technique ensures that the models maintain their adaptability and accuracy over time without succumbing to the impracticalities of unbounded data storage and processing requirements.

The potential for future research includes exploring more sophisticated approximation methods for the tail loss, enhancing the efficiency of the approximation in high-dimensional settings, and investigating the integration of EWMMs with online learning algorithms to further bolster their applicability in real-time analysis scenarios.

Acknowledgements

The contribution of peers and notable scholars in the field for their invaluable feedback and suggestions during the development of this work is notably appreciated, cementing the foundational premise of EWMMs as a potent and flexible tool for time-varying model estimation in time series analysis.