Exponential Smoothing of Time Series and Kernel Regression

Kathryn Richardson

Department of Statistics
University of British Columbia
Vancouver, B.C.
Canada, V6T 1Z2

April 2003

“Understanding exponential smoothing via kernel regression”, I. Gijbels, A. Pope, M.P. Wand (1999)

Introduction

•	What are the properties of a time series?

- A set of observations {Yt}, indexed by time (t=1,2,…,T).
- The T time points x1,…,xT are equally spaced (with time “d” between each point e.g. x4-x3=d).
- Here, we assume that there are no missing times.
- Observations are correlated; we define the autocorrelation function ρs,t=corr(Ys,Yt). 
- Of major interest is predicting future values of the time series.

•	
What is exponential smoothing?

The EWMA (Exponentially Weighted Moving Average) is a simple method of time series smoothing. 

We predict the next observation by applying exponentially decreasing weights to the past observations, where λ is the discount factor:
                                      
It can be easily shown that the above equation be rewritten as recurrence relations (for t=1,2,…,T):

Kernel Smoothing

•	EWMA is virtually identical to the Nadaraya-Watson kernel estimator with a “half-kernel” (Gijbels, Pope & Wand, 1999).
                                       
If we look at the sum of the weights:

     

so              as  

We can adjust the weights so they sum to one for all T.
                                    (1)

                                    (1)
If we define 

and use “half-kernel” K(u) =exp(u)1{u≤0} then we can rewrite equation (2) as:

             (2)

Therefore, the normalized EWMA forecast is equivalent to a Nadaraya-Watson, or zero-degree local polynomial, kernel estimate at xT+1.

Optimal smoothing
•	The optimal discount factor, λ, in EWMA is the one that minimises the average of the squared residuals (ASR).   

This is optimal when the data follows a state space model: 

•	For the Nadaraya-Watson kernel estimate: 

So the optimal bandwidth that minimises the ASR,

where is the estimate of Yt with observation (xt,Yt) omitted. Therefore, the optimal bandwidth is that from cross-validation with a half-kernel.
General degree polynomial

             (2)
•	The Nadaraya-Watson kernel estimator is a zero-degree local polynomial (so estimates a local constant).
•	Local linear estimates are better at the boundary than local constants.
•	Effectively, time series prediction involves smoothing near the boundary for each time point. 
•	Equation (2) can be extended to a general p degree polynomial kernel estimate:
where

Asymptotics

•	The previous general polynomial kernel estimate can be applied to time series data if it is of the form:

Here the errors can be correlated, with covariance function .

•	Define:

Mp(u) = Np, but with the 1st column as (1,u,…,up)T.

The asymptotics of our predictions are made through defining the half-kernel: 

•	Given various assumptions, such as the continuity of m(p+1) and K(p+1) (on [-x,0]), and assuming

Asymptotically optimal bandwidth 

•	The ASR(h) was defined previously as an optimality measure.

•	The global error of the estimate of m is a common optimality measure. We choose h to minimise the average square error (ASE) where:

And MASE(h) = Е{ASE(h)}.

Let

and

•	
The optimal bandwidths (under the respective measures) are:

Consider 3 cases:
•	Errors are independent
o	
•	Errors are positively correlated
o	
o	Cross-validation (ASR) is appropriate for prediction as averaging over a small number of previous observations is good when they are positive correlated.
•	Errors are negatively correlated
o	
o	CV (ASR) smooths out the extra variability due to the negative correlations.