Time Series and Stochastic Processes
This domain covers models for data observed in sequence, where dependence between successive observations is the whole point rather than an inconvenience.
Stationarity and dependence
- Weak stationarity means constant mean and variance, with autocovariance depending only on the lag. A random walk has variance growing linearly in time and must be differenced.
- The autocorrelation function rho_k = gamma_k / gamma_0 measures linear dependence at lag k; the partial autocorrelation function isolates the direct effect at each lag.
- White noise has zero autocorrelation at all non-zero lags and is the building block of ARMA models.
ARMA and ARIMA models
- AR(p): current value depends on past values. For AR(1) with parameter phi, the ACF decays geometrically as phi^k while the PACF cuts off after lag 1.
- MA(q): current value depends on past shocks. For MA(1), the ACF cuts off after lag 1 while the PACF decays.
- ARMA(p,q) combines both; ARIMA(p,d,q) differences the series d times to reach stationarity.
- Box-Jenkins: identify orders from ACF and PACF, estimate, check residual autocorrelation with Ljung-Box, then forecast. AIC and BIC compare candidate models; genuine forecast skill is confirmed out of sample.
Worked example
An AR(1) series x_t = 0.6 x_(t-1) + e_t with innovation variance 4 is stationary since |0.6| < 1. Its variance is 4/(1 - 0.36) = 6.25, and its autocorrelations are 0.6, 0.36, 0.216 for lags 1 to 3. The one-step forecast from a current value of 10 is 6, and forecast variance rises towards the unconditional 6.25 as the horizon lengthens.
Traps that cost marks
- Regressing one non-stationary series on another and interpreting a high R-squared (spurious regression).
- Differencing a series that is already stationary, which induces a spurious negative MA term.
- Judging models on in-sample fit rather than out-of-sample forecast accuracy.