4.5 Time Series

This subsection follows the conceptual arc of UC Berkeley STAT 153/248 [10] and parts of [18]. We will not repeat linear regression or general model selection. Instead, we focus on what is special about ordered data: dependence, forecasting from the past, and models whose structure respects time.

4.5.1 Dependence, stationarity, and forecasting

A time series is one observed path \(X_1,\ldots ,X_n\) of a stochastic process \(\{X_t\}_{t\in \mathbb {Z}}\). Unlike an iid sample, the ordering is part of the data. Usually \(X_t\) is informative about \(X_{t+1}\), and often we observe only one realization rather than many independent copies of the whole process.

Let \[ \mathcal {F}_t=\sigma (X_s:s\leq t) \] denote the information available at time \(t\). Under squared error loss, the best \(h\)-step-ahead forecast is the conditional mean \[ \widehat {X}_{t+h\mid t} =\mathbb {E}[X_{t+h}\mid \mathcal {F}_t]. \] This formula is deliberately abstract. Every forecasting model—ARMA, a state space model, a convolutional network, or a recurrent network—is a different way to summarize \(\mathcal {F}_t\) well enough to approximate this conditional expectation.

The simplest benchmark is white noise. A sequence \(\{\varepsilon _t\}\) is white noise with variance \(\sigma ^2\), written \(\varepsilon _t\sim \mathrm {WN}(0,\sigma ^2)\), if \[ \mathbb {E}[\varepsilon _t]=0,\qquad \operatorname {Var}(\varepsilon _t)=\sigma ^2,\qquad \operatorname {Cov}(\varepsilon _t,\varepsilon _s)=0\quad (t\neq s). \] White noise need not be independent. Gaussian white noise is special: because a Gaussian vector is independent exactly when it is uncorrelated, it is an iid Gaussian sequence.

To learn dependence from a single path, we need some stability across time.

Stationarity says that the covariance between two observations depends on their separation, not their calendar location. This lets different time windows act as comparable replicates. For a stationary series, natural estimates are \[ \widehat {\gamma }(h) =\frac {1}{n}\sum _{t=1}^{n-h}(X_{t+h}-\overline {X})(X_t-\overline {X}), \qquad \widehat {\rho }(h)=\frac {\widehat {\gamma }(h)}{\widehat {\gamma }(0)}. \] Strict stationarity requires the entire joint distribution to be invariant under time shifts; weak stationarity only asks for invariance of the first two moments. For linear prediction, the weaker notion is often enough. Stationarity alone does not guarantee that time averages converge to population averages; that additional role is played by ergodicity or suitable mixing conditions.

4.5.2 The time and frequency views

The ACF describes dependence by lag. The spectral density describes the same second-order structure by frequency. If \(\sum _h|\gamma (h)|<\infty \), define \[ f(\omega ) =\frac {1}{2\pi }\sum _{h=-\infty }^{\infty }\gamma (h)e^{-{\mathrm{i}} \omega h}, \qquad -\pi \leq \omega \leq \pi . \] Fourier inversion gives \[ \gamma (h)=\int _{-\pi }^{\pi }f(\omega )e^{{\mathrm{i}} \omega h}\,{\mathop{}\!\mathrm{d}} \omega , \qquad \operatorname {Var}(X_t)=\gamma (0)=\int _{-\pi }^{\pi }f(\omega )\,{\mathop{}\!\mathrm{d}} \omega . \] Thus \(f(\omega )\) decomposes variance by frequency: a peak near zero means slow, persistent variation, while a peak away from zero suggests oscillation. The time and frequency domains are not rival explanations; they are Fourier transforms of the same covariance information.

For observed data, the periodogram \[ I_n(\omega ) =\frac {1}{2\pi n} \left |\sum _{t=1}^n(X_t-\overline {X})e^{-{\mathrm{i}} \omega t}\right |^2 \] reveals prominent frequencies. A raw periodogram remains noisy even when \(n\) is large, so estimating a smooth spectrum usually requires averaging or smoothing nearby frequencies.

Filtering also becomes transparent in this view. If \(Y_t=\sum _j a_jX_{t-j}\) and \(A(z)=\sum _j a_jz^j\), then \[ f_Y(\omega )=|A(e^{-{\mathrm{i}} \omega })|^2f_X(\omega ). \] A moving average suppresses frequencies at which its transfer function \(A\) is small. This observation will later connect classical linear filters to one-dimensional convolutional networks.

4.5.3 ARMA models

We now turn to the central model class. Define the backshift operator \(BX_t=X_{t-1}\). For polynomials \[ \phi (z)=1-\phi _1z-\cdots -\phi _pz^p, \qquad \theta (z)=1+\theta _1z+\cdots +\theta _qz^q, \] an \(\operatorname {ARMA}(p,q)\) process satisfies \[ \phi (B)(X_t-\mu )=\theta (B)\varepsilon _t, \qquad \varepsilon _t\sim \mathrm {WN}(0,\sigma ^2). \] Equivalently, \[ X_t-\mu =\sum _{j=1}^p\phi _j(X_{t-j}-\mu ) +\varepsilon _t+\sum _{j=1}^q\theta _j\varepsilon _{t-j}. \] The AR part propagates past states; the MA part propagates past shocks. Notice that a time-series MA model averages latent innovations, not observed data. It should not be confused with the moving-average smoother used to remove noise.

The AR(1) model Consider \[ X_t-\mu =\phi (X_{t-1}-\mu )+\varepsilon _t. \] Iterating backward gives \[ X_t-\mu =\varepsilon _t+\phi \varepsilon _{t-1}+\phi ^2\varepsilon _{t-2}+\cdots . \] This causal stationary representation exists when \(|\phi |<1\). In that case, \[ \operatorname {Var}(X_t)=\frac {\sigma ^2}{1-\phi ^2}, \qquad \rho (h)=\phi ^{|h|}. \] Positive \(\phi \) creates persistent motion, negative \(\phi \) creates alternating motion, and \(|\phi |\) determines how quickly memory decays. At \(\phi =1\), shocks never decay and the model becomes the nonstationary random walk.

The \(h\)-step forecast and its error variance are especially revealing: \[ \begin {aligned} \widehat {X}_{t+h\mid t} &=\mu +\phi ^h(X_t-\mu ),\\ \operatorname {Var}(X_{t+h}-\widehat {X}_{t+h\mid t}\mid \mathcal {F}_t) &=\sigma ^2\sum _{j=0}^{h-1}\phi ^{2j} =\sigma ^2\frac {1-\phi ^{2h}}{1-\phi ^2}. \end {aligned} \] The point forecast forgets the present at rate \(|\phi |^h\), while forecast uncertainty grows toward the unconditional variance. These are two sides of the same loss of memory.

The MA(1) model Now consider \[ X_t=\varepsilon _t+\theta \varepsilon _{t-1}. \] It is stationary for every finite \(\theta \), with \[ \gamma (0)=\sigma ^2(1+\theta ^2),\qquad \rho (1)=\frac {\theta }{1+\theta ^2},\qquad \rho (h)=0\quad (|h|>1). \] The shock \(\varepsilon _t\) affects only \(X_t\) and \(X_{t+1}\), so the ACF cuts off after one lag. By contrast, an AR(1) shock has a geometrically decaying effect forever.

There is an identifiability issue: \(\theta \) and \(1/\theta \) generate the same ACF, and a rescaling of \(\sigma ^2\) can make their autocovariances identical. We choose the invertible representation \(|\theta |<1\), for which the innovations can be recovered from the observed past: \[ \varepsilon _t=(1+\theta B)^{-1}X_t =\sum _{j=0}^{\infty }(-\theta )^jX_{t-j}. \] Invertibility is therefore both a stability condition and a convention that selects one observable representation among equivalent ones.

General ARMA models For \(\phi \) and \(\theta \) with no common roots, the usual forecasting model is causal when every root of \(\phi (z)=0\) lies outside the unit circle, and invertible when every root of \(\theta (z)=0\) lies outside the unit circle. Then \[ X_t-\mu =\frac {\theta (B)}{\phi (B)}\varepsilon _t =\sum _{j=0}^{\infty }\psi _j\varepsilon _{t-j}, \qquad \sum _{j=0}^{\infty }|\psi _j|<\infty . \] Its spectral density is the rational function \[ f_X(\omega ) =\frac {\sigma ^2}{2\pi } \frac {|\theta (e^{-{\mathrm{i}} \omega })|^2}{|\phi (e^{-{\mathrm{i}} \omega })|^2}. \] Poles created by the AR polynomial produce persistent or oscillatory modes; zeros created by the MA polynomial suppress selected frequencies. This is the frequency-domain meaning of an ARMA model.

“Purely nondeterministic” rules out a nonzero component perfectly predictable from the infinite past. Wold’s theorem explains why linear filters of innovations are fundamental: they are not merely a convenient parametric guess. ARMA models replace the infinite filter \(\sum _j\psi _jz^j\) by the parsimonious rational approximation \(\theta (z)/\phi (z)\).

The ACF and partial ACF (PACF) provide a useful first diagnostic. The PACF at lag \(h\) measures the remaining linear association between \(X_t\) and \(X_{t-h}\) after accounting for the intermediate lags. In ideal population models, \[ \begin {array}{c|ccc} & \operatorname {AR}(p) & \operatorname {MA}(q) & \operatorname {ARMA}(p,q)\\ \hline \text {ACF} & \text {tails off} & \text {cuts off after }q & \text {tails off}\\ \text {PACF} & \text {cuts off after }p & \text {tails off} & \text {tails off} \end {array} \] This pattern suggests candidate orders; it does not prove that a model is correct. After fitting, the estimated innovations should resemble white noise. Visible autocorrelation in the residuals means that predictable structure remains. Among models with adequate residuals, criteria such as AIC or BIC trade improved fit against unnecessary order.

4.5.4 Nonstationarity and hidden state

ARMA models describe stationary series. For a series whose differences are stationary, an \(\operatorname {ARIMA}(p,d,q)\) model is \[ \phi (B)(1-B)^dX_t=\theta (B)\varepsilon _t. \] The random walk is \(\operatorname {ARIMA}(0,1,0)\). Seasonal differencing replaces \(1-B\) by \(1-B^s\) when a pattern repeats every \(s\) observations. Differencing is not a ritual: it asserts that changes are stable even when levels are not. Too much differencing injects artificial dependence and increases forecast variance, so one should use the smallest \(d\) that makes a stationary description plausible.

Another viewpoint separates an unobserved state from noisy measurements: \[ \begin {aligned} Z_{t+1}&=AZ_t+\eta _{t+1}, &&\text {state evolution},\\ X_t&=CZ_t+\varepsilon _t, &&\text {observation}. \end {aligned} \] In a linear Gaussian state space model, the Kalman filter alternates between predicting \(Z_t\) from the previous state and correcting that prediction using the new forecast error \(X_t-C\widehat {Z}_{t\mid t-1}\). The model distinguishes uncertainty about a changing signal from measurement noise. ARMA models admit state space representations, but the state space language also naturally handles missing observations, time-varying latent trends, and multivariate signals.

4.5.5 CNNs and RNNs as time-series models

Classical models make their memory explicit and linear. Neural models can learn nonlinear summaries of the past, but the same design question remains: what part of \(\mathcal {F}_t\) should be retained?

A one-dimensional causal convolutional network begins with filters such as \[ H_t=\sigma \!\left (b+\sum _{j=0}^{k-1}W_jX_{t-j}\right ). \] With the identity activation, this is simply a finite linear filter. Nonlinear activations and stacked filters learn local motifs; dilation allows a large time window without an equally deep network. Shared weights encode the belief that a pattern has the same meaning wherever it appears in time.

A recurrent neural network instead updates a learned state, \[ H_t=\sigma (UH_{t-1}+VX_t+b), \qquad \widehat {X}_{t+h\mid t}=g(H_t). \] The hidden state is a nonlinear, data-adaptive analogue of a state space model. If the activation is the identity and the powers of \(A\) decay, then \(H_t\) becomes an infinite linear filter of past observations. Thus CNNs resemble finite impulse response filters, while stable RNNs resemble infinite impulse response or state space models.

CNNs and RNNs are useful examples, not automatic replacements for ARMA. A flexible model can fit dependence that is too weak to estimate reliably from a single series. It should therefore be compared with simple forecasts, evaluated only on future observations, and judged by calibration as well as point accuracy. The essential question is unchanged: after conditioning on the model’s summary of the past, is there any predictable structure left in the forecast errors?