Residuals of regression models with ARIMA noises

Description of the model

We consider the following regression model with stationary ARMA noises

$y = X \beta + \mu$

with $\mu \sim N(0, \sigma^2\Omega)$ , $y \sim n \times 1$ , $X \sim n \times n_X$

The GLS estimators of $\beta$ are defined by

$\hat\beta= \left(X'\Omega^{-1}X\right)^{-1} X'\Omega^{-1}y$

An alternative solution will consist in applying the inverse of the Cholesky factor of the covariance matrix on the model and than in applying OLS.

More exactly, we suppose that $\Omega = LL’$ . Than, we consider the regression model

$L^{-1}y = L^{-1} X \beta + L^{-1}u$

$y_l = X_l \beta + \epsilon$

with $\epsilon ~ N(0, \sigma^2 I)$

We define the raw residuals as $r_{raw}=y-X \hat\beta$

We define the full residuals as $r_{full}=y_l-X_l \hat\beta= L^{-1}r_{raw}$

The leas squares problem $y_l-X_l\beta$ is usually solved by means of the QR algorithm.

More exactly, we can find an orthogonal matrix Q and an upper triangular matrix R such that $X=QR$ .

We have then:

$\mid\mid y_l-X_l \beta\mid\mid=\mid\mid Q'y-R\beta \mid\mid$

which is minimized for

$\hat\beta=R_I^{-1}(Q'y)_I$

We can consider then the qr residuals defined by $r_{QR}=(Q’y)_{II}$ . $I$ refers to the first $n_X$ rows and $II$ to the next rows.

$r_{full} \sim n \times 1$ and $r_{QR} \sim (n-n_X) \times 1$ . We have of course that $\sum_i{r_{full, i}^2}=\sum_i{r_{QR, i}^2}$

Finally, we can consider the one-step-ahead forecast errors : $r_{KF,i}=y_i-E(y_i\mid i-1)$ . Those residuals can be generated by the Kalman filter.