Numerical analysts frown upon regression calculations which manipulate the
cross product matrix X'X. Why? In some so-called ill-conditioned problems,
the inaccuracies of computer implementations of arithmetic lead to calculation errors.
One case can be introduced in polynomial regression. The hugh collinearity eventually
overwhelmes the numerical range of the computer and it gets the wrong answer. Avoiding
the X'X calculation improves the computers accuracy.
Perhaps a more important reason to look at alternative calculation strategies is the
understanding that they produce. We will look briefly at five matrix factorizations and
discuss how each factorization (or decomposition as they are also called)
plays an important role in statistics:
- LU decompositions
- These lead to a system of recursive solutions of linear equations and a first
look at factorization. They get you starting thinking about what is involved in
solving systems of equations, such as backsubstitutions.
- Cholesky factorization
- The most common matrix square root, this is basically an LU decomposition
in which the factors are equal, but transposed. The statistical application is
to generating correlated normal samples, as in regression or time series.
- QR factorization
- Rather than factor the matrix into triangular forms, the QR forms a Gram-Schmidt
reduction into a triangular and orthogonal system. This is the first that we
have seen that works with non-square matrices and a popular choice for regression
calculations.
- Spectral representation
- All square matrices can be represented through a nice cannonical form using
eigenvectors and eigenvalues. The spectral representation leads to a different
version of the square root of a matrix. The version in LispStat is limited to
symmetric matrices.
- Singular value decomposition
- This representation generalizes the notion of eigenvalues/vectors to non-square
matrices. The resulting representation leads to a different type of regression.