[R] ridge regression - covariance matrices of ridge

[Ravi Varadhan]

Hi Michael,



The coefficients of ridge regression are given by:



\beta^* = (X'X + k I)^{-1} X' y,  ---------------- (1)



where k > 0 is the penalty parameter and I is the identity matrix.



The ridge estimates are related to OLS estimates \beta as follows:



\beta^* = Z \beta,  ---------------------- (2)



where Z = [I + k(X'X)^{-1}]^{-1} , ------------------ (3)



Let \Sigma and \Sigma^* be the variance-covariance matrices of \beta and \beta^*, resply.  Therefore,



\Sigma^* = Z \Sigma Z'  ----------------- (4)



In other words, for a fixed k, you can obtain the covariance matrix of \beta^* by making use of (3) and (4).



You can read the original paper by Hoerl and Kennard (Technometrics 1970) for more details.



Hope this is helpful,

Ravi.