2.1 Plain vanilla

The plain residual e_i and its plot is useful for checking how well the regression line fits the data, and in particular if there is any systematic lack of fit, for example curvature.

But, what value should be considered as a big residual?

Problem: e_i retains the scale of the response variable (Y).

Answer: standardize by an estimate of the variance of the residual.

Know, $Var(y_i) = \sigma^2$ estimated by (RMSE)².

But, $e_i = (y_i - \hat y_i)$, which is more than just y_i.

Turns out, $Var(e_i) = \sigma^2 (1 - h_{ii})$.

Use standardized residual, s_i.

The quantity, h_ii is fundamental to regression.

An heuristic explanation of h_ii (visually we are dragging a single point upward and measuring how the regression line follows):

Think about y_i the observed value, and $\hat y_i$ the estimated value (ie the point on the regression line).

For a fixed x_i perturb y_i a little bit, how much do you expect $\hat y_i$ to move?

If $\hat y_i$ moves as much as y_i then clearly y_i has the potential to drive the regression - so y_i is leveraged.

If $\hat y_i$ hardly moves at all then clearly y_i has no chance of driving the regression.

In other words h_ii is the measure of ``leverage''.

More precisely

$$h_{ii} = \frac{d\hat y_i}{d y_i},$$

and it depends only on the x-values.

Understanding leverage is essential in regression because leverage exposes the potential role of individual data points. Do you want your decision to be based on a single observation?