The general linear model

Least squares is orthogonal projection. Seeing it that way gives every distribution and test in regression at once.

1. The hat matrix (Prop. 1.1) and Cochran's theorem (Lemma 1.2), giving the sampling distribution of $\hat\beta$ and $s^2$ (Thm 1.3).
2. The F test for nested models (Thm 1.4), with one-way ANOVA (Example 1.5).
3. Confidence and prediction intervals (Prop. 1.6), collinearity (Example 1.7) and model selection (Def. 1.8).

$H = X(X^{\mathsf T}X)^{-1}X^{\mathsf T}$ is the orthogonal projection onto the column space $\mathcal C(X)$: $H^{\mathsf T} = H$, $H^2 = H$, $\operatorname{rank}H = \operatorname{tr}H = p$. The fitted values are $\hat Y = HY$, the residuals $e = (I - H)Y$, and $\hat Y\perp e$.

Let $Z\sim N(\mu, \sigma^2I_n)$ and let $P_1, \dots, P_k$ be orthogonal projections onto mutually orthogonal subspaces of dimensions $r_1, \dots, r_k$. Then $P_1Z, \dots, P_kZ$ are independent, and if $P_j\mu = 0$ then $\|P_jZ\|^2/\sigma^2\sim\chi^2_{r_j}$.

In the normal linear model, $\hat\beta\sim N_p(\beta, \sigma^2(X^{\mathsf T}X)^{-1})$, $\mathrm{RSS}/\sigma^2\sim\chi^2_{n-p}$, and $\hat\beta$ and $\mathrm{RSS}$ are independent. Hence $s^2 = \frac{\mathrm{RSS}}{n - p}$ is unbiased for $\sigma^2$ and

Let $\mathcal C(X_0)\subseteq\mathcal C(X)$ with dimensions $p_0 < p$, and test $H_0$: $\mathbb EY\in\mathcal C(X_0)$. With $\mathrm{RSS}_0$, $\mathrm{RSS}$ the residual sums of squares and $q = p - p_0$,

Reject $H_0$ for large $F$. For $q = 1$, $F = t^2$ for the corresponding coefficient.

With $k$ groups, $Y_{ij} = \mu_i + \varepsilon_{ij}$ ($j = 1, \dots, n_i$, $n = \sum n_i$) is a linear model with $p = k$ indicator columns; $\hat\mu_i = \bar Y_{i\cdot}$. Testing $H_0$: $\mu_1 = \cdots = \mu_k$ (the intercept-only model) gives

the ratio of between-group to within-group mean squares.

At a new covariate vector $x_0$, with $v_0 = x_0^{\mathsf T}(X^{\mathsf T}X)^{-1}x_0$:

• a $100(1 - \alpha)\%$ confidence interval for the mean $x_0^{\mathsf T}\beta$ is $x_0^{\mathsf T}\hat\beta\pm t_{n-p, \alpha/2}\,s\sqrt{v_0}$;
• a prediction interval for a new response $Y_0 = x_0^{\mathsf T}\beta + \varepsilon_0$ is $x_0^{\mathsf T}\hat\beta\pm t_{n-p, \alpha/2}\,s\sqrt{1 + v_0}$.

$\operatorname{Var}\hat\beta_j = \frac{\sigma^2}{(1 - R_j^2)\sum_i(x_{ij} - \bar x_j)^2}$, where $R_j^2$ is the $R^2$ from regressing column $j$ on the other columns. The factor $\mathrm{VIF}_j = \frac1{1 - R_j^2}$ is the variance inflation factor. Nearly collinear predictors ($R_j^2\approx1$) make individual coefficients unstable and insignificant even when the overall fit is good; values above about $10$ are a warning.

Adding predictors never increases $\mathrm{RSS}$, so fit alone favours the largest model. Criteria that penalise size:

• adjusted $R^2_{\mathrm{adj}} = 1 - \frac{\mathrm{RSS}/(n - p)}{\mathrm{TSS}/(n - 1)}$;
• AIC $= n\log(\mathrm{RSS}/n) + 2p$ and BIC $= n\log(\mathrm{RSS}/n) + p\log n$ (smaller is better);
• Mallows' $C_p$ $= \frac{\mathrm{RSS}_p}{\hat\sigma^2} - n + 2p$, with $\hat\sigma^2$ from the full model. Search strategies: all subsets, forward or backward stepwise.

• Count parameters $p$ (including the intercept) and residual degrees of freedom $n - p$.
• One coefficient: $t$ test. Several at once: F test comparing nested fits.
• Mean response: confidence interval; new observation: prediction interval.
• Insignificant coefficients with a good overall fit: check collinearity (VIFs).