Linear Models
CS1 tests linear models: specification, ordinary least squares, inference and diagnostics. You should be able to write the model matrix, interpret coefficients, and distinguish fitted values from predictions.
Core rules
- Simple linear regression: Y_i=β0+β1x_i+ε_i, with ε_i iid N(0,σ²). OLS minimises Σ(y_i−β0−β1x_i)².
- With x centred, β̂1=Sxy/Sxx and β̂0=ȳ−β̂1x̄. Residuals e_i=y_i−ŷ_i satisfy Σe_i=0 and Σx_i e_i=0.
- Under normality, β̂1~N(β1,σ²/Sxx); use t tests for individual coefficients and an F test for the overall model.
- R²=1−SSE/SST measures explained variation but does not validate the model.
Worked example
For x=(1,2,3,4), y=(2,4,5,8), x̄=2.5, ȳ=4.75. Sxx=Σ(x−x̄)²=5, Sxy=Σ(x−x̄)(y−ȳ)= (−1.5)(−2.75)+(−0.5)(−0.75)+(0.5)(0.25)+(1.5)(3.25)=4.125+0.375+0.125+4.875=9.5. Thus β̂1=9.5/5=1.9 and β̂0=4.75−1.9×2.5=0.0. The fitted line is ŷ=1.9x. Check: predictions are 1.9, 3.8, 5.7, 7.6, with residuals 0.1, 0.2, −0.7, 0.4 summing to 0.
Traps
- Extrapolating beyond the observed x range.
- Reading R² as proof of causality or correct functional form.
- Forgetting that adding predictors never decreases R²; use adjusted R².
- Ignoring residual plots that show curvature or heteroscedasticity.