Linear Models and Regression
GATE ST examines linear models through matrix formulation, least squares, inference, and diagnostics. You should be able to derive OLS estimates, interpret coefficients, construct ANOVA tables, and recognize when assumptions fail.
Core rules
- Write y=Xβ+ε with E(ε)=0 and Var(ε)=σ²I. The OLS estimator is β̂=(X′X)⁻¹X′y when X has full column rank.
- Gauss–Markov says OLS is the best linear unbiased estimator under these assumptions; normality is needed for exact t and F tests, not for unbiasedness.
- The hat matrix H=X(X′X)⁻¹X′ gives fitted values ŷ=Hy and residuals e=(I−H)y. Leverage is the diagonal of H.
- R² measures explained sample variation and does not prove causation. Adjusted R² penalizes additional predictors.
- Multicollinearity inflates standard errors; variance inflation factors above about 10 signal serious problems. Ridge regression trades bias for variance.
Worked example
For points (1,2), (2,4), (3,5), x̄=2 and ȳ=11/3. Sxy=Σ(xi−x̄)(yi−ȳ)=5/3+0+4/3=3, and Sxx=2. The slope is b=Sxy/Sxx=1.5, and the intercept is a=ȳ−bx̄=11/3−3=2/3. The fitted line is ŷ=0.667+1.5x. The slope means predicted y increases by 1.5 units per one-unit increase in x within the observed range.
Traps
Do not extrapolate beyond the data range. A high R² can coexist with biased coefficients when an important variable is omitted. Do not interpret a regression coefficient as causal without design or identification. Check residual plots for nonconstant variance and curvature, and remember that influential points can change the slope dramatically.