Regression Analysis and Multivariate Methods
Core concepts
Understand simple and multiple linear regression: the least squares criterion, interpretation of coefficients, the coefficient of determination R², adjusted R², standard errors, t-tests for coefficients, and the overall F-test. Know the Gauss–Markov assumptions and the consequences of their violation: heteroscedasticity, autocorrelation, multicollinearity, and specification error. Be familiar with dummy variables, interaction terms, and transformations. For multivariate methods, know principal component analysis, factor analysis, discriminant analysis, cluster analysis, and canonical correlation, and when each is used.
Traps
- Interpreting a regression coefficient as a causal effect without a design that supports it.
- Reading R² as a measure of model adequacy; a high R² can coexist with biased coefficients.
- Forgetting that multicollinearity inflates standard errors but does not bias estimates.
- Using R² to compare models with different numbers of predictors; use adjusted R².
- Misreading dummy variable coefficients when the reference category is not stated.
- Confusing PCA (unsupervised dimension reduction) with discriminant analysis (supervised classification).
Worked example
In a model with intercept, one continuous predictor, and a two-level dummy, the dummy coefficient of 4.2 means the group coded 1 has an expected response 4.2 units higher than the reference group, holding the continuous predictor constant. Rule of thumb: a t-statistic for a coefficient is the estimate divided by its standard error, and with large samples values beyond about 2 are typically significant at 5%.