Overdispersion in group health claim counts and the exposure offset
A health insurer models annual claim counts for 12,000 group policies. The Poisson GLM uses a log exposure offset in policy-years and covariates for age band, region and a chronic-condition flag. Output shows residual deviance 18,450 on 11,980 degrees of freedom, Pearson chi-square 22,100, and AIC 24,300. The chronic-condition coefficient is 0.85 with standard error 0.12. A negative binomial model gives the same mean structure, AIC 23,100, and a dispersion parameter theta of 2.1 with standard error 0.3. The chronic-condition coefficient becomes 0.88 with standard error 0.18. A colleague suggests using a quasi-Poisson model instead and asks whether the offset can be dropped because policy-years are similar across policies. You must advise on model choice, interpretation and checking.
1. Diagnose overdispersion from the output. Explain the consequences for the Poisson standard errors and for the reported chronic-condition effect.
Read the worked answer
The residual deviance divided by degrees of freedom is 18,450/11,980 = 1.54, and the Pearson chi-square divided by degrees of freedom is 22,100/11,980 = 1.84. Both exceed 1, which is evidence of overdispersion relative to the Poisson variance equal to the mean. The Poisson standard errors are therefore too small. A rough quasi-Poisson correction multiplies them by sqrt(1.54) to sqrt(1.84), so the chronic-condition z-statistic of 0.85/0.12 = 7.08 is overstated. Under the negative binomial fit the standard error rises to 0.18 and z = 0.88/0.18 = 4.89. The effect remains significant, but the strength of evidence is lower and the Poisson confidence interval is too narrow. AIC from the Poisson fit is also unreliable because the Poisson likelihood is misspecified.
2. Interpret the negative binomial dispersion parameter and the chronic-condition coefficient. Why might the coefficient and its standard error both change when moving from Poisson to negative binomial?
Read the worked answer
In the NB2 parameterisation, Var(Y) = mu + mu^2/theta. With theta = 2.1, at mu = 1 the variance is 1 + 1/2.1 = 1.476, and at mu = 3 it is 3 + 9/2.1 = 7.29. A smaller theta means more overdispersion; as theta tends to infinity the model tends to Poisson. The chronic-condition coefficient 0.88 means that, holding age, region and exposure fixed, the expected claim rate for a policy with the chronic condition is exp(0.88) = 2.41 times that for a policy without it. A 95% interval is exp(0.88 +/- 1.96 times 0.18) = 1.69 to 3.43.
The coefficient can move because the Poisson and NB likelihoods weight observations differently. The NB model downweights very high counts that are poorly explained by a Poisson mean, so the estimated effect can shift. The standard error rises because the model recognises extra-Poisson variation. If the coefficient changes substantially, check for influential policies, zero-inflation, and whether the mean-variance relationship is really quadratic.
3. Advise on quasi-Poisson versus negative binomial, and on dropping the offset. Include a model-checking plan in R.
Read the worked answer
Quasi-Poisson assumes Var(Y) = phi mu, while NB2 assumes Var(Y) = mu + mu^2/theta. Plot squared Pearson residuals or the empirical variance against the fitted mean. If variance grows roughly linearly with the mean, quasi-Poisson is adequate; if it grows quadratically, NB2 is more natural. Quasi-Poisson gives robust standard errors but no proper likelihood, so AIC cannot be used and model comparison is limited to weighted or F tests. Here the Pearson statistic of 1.84 and the AIC improvement of 1,200 favour the negative binomial, but confirm with residual plots and out-of-sample validation.
Do not drop the offset. Policy-years vary across policies, and dropping the offset models counts rather than rates, confounding exposure with risk and biasing coefficients. In R, fit glm(count ~ age + region + chronic + offset(log(exposure)), family = poisson), compute sum(residuals(type = 'pearson')^2)/df.residual, then fit MASS::glm.nb with the same mean structure. Check theta and its standard error, use DHARMa or simulated residuals for zero-inflation and dispersion, and compare AIC only among proper likelihoods. If quasi-Poisson is preferred, use sandwich::vcovHC for robust standard errors and report that AIC is unavailable.