Calibration Curve: Extrapolating Beyond the Standards and Trusting R²
A clinical chemistry laboratory validates a glucose assay. Six calibrators at 0, 100, 200, 300, 400 and 500 mg/dL are each measured once in a single run. Ordinary least squares of absorbance on concentration gives y = 0.00312x + 0.041 with R² = 0.9987. A proficiency sample returns absorbance 1.98, which back-calculates to 621 mg/dL, and the supervisor wants to report it. A reviewer notices two things. First, residuals are tiny at low concentration and grow steadily with concentration; repeat measurements of a 200 mg/dL pool (n = 10) give SD 0.021 absorbance units, while ten repeats of the 500 mg/dL standard give SD 0.048. Second, the zero calibrator was measured once, so the intercept is unverified. No certified reference material above 500 mg/dL is available, and the instrument manual warns that absorbance above 2.0 may leave the linear response range.
1. Should the laboratory report 621 mg/dL? Support your answer with the leverage/uncertainty calculation.
Read the worked answer
No. 621 mg/dL lies outside the calibrated range (0–500) and cannot be supported by these data.
The regression is only validated over the span of the standards. Above 500 mg/dL the relationship may bend for physical reasons - detector saturation, reagent exhaustion, path-length nonlinearity - and nothing in the data constrains it. The manual's warning about absorbance above 2.0 is directly relevant: the measured absorbance is 1.98, at the edge of the stated linear range.
Quantify the widening. With x = 0, 100, 200, 300, 400, 500: x̄ = 250 and Sxx = 2(250² + 150² + 50²) = 175,000.
The confidence-band multiplier is √[1/n + (x₀ − x̄)²/Sxx]:
- at the centroid x₀ = 250: √(1/6) = 0.408
- at x₀ = 500: √(1/6 + 62,500/175,000) = √0.5238 = 0.724
- at x₀ = 621.5: (371.5)² = 138,012, so √(1/6 + 138,012/175,000) = √0.9553 = 0.977
The interval at 621 mg/dL is about 0.977/0.408 = 2.4 times as wide as at the centroid, and that is before allowing for any curvature or for the heteroscedasticity noted by the reviewer.
Action: dilute the sample into the calibrated range (e.g. 1 + 1 with the same matrix), measure in duplicate, apply the dilution factor, and document dilution integrity and recovery. In the interim report ">500 mg/dL, dilution pending". Reporting 621 would present an extrapolated, model-dependent number as a measurement.
2. The supervisor argues that R² = 0.9987 proves the model is correct. Explain why this reasoning is unsound and what the residual pattern implies for inference.
Read the worked answer
R² = 0.9987 means the straight line accounts for 99.87% of the variance in absorbance. But the calibrators span 0 to 500, so the variance to be explained is enormous and almost any monotone function scores highly. R² is inflated by a wide range of x and is insensitive to the shape of the residuals; it says nothing about whether the functional form or the error assumptions are right.
The residual pattern is the diagnostic that matters. Residual spread increases with concentration, matching the replicate SDs (0.021 at 200, 0.048 at 500). Constant variance is therefore violated.
Consequences:
- OLS standard errors, confidence intervals, detection limits and prediction intervals are all wrong: too optimistic at high concentration, too conservative at low.
- Unweighted regression is dominated by the high-concentration points, which is acceptable only if the variance really is constant. Here it is not, so the fit is not efficient.
Remedies:
- Plot residuals against fitted values and against concentration; never report R² alone.
- Refit by weighted least squares with weights ∝ 1/x² (or 1/s² from the replicate SDs), or model log(absorbance) if the spread is proportional to the mean.
- Run replicates at each level and perform a lack-of-fit test comparing pure error with residual (between-level) variation - this tests curvature directly, which R² cannot.
- Report the intercept with its confidence interval. With a single zero calibrator the intercept 0.041 is unverified; a non-zero intercept indicates a blank/background problem and must be carried through the calculation, not dropped.
3. Compute the leverage of the 500 mg/dL calibrator and explain what it implies for this calibration.
Read the worked answer
Leverage for a simple linear regression: hᵢ = 1/n + (xᵢ − x̄)²/Sxx.
With n = 6, x̄ = 250, Sxx = 175,000:
- x = 500: h = 0.1667 + 250²/175,000 = 0.1667 + 0.3571 = 0.524
- by symmetry, x = 0 has the same h = 0.524
- an interior point at 250 would have h = 0.1667
The two end calibrators each carry 52% of the leverage in the fitted line; the four interior points share the remaining ~48% between them.
Implications:
- The slope is largely determined by the 0 and 500 standards. A single bad measurement at 500 rotates the entire line - and 500 is the very region from which the laboratory is extrapolating. The two hazards compound.
- The common screening rule 2p/n = 4/6 = 0.667 would not flag these points. That rule is unreliable in tiny designs; the numeric leverage matters, not just a flag.
- Mitigations: run the 0 and 500 standards in replicate and average; compute a covariance/studentized residual check; refit with each point deleted and report how far the slope moves (deleting the 500 standard will move it most).
- If the assay routinely reports near or above 500 mg/dL, extend the calibration upward rather than relying on the current model.
Leverage is not a defect of the data - a well-spread calibration should have high-leverage endpoints. It is a warning about where the fit is fragile and where answers must be treated as model-dependent.