Assessment Sampling That Hides a Failure
An assessor is evaluating a Moderate system with 325 applicable controls. To fit a 40-hour assessment window, the team samples 60 controls and reports 57 satisfied, 3 other-than-satisfied. The three failures are all in the access-control family. The assessor's report concludes the system is 'substantially compliant' and recommends authorization. The system owner is pleased. However, the assessment plan shows that the sampling method was convenience-based: assessors reviewed controls for which documentation was readily available, and the three access-control failures were discovered only incidentally. Two of the failed controls relate to account provisioning approvals; the third concerns periodic access reviews. The authorizing official's staff notices that the system has 480 privileged accounts against 210 total users and asks whether the sampling supports the conclusion.
1. Critique the assessment methodology and the 'substantially compliant' conclusion. What should the authorizing official do with this package?
Read the worked answer
The methodology is not defensible. Convenience sampling selects controls based on documentation availability, which systematically biases the sample toward well-documented, well-implemented controls and away from the controls most likely to be weak. A 60-of-325 sample can be legitimate only if it is risk-based and representative - stratified to ensure coverage of high-risk families, prior findings, and controls with known implementation difficulty. Here the only failures found were incidental, which suggests the sample was not designed to detect access-control weaknesses at all.
The 'substantially compliant' framing is also unsupported. Three failures concentrated in access control, on a system with 480 privileged accounts for 210 users, is a signal of a systemic provisioning and review problem, not three isolated exceptions. The authorizing official should reject the package for decision and require: (1) a redesigned assessment plan with stratified, risk-based sampling and explicit coverage of access-control controls; (2) targeted examination of the provisioning-approval and access-review controls across the full privileged population, not a sample; (3) reconciliation of the 480-versus-210 account discrepancy; and (4) a revised report that does not characterize the system as substantially compliant absent that evidence. Authorization should not proceed on this basis.
2. What would a defensible sampling approach look like for this system, and how should the assessor report results so the authorizing official can make a risk-based decision?
Read the worked answer
A defensible approach starts with stratification by risk: every control in the access-control, audit-and-accountability, and configuration-management families should be assessed in full or with high coverage, because those are where prior findings and the privileged-account anomaly point. Lower-risk families can be sampled, but the sample should be randomly selected within strata, sized to support the claimed confidence, and documented in the assessment plan before fieldwork. Any control that fails should trigger expansion of testing in its family rather than being reported as an isolated exception.
Reporting should separate: (1) controls assessed and satisfied, with evidence; (2) controls assessed and not satisfied, with the specific deficiency; (3) controls not assessed, explicitly labeled as coverage gaps rather than implied compliance; and (4) a risk summary connecting deficiencies to mission and data impact. The report should state the sampling design and its limitations so the authorizing official understands what the evidence does and does not support. The decision package should include a plan of action and milestones for the access-control deficiencies and a clear statement that authorization is being requested with known, documented risk - not on a claim of substantial compliance.