Vantage Insurance: 1,240 findings, a managed platform that cannot be scanned, and a board asking for one number
Vantage Insurance runs a monthly cloud security posture scan. Last month it produced 1,240 findings across three subscriptions and 470 accounts; 58% were never triaged and 31% were duplicates of four recurring rules. Mean time to remediate a publicly exposed storage bucket is 19 days. The team now wants an external penetration test of its customer-facing API, which runs on the provider's managed container platform and authenticates through the provider's identity service. The provider's acceptable-use policy requires 48 hours' written notice before any active scanning and prohibits testing the underlying managed services. The board has asked the CISO for one number that shows security is improving, and the CISO has a workshop next week to agree the test scope and the metric.
1. How should the team structure the penetration test given the provider's restrictions, and what assurance fills the gaps?
Read the worked answer
Scope honestly. The API's own logic is testable: authentication and session handling as implemented, object-level authorisation (can tenant A read tenant B's claim?), input validation, rate limiting, error handling and business-logic abuse. The identity service and the container platform's internals are not testable by Vantage under the acceptable-use policy. Write those exclusions into the test plan with the reason, so the report is not later read as 'no findings, therefore secure'.
Contractual steps before any traffic. Written notice inside the 48-hour window, naming source IP ranges, time windows, target endpoints and a rollback contact; the tester's written confirmation that it will not test the managed services; confirmation that the API's own traffic will not trip provider abuse detection and lock the account.
Filling the gaps. For the untestable layers, rely on the provider's own penetration-test attestation and assurance reports, confirm the period and scope, and read them for the platform and identity components specifically. Add configuration review of the identity integration, log review for anomalous authentication, and a review of how the application consumes tokens.
Judgement point. Proceeding without the notice because 'it is only a scan' is the error the exam is testing: it breaches the acceptable-use terms, may breach the customer contract, and can terminate the account.
2. Redesign how the 1,240 monthly findings are triaged and remediated.
Read the worked answer
The numbers as a diagnosis. 58% untriaged and 31% duplicates means the pipeline, not the cloud, is the problem. Volume with no ownership produces exactly the 19-day exposure window.
Redesign
- Collapse duplicates at source. Four rules generate a third of the findings; group them by rule and resource so the queue is counted in distinct issues, not rows.
- Route by ownership. Map every account to a team with a named owner. Findings with no owner never get fixed.
- Score on exposure and data class, not on scanner severity alone. An internet-reachable store with customer data outranks a hundred medium findings on an internal test account.
- Automate the repeats. Public bucket and unencrypted volume rules should trigger auto-remediation or auto-ticketing with a deadline, not a human read.
- Tune or suppress with evidence. A rule that cannot apply in this architecture should be disabled with a documented justification, so the noise floor drops honestly.
- Set SLAs by tier and measure against them: exposure-tier findings in hours, the rest in days.
Board-level consequence. Track the trend of distinct, exposure-tier issues and their age, not raw finding counts. Raw counts reward suppression and punish detection, which is the opposite of improvement.
3. The CISO wants 'one number that shows we are improving'. What should be presented instead, and why?
Read the worked answer
Why one number is dangerous. A single aggregate invites the wrong behaviour: teams reduce findings by disabling rules or by not scanning new accounts, and the number improves while risk rises. It also conceals the thing the board actually cares about - how long a real exposure stays open.
Present a small, defensible set
- Median and 90th-percentile time to remediate exposure-tier findings (internet-reachable, sensitive data), with the current 19 days as the baseline.
- Percentage of exposure-tier findings remediated inside SLA, which is easier to govern than an average.
- Recurrence rate for the four recurring rules - a direct measure of whether root causes are being fixed.
- Asset and account coverage: what proportion of accounts are scanned at all, because coverage is the hidden weakness behind a low finding count.
If forced to one number, use median time to remediate internet-exposed findings on data-bearing assets, and state its limits in one sentence on the slide: it measures responsiveness, not the absence of unknown exposure.
Governance. Publish the definitions, keep the calculation stable across quarters, and review the trend at the same forum that accepts risk. A metric that changes definition whenever it worsens is worse than no metric, because it corrodes trust in the whole reporting line.