Least Privilege and Customer-Controlled Keys in a 140-Engineer Payments Account
Meridian Health Pay adjudicates payments for 300 clinics. Its IBM Cloud account holds 140 engineers in six product teams, three environments (dev, staging, prod) collapsed into a single resource group, and 62 service IDs whose API keys were pasted into Git history over two years. Contractors rotate every six months. An external auditor issued three findings: there is no evidence permissions are least-privilege, the payments database uses provider-managed encryption keys, and there is no reliable record of who deleted objects from the claims bucket. A contract clause requires FIPS 140-2 Level 4 validated hardware security modules with customer-controlled key material for the payments workload. Developers need read access to production logs during incidents, but not permanently. The head of security wants CI/CD pipelines to keep working with minimal change.
1. Redesign identity and access so that the audit finding on least privilege is closed without breaking pipelines. Address the 62 leaked keys explicitly.
Read the worked answer
Start with the leaked keys: they are compromised the moment they are committed, so step one is inventory and revocation, not policy design. Use the account's access and key inventory to enumerate service IDs and API keys, map each to a consumer by checking CI/CD variables and deployment manifests, then rotate in place and confirm pipelines still run. Removing the strings from the current branch while they remain in history is not remediation.
Structure. Split resource groups by environment first (dev, staging, prod) and by workload second. Today one resource group means every policy is account-wide in effect. Then define access groups by role - viewer, operator, developer, pipeline deployer, security admin - and attach policies scoped to a resource group and service, never account-level editor. Contractors belong to identity-provider groups whose membership expires at the IdP, since IAM policies themselves do not expire; quarterly access reviews become a group-membership query rather than a spreadsheet of 140 people.
Pipelines. Replace long-lived keys where possible with trusted profiles that grant compute identities to workloads based on conditions such as the VPC or compute resource they run on, so the pipeline obtains short-lived credentials. Where a service ID must remain, keep its key in a secrets manager, rotate on a schedule under 90 days, and inject at runtime. Pipelines change configuration, not architecture, which satisfies the security lead's constraint.
2. The payments contract demands FIPS 140-2 Level 4 HSMs with customer-controlled key material. Which key service satisfies it, and how do you design separation of duties and recovery?
Read the worked answer
The clause rules out a shared, multi-tenant key management service validated to a lower level. A dedicated, single-tenant cloud HSM offering at FIPS 140-2 Level 4 is the match, and it is also the mechanism for keep-your-own-key: key material is generated in the customer's dedicated HSM partition, and the provider cannot reconstruct it.
Design.
- One key store or key ring per environment; never reuse a production root key in staging.
- Separate access groups: key custodians who administer the HSM and keys, and data administrators who can attach a key to a database or bucket but cannot read or delete the root key. The auditor's real question is whether one person can both exfiltrate data and destroy the key; the answer must be no.
- Dual control and quorum for high-risk operations such as key deletion or initialisation, implemented with signing keys held by different people.
- Envelope encryption: the data service uses a data key wrapped by the root key, so rotation of the root key does not require re-encrypting 900 TB.
- Rotation schedule with documented rewrap of data keys, and tested recovery from a backup of the key material.
Recovery. State plainly that destroying the root key renders the payments data unrecoverable - that is the control - and that the recovery plan therefore rests on custodian backups held under dual control. Test a restore into a non-production partition before the auditor asks.
3. How do you produce reliable evidence of who deleted objects from the claims bucket, and how do you handle temporary production log access?
Read the worked answer
Evidence requires that logging be turned on, retained, and out of reach of the people it records.
Deletion evidence. Object-level read and write events are not captured by default in the same way management events are, so explicitly enable data event auditing for the claims bucket and verify with a controlled delete that the event appears. Route Activity Tracker events to a destination in a separate, tightly controlled account or to an event stream feeding the SIEM, with retention at least as long as the audit cycle. Grant bucket-delete permission only to a break-glass identity with dual approval, and enable versioning plus retention on the bucket so a delete is recoverable while the investigation proceeds.
Temporary production access. There is no native expiring IAM policy, so do not claim one exists. Implement just-in-time access through an identity-provider group that a workflow adds a user to on approval and removes automatically at the end of the window - for example two hours - with the change itself logged. Record the approval, the membership change, and the session. Quarterly, reconcile group membership against the approved roster.
Posture and support. Use a security and compliance posture tool for continuous checking of these controls, and align the support tier with the incident response commitments the clinics' contracts require. The audit answer is a repeatable query, not a one-off screenshot.