Envelope Encryption With Copied Keys and No Custodian Paper Trail
Arcadia Pay processes card-not-present payments and stores encrypted PAN in a 400 TB S3 data lake. It uses AWS KMS customer-managed keys in three regions with envelope encryption: each object has an AES-256 data key wrapped by a regional CMK. The original key ceremony three years ago had two engineers in a room; one generated the CMK, both wrote the same paper backup, and it went into a single safe. Rotation is manual and scheduled every 36 months; the next is in six months, and nobody has modelled how long re-encrypting 400 TB would take. Disaster recovery depends on imported key material copied from the primary region into the DR region, and two contractors hold a role with kms:Decrypt across the data lake. The head of platform states that “AWS manages the HSM, so Requirement 3.6 is inherited” and that the QSA should not need key management procedures.
1. Identify the key management gaps and the specific requirements they touch.
Read the worked answer
The gaps cluster around custody, distribution and rotation:
- 3.6.1.1 / 3.6.1.2: exporting and re-importing key material between regions is a key-distribution event. It needs a documented, dual-controlled procedure with logging; copying material is not a controlled distribution.
- 3.6.1.3 / 3.6.1.4: the same key material exists in the paper backup, the safe, the source region and the DR region. That is not the fewest possible locations, and storage is not demonstrably secure.
- 3.6.1.5: there is no written acknowledgement from key custodians.
- 3.7.6: split knowledge and dual control fail in substance - two people attended, but both produced the same backup and one safe means any single person with the combination controls the material.
- 3.7.7 / 3.6.2.1: a 36-month cryptoperiod is defensible only with a documented risk assessment and a tested rotation procedure. Manual rotation with no rewrap plan is neither.
- 3.7.9: no key-compromise procedure, despite two contractors holding broad decrypt rights.
- 7.2 / 8.2: human decrypt rights across the entire lake are excessive; decryption should be brokered by the application role, with human access exception-based and logged.
- 3.7.3 / 3.7.4: procedures exist informally but are not documented or enforced.
2. How would you respond to the claim that Requirement 3.6 is inherited from AWS?
Read the worked answer
Partly true and mostly irrelevant to the finding.
What is genuinely inherited: the physical HSM, its FIPS validation, the durability and physical security of the key store, and provider-side separation of duties. Those can be evidenced from the provider's attestation and configuration documentation, and the assessor can accept them as inherited controls for the infrastructure layer.
What is not inherited: the key policy and IAM grants, the decision about who may decrypt, custodian acknowledgements, rotation decisions and cryptoperiods, logging of kms:Decrypt into the SIEM (10.2), the compromise and revocation procedure, and the DR copies. Requirements 3.6 and 3.7 have no “provider does it” checkbox - the assessor must see Arcadia's procedures and evidence that they operated during the period.
A practical test of the provider claim: ask whether the provider's AOC covers the specific service and the specific regions used here. A KMS coverage statement for one region does not automatically extend to a region launched later, and imported key material is a customer-managed activity that sits squarely on Arcadia's side of the line.
So the answer to the platform head is: the HSM is theirs, the key lifecycle is yours, and the QSA will ask for exactly that lifecycle.
3. Design a rotation approach for the 400 TB lake that satisfies 3.6/3.7 without a five-week outage.
Read the worked answer
Rotate the key-encryption key, not the payload.
With envelope encryption, each object's data key is wrapped by the CMK. Rotation means creating a new CMK version and re-wrapping the stored data keys, then retiring the old version once nothing references it. The 400 TB of ciphertext never moves.
- Automatic rotation where supported: enable annual rotation for customer-managed keys. For imported material, automatic rotation is unavailable, so define a controlled re-import ceremony with new material, dual control, logging, and a new alias.
- Rewrap job: walk the object metadata and re-encrypt only the wrapped data keys in batches (for example five million objects per run). This is hours to days, not five weeks, because only metadata is rewritten.
- Retirement: keep the old CMK enabled for decrypt until the rewrap coverage metric reaches 100%, then schedule deletion with the provider's waiting period and document the rollback.
- Fix DR: stop copying key material between regions. Use region-specific CMKs and rewrap, or multi-region keys if available with the same custody controls.
- Document and test: record the cryptoperiod and rationale in the key management policy, run a targeted risk analysis if 36 months is retained, and rehearse the full rotation in DR before the scheduled date.
The point of the exercise is that an untested manual rotation is not a control; a rehearsed, measured, dual-controlled one is.