The Key Policy Rewrite That Broke Decrypt, Snapshot Copy, and a Seven-Year Archive
Veridian Analytics ingests telemetry into account A, transforms it in account B, and archives to account C. S3 objects are encrypted with a customer managed key, key-producer-A, whose policy originally contained a single statement allowing kms:* to the account root principal. A security review replaced that statement with two statements naming the exact roles in A and B. The next nightly run failed: role-transform in account B received AccessDenied on kms:Decrypt. The RDS snapshot copy job to the DR region began failing at the same time. The team also found that key-legacy-2019 is scheduled for deletion in 21 days and still encrypts 90 TB of Glacier Deep Archive that finance must retain for seven years. Compliance now asks for proof that all data is re-encrypted every 12 months with fresh key material, and storage says re-encrypting 400 TB is not feasible.
1. Explain why the Decrypt calls fail and what must change. Cover key policy delegation, identity policies, and grants, and say how you would verify the fix.
Read the worked answer
Mechanism. In AWS KMS the key policy is the root of trust. The original Principal: arn:aws:iam::A:root statement delegated administration to IAM in account A, so identity policies could grant kms:Decrypt. Replacing it with two named principals removed that delegation: no IAM policy in any account can now grant access unless the principal is named in the key policy itself, or the account root is named so IAM can delegate, or a grant exists.
Why role-transform specifically fails:
- It is not one of the two named principals, so the key policy denies before its identity policy is consulted.
- Cross-account use needs both sides: the key policy in A must allow the caller, and role-transform's identity policy in B must allow
kms:Decrypt(andkms:DescribeKey). Fixing only one side changes nothing. - Check for a live grant before assuming: grants created earlier keep working even after a key policy change, so a hard AccessDenied confirms no grant is in place for that role. Grants cannot be self-created either, because creating a grant itself requires permission the key policy no longer gives.
Fix: restore delegation narrowly - add a statement naming the account B root or, better, the exact role ARN, with conditions kms:ViaService: s3.eu-central-1.amazonaws.com and an encryption-context condition where practical. Avoid reinstating kms:* to root; that reopens the finding.
Verification: reproduce with a single aws kms decrypt from an assumed role-transform session, read the exact errorMessage in CloudTrail (it distinguishes key-policy denial from identity-policy denial), diff get-key-policy against the intended design, and re-run the nightly job in a staging path before production. Keep the CloudTrail evidence of the successful call for the audit.
2. Why did the cross-region snapshot copy start failing at the same time, and what does the copy actually require?
Read the worked answer
It is a different permission problem that surfaced on the same change. A cross-region copy of an encrypted RDS snapshot is not a byte-for-byte copy: the snapshot is decrypted in the source region and re-encrypted with a KMS key in the destination region. So it needs:
- Source side: the calling principal needs
kms:Decryptandkms:CreateGranton key-producer-A, and the key policy must allow the RDS service to use the key for the snapshot. The key policy rewrite removed the principal the copy job and the RDS service relied on, which is exactly why the failure coincides with the Decrypt failure. - Destination side: a customer managed key must exist in the DR region whose key policy allows the caller
kms:GenerateDataKeyWithoutPlaintext,kms:Encrypt, andkms:CreateGrant, and allows the RDS service to use it. If the job previously used the AWS-managedaws/rdskey it cannot be shared cross-account, and it cannot be chosen explicitly - you cannot specifyaws/rdswhen copying an encrypted snapshot cross-account. - Account topology: if source and destination are different accounts, the snapshot must be shared first and encrypted with a customer managed key, because snapshots encrypted with the default key cannot be shared at all.
Diagnosis path: look at CloudTrail KMS events in the destination region for calls made by the RDS service principal; the region matters and is the most commonly missed place to look. Then confirm the destination key policy and whether the copy job's role is named there. Then add the missing destination key statement and re-test with a small snapshot rather than the production database, so a failed copy does not consume the window.
3. Address the deletion schedule, the rotation requirement, and the re-encryption demand together. What do you actually commit to?
Read the worked answer
1. Cancel the deletion now. CancelKeyDeletion is available during the 7–30 day waiting period and this is the last moment. If key-legacy-2019 is deleted, the 90 TB in Deep Archive is unrecoverable - there is no support path and no backup of key material. Restoring from Deep Archive takes up to 48 hours and costs real money, so re-wrapping is not a quick fix either. Add an alarm on ScheduleKeyDeletion events and require dual approval for any future key deletion.
2. Correct the compliance claim. For symmetric customer managed keys with AWS-generated material, enabling automatic annual rotation rotates the backing key material without changing the key ID or ARN, and old material is retained so existing ciphertext keeps decrypting. So 'fresh key material annually' is satisfied for the key, and the 400 TB does not need to be rewritten. Note the exceptions the auditor will ask about: imported key material does not auto-rotate (manual re-import and alias switch), and asymmetric and HMAC keys do not support automatic rotation.
3. Commit to re-encryption only where the requirement is genuinely about ciphertext. If the requirement is that every object is rewritten under the current key, do it lazily for everything that is rewritten anyway, and use S3 Batch Operations with an in-place copy for the rest - an in-place copy re-encrypts under the key's current material. Exclude Deep Archive from the first wave and handle it as a separately funded, scheduled project; document the residual risk and compensating keys for new data.
4. Design guardrails. Deny kms:ScheduleKeyDeletion at organization level except for a break-glass role, tag every key with owner, data class, and review date, and add a Config rule that flags keys without rotation enabled.