Envelope Encryption and a Restored Production Snapshot
A healthcare analytics company stores PHI in a multi-cloud architecture: primary on AWS us-east-1, disaster recovery on Azure West Europe. The application uses envelope encryption: AWS KMS customer-managed keys wrap data keys; Azure Key Vault stores a separate wrapping key for DR. Annual rotation is manual. Developers have kms:Decrypt on the production key because they debug production issues. Last week a production snapshot was restored into a developer AWS account for testing; because the snapshot contained encrypted data and the developer account had a grant to the production key, PHI was readable. The incident was caught by a DLP alert after 6 hours. The CISO asks for a key management redesign that prevents this class of error without making DR impossible.
1. Propose a key hierarchy and cross-cloud key strategy that supports DR while limiting blast radius from restored backups.
Read the worked answer
Key hierarchy:
- Root KEK per cloud and environment: an AWS KMS customer-managed key in the production account and an Azure Key Vault Managed HSM key in the DR subscription. No key material export between clouds.
- Data keys per object or tenant: generated locally by the application, used for AES-256-GCM, then wrapped twice: once by the AWS production KEK and once by the Azure DR KEK. Store both wrapped copies alongside the ciphertext.
- Backup KEK separate from the live KEK: backups are encrypted under a backup key whose policy allows only the backup service and a break-glass role, not developers.
Why this supports DR: Azure can unwrap its copy of the data key during a regional disaster without needing AWS KMS. It avoids cross-cloud key import, which is often unsupported or weakens key custody.
Blast-radius controls: separate production and non-production keys; key policies deny kms:Decrypt to all non-production accounts and to human roles except break-glass; require encryption context (tenant, purpose, environment) in key policy conditions; enable CloudTrail and alert on decrypt from unexpected accounts. A restored snapshot remains encrypted under the production backup key, so a developer account cannot read it unless the key policy grants access.
2. Assess the developers' kms:Decrypt permission and propose an alternative that preserves debugging capability.
Read the worked answer
Current permission is too broad. kms:Decrypt on the production key lets any developer decrypt any ciphertext the key protects, including restored snapshots, if they can reach the ciphertext. Debugging does not require that.
Alternative:
- Remove kms:Decrypt from the developer role. Use a break-glass role with MFA, approval, and a one-hour session; all use is logged and reviewed.
- For debugging, use tokenization or format-preserving masking in non-production so developers see realistic but non-PHI values. If real data is required, use a just-in-time data-access service that decrypts one record at a time with an audit trail.
- Enforce encryption context in the key policy, for example environment=prod, and grant decrypt only when the context matches the caller's purpose. This narrows accidental use but does not replace authorization.
- Use per-tenant data keys so a developer granted one tenant's key cannot read all tenants.
- Alert on kms:Decrypt volume anomalies and on decrypt calls from non-production accounts.
Trade-off: incident response becomes slower; pre-approve a small on-call group for break-glass.
3. What controls would prevent a restored snapshot from being readable in a developer account?
Read the worked answer
Prevent restore-to-wrong-account readability with layered controls:
- Key policy boundary: the production backup KMS key policy must not grant kms:Decrypt to any principal in non-production accounts, even if the snapshot is shared. Use aws:PrincipalOrgID and account conditions, and deny kms:CreateGrant and kms:PutKeyPolicy except to a break-glass admin role.
- Snapshot sharing controls: block sharing of production snapshots outside the production account using AWS RAM or Organizations policies and SCPs; require a ticket and a re-encryption step for any restore.
- Re-encrypt on restore: the restore runbook must copy the snapshot into a non-production account, re-encrypt with a non-production key, and run a masking job before developers can access it. The restore role should not have decrypt on the production key.
- Detective control: CloudTrail alerts on ModifyDBSnapshotAttribute, CopyDBSnapshot, and kms:Decrypt from non-production accounts.
- DLP and access review: keep the DLP alert that caught the six-hour exposure, but add a 15-minute alert on production-key decrypt from a developer account.
Reasoning: defense in depth means that even if a snapshot is copied, the key policy makes it unreadable without separate approval.