Cross-Account Snapshot Sharing and Credential Hygiene Under Audit
Halcyon Health has prod, analytics and dev accounts under one AWS Organization. Analytics needs a nightly read-only copy of a 2 TB Aurora PostgreSQL cluster in prod to run models. Today an engineer takes a manual snapshot, shares it with analytics, and analytics restores it. Last month the same snapshot ARN was also shared with dev by mistake and sat there for 11 days. Compliance requires: all data at rest encrypted with customer-managed KMS keys, annual rotation, no long-lived database passwords in code, and a demonstrable audit trail of who accessed what. Forty application pods connect with a password stored in a Kubernetes secret created 18 months ago. Developers occasionally connect to prod with the same shared application user to debug. A separate platform-security team owns the KMS keys and will not delegate key administration to the data team.
1. Redesign the cross-account copy so dev cannot receive prod data, and explain the KMS mechanics that make it work.
Read the worked answer
KMS mechanics. An encrypted snapshot cannot be shared unless the snapshot's customer-managed key policy allows the target account to use the key. The receiving account needs kms:DescribeKey, kms:CreateGrant and kms:Decrypt (or a grant marked as for an AWS resource) on the source CMK. Sharing the snapshot itself is then done with modify-db-cluster-snapshot-attribute setting restore to the target account ID.
Two consequences to design around:
- The receiving account should immediately re-encrypt on restore using its own CMK, so prod's key never protects analytics' long-lived data and prod's key policy does not have to keep granting analytics access.
- Cross-region sharing is not direct: copy the encrypted snapshot into the target region first (still under prod control), then share.
Preventing dev from ever receiving it. Relying on nobody making a mistake is not a control.
- Write the key policy so the grant/decrypt statement names the analytics account principal specifically - not the Organization root, which would silently include dev.
- Add an SCP on the dev account denying
rds:RestoreDBClusterFromSnapshot,rds:CopyDBClusterSnapshotandrds:ModifyDBClusterSnapshotAttributeexcept for an explicit allowlist of snapshot ARNs that contains no prod data. - Prefer AWS RAM for the sharing relationship: the resource share has an explicit principal list, is visible in one place, and is auditable, instead of a per-snapshot attribute that anyone with the API permission can edit.
- Remove human share ability: permission-boundary the data team so they can invoke a pipeline, not call the share API. The pipeline (EventBridge schedule → Step Functions → RDS/KMS) takes the snapshot, shares it to analytics only, and logs the action. Re-encrypt on restore is a pipeline step, not a human step.
- Alarm in real time: EventBridge rule on
ModifyDBClusterSnapshotAttributeandRestoreDBClusterFromSnapshotevents, plus an AWS Config custom rule asserting therestoreattribute equals the analytics account only.
The security team keeps control by issuing the data team a scoped grant with a grant constraint (for example restricted by encryption context to the specific cluster ARN) rather than key administration.
2. Choose an authentication model for the pods and for developers, and describe the migration off the shared user.
Read the worked answer
Pods: two viable models, pick based on what breaks least.
- IAM database authentication with IRSA: the pod's service account assumes an IAM role, and the driver exchanges it for a 15-minute token. No password exists anywhere. Constraints to design for: each new connection needs a fresh token (fine with pooling, awkward without), the instance limits new IAM-authenticated connections per second, and IAM auth does not replace database roles - you still map IAM identities to database users and grants. Pair with RDS Proxy so token churn and failover are handled in one place.
- Secrets Manager with automatic rotation (Aurora rotation Lambda, 30-day schedule) delivered via the External Secrets Operator or the Secrets Manager CSI driver. Works with every driver and every tool, and rotation is a solved problem. Slightly more moving parts than IAM auth, and a pod restart is usually needed to pick up a rotated value unless the app re-reads the secret.
Recommendation: IAM auth if the pods already have IRSA and a modern driver; Secrets Manager otherwise. Either way, the Kubernetes secret created 18 months ago is a finding - treat it as compromised and rotate the credential as part of the change, not after.
Developers: stop sharing the application user. The application user should be usable only by the application. Give humans their own identities: IAM Identity Center federated roles, connecting through SSM Session Manager port forwarding or a bastion, with per-person database users or a shared read-only role plus pgaudit logging of the individual login. Prod write access for debugging requires a break-glass role with an approval step and an alert on use.
Migration order: create the new roles and grants → deploy the pod change to a canary → rotate the old password and confirm nothing breaks → revoke CONNECT on the application user from every human identity → verify with connection logs.
3. What evidence would you assemble to demonstrate compliance, and how does key and secret rotation interact with it?
Read the worked answer
Access to data
- CloudTrail organization trail with log file validation, covering
RestoreDBClusterFromSnapshot,CopyDBClusterSnapshot,ModifyDBClusterSnapshotAttribute,CreateGrantand the KMSDecryptcalls (KMS cryptographic operations appear in CloudTrail, which is how you show use of the key, not just changes to the key policy). pgauditenabled via a cluster parameter group, withlog_connectionsandlog_statementscoped sensibly, exported to CloudWatch Logs with a retention policy. This is the record of who connected and what they ran.- VPC Flow Logs for the database subnets, and IAM Access Analyzer findings for any resource shared outside the organization.
Configuration state
- AWS Config rules:
rds-storage-encrypted,rds-in-backup-plan,encrypted-volumes, plus a custom rule for snapshot sharing scope. Config gives point-in-time compliance history, which is what auditors actually ask for.
Rotation
- KMS automatic annual rotation enabled on each CMK; the rotation history is visible in the console/API. Explain the nuance: rotation creates new backing key material while retaining old material so existing ciphertext still decrypts - that is expected, not a gap.
- Secrets Manager rotation with the documented Lambda and the
AWSPENDING/AWSCURRENTversion stages, plus CloudWatch Logs of each rotation invocation. Evidence is the rotation schedule, the version history and successful invocations - not a screenshot of a secret value, which you should never produce.
Process evidence
- The default encryption policy in the snapshot-sharing pipeline, the SCP text on dev, and the alarm/EventBridge rules with their notification targets, plus records of the alarm firing in tests.
Present it as a control-to-evidence matrix, one row per compliance requirement, with the automated source for each and the owner. Anything that requires a human to take a screenshot six months from now is a control that will fail the next audit.