Cross-Account Read Fails After Replication - and the KMS Bill Is Climbing
A SaaS company stores customer documents in bucket corp-docs-prod (us-east-1), encrypted with SSE-KMS using key alias/corp-docs. Objects replicate to corp-docs-prod-eu in eu-west-1 using Cross-Region Replication with a separate replica key, alias/corp-docs-eu (single-Region). An analytics team in account 333344445555 needs read-only access to the EU bucket. You granted s3:GetObject and s3:ListBucket on the bucket to their role and added a matching bucket policy. They report: ListBucket works; HeadObject on the sample object returns 200 with the expected size; GetObject returns 403 with KmsAccessDeniedException in the response body. Finance separately flags a monthly KMS bill above $9,000, dominated by GetObject calls on the EU bucket. Compliance insists EU customer data must never leave eu-west-1. Bucket keys are disabled on both buckets.
1. Explain why ListBucket and HeadObject succeed while GetObject fails, and state precisely which policy must change and in which account.
Read the worked answer
S3 satisfies ListBucket and HeadObject from its own metadata - object key, size, storage class, encryption algorithm. Decrypting plaintext requires a KMS call, and HeadObject never makes one. That is why a 200 here is not evidence that access is correct; it only proves the S3-layer permissions work.
GetObject needs kms:Decrypt on corp-docs-eu. For cross-account use, two independent grants must both allow it:
- The key policy in the key's account must delegate to account 333344445555 - e.g. a statement with Principal
arn:aws:iam::333344445555:root(or, better, the exact analytics role ARN) allowingkms:Decryptandkms:DescribeKey, conditioned onkms:ViaService=s3.eu-west-1.amazonaws.comandkms:CallerAccount=333344445555. - An IAM policy in account 333344445555 must grant the same actions on the key ARN to the role.
The S3 bucket policy cannot substitute for either. A frequent and expensive mistake is trying to fix this by widening the bucket policy - the request fails at the key policy before the object is ever decrypted. Note also that KMS returns the error inside the S3 response body (KmsAccessDeniedException), so the HTTP 403 alone will mislead you toward S3 permissions; check the body.
Judgement on mechanism: edit the key policy for durable, auditable cross-account access. Use a grant if the access is temporary or per-object, accepting that grants are easy to lose track of and must be retired deliberately. For a standing team, the key policy with kms:ViaService is the safer choice.
2. How do you cut the KMS cost without weakening the requirement that EU data stays in eu-west-1? Identify the specific trade-off bucket keys introduce.
Read the worked answer
Cost fix: enable S3 Bucket Keys on the EU bucket. With bucket keys, S3 obtains a short-lived bucket-level key from KMS and derives per-object data keys locally, cutting KMS requests by up to 99%. At your call volume that turns a four-figure bill into a small one, with no application change and no re-encryption of existing objects.
The trade-off, stated honestly: with bucket keys, KMS no longer sees an API call per object read. Your CloudTrail KMS log stops being a per-object access record, and kms:ViaService/kms:CallerAccount conditions on the key policy are evaluated when the bucket key is obtained, not on every GetObject. If a compliance obligation requires per-object proof of who decrypted what, bucket keys weaken that evidence - and you should compensate with S3 server access logs or CloudTrail S3 data events on that bucket, which record per-object reads even though KMS does not.
Residency: bucket keys do not affect where anything lives. The bucket key is materialised by KMS in eu-west-1, from a single-Region eu-west-1 key. Do not "solve" the replication friction by switching to a multi-Region key replicated into us-east-1 - that would put the EU data key material in a US region and break the residency requirement outright. Also verify the replica bucket's default encryption references the EU key, not the us-east-1 key ARN; if the default still points at the us-east-1 key, direct PUTs into the EU bucket will fail replication or encryption, and someone will be tempted to 'fix' it by shortening the key policy.
Finally, confirm the replication configuration's ReplicaKmsKeyID is corp-docs-eu and not the source key - a misconfigured replica key is the second-most-common cause of these bills.
3. The analytics team's role lives in account 333344445555 and may be in any Region. Compliance says EU data must never leave eu-west-1. What do you add to enforce that technically, and what limitation should you disclose?
Read the worked answer
Enforce residency on the read path, not just the storage path. With SSE-KMS, the plaintext is decrypted in the bucket's Region and returned over TLS to the caller - so 'data at rest is in the EU' is true, but a caller in us-east-1 reading the bucket pulls EU plaintext into a US region. At-rest encryption does not give you residency of access.
Add to the EU bucket policy: a deny for any principal outside the permitted set when aws:RequestedRegion is not eu-west-1. Condition on aws:RequestedRegion and test it - validate with the IAM policy simulator and a real cross-Region call from a scratch role before promising it to compliance. Pair it with aws:SourceVpce if the analytics team reaches S3 through a VPC endpoint, which is a stronger, network-level guarantee than a request-level condition, and with a deny on non-TLS requests.
Limitations to disclose, in writing:
- A request-level condition constrains where the API call is made, not where the returned bytes are subsequently copied. Nothing prevents the analytics team from writing the result to a us-east-1 bucket in their own account - that is an account-level and contractual control, not an S3 one.
aws:RequestedRegiondepends on correctly signed requests; a principal determined to bypass it in their own account is constrained only by their own SCPs, so mirror the deny in the analytics account's SCP.
Complementary controls: SCP in account 333344445555 restricting S3 access to eu-west-1 endpoints; Macie on the analytics account's buckets to catch EU-tagged data landing outside the region; key policy condition kms:ViaService: s3.eu-west-1.amazonaws.com so the key cannot be used through any other service or Region.