Private S3 Access Across 61 VPCs and a 90-Day Contractor Role
Northwind Payments runs 38 accounts in AWS Organizations (all features enabled) across us-east-1 and eu-west-1, with 61 VPCs and a 1 Gbps Direct Connect from two colo sites. Security has three open findings: (1) S3 traffic from on-premises and from VPC workloads must never cross the public internet; (2) a contractor team needs read access to 12 buckets in 4 accounts for 90 days, currently handled by per-account IAM users with access keys averaging 400 days old; (3) there is no central record of S3 object-level access. The network team has budgeted for VPC endpoints but notes that interface endpoints cost roughly $7.30 per AZ per month and every VPC runs two AZs. Applications use the standard regional S3 hostnames and cannot be changed to custom endpoints.
1. Design the private S3 path for both the 61 VPCs and the on-premises data centers, and justify the endpoint types and monthly cost.
Read the worked answer
Split the problem: VPC-origin traffic and on-premises-origin traffic are solved by different tools.
VPC workloads -> gateway endpoint (free). A gateway endpoint is a prefix-list route target in a route table. No ENI, no hourly charge, no per-GB charge, and it keeps the standard s3.<region>.amazonaws.com hostname working, which satisfies the 'cannot change endpoints' constraint. Attach one to every route table that currently sends 0.0.0.0/0 to a NAT gateway or IGW: 61 VPCs x 2 AZs x $0 = $0/month. Replace the same footprint with interface endpoints and you pay 61 x 2 x $7.30 = $890/month before data processing.
On-premises -> interface endpoint (PrivateLink), but only one. A gateway endpoint is reachable only from the VPC whose route table references it, so it is not a general solution for traffic arriving over DX via a Transit Gateway. Instead, place a single S3 interface endpoint in the shared-services VPC, reachable from on-premises over the DX virtual private gateway and from every spoke VPC over TGW. Cost: 2 AZs x $7.30 = $14.60/month plus ~$0.01/GB processed.
The part that usually breaks: DNS. Interface endpoints are useless to on-premises clients unless their resolvers return the endpoint ENI addresses. Deploy a Route 53 Resolver inbound endpoint in shared services and forward the s3.<region>.amazonaws.com zone from the on-premises DNS servers.
Rejected alternative: an S3 public virtual interface over DX is cheaper (no endpoint charges) and keeps traffic off the internet, but it cannot be constrained by an endpoint policy, so it gives you connectivity without the defence-in-depth the finding asks for. Use the interface endpoint and add an endpoint policy that allows only the 12 relevant bucket ARNs.
2. Replace the long-lived access keys for the contractor team. What is the access design, and what stops the same problem recurring?
Read the worked answer
Stop issuing IAM users entirely. Federate the contractor identity provider into IAM Identity Center, put the contractors in a dedicated group, and assign a ContractorReadOnly permission set to only the 4 target accounts. The contractors then assume short-lived roles per account; no access key ever exists on disk.
Scope the data, not just the action. A permission set granting s3:GetObject on * is too wide. Attach a bucket policy in each of the 4 accounts that names the exact role ARNs Identity Center issues and requires aws:PrincipalOrgID plus an explicit aws:PrincipalArn allow-list. Do not rely on aws:PrincipalOrgID alone - every principal in the organization satisfies it, including workloads that should not read these buckets.
Bound the blast radius in time. Cap the permission set session duration (8 hours is ample for a working day; the maximum is 12) and require MFA through the IdP. Set a calendar-driven deprovisioning task for day 90 - the group removal is the revocation, so there is nothing to hunt down across 4 accounts.
Prevent recurrence with an SCP at the organization root: deny iam:CreateUser and iam:CreateAccessKey in all accounts except a named break-glass account, and deny s3:PutBucketPolicy where s3:PublicAccessBlock configuration permits public access. SCPs apply to every principal below the root, including future accounts, which a runbook does not.
Verify, don't assume: run IAM Access Analyzer after the change; the finding should clear within 24 hours. Until it does, the old keys are still live - disable them, do not delete on day one, so you can prove nothing depended on them.
3. You must produce a central audit trail of S3 object-level access without an unbounded CloudTrail bill. Design it.
Read the worked answer
Organization trail, not per-account trails. Create one trail in the management account covering all 38 accounts, delivering to a dedicated log-archive account. A per-account design gives you 38 delivery configurations to keep in sync and 38 opportunities for one to be silently disabled.
Cost control lives in the event selector. S3 data events are the expensive part (roughly $0.10 per 100,000 events), and logging all buckets would be dominated by the high-read buckets. Use advanced event selectors scoped to the 12 bucket ARNs in the finding, and add a separate selector for DeleteObject/PutBucketPolicy across all buckets, since destructive and configuration events are low-volume and high-value. Management events stay on for everything.
Make the log store tamper-evident. Enable S3 Object Lock (compliance mode) on the log-archive bucket with a retention that exceeds your longest investigation window, enable versioning, and grant the log-archive bucket policy write-only access to the trail's service principal so no human role can delete evidence. Restrict read access to the security account only.
Answer the 'who read what' question directly by also turning on S3 Server Access Logs or CloudTrail Lake for the 8% of buckets that carry regulated data - CloudTrail data events alone do not record the HTTP status or bytes returned, which is usually what an auditor asks for next.
Close the loop: pair the trail with IAM Access Analyzer for external-access findings on the same 12 buckets, so the control detects misconfiguration rather than only recording access after the fact.