A 47-Minute Hole in an Organization Trail
Your organization has 40 accounts in a single OU. An organization trail delivers management events to a centralized bucket in the log-archive account (222233334444), encrypted with a customer managed KMS key, with log file validation enabled. A monthly review finds a 47-minute gap for account 555566667777: that account's event history shows a StopLogging call followed 47 minutes later by StartLogging, and no S3 objects exist for that window. The account owner says a contractor holding AdministratorAccess was 'testing something'. CloudTrail is not integrated with CloudWatch Logs. Organization-wide S3 data events are estimated at 40 TB/month and were rejected on cost, but three buckets - payroll, contracts, clinical-trials - hold regulated data. Leadership asks two questions: how do we know within five minutes next time, and how do we stop it happening at all?
1. Design detection for StopLogging, DeleteTrail and UpdateTrail that fires in under five minutes, and explain why EventBridge rules alone are not sufficient as the primary control.
Read the worked answer
Detection layer: create an EventBridge rule on the default bus in every account (StackSets or Control Tower), matching AWS API Call via CloudTrail for cloudtrail:StopLogging, cloudtrail:DeleteTrail, and cloudtrail:UpdateTrail. Route matches to a central event bus in the security account, whose resource policy permits the organization, and alarm from there. Deploying per account makes delivery location irrelevant, and the organizational bus gives you one place to alert and one place to audit.
Also ingest the organization trail into CloudWatch Logs in the log-archive account for metric filters - but note that this is a logging sink, not a tamper-proof detector.
Why detection alone fails: the rule lives in the same account whose administrator you are trying to constrain. A contractor with AdministratorAccess can disable the EventBridge rule, delete the CloudWatch log group, or edit the metric filter before touching CloudTrail. Detection that the adversary can silence is a monitoring convenience, not a control.
The primary control is preventive: an SCP on the OU denying cloudtrail:StopLogging, cloudtrail:DeleteTrail, cloudtrail:UpdateTrail, and cloudtrail:PutEventSelectors to every principal, with a aws:PrincipalArn exception for one break-glass role in the management account. SCPs cannot be overridden by any IAM principal inside the member account - that is exactly the property EventBridge lacks.
2. The 47 minutes of log objects are gone. What evidence can still be recovered, and what configuration changes would make the bucket itself tamper-evident going forward?
Read the worked answer
Recovery: CloudTrail event history retains 90 days of management events for an account independently of whether a trail is logging and independently of the trail's S3 destination. Pull the window directly from account 555566667777's event history (console or aws cloudtrail lookup-events) - you will recover the API calls the contractor made during the gap, which is precisely the window you care about. Do this first, before anything else, and export it to the log-archive bucket.
Making the bucket tamper-evident:
- Log file validation is already on - keep it, and verify digest files for the archived range so you can prove no retained object was altered.
- S3 Object Lock in compliance mode on the log bucket, with a retention period matched to your regulatory minimum (e.g. 7 years). Compliance mode cannot be shortened or removed by any principal, including the root user - that is the property versioning and MFA delete do not give you.
- KMS key policy that does not delegate to member accounts. The key policy should grant decrypt/encrypt only to the CloudTrail service principal scoped by
kms:ViaServiceandaws:SourceArn, plus the log-archive role. Cross-account delegation to the organization root re-opens the door you are closing. - Bucket policy denying
s3:DeleteObject,s3:PutBucketPolicy,s3:PutLifecycleConfiguration, ands3:DeleteBucketto every principal except the log-archive break-glass role, and denying any request whereaws:PrincipalOrgIDmatches but the account is not the log-archive account.
The honest framing for leadership: you cannot recover the 47 minutes of S3 objects, but you did not lose the events.
3. How would you satisfy the regulated-data requirement without paying for 40 TB/month of organization-wide data events?
Read the worked answer
Scope data events to where regulated data actually lives, and layer cheaper evidence underneath.
- A second trail with advanced event selectors, limited to
AWS::S3::Objectevents for the three buckets only, with prefix filters where only a subtree is sensitive (e.g.arn:aws:s3:::payroll/prod/). This is typically a rounding error next to organization-wide data events because volume scales with object count, not with bucket count. - Tag-driven selection - put
data-classification=regulatedon the three buckets and maintain the selector as they change. Write a Config rule that flags any new bucket taggedregulatedthat is not covered by the selector; otherwise the coverage silently rots, which is the actual failure mode of every selective-logging design. - S3 server access logs into the log-archive bucket for the remaining buckets as a low-cost, lower-fidelity backstop - they give you requestor, key, and status without CloudTrail's per-event price.
- Amazon Macie on the three buckets for discovery and for evidence that classification is accurate, rather than guessing which buckets hold PII.
State the trade-off explicitly to leadership: buckets outside the selector lose object-level CloudTrail events, so a future data-classification mistake becomes a blind spot. The Config guardrail in step 2 is what makes that trade-off defensible - it converts an unmonitored assumption into an alert.