GCP Network Engineer: Cloud Armor Under DDoS and a BigQuery Exfiltration Path

Cloud Armor Under DDoS and a BigQuery Exfiltration Path

Fintrail runs a PCI-scoped payment platform behind a global external Application Load Balancer fronting managed instance groups in us-central1. Cloud Armor applies one preconfigured WAF rule at default sensitivity, plus a global allowlist of 200 partner /32 addresses placed at priority 2147483647. During a week-long DDoS attack from roughly 12,000 addresses in three countries, the load balancer stayed up but legitimate users in those same countries saw 9-second page loads, and one newly onboarded partner received 403 responses. Separately, a red-team exercise showed that a compromised VM in a non-production project could read a BigQuery dataset of tokenized card data and copy it to a bucket in a project the team does not own, even though the subnet has no custom internet route. Auditors want preventive controls and evidence that they work before the next audit in six weeks.

1. Why is the partner getting 403s while legitimate users in the attacked countries slow down, and how would you restructure the Cloud Armor policy?

Read the worked answer

Cloud Armor is first-match by priority, and the partner allowlist is the last rule that can possibly match.

Priority 2147483647 is the lowest-precedence value. Every WAF, rate-limit, and deny rule above it is evaluated first, so if a preconfigured WAF signature or a rate-based ban matches the partner's traffic, evaluation stops with a deny before the allowlist is ever reached. Two separate faults follow from the same design:

Partner 403s: move the allowlist to a low priority number (for example 1000) so those source addresses short-circuit the rest of the evaluation. Also verify the partner's address is genuinely the one arriving: enterprises often egress through NAT or a proxy, and an allowlist entry can be IPv4-only when the partner also uses IPv6. Confirm with Cloud Armor logs (enforcedSecurityPolicy.priority and outcome) which rule actually matched - never guess from the client's claim.
Nine-second page loads for legitimate users in the attacked countries: the likely cause is the rate-based ban, or the default-sensitivity WAF rule. A rate-limit rule keyed on client IP with a long ban_duration will punish everyone behind a carrier-grade NAT or corporate proxy that shares an address with bots. Default sensitivity on SQLi/XSS signatures also produces false positives on ordinary form traffic.

Restructured policy:

priority 1000: allow the partner /32s (and their IPv6 equivalents).
priority 2000: allow a narrow set of verified internal or monitoring sources if needed.
priority 3000: rate-based ban, --rate-limit-threshold-count sized from measured peak, --ban-duration=600, --enforce-on-key=HTTP-HEADER (a session cookie or a partner header) rather than IP wherever possible, with --exceed-action=deny-429 instead of a hard deny.
priority 4000+: preconfigured WAF rules, deployed first with --preview at the tuned sensitivity, excluding signatures that fire on legitimate traffic, then enforced.
Enable Adaptive Protection for L7 DDoS: it builds per-service traffic baselines and proposes signatures. Run it in preview, review, then enforce.

Do not geo-block. The attack comes from the same three countries as paying customers. Also confirm scope: Cloud Armor only protects proxied load balancers, so any path that reaches a VM or a passthrough Network Load Balancer directly is unprotected.

2. The compromised VM reached BigQuery from a subnet with no internet route. Explain the path, the preventive control, and how you produce audit evidence within six weeks.

Read the worked answer

The egress did not need the internet, which is why the "no route" design gave no protection.

Google APIs are reachable from any VM over Google's network through Private Google Access to the API VIPs (199.36.153.8/30 for private, 199.36.153.4/30 for restricted), and failing that, through Cloud NAT. Firewall rules and route tables govern IP reachability; they say nothing about which Google Cloud APIs a workload's identity may call. Copying a BigQuery dataset to another project's bucket is an API-to-API operation performed by a service account with roles/bigquery.dataViewer, not an outbound TCP connection you can firewall.

Preventive control: VPC Service Controls.

Create an access policy and a regular perimeter whose restricted_services include bigquery.googleapis.com and storage.googleapis.com, containing the cardholder-data project, the projects whose workloads legitimately need it, and any bucket destinations inside the trust boundary.
With the perimeter in place, the red-team write to an out-of-perimeter bucket fails with a violation (RESOURCES_NOT_IN_SAME_PERIMETER / egress violation) regardless of IAM permissions.
Add access levels for the identities and networks that legitimately cross the boundary (specific service accounts, a corporate CIDR), and use ingress/egress rules for narrow exceptions rather than bridges, which defeat the purpose.
Reinforce with IAM and org policy: remove project-level roles/bigquery.dataViewer in favour of dataset-level access, use authorized views, disable service-account key creation (constraints/iam.disableServiceAccountKeyCreation), and prefer Workload Identity Federation.

Evidence inside six weeks:

Deploy the perimeter in dry-run mode first, collect violation logs for roughly a week, resolve false positives, then enforce.
Retention: export VPC Service Controls audit logs to Cloud Logging sinks plus BigQuery, and store the Access Context Manager perimeter definition as configuration in source control so an auditor can see exactly what is enforced.
Re-run the exact red-team procedure after enforcement and record the blocked API error, the matching audit log entry, and the change ticket. Auditors accept a demonstrated before/after with a log entry far more readily than an architecture diagram, and dry-run evidence alone will not satisfy a "prevention, not detection" requirement.