OKE cost overrun: NAT egress, idle GPU, and missing logs
Lumen SaaS runs a customer portal on OKE in the Ashburn region with three node pools: 6 general-purpose nodes, 4 memory-optimized nodes, and 2 GPU nodes used only for weekly batch scoring. The cluster has 140 pods and pulls images from OCI Container Registry. Application pods read and write 30 TB per month to Object Storage through a NAT gateway. The platform team has no centralized pod logs, uses only node-level CPU alarms, and spends 38% above the monthly budget. Finance sees one untagged invoice line for all environments. Investigation finds 60 unattached block volumes, 12 old Object Storage buckets without lifecycle rules, and a GPU node pool running continuously even though batch jobs finish in 90 minutes. Security wants audit logs and VCN flow logs retained for 180 days, and the team wants per-team cost visibility without giving finance access to workloads.
1. Diagnose the main cost drivers and propose fixes in priority order, with the expected impact of each.
Read the worked answer
The largest avoidable driver is likely Object Storage traffic through the NAT gateway. At 30 TB per month, pods are paying NAT data-processing charges and taking an unnecessary network hop. Create a service gateway in the VCN, add a route rule for the Object Storage service CIDR, and update pod network policies so Object Storage traffic uses the service gateway. This is usually the fastest high-impact fix and also improves security by removing internet egress.
The second driver is the always-on GPU pool. Two GPU nodes running 24x7 for a 90-minute weekly job are mostly idle. Move batch scoring to a scheduled node pool or a separate OKE node pool that scales to zero when the job finishes. If the workload can tolerate interruption, use preemptible nodes. Third, reclaim 60 unattached block volumes after verifying they are not backups, and apply lifecycle rules to the 12 old buckets so obsolete objects move to lower-cost tiers or expire.
Then fix visibility: require defined tags for team, environment, and application; enable budgets and cost reports by tag. Finance should see cost dashboards, not cluster access. Quotas on GPU OCPUs and block volume count prevent recurrence. The order matters because NAT and GPU fixes reduce recurring spend immediately, while tagging and quotas prevent the next overrun.
2. Design centralized observability for OKE, including log retention and per-team cost visibility, without giving finance workload access.
Read the worked answer
Enable OKE logging and collect pod and node logs into OCI Logging, using a Fluent Bit or logging agent DaemonSet with namespace-based routing. Send Kubernetes audit logs, application logs, and VCN flow logs to OCI Logging. Use a Service Connector Hub to archive the required logs to Object Storage for 180 days, with an Object Storage lifecycle policy that expires them after the retention period. Restrict bucket delete access to the security team.
For metrics, use OCI Monitoring for node and pod resource usage, OKE control-plane metrics, and alarms on pending pods, node pressure, GPU utilization, and NAT gateway throughput. Add APM for request tracing where the portal calls downstream services. Create dashboards per namespace and per team. Use OCI Logging search with saved queries so an on-call engineer can find errors by namespace without scanning every pod.
For finance, do not grant OKE or compartment workload access. Instead, expose cost through OCI Budgets and cost-analysis reports grouped by defined tags, plus a read-only dashboard for finance. Tag namespaces and workloads by team and environment so chargeback is possible. If finance needs a daily extract, send aggregated cost reports to a separate bucket; do not give finance read repos or cluster RBAC roles.
3. What governance controls would prevent the same cost and observability gaps from recurring?
Read the worked answer
Start with tag defaults at the compartment level and require defined tags for team, environment, application, and cost center. Reject or flag untagged resources through Cloud Guard or a weekly report. Create budgets per team and per environment, with alerts at 50, 80, and 100 percent, and use quotas to cap GPU OCPUs and block volume counts. The GPU quota is the specific control that would have stopped the always-on batch pool.
For storage, enforce lifecycle policies on Object Storage buckets through Security Zones or infrastructure-as-code templates. Require block volumes to have a backup policy and an owner tag; run a scheduled cleanup report for unattached volumes. For OKE, make node-pool autoscaling and scale-to-zero the default for batch workloads, and require a namespace label for logging and cost allocation. Use OCI Logging and VCN flow logs as mandatory add-ons for every cluster, with Service Connector Hub configured by default.
Finally, make these controls part of the landing zone. A new team should get a compartment, budget, quota, tag defaults, logging route, and service gateway automatically. Governance that depends on a monthly cleanup effort will fail; governance built into the provisioning path is what keeps cost and observability under control.