Undefended GPU nodes, a shared base image, and a serverless team saying no

Separate the two failure classes first, because they have different fixes.

Check A - is there a Defender pod on those nodes at all? kubectl get pods -n twistlock -o wide and filter by node. If no pod is scheduled on the GPU nodes, the DaemonSet is not tolerating their taints. A DaemonSet only places pods on tainted nodes if it declares the matching toleration (nvidia.com/gpu=present:NoSchedule), and the Console-generated Helm values frequently omit it. Confirm the taint value and effect exactly - a toleration for NoExecute does not satisfy NoSchedule. Also check node selectors/affinity in the values file and whether an autoscaled managed node group uses a custom AMI or runtime the Defender build does not support.

Check B - pod present but not reporting. If pods exist on those nodes, look at their status and logs. Typical causes: CPU/memory requests exceeding what the node has allocatable for pods (GPU nodes often reserve nearly everything for the accelerator workload), and no network path to the Console - private-subnet nodes without NAT or an HTTP proxy cannot reach the Console's Defender communication port. Proxy settings must be in the Defender deployment, not only in the node's environment.

Remediation without draining. Patch the DaemonSet tolerations (and, if needed, lower resource requests) and let the DaemonSet controller roll new pods onto the tainted nodes - this does not evict training pods, it only adds the Defender pod. For connectivity, add proxy configuration to the deployment and verify with a Defender log check. Confirm success by watching the Console's defended-node count rise from 208 and by checking each new pod reports in, then set a monitor that alerts when defended nodes < total nodes for more than 30 minutes, so this never drifts silently again.

Defence in depth, with different jobs for each layer.

CI layer (primary gate). Scan the image in the pipeline as soon as it is built, before it is pushed to the registry, using the Prisma Cloud Compute scanner invoked from Jenkins/GitHub Actions. Fail the build on a defined vulnerability threshold (critical with a fix available, or an explicit severity/score gate). This is the right primary control because it fails fast, is attributable to a commit, and cannot block the whole cluster. Caveat: the base image is an upstream artefact you do not build - so add a scheduled rebuild/rebase job for the shared base image and a registry scan, otherwise the same vulnerable layer keeps reappearing under a new tag.

Admission layer (backstop). Prisma Cloud Compute admission control is a validating webhook that evaluates rules at pod create/update. Configure rules for "critical vulnerabilities with a fix" and "disallow latest tag", plus whatever the platform standard requires. It exists to catch images that bypassed the pipeline - vendor charts, manual kubectl apply, hotfixes.

Rollout sequence. (1) Deploy the admission rules in monitor/audit mode for two weeks and export violations. (2) Fix or rebase the workloads that would have been blocked. (3) Enforce namespace by namespace, starting with non-production. (4) Allowlist only with a ticket, an owner, and an expiry.

Operational risks to state out loud:

• Failure policy. Fail-closed protects you but a Console outage or network partition then blocks every deployment in the cluster. Start fail-open with alerting on webhook errors, and move to fail-closed only when the Console is highly available and the team accepts the blast radius.
• Scope creep. A rule that blocks "any high CVE" will stop the business within a day; be narrow and explicit.
• Retroactivity. Admission acts on create/update only. Existing running pods with the bad image keep running until redeployed, so pair it with a live image-vulnerability policy on running workloads.
• Break-glass. Agree in advance how an incident commander deploys an unapproved image at 3am, and make sure that path is logged.

Frame the tradeoff as coverage versus runtime depth, and stop treating it as one decision for all 300 functions.

Option 1 - scan without a runtime agent. Prisma Cloud can inventory serverless functions through the cloud account's read role and analyse their code, dependencies, and configuration for vulnerabilities and misconfigurations. No package change, no cold-start impact, no team friction. Limitation: it is point-in-time and pre-execution. It will not tell you that a function just made an outbound connection to an unknown host.

Option 2 - Serverless Defender embedded in the function (layer/package). Gives runtime protection and behaviour visibility inside the function: process and network events, anomalous invocation patterns. Costs: added package size and initialisation time (which is exactly the cold-start concern), per-language support limits, and a redeploy cycle for every function. On a latency-sensitive function the objection is legitimate, not obstructionist.

Option 3 - platform-level runtime signal. Use cloud-native audit and network telemetry (CloudTrail, VPC flow logs, GuardDuty findings) for detection on the functions that have no agent, and correlate in Prisma Cloud. Cheaper, shallower, but real.

Recommendation. Turn on Option 1 for all 300 immediately - it needs no cooperation from the serverless team and closes the vulnerability and misconfiguration gap this quarter. Layer Option 3 on top for behavioural coverage. Then negotiate Option 2 for the small set of functions that handle payment data and have the highest impact if compromised - perhaps 10 to 15 functions - where the cold-start cost is defensible and can be measured with a load test rather than argued about.

Make it a risk conversation, not a policy fight. Write down the residual risk for the functions left on scan-only, get the serverless owner to accept it, and revisit when the latency budget changes. Blanket mandates get overridden informally; explicit, scoped acceptances get honoured.