Tenant Isolation in a Shared AI Assistant Platform

The root cause is a flat trust model: shared identity and unrestricted east-west network access across namespaces. The first action is to revoke the cluster-admin service account and replace it with per-namespace service accounts scoped to the minimum verbs each workload needs. Rotate all credentials and tokens that the shared account could have accessed. Next, apply a default-deny network policy in each tenant namespace and use Flow to allow only explicitly required traffic, such as ingress from the approved gateway to that tenant's inference endpoint. Cluster DNS resolution alone should not grant reachability.

For Objects, remove shared access keys and create per-tenant buckets or prefixes with bucket policies that bind each tenant's identity to its own data. Audit object access logs for the exposure window. For NAI endpoints, require authenticated calls and per-tenant authorization so a compromised pod cannot invoke another tenant's model. I would preserve logs and snapshots for forensics before rotating everything. The containment principle is to break the shared identity and the east-west path first, then rebuild access from least privilege. Adding more tenants before this is fixed would multiply the exposure.

I would define a tenant onboarding standard with four enforced layers. On compute and identity, each tenant gets an NKP namespace bound to a Prism Central project, a dedicated service account, RBAC roles limited to that namespace, and ResourceQuotas for CPU, memory, and GPU. On network, each namespace gets a default-deny NetworkPolicy plus Flow policies that allow only gateway-to-endpoint and required telemetry paths. On data, each tenant gets its own Objects bucket or prefix with per-tenant access keys or IAM binding, SSE-KMS encryption, and for tenant C a customer-managed key with documented key custody. On model access, each tenant gets a separate NAI endpoint or workspace with authentication and authorization tied to the tenant identity.

I would add admission control to block pods that request host networking, privileged mode, or cluster-admin bindings, and a private registry with signed images. Tenant C's prompts and embeddings should be encrypted at rest with its key, and key revocation should be tested. Shared platform services should be reachable only through explicit policies. The standard should be codified as manifests and policy checks, not a runbook, so onboarding cannot skip a layer. Evidence for the new customer would include network policy tests, RBAC review, bucket policy tests, encryption configuration, and audit log samples.

I would require a model registry with immutable version records, including source, scan results, approval, and deployment history. Only approved artifacts can be promoted, and promotion requires a role separate from the data-scientist role. Deployments should reference signed images and versioned model artifacts from Objects. Each tenant's endpoint configuration should record which model version it uses, so access can be audited per tenant and per model.

Audit logging should capture authentication, authorization decisions, endpoint invocation metadata, model version, and administrative changes, and forward to a SIEM with tenant-specific retention. I would avoid logging raw prompts by default unless the tenant contract requires it, because prompts may contain protected health information. Instead, log request IDs and hashed tenant identifiers so access can be reconstructed without exposing content. For tenant C, key usage and key-management events must be auditable. Finally, I would run periodic isolation tests and access reviews, and make evidence available to tenants. Governance is not a one-time policy; it needs drift detection, because a single shared service account or broad Flow rule can undo the entire isolation model.