Production sizing and HA for a 45 TB analytics platform
A regional insurer is deploying IBM Cloud Pak for Data 4.8 on Red Hat OpenShift 4.12 in its own data center. It has 3 control-plane nodes (16 vCPU/64 GB) and 6 worker nodes (32 vCPU/256 GB) with 12 TB of NFS-backed persistent storage. Planned services are Watson Studio, Watson Machine Learning, Db2 Warehouse, Data Virtualization, and Analytics Engine powered by Apache Spark. There are 140 business analysts, 35 data scientists, nightly ETL loading 6 TB, 45 TB of total curated data, and peak concurrency of 90 users. RPO is 15 minutes and RTO is 4 hours. The existing NFS array is shared with other applications and delivers 800 IOPS at 4 KB. Security requires encryption at rest and in transit plus namespace-level isolation between dev, test, and prod. The CIO asks whether the current cluster can host production and what must change before go-live.
1. Assess whether the current cluster and storage can support production. Identify the first three changes you would make and justify each.
Read the worked answer
The cluster cannot go live as designed. The clearest gap is storage: 45 TB of curated data plus Db2 Warehouse temporary space, Spark scratch, and backups cannot fit in 12 TB. Even with compression, the NFS array is shared and delivers only 800 IOPS at 4 KB, which is roughly 3.2 MB/s of random throughput; Db2 Warehouse and Spark shuffle will contend badly. Six 32-vCPU workers are also marginal for 90 concurrent users plus ETL. First, add dedicated storage classes: block-backed RWO storage for Db2 Warehouse, a separate RWX file class for shared services, and object storage for backups and large files. Size raw capacity for replication and growth, not just usable data. Second, isolate dev, test, and prod into separate OpenShift projects and dedicated worker pools with taints and tolerations so a dev Spark job cannot starve production. Third, replace or QoS-limit the shared NFS for CP4D; validate IOPS and latency with a benchmark against Db2 and Spark before migration. Encryption should be enforced through storage-class encryption and TLS, but encryption does not fix capacity or IOPS.
2. Design HA and DR to meet RPO 15 minutes and RTO 4 hours. Include what must be backed up and why OpenShift node HA alone is insufficient.
Read the worked answer
OpenShift node HA restarts pods after a worker failure, but it does not protect against storage-array loss, namespace deletion, corrupted metadata, or site loss. The design needs two layers. For HA, spread control-plane and worker nodes across racks, use pod anti-affinity for Db2 and WML, configure multiple Db2 Warehouse pods or replicas where supported, and keep etcd on low-latency disks. For DR, replicate backups off-site and pre-build a recovery cluster. Back up etcd, the CP4D control plane with the supported backup utility, Db2 Warehouse to object storage, PVC snapshots where the driver supports them, Watson Machine Learning model artifacts, and catalog/project metadata. RPO 15 minutes cannot be met by nightly Db2 backups alone; use Db2 transaction-log shipping or storage replication to a secondary site and monitor lag. RTO 4 hours requires tested runbooks, pre-created projects and secrets, and a restore rehearsal. Document which services are rebuildable from source control versus which require data restore.
3. Security wants namespace isolation and encryption. Which OpenShift and CP4D controls do you use, and what is a common design mistake?
Read the worked answer
Use one OpenShift project per environment and per CP4D project, RBAC mapped to LDAP groups, NetworkPolicies between namespaces, separate service accounts, and resource quotas. For encryption, enable storage-class encryption, enforce TLS for routes and internal service traffic, and manage certificates centrally. In CP4D, use platform roles and project roles rather than giving users cluster-admin. The common mistake is treating a CP4D project as a hard security boundary. Projects isolate collaboration assets, but many CP4D services are installed at the instance or cluster scope, and a platform administrator can still reach data across projects. If two departments require strong administrative separation, use separate CP4D instances or clusters, not just separate projects. Also, namespace isolation without NetworkPolicy and quota enforcement is only naming isolation, not security isolation.