IBM Cloud Pak for Data v4.7 Architect cheat sheet

Cloud Pak for Data runs on Red Hat OpenShift; every service is a containerized operator-managed workload, so sizing is driven by OpenShift worker capacity, not by per-service VMs.

Control plane components: zen (home page, IAM integration, licensing), the Cloud Pak for Data operator, and the common core services (ccs) that provide metadata, search, and RabbitMQ/MongoDB backing stores.

Analytics Engine powered by Apache Spark provides Spark jobs and notebooks; Watson Studio hosts projects and assets; Watson Knowledge Catalog provides the governance catalog; Watson Query provides virtualization; DataStage provides ETL/ELT.

Separate 'instances' (service instances) from 'services': one installed service can back multiple instances, and each instance consumes its own PVCs and resource quota.

Reference topology decision rule: small (proof of concept) uses a single worker or three small workers; production requires at least three master and three worker nodes with anti-affinity across zones.

Storage decision rule: use ReadWriteMany (RWX) file storage (NFS/OCS/Portworx) for shared services; block storage is only acceptable for single-pod ReadWriteOnce workloads.

Minimum production baseline commonly cited: 3 masters, 3+ workers, each worker 16 vCPU / 64 GB RAM, plus dedicated storage nodes; verify against the official sizing tool for the exact services installed.

GPU nodes are required only for Watson NLP/Watson Speech/foundation-model inferencing; schedule them with node selectors and taints so non-GPU workloads do not consume them.

Cloud Pak for Data as a Service (SaaS) versus software: the same control plane API and catalog concepts, but SaaS removes installation and upgrade responsibilities and constrains customization.

Installation is driven by the cpd-cli (cpd-cli manage) tool: generate the install-config, then run commands such as 'cpd-cli manage setup-instance', 'install-components', and 'create-cartridge' in a fixed order.

The global pull secret must be updated with the IBM Entitled Registry key (cp.icr.io) before any component install; a missing or expired entitlement key produces ImagePullBackOff on component pods.

The Cloud Pak for Data operator must be installed in the cpd-operators namespace; service instances are created in the cpd-instance namespace by default, and these are separate projects.

Installation order: OpenShift cluster -> prerequisites (storage classes, catalog sources, node tuning, certificate manager) -> Cloud Pak for Data platform (zen) -> services -> service instances.

cpd-cli manage requires environment variables (CPD_URL, CPD_USERNAME, CPD_PASSWORD or API key, CPD_APIKEY) and a portable client container image; run it from a workstation with oc/kubectl access.

Upgrade path rule: upgrade the Cloud Pak for Data platform first, then the services; the operator supports sequential minor upgrades and you must not skip a required intermediate release.

Before upgrade, back up MongoDB, Zen metadata, and the catalog database; the cpd-cli 'backup' and 'restore' commands use an object store (S3-compatible) for the backup payload.

Verify health with 'cpd-cli manage get-cr-status' and 'oc get pods -n cpd-instance'; a component is not ready while its operand CR reports a non-Completed status.

Offline/air-gapped installs use a private container registry mirror; all images must be copied with 'cpd-cli manage mirror-images' before installing.

Identity: Cloud Pak for Data integrates with an enterprise LDAP (or SAML/OpenID Connect) identity provider; the platform does not maintain its own user directory for production.

Authorization model: platform roles (Administrator, User, Viewer) plus per-service roles plus per-resource roles in the catalog; a user needs the intersection of all three to act.

Cloud Pak for Data uses Zen/Identity and Access Management (IAM) to map IdP groups to roles; group membership changes propagate on token refresh, not instantly.

Watson Knowledge Catalog governance artifacts: business terms, data classes, classifications, policies, rules, and reference data; policies are enforced by rules attached to assets.

Data protection rules can mask, filter rows, or deny access; masking rules are applied at query time for governed assets through Watson Query and integrated services.

Encryption: data at rest via storage-class encryption and etcd encryption; in transit via TLS; the platform certificate can be replaced with an enterprise CA-signed certificate using the cpd-cli certificate commands.

Audit: platform and service audit logs are exported to a SIEM via the activity tracker / audit logging configuration; catalog activity is recorded per asset.

Secrets: credentials for data sources are stored in the platform credential store or an external vault; never embed credentials in notebook or job code.

Network: use OpenShift NetworkPolicy and an egress proxy for outbound calls; private connectivity (IBM Cloud Private Path / VPC) is preferred over public endpoints for data sources.

Watson Query (formerly Db2 Big SQL) virtualizes remote sources: it creates a catalog entry and nickname, and queries are pushed down when the source supports the operation, otherwise processed in the virtualizer.

Watson Query connection types include Db2, Oracle, SQL Server, Netezza, Teradata, PostgreSQL, MongoDB, S3, and Kafka; each requires a JDBC driver bundled with the service or supplied by the administrator.

DataStage in Cloud Pak for Data runs parallel jobs on Spark or on the classic parallel engine; jobs, stages, and parameter sets are the core reusable assets.

Data Virtualization versus DataStage decision rule: use virtualization for read-mostly federation and rapid access without copying; use DataStage when you need transformation, cleansing, or persisted target tables.

Knowledge Catalog lineage is captured automatically for DataStage, Watson Query, and notebook-based data flows; lineage is not captured for arbitrary external tools unless metadata is imported.

Metadata import: use discovery/import jobs to profile and enrich assets; data classes assign automatically from column names and values, and the results drive quality and governance rules.

Replication: Data Replication captures change data capture from sources such as Db2, Oracle, SQL Server, and Kafka into targets; it is the choice for near-real-time synchronization, not batch ETL.

Performance rule: pushdown reduces data movement, but cross-source joins in Watson Query require data to be staged locally, so size the virtualizer's temporary storage accordingly.

DataStage parallel job tuning levers: partitioning method, buffering, node configuration file, and 'combine/redistribute' stages to avoid skewed partitions.

Design method: capture non-functional requirements (volume, latency, concurrency, retention, RPO/RTO) first, then map each to a service capability; do not select services before requirements are quantified.

Architecture pattern: separate ingestion (DataStage/Replication), storage (lakehouse on object storage), governance (Knowledge Catalog), and consumption (Watson Studio, Watson Query, dashboards) into layers with defined interfaces.

High availability: three or more workers across zones, RWX storage replicated, and MongoDB/Zen backed up; a single-zone cluster is a single point of failure regardless of pod replicas.

Common failure: pod stuck in ImagePullBackOff means entitlement key or pull secret problem; CrashLoopBackOff usually means misconfigured environment variables, resource limits, or storage permissions.

Common failure: PVC pending means no storage class supports the requested access mode; check 'oc get pvc' events and the storage class's volumeBindingMode.

Common failure: 'instance not ready' after install is often a resource quota shortage; check 'oc describe quota' and the operand CR status message before reinstalling.

Diagnostics: 'oc logs', 'oc describe pod', 'oc get events', and the cpd-cli 'collect-logs' command gather the evidence IBM support requires; collect before restarting pods.

Scaling rule: scale workers horizontally before increasing pod resources; vertical scaling of individual services is limited by the service's documented maximums.

Cost/performance trade-off: virtualization avoids duplicate storage but adds query latency; persisted copies cost storage but deliver predictable performance for heavy repeated analytical workloads.

Cloud Pak for Data is an OpenShift-native platform: services are operators and custom resources, so architecture decisions are OpenShift cluster design decisions.

Zen is the control plane service that provides the home page, IAM integration, and the platform API; if Zen is unhealthy, the whole platform UI is unavailable even when data services run.

Common core services (ccs) supply shared MongoDB, RabbitMQ, and Elasticsearch-like search; they are shared dependencies, so their failure affects many services at once.

Watson Studio is the authoring environment (notebooks, projects, assets); Watson Machine Learning deploys and serves models; they are separate services with separate instances.

Watson Knowledge Catalog is the governance layer; Watson Query is the virtualization layer; DataStage is the integration layer; each can be installed independently but governance value depends on cataloged assets.

Sizing inputs: number of concurrent users, data volume scanned per query, number of parallel jobs, and model inference throughput; convert each into CPU, memory, and storage per service.

Node roles: masters run control plane, workers run workloads, and dedicated nodes (GPU, storage) are isolated with taints and tolerations.

Air-gapped deployments require a mirrored registry and a bastion host; plan registry storage capacity before mirroring images.

The cpd-cli manage tool wraps oc/kubectl operations; commands are idempotent and re-runnable, so failed steps are retried rather than reinstalled from scratch.

Prerequisites before platform install: a default storage class, the IBM catalog sources, cert-manager, node tuning profiles, and the entitlement key in the global pull secret.

The platform (zen) must be installed and healthy before any service; the operator reports readiness through the operand request CR status.

Namespace convention: cpd-operators for operators, cpd-instance for the platform and service instances; keeping them separate simplifies RBAC and backup.

Service instances are created through the web UI or the platform API; each instance has its own PVCs and resource requests, so plan quota per instance.

Upgrade sequence: back up, upgrade the platform operator and platform, then upgrade each service operator and instance; check the support matrix for compatible versions.

Air-gapped installs mirror images to a private registry and set the image registry prefix in the install options; the entitlement key is still required at mirror time.

Health checks: 'cpd-cli manage get-cr-status' for operand status, 'oc get pods' for runtime state, and the platform's own diagnostics page for service-level errors.

Rollback is not automatic; restoring a backup to the pre-upgrade version is the documented recovery path, so validate backups before upgrading.

Authentication is delegated to an external IdP (LDAP, SAML, OIDC); Cloud Pak for Data stores the mapping, not the passwords, for enterprise users.

Authorization is layered: platform role, service role, and resource (catalog/project) role; access requires the effective intersection of all applicable layers.

Platform roles: Administrator manages users and services; User can create projects and instances; Viewer has read-only access to the platform UI.

Watson Knowledge Catalog enforces governance through policies and rules; a data protection rule can deny, mask, or filter data at query time for governed assets.

Business terms and data classes enrich assets; classifications and reference data support quality rules and automated policy assignment.

Certificates: replace the platform's self-signed certificate with an enterprise CA certificate via cpd-cli; clients then trust a single CA chain.

Audit and monitoring: export audit records to a SIEM, and use the platform monitoring service to track service health and resource consumption.

Secrets management: store data source credentials in the platform credential store or an external vault; rotate keys and never hardcode them in notebooks or job parameters.

Network hardening: apply NetworkPolicies, restrict egress through a proxy, and prefer private endpoints for cloud data sources.

Watson Query federates sources through nicknames; pushdown happens when the source can execute the operation, and cross-source joins require local staging.

Supported Watson Query sources include Db2, Oracle, SQL Server, Netezza, Teradata, PostgreSQL, MongoDB, S3, and Kafka; JDBC drivers must be available to the service.

DataStage jobs run on the parallel engine or on Spark; partitioning and buffering determine throughput, and skewed partitions are the usual performance culprit.

Data Replication provides change data capture for near-real-time synchronization; use it instead of batch ETL when latency requirements are seconds to minutes.

Choose virtualization when data must stay in place or freshness matters; choose DataStage when transformation, cleansing, or a persisted target is required.

Lineage is captured automatically for DataStage, Watson Query, and notebooks; external tools need metadata import to appear in lineage.

Metadata import and discovery jobs profile assets, assign data classes, and feed quality analysis; results drive governance rules and catalog enrichment.

Tuning levers for Watson Query: increase virtualizer resources, enable pushdown-friendly predicates, and pre-stage large cross-source joins.

DataStage parameter sets externalize connection and path values so the same job runs across environments without edits.

Start from quantified non-functional requirements (data volume, latency, concurrency, retention, RPO/RTO); each requirement maps to a specific service capability or cluster sizing decision.

Layer the architecture: ingest, store, govern, consume; define the interface between layers so services can be swapped without redesign.

High availability needs multi-zone workers, replicated RWX storage, and tested backups of MongoDB and Zen metadata; pod replicas alone do not survive a zone outage.

ImagePullBackOff points to an entitlement key or pull-secret problem; CrashLoopBackOff points to configuration, resource limits, or storage permissions.

PVC stuck in Pending means no storage class satisfies the access mode; inspect events and the storage class's binding mode.

A service instance that never becomes ready is often blocked by resource quota; check quota and the operand CR status message before reinstalling.

Collect evidence with oc logs, oc describe, oc get events, and cpd-cli collect-logs before restarting pods, because restarts destroy the failing state.

Scale workers horizontally before raising per-pod limits; service-level maximums cap vertical scaling.

Virtualization lowers storage cost but adds query latency; persisted copies cost storage but give predictable performance for repeated heavy analytics.

Cloud Pak for Data is deployed on Red Hat OpenShift; services are operators managing custom resources, so platform architecture equals cluster architecture.

Zen provides the control plane (home page, IAM integration, platform API); common core services provide shared MongoDB, RabbitMQ, and search backing stores.

Watson Studio authors assets and notebooks; Watson Machine Learning deploys and serves models; Watson Knowledge Catalog governs; Watson Query virtualizes; DataStage integrates.

One installed service can host multiple instances; each instance consumes its own PVCs and resource quota, so capacity planning is per instance.

Production topology: at least three masters and three workers spread across zones, with anti-affinity so a zone loss does not take down a service.

Use RWX file storage for shared services; block storage is only suitable for single-pod ReadWriteOnce workloads.

GPU nodes are needed only for NLP, speech, and foundation-model inferencing; isolate them with taints and node selectors.

Sizing inputs are concurrent users, data scanned per query, parallel job count, and inference throughput; convert each to CPU, memory, and storage.

cpd-cli manage drives installation; typical order is prerequisites, platform (zen), services, then service instances, and commands are safely re-runnable.

The IBM entitlement key must be added to the global pull secret before installing; without it, component pods fail with ImagePullBackOff.

Operators live in cpd-operators and instances in cpd-instance; separating namespaces simplifies RBAC, backup, and troubleshooting.

Prerequisites include a default storage class, IBM catalog sources, cert-manager, and node tuning profiles.

The platform must report healthy before services are installed; check the operand request CR status with cpd-cli manage get-cr-status.

Upgrade the platform before services, never skip a required intermediate release, and confirm versions against the support matrix.

Back up MongoDB, Zen metadata, and catalog data to S3-compatible storage before upgrading; rollback means restoring that backup.

Air-gapped installs mirror images to a private registry with cpd-cli manage mirror-images, then set the image registry prefix.

Verify health with cpd-cli manage get-cr-status and oc get pods -n cpd-instance before declaring the install complete.

Authentication uses an external IdP (LDAP, SAML, OIDC); the platform stores identity mappings, not enterprise passwords.

Authorization layers: platform role, service role, and resource role; a user needs the effective intersection of all three.

Platform roles are Administrator, User, and Viewer; group-to-role mapping is managed through Zen IAM.

Watson Knowledge Catalog enforces policies through rules; data protection rules can deny, mask, or filter rows at query time.

Business terms, data classes, classifications, and reference data enrich assets and drive automated governance.

Replace the self-signed platform certificate with an enterprise CA certificate via cpd-cli so clients trust one chain.

Export audit records to a SIEM and use platform monitoring to track service health and consumption.

Store data source credentials in the credential store or an external vault; never hardcode them in notebooks or jobs.

Harden the network with NetworkPolicies, an egress proxy, and private endpoints for cloud data sources.

Watson Query creates nicknames over remote tables; operations push down when the source supports them, and cross-source joins require local staging.

Watson Query supports Db2, Oracle, SQL Server, Netezza, Teradata, PostgreSQL, MongoDB, S3, and Kafka, each needing a JDBC driver.

DataStage runs parallel jobs on the parallel engine or Spark; partitioning, buffering, and skew control throughput.

Data Replication provides change data capture for second-to-minute latency; batch ETL is the wrong tool for that requirement.

Use virtualization for read-mostly federation without copying; use DataStage for transformation, cleansing, or persisted targets.

Lineage is automatic for DataStage, Watson Query, and notebooks; other tools require metadata import.

Discovery and metadata import jobs profile assets, assign data classes, and feed quality and governance rules.

Tune Watson Query by adding virtualizer resources, keeping predicates pushdown-friendly, and pre-staging large cross-source joins.

DataStage parameter sets externalize connection and path values so one job runs across environments.

Quantify non-functional requirements first (volume, latency, concurrency, retention, RPO/RTO); each maps to a service capability or sizing decision.

Layer the design into ingest, store, govern, and consume, with explicit interfaces so services can be replaced without redesign.

High availability requires multi-zone workers, replicated RWX storage, and tested backups; pod replicas alone do not survive a zone outage.

ImagePullBackOff indicates entitlement key or pull-secret problems; CrashLoopBackOff indicates configuration, limits, or storage permissions.

A Pending PVC means no storage class matches the access mode; check events and the storage class binding mode.

An instance that never becomes ready is often blocked by resource quota; check quota and operand CR status before reinstalling.

Collect logs with oc logs, oc describe, oc get events, and cpd-cli collect-logs before restarting pods, since restarts destroy evidence.

Scale workers horizontally before raising pod limits, because service maximums cap vertical scaling.

Virtualization saves storage but adds latency; persisted copies cost storage but give predictable performance for repeated heavy analytics.