Unified Data Fabric Across Environments
The Unified Data Fabric domain tests how CDP provides consistent security, governance, metadata, and data movement across hybrid and multi-cloud environments. Expect questions on SDX scope, Ranger versus Atlas responsibilities, replication, and catalog/lineage across clusters.
Core concepts
- Unified Data Fabric is an architectural outcome, not a single product. It is delivered through SDX (shared data context), Replication Manager, Data Catalog, and consistent identity and security.
- SDX centralizes Ranger policies, Atlas metadata and lineage, and audit context across data services. In CDP Public Cloud, a Data Lake service hosts SDX components; in Private Cloud Base, Ranger and Atlas are deployed per cluster but can be integrated.
- Ranger handles authorization policies for HDFS, Hive, Impala, Kafka, and other services, including tag-based policies, row filtering, and column masking. Atlas handles metadata, classification, lineage, and search. Knox provides perimeter security and gateway services.
- Replication Manager applies policy-based replication of HDFS and Hive data and metadata between clusters, with overwrite or append options.
- Data Catalog provides business discovery, tagging, and collaboration and can harvest metadata from Atlas.
- Identity: Kerberos, Active Directory, or LDAP; IDBroker for cloud; consistent principals across environments.
Worked example
A company replicates a Hive table from on-prem to cloud and must keep PII masked for cloud analysts. Use Atlas to classify PII columns on the source; Ranger tag-based masking policies synced via SDX; Replication Manager to copy Hive data and metadata; verify that the replicated table inherits Atlas tags and Ranger policies. Reasoning: replication moves data and metadata, but consistent masking depends on shared tags and policies. If policies are cluster-local, export/import them or use SDX.
Traps
- Treating SDX as a data storage layer; it stores policies and metadata, not the data itself.
- Confusing Ranger (authorization) with Atlas (metadata and lineage); they are complementary.
- Assuming replication automatically replicates Ranger policies; policy consistency needs SDX or manual sync.
- Forgetting that tag-based policies require Atlas classification to be present and propagated.
- Believing a data fabric requires a single cluster; it spans clusters and clouds by design.
- Overlooking Knox for edge and perimeter security in public cloud.