Atlas Tag Propagation and Ranger Masking Gap
A CDP Private Cloud Base cluster uses Ranger 2.1 and Atlas 2.2. The Hive service has 180 external tables with columns ssn, email, and dob. Atlas has a classification PII on the table entity cust_profile; only 40 of 180 tables have column-level classifications. A Ranger tag-based policy uses PII with a column mask MASK_SHOW_LAST_4 and a row filter region='US'. Analysts in the AD group analysts query through HiveServer2 and Impala; ETL service accounts svc_etl and svc_bi use JDBC. During an audit, analysts see unmasked ssn for cust_profile in both Hive and Impala, while email is masked. Ranger audit shows 'no matching policy' for the ssn column. The Atlas hook is installed on Hive but not on the HMS. A recent Atlas restart took 38 minutes, and Ranger tag sync lag is reported. You must restore masking without breaking ETL and without granting broad table access.
1. Why does a table-level PII tag fail to mask `ssn` while `email` is masked? What would you inspect in Atlas and Ranger, and what is the fix?
Read the worked answer
Ranger tag-based policies evaluate the tag attached to the specific resource named in the policy. If the policy resource is the column ssn, the PII classification must be attached to the Atlas column entity for ssn. A classification on the table entity cust_profile is not automatically inherited by every column. That explains the pattern: email has a column-level PII classification and is masked, while ssn does not and Ranger reports no matching policy.
Inspect Atlas first. Query the table entity and its column entities, and confirm whether ssn has the PII classification directly. Check whether the classification is on a column with the same name in the correct database and table, not a similarly named table. Then inspect Ranger TagSync: ranger.tagsync.source.atlas, the Atlas REST address, sync interval, and recent TagSync logs. A 38-minute Atlas restart can leave Ranger's tag store stale, so even a correct column tag may not have reached Ranger. Also confirm the Ranger tag service is enabled for Hive and Impala and that the policy resource is column with the exact column name.
The missing Atlas hook on the HMS matters for completeness: DDL or column entities created through the metastore may never reach Atlas, so some tables will have no column tags. Install or repair the Atlas Hive hook on the correct service, then bulk-classify the missing column entities through the Atlas API. Do not just tag the table and hope for inheritance. After classification, wait for TagSync and verify by running a query as a member of analysts and checking Ranger audit for a tag-policy match on ssn.
2. Design a phased remediation for all 180 tables and the ETL service accounts. How do you avoid broad table grants and prove the control works?
Read the worked answer
Treat this as a data-classification project, not a single policy edit. First inventory all columns named ssn, email, and dob through the Atlas API, including tables without column classifications. Classify the column entities as PII, and add a second tag such as PII_RAW only where ETL legitimately needs raw values. Fix the Atlas hook gap so new columns are classified at creation time. Then let Ranger TagSync propagate the tags and confirm the tag store is current before changing access.
Keep the Ranger tag policy scoped to the analysts group rather than global. That avoids needing a masking exception for ETL: svc_etl and svc_bi can receive narrow resource-based policies for specific databases and columns, with no mask, while analysts hit the tag policy. Do not grant SELECT on whole tables to service accounts; grant column-level access and keep row filters where required. Separate svc_etl from svc_bi if one only needs masked data. Check AD group nesting so a service account is not indirectly a member of analysts.
Roll out in waves: 10 tables, then 40, then the rest. For each wave, test as an analyst, as svc_etl, and as svc_bi using both HiveServer2 and Impala. Verify masked values, row-filter behavior, and that ETL row counts are unchanged. Use Ranger audit to prove the tag policy matched the ssn column, and use Atlas lineage to find downstream views that might expose raw data. Keep a rollback policy and a list of tables with unresolved ownership. The control is proven only when audit shows the intended policy for each persona and no unintended table-wide grants exist.