The table that returns zero rows in the session

The SQL is almost certainly not the problem; the identity executing it probably is. Work from cheapest to most expensive:

1. Confirm the effective user in both places. In the session run SELECT current_user(), effective_user() and compare with beeline. If the session presents a per-user Kerberos identity while beeline used svc_analytics, the row-filter policy resolves differently and the two queries are not the same query at all.
2. Inspect the data connection. Check whether the project data connection uses the caller's principal or a shared keytab/service account. This is the single most common cause of "works in beeline, empty in the notebook".
3. Read the Ranger audit log for both queries. It records the policy that applied and the filter predicate injected. That converts a guess into evidence.
4. Test the hypothesis directly: run SELECT region, COUNT(*) ... GROUP BY region in the session. If only one region appears, the row filter is confirmed. Combined with WHERE region = 'APAC', an analyst filtered to EMEA gets exactly zero rows - the observed symptom.
5. Rule out the boring alternative: confirm both clients point at the same cluster and the same database (prod versus dev endpoints are easy to confuse). If current_user() matches and Ranger shows no filter, this becomes the leading hypothesis.

The fix is not to bypass the policy. Either grant the analyst a legitimate policy for the regions they need, or rewrite the work to use their own region's data and document that the model is region-scoped.

What is happening. Ranger column masking substitutes NULL (or a masked literal) at query time. The value is not missing in storage; it is being withheld from this principal. The scientist cannot detect this from the dataframe alone - a NULL looks identical to a genuine data-quality gap.

Consequences

• Modelling: a derived key built on a masked column collapses to a single NULL bucket. The feature carries no signal, and if it is used carelessly (for example, filling NULL with a constant) the model silently learns from a constant. Worse, the training pipeline may still report respectable metrics driven by other features, so nobody investigates.
• Compliance: asking for the mask to be lifted for convenience is the wrong move and will be refused. Storing raw national IDs in a notebook or a feature table would create a new uncontrolled copy of regulated data.

What to do instead

1. Do not request unmasking. Raise a request for a governed, de-identified derivation: a salted hash or surrogate household key produced by an approved pipeline into a curated table, where the salt is managed centrally and the mapping is not exposed.
2. If that cannot be delivered in two weeks, drop the feature and substitute an approved proxy (postal district, account tenure band) and record the decision.
3. Add a data-quality gate that measures the null rate of every model input feature against a threshold before training. A 100% null column should fail the pipeline, not become a constant.
4. Record in the model documentation which fields were masked for the training principal - this is exactly the kind of thing that is invisible until an audit asks why the model never uses a field everyone believes is present.

Pulling 1.4 billion rows into a driver-side dataframe is the wrong tool at every layer: it moves bytes over a single connection, ignores partition pruning after the fact, and is bounded by session memory rather than by cluster capacity.

Push the work down

• Aggregate in Impala or Spark and return a compact result - one row per account per period rather than raw events. Model training usually needs tens of features per entity, not a billion event rows.
• Select explicit columns. SELECT * on a wide event table is the fastest way to blow a memory limit.

Make pruning effective

• Always filter on the partition columns dt and region. Note that the region predicate must be compatible with the Ranger row filter; filtering to a region the principal cannot see returns nothing.
• Confirm pruning in the query profile rather than assuming it.

Curate once, then reuse

• Build a curated table in partitioned Parquet via a Spark job, with target file sizes of roughly 128–256 MB. 220,000 small files cost name-node memory and turn every scan into a metadata storm.
• Compact by rewriting affected partitions (INSERT OVERWRITE a partition from a compacted result), then refresh table and column statistics so the optimizer has current cardinalities.
• Refresh incrementally: process only new dt partitions, and make reruns idempotent by overwriting the target partition rather than appending.
• Point the notebook at the curated table. Training then reads a few GB, not a terabyte, and the refresh is a scheduled job rather than an interactive accident.