Claims Document Pipeline: PHI, Duplicates, and Temporal Leakage
An insurer is building a document classifier to route 4.2 million scanned claims PDFs per year into 27 queues. Documents arrive as 60–400 page packets; 12% are faxes with skew and poor OCR. Each packet may contain multiple claim forms, and 8% of packets are near-duplicate resubmissions. The label is derived from an operational queue assignment that changed 18 months ago when the company reorganized its claims taxonomy. The team plans to train on 240,000 packets from the last three years and evaluate on a random 20% split. PHI includes names, policy numbers, and clinical notes. Data resides in Cloud Storage and BigQuery; the pipeline runs in Dataflow. A privacy review requires de-identification before annotation and a documented lawful basis for re-identification keys. The model must be retrained quarterly, and the business wants a 30-day audit trail for every training example.
1. Design the data preparation pipeline from ingestion to a training-ready dataset. Specify where de-identification, OCR, deduplication, and label normalization occur.
Read the worked answer
Ingest PDFs into a landing Cloud Storage bucket with Pub/Sub events. Run a Dataflow pipeline that applies Cloud DLP de-identification before any human annotation, then uses Document AI OCR and packet segmentation to identify individual claim forms. Write raw and curated layers to BigQuery with CMEK and VPC-SC. Deduplicate with content hashes plus embedding similarity, grouping near-duplicates by packet and claimant so they cannot cross splits. Normalize labels through a versioned mapping table that translates the pre-reorganization queue taxonomy into the current 27 queues; store the mapping version with every example.
Keep re-identification tokens in a separate, tightly restricted dataset with Cloud KMS-managed keys and break-glass IAM. Training data should contain only de-identified text and stable pseudonymous IDs. Record lineage in Data Catalog and ML Metadata: source URI, OCR version, DLP transformation, dedupe group, label-map version, and annotation batch.
Order matters. De-identifying before annotation reduces PHI exposure; deduplicating before splitting prevents leakage; normalizing labels before training avoids learning a retired taxonomy.
2. The team wants a random 80/20 split. Why is that risky, and what split and metrics would you use instead?
Read the worked answer
A random split is risky for two reasons. First, near-duplicate packets and multiple forms from the same claimant can land in both train and test, so the model memorizes document layouts or claimant-specific phrases and the test score is optimistic. Second, the label taxonomy changed 18 months ago; a random split mixes old and new queue semantics, so the model is evaluated on a distribution the business no longer uses.
Use a temporal holdout, for example the most recent six months after the taxonomy change, and group by claimant and packet before splitting. Deduplicate before the split, not after. If enough history exists, backtest on multiple time windows to measure stability.
Report macro F1 across all 27 queues, per-queue precision and recall, calibration error, and confusion among queues that were split or renamed during the reorganization. Include a separate slice for fax-quality documents and near-duplicate resubmissions. A single aggregate accuracy number would hide poor performance on rare but operationally important queues.
3. How should the pipeline protect re-identification keys and provide the required 30-day audit trail?
Read the worked answer
Separate re-identification from training entirely. Store tokens and mapping keys in a dedicated project or dataset with CMEK, VPC-SC, and least-privilege IAM. Only a narrow break-glass role can join a token back to a claimant, and every use should require a documented purpose and approval. Use Cloud KMS key rotation and audit key usage.
For the audit trail, log every training example’s lineage: source object, ingestion time, DLP transformation, OCR and segmentation version, dedupe group, annotation batch, label-map version, and the dataset snapshot used. BigQuery audit logs, Data Catalog lineage, and ML Metadata can provide this. Retain logs for at least 30 days, preferably longer, and make them immutable or export-only.
Do not place re-identification keys in the training bucket, even hashed, if the hash can be reversed by joining against a small claimant population. The privacy review should see that annotation, training, and evaluation all use de-identified data and that re-identification is an exception path, not a routine pipeline step.