Certifying Lineage When a SELECT * View Hides the Path
A logistics company has 1,800 assets in CDGC. Column lineage covers 68% of columns. A legacy source field, SHIP_LEGACY_ID, will be dropped next quarter. The business term “Shipment” is linked to five assets, one of which is the legacy column. Column lineage shows four downstream columns, but a critical regulatory report reads from a view defined as SELECT * FROM shipment_legacy. The scanner has not resolved column-level lineage through that view. A steward wants to certify the lineage as complete because the dashboard displays a green lineage indicator. Manual lineage entries created in 2022 have not been reviewed. The team has six weeks before the drop.
1. How should the team assess the true impact of dropping SHIP_LEGACY_ID given 68% column coverage, the SELECT * view, and unreviewed manual lineage?
Read the worked answer
Treat the green indicator as metadata coverage, not proof of completeness. The impact assessment should combine automated lineage, SQL inspection, and manual verification.
Step 1: Bound the known impact The four downstream columns from automated lineage are confirmed. Trace each to its report or application owner. These are the minimum impacted set.
Step 2: Resolve the SELECT * view A SELECT * view obscures column mapping. Inspect the view definition and the report’s SQL to see whether SHIP_LEGACY_ID is actually referenced. If the report selects specific columns, the field may pass through the view without being used. If the report selects *, the field reaches the report and must be treated as impacted. The scanner’s failure to resolve column lineage here is a known gap, not evidence of no impact.
Step 3: Treat 2022 manual lineage as unverified Manual entries may be outdated or may have been created for a previous schema. Review each entry that touches Shipment or shipment_legacy, confirm it against current SQL, and mark it verified or retire it.
Step 4: Search beyond lineage Query the catalog for assets linked to the Shipment term, assets with the legacy column name in views/procedures, and report owners who reference the field. Lineage is one evidence source, not the only one.
Conclusion The true impact is at least four columns, potentially more through the view and manual paths. Six weeks is enough to inspect the view and the 2022 entries, but not to assume completeness.
2. Should the steward certify the lineage as complete? What certification criteria and caveats should apply?
Read the worked answer
No. Certification should mean the lineage is fit for the stated purpose, not that a dashboard is green. Here the stated purpose is impact analysis for a field drop, and there is a known unresolved view plus unreviewed manual entries.
Certification criteria
- All downstream paths from the field are either resolved by the scanner or manually verified against current SQL.
- Known gaps are documented with owner, reason, and residual risk.
- Manual lineage entries in scope are reviewed and dated.
- A named steward and a technical reviewer sign off.
- The certification states its scope: for example, “Shipment domain, report-facing columns, as of [date].”
Caveats to record now
- SELECT * view not resolved at column level; report SQL inspection pending.
- 2022 manual entries not yet reviewed.
- Overall column coverage is 68%, so absence of lineage is not proof of absence of impact.
What to do instead Certify the verified subset and mark the remainder “under review.” A partial certification with explicit gaps is more useful than a false complete status. If the organization requires a binary status, the correct status is “not certified - known gaps.” The steward should not let the green indicator override the evidence.
3. What governance actions should follow for the Shipment term link and the unreviewed manual lineage before the field is dropped?
Read the worked answer
Use the field drop as a governance trigger, not just a technical migration.
Term link actions
- Review the five assets linked to Shipment. Confirm which ones truly represent the shipment concept at the stated granularity. The legacy column may be a technical identifier, not the business term itself.
- If SHIP_LEGACY_ID is being replaced, link the successor field to Shipment and record the relationship or replacement in the term’s context.
- Do not leave the term linked only to the retiring field; that would make the glossary point to a dead asset after the drop.
- Assign a term owner and review date for the change.
Manual lineage actions
- Review each 2022 manual entry that touches Shipment or shipment_legacy. Verify against current view and report SQL.
- Mark entries as verified, corrected, or retired. Record who reviewed them and when.
- For the SELECT * view, add manual lineage only after inspecting the report SQL, and state the evidence source.
Broader actions
- Add the unresolved SELECT * pattern to the scanner remediation backlog.
- Notify report owners of the drop and the current impact list, including the caveat that the view path is unresolved.
- Schedule a post-drop validation to confirm no report breaks and to update lineage and term links.
These actions close the governance loop: the glossary reflects the future state, manual lineage becomes trustworthy, and the impact assessment has an auditable owner.