Cobalt Health: A Field Rename Breaks Production Content
Cobalt Health has 210 Looker users and 45 production dashboards. A developer merges a LookML change that renames the dimension patient_id to member_id and changes the status dimension's values from 'Active'/'Inactive' to 'A'/'I'. The change passes code review because the SQL is valid. On Monday, eight dashboards show 'Field not found' errors, three scheduled deliveries send blank PDFs, and a clinical operations team reports that a filter for 'Active' returns zero rows. The developer had used hidden: yes on the old patient_id field but did not add an alias. The project uses Git with development and production modes. The team has no content validation step in its release checklist and no named owners for dashboards.
1. What immediate remediation steps should the team take, in order, to restore trust and fix the broken content?
Read the worked answer
Stop the bleeding first. Pause or disable the three failing scheduled deliveries so blank PDFs do not continue to reach users. Revert the production commit to the last known-good version, or deploy a hotfix that restores patient_id and the old status values immediately. Communicate a short incident note to affected teams: what broke, what is restored, and when the next update will come. Then validate the eight broken dashboards and any schedules that depend on them; fix references to the old field or add a compatibility alias. Check whether the clinical operations filter used a saved value 'Active' or a dashboard filter default; restoring the field name may not restore a saved dashboard filter value, so inspect the dashboard filters directly. After service is restored, use content validation and System Activity to find other dashboards that silently changed, not just the ones that errored. Record the incident and add a rollback step to the release checklist.
2. How should the team use LookML field deprecation, aliases, and content validation to prevent this? Be specific about what should have happened before the merge.
Read the worked answer
Renaming a field in place is a breaking change. Before the merge, add the new member_id dimension as an alias of the existing SQL, with a clear label, while leaving patient_id in place. Mark patient_id as deprecated in its description and label, but do not hide it until content validation shows no production references. For status, do not overwrite the old values; add a new status_code dimension or use a case expression that maps both old and new values during the transition. Keep the old dimension for saved filters until dashboards are updated. Before deploying, run content validation against the production content in a staging environment, validate scheduled content separately, and search for field usage in dashboards and Looks. Require a second reviewer for changes to shared fields, not just SQL validity. After deploy, monitor validation errors and failed schedules. The goal is expand-and-contract migration: add new, migrate content, then remove old.
3. What ownership and release-process changes would you recommend for a team of this size? Explain how to measure whether the process is working.
Read the worked answer
Assign an owner to every production dashboard and a maintainer for each shared Explore or view. Owners approve changes that affect their content and receive validation failures. Put a lightweight release checklist in the pull request: impact assessment, content validation result, scheduled-delivery check, rollback plan, and communication plan. Use Git branches and production mode consistently; no direct production edits. For changes to shared fields, require a named reviewer from the analytics engineering team plus the business owner of the most affected dashboard. Keep a short list of certified dashboards and run validation before every deploy. Measure the process with four indicators: number of content validation errors at deploy, failed or blank scheduled deliveries per week, time to restore service after a content incident, and percentage of production dashboards with a named owner. Review these monthly. If validation errors stay near zero but owners are missing, the process is fragile; if validation blocks every release, simplify the checklist rather than skipping it.