Provisioned dashboards, lost UI edits, and duplicate UIDs
Your team runs Grafana 11 OSS for 40 engineers. Dashboards are provisioned from Git with the file provider: the provider YAML points at /var/lib/grafana/dashboards, sets updateIntervalSeconds: 10, allowUiUpdates: false, and foldersFromFilesStructure: true. Last month two engineers edited the same Checkout Latency dashboard in the UI and saved; their changes vanished after the next pod restart. Since then a provisioned dashboard sometimes returns Dashboard not found when opened from a saved link, and a duplicate copy appears in the General folder after deploys. The repo contains 60 JSON files; three files use the same uid checkout-latency, and one file has no uid at all. Deploys copy files into the directory but never delete files that were removed from Git. The team uses a GitOps pipeline and expects no manual UI edits in production.
1. Explain why the UI edits disappeared and what the missing and duplicate uid symptoms indicate. What is the file provider doing on each sync?
Read the worked answer
With allowUiUpdates: false, the file provider treats disk as authoritative. A UI save can still write a row to Grafana's database in some versions, but the next provisioning sync compares or overwrites the dashboard from the JSON file, so the edit is lost. The real issue is a workflow mismatch: engineers are editing a read-only artifact.
The duplicate and 'Dashboard not found' symptoms point to identity problems. Provisioning matches dashboards by uid; three files sharing checkout-latency fight over one database row, so the last file processed wins and links can resolve to a dashboard whose content or folder changed. A file with no uid has no stable identity, so Grafana may create a new dashboard on a later sync instead of updating the old one, leaving an orphan in General. That is why the duplicate appears after deploys rather than on a fixed schedule. Check provider log lines for 'updating dashboard' and compare dashboard id and uid values via /api/search.
2. Design a provisioning and review workflow that lets engineers change dashboards without losing work. Include what must be true in Git and in Grafana.
Read the worked answer
Treat Git as the only source of truth and make the production provider read-only. Engineers should either edit a local or dev Grafana where allowUiUpdates: true, export the JSON, and open a PR, or use a dashboard-as-code tool or API to pull the current JSON into the repo. The PR should run CI that validates JSON, rejects duplicate uid values, and enforces a naming convention such as <team>-<dashboard>. Add a required uid to every file and store folders as directories so foldersFromFilesStructure is deterministic.
Keep allowUiUpdates: false in production and set disableDeletion: false only if you want removals from Git to delete dashboards; otherwise document that deleted files leave orphans. Add a deploy step that prunes files removed from Git. Use a short updateIntervalSeconds only for fast reloads. Finally, give engineers a documented 'Save as' path for exploratory work, and review exports in PRs rather than allowing direct production saves.
3. What immediate remediation would you perform for the duplicate and not-found dashboards, in order?
Read the worked answer
First stop new damage: freeze deploys, then inventory all provisioned JSON for uid with jq, and assign unique, stable UIDs and folder paths. Decide which of the three checkout-latency files is canonical, rename the others, and add a UID to the file missing one.
Second, remove the orphaned database row created by the uid-less file. Because the file is provisioned, deleting the file alone may not remove the row if disableDeletion is true; use the dashboard API (DELETE /api/dashboards/uid/<uid>) or the UI as an admin, then re-provision. Third, fix saved links by updating them to the canonical UID and checking folder permissions, since a link can return 'not found' if the viewer lacks access to the destination folder. Fourth, reload provisioning and verify /api/search returns one entry per intended dashboard, and that a restart does not recreate duplicates. Only then re-enable deploys with CI checks.