Major upgrade with an Angular plugin and an anonymous public dashboard
You are upgrading Grafana OSS from 9.5.21 to 11.x on virtual machines. Plugins include grafana-clock-panel, grafana-piechart-panel, an internal Angular-based plugin built in 2020, and grafana-simple-json-datasource. Config has [auth.anonymous] enabled = true for a public status dashboard, [security] admin_user = admin, and [dashboards] versions_to_keep = 20. The database is MySQL 8.0. You have 800 dashboards, 120 alert rules, 60 data sources, and 900 users. A staging instance with a copy of the production database upgraded successfully, but the internal plugin fails to load and the public dashboard shows 'Plugin Error'. Leadership wants a maintenance window this month, and you need a rollback story.
1. What are the likely upgrade blockers, and why is the staging success misleading?
Read the worked answer
The internal 2020 plugin is almost certainly Angular-based. Grafana deprecated Angular in 9, disabled it by default in 10, and removed Angular support in 11, so a plugin that loaded in staging may still fail in production if staging has an old cached plugin build or a different angular_support_enabled setting. The grafana-simple-json-datasource plugin is also old and may not be maintained for 11.
Staging 'success' can be misleading because a database copy upgrades but plugin binaries, provisioning files, and anonymous-auth behavior come from the filesystem and config, not the database. If staging did not exercise the public dashboard anonymously, it would not catch a plugin error that only appears for anonymous users. Also, Grafana 9.5 to 11.x crosses two major versions; config options and feature toggles changed, and database migrations are irreversible. Check the plugin list with grafana-cli plugins ls, inspect each plugin's plugin.json for Angular, and read the 10 and 11 upgrade notes before touching production.
2. Lay out an upgrade and rollback plan for the VMs.
Read the worked answer
Back up MySQL with mysqldump --single-transaction, copy /etc/grafana, /var/lib/grafana/plugins, and provisioning directories, and record exact package versions. Inventory plugins and remove or replace unsupported ones before the window; do not rely on Grafana 11 to run Angular. Build a staging VM from the same backup and run the same config, including anonymous access, and test login, dashboards, alert rules, data sources, and public dashboards.
In production, take a maintenance window; Grafana does not support mixed versions against one database, so upgrade all nodes together behind the load balancer rather than one at a time. Upgrade 9.5 to 10.x to 11.x if the notes require intermediate steps, checking /api/health and logs after each. Rollback means restoring the MySQL dump and reinstalling the 9.5 binaries; you cannot downgrade the database in place. Keep the old packages and config until validation is complete.
3. What specific checks would you run for the anonymous public dashboard and admin security after upgrade?
Read the worked answer
For the public dashboard, confirm [auth.anonymous] enabled = true, org_name points to the intended org, and org_role is Viewer, not Editor or Admin. Test from an incognito browser with no cookies, because an admin session can mask an anonymous failure. Check which plugin renders the public dashboard and whether that plugin loads for anonymous users; if the plugin is unsupported, replace the panel with a core panel or move the dashboard to a static export.
Confirm public dashboards are enabled in the version and that the dashboard's data source is accessible to anonymous viewers, since data source permissions still apply. For admin security, verify admin_user is not a shared default, change the admin password, restrict the admin endpoint by network or SSO, and confirm anonymous users cannot reach Explore or the admin UI. Also check versions_to_keep is still applied and that old dashboard versions do not expose secrets in JSON.