Three pods, three SQLite files, and alerting that fires three times
Grafana 10.4 OSS runs as three Kubernetes pods behind an AWS ALB. Each pod writes to its own SQLite database on an emptyDir volume at /var/lib/grafana/grafana.db, so the pods have independent state. Users report that dashboards disappear and reappear, saved changes sometimes revert, and logs show 'database is locked'. Sessions log users out whenever the ALB routes them to another pod. Alert rules created on one pod are missing on another, and some alerts fire multiple times while others do not fire at all. The ALB has no sticky sessions. There are 150 daily users, 400 alert rules, and a single Prometheus data source. The platform team wants to keep three replicas for availability during node upgrades.
1. Diagnose why users see inconsistent state, locked database errors, and duplicate or missed alerts.
Read the worked answer
Each pod has its own SQLite file, so there is no shared state. A dashboard saved on pod A is invisible on pod B; the load balancer sends the next request to B, so the UI appears to lose changes. SQLite also serializes writes at the file level; concurrent writes from Grafana's own processes can produce 'database is locked', and it is not a supported HA backend.
Sessions are stored per pod (the default file provider), so a user authenticated on A is logged out when routed to B; even with a shared session store, all pods must share [security] secret_key or cookies fail validation. Alerting is worse: every replica runs the alerting scheduler independently against its own database. Three replicas evaluate the same rules, causing duplicate or near-duplicate notifications, while rules created on one replica are absent on the others, so some alerts never fire. The ALB has no stickiness, which also breaks live and WebSocket features. This is not a tuning problem; the deployment topology is wrong for Grafana HA.
2. Propose a supported HA architecture and a migration plan that minimizes data loss.
Read the worked answer
Use a shared MySQL 8 or PostgreSQL managed database with Multi-AZ, and point all replicas at it. Set [database] type, host, and credentials from a secret. Set [security] secret_key identically on every pod, and set [session] provider = database (or Redis) so sessions survive pod routing; enable ALB stickiness as a belt-and-braces measure for live connections. For alerting, configure [unified_alerting] ha_listen_address, ha_advertise_address, and ha_peers so replicas form a cluster and rules are evaluated once, not three times.
Migration: rehearse on staging, export dashboards, folders, data sources, and alert rules via the HTTP API as code, then stand up the new Postgres-backed instance and import. Do not assume a risk-free automatic SQLite-to-Postgres migration for every version; either use a tested community conversion or rebuild configuration as code. Keep the old instance read-only for rollback, cut DNS or ALB over, and verify one notification per alert by killing a pod.
3. What would you verify after cutover to prove HA is actually working?
Read the worked answer
Verify state is shared: save a dashboard on one pod and read it through the load balancer repeatedly, then restart a pod and confirm no loss. Check /api/health on every replica and confirm the database section reports the shared backend. Check logs for 'database is locked' and for alerting cluster membership; Grafana's alerting HA logs should show peers discovered.
Test sessions by logging in, forcing requests to each pod with curl and the same cookie, and confirming no re-login. Test alerting by creating a test rule with a short interval and a contact point that records deliveries; kill one replica during evaluation and confirm exactly one notification, not zero or two. Confirm the ALB target group has stickiness enabled if you rely on it, and that WebSocket and live dashboards stay connected. Finally, check that MySQL or Postgres connection limits and max_open_conns are sized for three replicas, because a shared database introduces a new bottleneck.