Acquia Back End Specialist (D10) study materials
Acquia Back End Specialist (D10) study resources

Config that drifts between environments after a hotfix

Acquia Back End Specialist (D10) · Put your knowledge to work in context.

All case studies

Config that drifts between environments after a hotfix

A site uses the standard config sync workflow: config/sync in Git, drush config:import on deploy, and Acquia Cloud with dev, stage, and prod environments. During an incident, an on-call engineer edits a view's filter and a block's visibility directly in the prod UI to unblock editors. The change is never exported. Two weeks later, a routine deploy imports the config/sync from main, and prod's view reverts to the old filter. Editors notice within minutes. A second engineer proposes running drush config:export on prod to capture the hotfix, then committing it. The repo's config/sync also contains a system.site UUID that differs from prod's.

1. Evaluate the second engineer's proposal to export from prod.

Read the worked answer

It's the wrong direction and risks making things worse.

Why it's tempting: it would capture the hotfix and make prod match the repo again.

Why it's dangerous:

  1. It exports whatever else is on prod, including any other drift introduced since the last deploy - not just the hotfix. You would be blessing unknown state into version control.

  2. It exports the system.site UUID and other environment-specific config, which is exactly the mismatch already flagged. Importing that back to dev/stage would either fail (UUID mismatch) or overwrite the other environments' identity.

  3. It inverts the workflow. The source of truth is config/sync in Git; prod is a deployment target. Exporting from prod makes prod the source of truth and guarantees the next person's deploy silently reverts something.

What to do instead: reproduce the hotfix in a local/dev environment, export from there, review the diff in a merge request, and deploy forward. If the hotfix must be preserved urgently, diff prod's active config against config/sync (drush config:status and a manual diff of the specific view/block), cherry-pick only those two changes into the repo, and commit them with a note explaining the incident.

2. How should the `system.site` UUID mismatch be handled, and what does it tell you about the workflow?

Read the worked answer

The UUID mismatch means the repo's config/sync was not seeded from an environment that shares prod's identity - typically because someone exported from a fresh install, or a config:import was run with --partial and skipped system.site. Drupal treats system.site as a singleton and will refuse to import a config that changes its UUID unless forced.

The correct handling:

  • Never force it. Do not use --partial to skip the UUID or hand-edit it to match prod. That masks the underlying problem.
  • Align the repo to prod's UUID once, deliberately: take prod's system.site UUID, set it in the config/sync/system.site.yml in the repo, and commit. From then on every environment shares the identity.
  • Exclude environment-specific config from sync where it legitimately differs (e.g. system.performance CSS/JS aggregation, acquia_connector.settings, search server config). Use $config['...']['...'] overrides in settings.php per environment rather than syncing values that must differ.

The mismatch is a symptom: it means the team has been importing config without checking drush config:status first. Add that check to the deploy script so drift is caught before it becomes an incident.

3. What process change would prevent this class of incident from recurring?

Read the worked answer

Three changes, each cheap:

  1. Make UI edits impossible in prod. Remove administer site configuration and administer views from roles used on prod, or use a config-read-only approach. If editors can't change config in prod, drift can't originate there. The incident's root cause is that prod was writable.

  2. Gate deploys on drush config:status. Run it in the deploy pipeline and fail the build if it reports differences between active config and config/sync. That turns silent drift into a red build before the import runs.

  3. Give the on-call a sanctioned hotfix path. A documented procedure: make the change in a branch, run the export locally, open a PR, and deploy through the normal pipeline - with an explicit "emergency" label that allows a fast-track review. People edit prod directly because the correct path feels too slow during an incident; make the correct path fast and the workaround unnecessary.

Optionally, add a post-deploy check that diffs active config against config/sync and alerts if they diverge, so drift is caught within minutes rather than two weeks.