RHCE: Port 8081 Appears on One Host, 8080 on Another

Port 8081 Appears on One Host, 8080 on Another

A platform team keeps ansible.cfg, inventory/prod/ and group_vars/ in one git repo. The web tier must listen on port 80 in production and 8080 in staging. After a merge, ansible-playbook site.yml -i inventory/prod reports http_port as 8081 on web01 and 8080 on web02, although group_vars/all.yml sets http_port: 80. Files show inventory/prod/group_vars/web.yml (8080), inventory/prod/host_vars/web01.yml (8081) and a play-level vars: block (8080) all mentioning the port. There is no host_vars/web02.yml. A new engineer ran the same playbook earlier with -e http_port=8081. The team wants one authoritative default, deliberate per-host overrides, and a repeatable way to explain a resolved value during incidents.

1. State which value wins for web01 and web02, and explain in what order you must reason about the four sources involved.

Read the worked answer

Order the sources by precedence, lowest to highest:

role defaults
inventory group_vars/all
inventory group_vars/web
inventory host_vars/web01
play-level vars:
extra vars (-e / --extra-vars)

Consequences:

In the run with -e http_port=8081, extra vars beat everything, so both hosts resolve to 8081. Nothing else is observable.
In a run without -e, the play-level vars: 8080 beats host_vars (8081) and group_vars, so both hosts resolve to 8080.

So the reported symptom (8081 on web01, 8080 on web02) is not a state any single invocation produces. Before editing files, establish which command produced which number (shell history, CI logs, --extra-vars in a wrapper script). The real defect is the play-level vars: entry: while it exists, the host_vars/web01.yml override is dead weight and will never take effect. Extra vars are for one-off debugging only and must never be baked into a deploy path.

2. How do you determine the value that a given host actually resolves to, without reading every vars file?

Read the worked answer

ansible-inventory --host web01 is useful but incomplete: it shows only inventory-derived variables (host_vars, group_vars), never play vars:, role defaults, facts or -e. A clean inventory report therefore does not prove the effective value.

To resolve through the same precedence chain as the real run, replay the same invocation context:

ansible web01 -i inventory/prod -m ansible.builtin.debug -a "var=http_port"

and then repeat with the same -e values used in production. Inside the playbook, a temporary debug: var=http_port (or debug: msg="{{ hostvars[inventory_hostname].http_port }}") records the resolved value into the run log. ansible-playbook -vvv confirms which config and inventory files were loaded, and ansible-config dump --only-changed confirms hash_behaviour is the default (replace, not merge) so no deep merging is silently combining the sources. Keep the debug task behind a -e show_vars=true guard so it can stay in the repo for incident use.

3. What structural change makes a per-host override possible while keeping a single default, and how do you protect it from silent drift?

Read the worked answer

Remove the play-level vars: entry for http_port entirely. Choose one owner for the default:

If a role owns the setting, put http_port: 80 in roles/<name>/defaults/main.yml (lowest precedence, intended to be overridden).
If the value is site-wide configuration rather than role behaviour, put it in inventory/prod/group_vars/all.yml.

Leave host_vars/<host>.yml as the only exception channel, and reserve -e for incident debugging that is recorded in the change ticket.

Add a guard so a bad merge fails loudly rather than silently changing the listener:

- name: Validate listener port
  ansible.builtin.assert:
    that:
      - http_port | int in [80, 8080]
    fail_msg: "http_port resolved to {{ http_port }} on {{ inventory_hostname }}"

Run the assert before the template task so a mis-set extra var or a stale override aborts before the service configuration is rewritten.