Storage Play Not Safe to Rerun in a Nightly Job

Command-based storage tasks are unconditionally changed and unconditionally fail on the second run. Replace each with a state-aware module:

- name: Create volume group
  community.general.lvg:
    vg: vg_data
    pvs: /dev/disk/by-id/<wwid>

- name: Create logical volume
  community.general.lvol:
    vg: vg_data
    lv: lv_data
    size: 200g
    resizefs: true

- name: Create filesystem
  community.general.filesystem:
    fstype: xfs
    dev: /dev/vg_data/lv_data

- name: Mount and persist in fstab
  ansible.posix.mount:
    path: /data
    src: /dev/vg_data/lv_data
    fstype: xfs
    state: mounted

state: mounted writes the fstab entry and mounts in one idempotent step, so the lineinfile hack disappears. creates: on a command is only a presence check on a path: it silences the second run, but it cannot validate sizes, filesystem type, mount options or fstab state, and it hides drift on the 12 hosts that report changed. It is a migration crutch, not a fix. Note filesystem will not reformat an existing filesystem unless force: true, which must never be set here.

Treat both as stop-and-report conditions, not auto-repair conditions. lvg given a pvs list can alter PV membership of an existing VG, and lvol will attempt to grow an LV to reach size; against the wrong disk that means destroying the wrong data. So gather read-only evidence first and assert:

- name: Inspect block devices
  ansible.builtin.command: lsblk -dnro NAME,SERIAL,SIZE
  register: disks
  changed_when: false
  failed_when: false

- name: Inspect existing VGs
  ansible.builtin.command: vgs --noheadings -o vg_name,pv_name
  register: vginfo
  changed_when: false
  failed_when: false

- name: Refuse to proceed on unexpected layout
  ansible.builtin.assert:
    that:
      - vginfo.stdout is not search('vg_data')
    fail_msg: "Pre-existing vg_data on {{ inventory_hostname }}; manual review required"

Also: reference disks by /dev/disk/by-id/<wwid> or /dev/disk/by-path instead of /dev/sdb, because kernel naming is not stable across a disk replacement. For the host with the leftover VG, the correct action is a human decision - vgexport/vgremove only after confirming the VG is not in use (vgs, lvs, pvs -a, mount check). Automate the detection, not the destruction.

The VG task fails on that host, so it is removed from the play; its remaining tasks never run and any handler already notified is discarded (handlers are not run for failed hosts unless force_handlers: true). If the failure aborts the play for others as well, look for any_errors_fatal or a max_fail_percentage that is too strict.

Restructure:

• Default behaviour (no any_errors_fatal) already lets the other hosts continue; with serial: 5 you get batches and can set max_fail_percentage: 20 so a systemic problem still stops the rollout while one bad host does not.
• Wrap the storage tasks in block/rescue per host. rescue logs the condition and can fail deliberately with a clear message, while the host is still reported individually rather than vanishing.
• Place meta: flush_handlers right after the config-change tasks so the restart happens while the host is known-healthy, instead of relying on end-of-play flushing that a later failure can discard.
• End with an explicit health assertion (systemctl is-active postgres) with changed_when: false, so a restarted-but-broken service fails the job rather than passing silently.

Decision rule: a data-affecting task should never be followed by tasks that assume success; assert on observed state before moving on.