HA node with a local filestore joins the shared cluster

Root cause

art-node-4 is running with a local filestore while its database metadata is shared. It can accept an upload, write the binary to local XFS, and commit the database row that all nodes see. Other nodes then return checksum or 404 errors, and the cluster has a split-brain filestore. The local df output is the strongest evidence; the NFS change is a second risk because all nodes must mount the same export with compatible locking and caching.

Evidence to collect before changing

• findmnt, mount, and /etc/fstab on all nodes; compare mount options, NFS version, and server export.
• stat and write test as the artifactory user on the NFS mount; check UID/GID ownership.
• Service logs around the first local write, plus the filestore checksum errors on nodes 1-3.
• Recent deployments and their checksums; identify artifacts that may exist only on node 4.
• binarystore.xml, system.yaml, join.key, and master.key hashes across nodes.

Do not simply delete node 4's local data. Isolate it from the load balancer and stop writes first; back up the local filestore, then reconcile unique blobs into the shared filestore under a controlled procedure.

Shared filestore requirements

Every node must mount the same POSIX-compliant filestore at the same path with the same options. NFSv4 is the safer choice; mixing v3 and v4 across nodes can change locking, attribute caching, and open-file behavior. The artifactory user must be able to create, read, rename, and delete files and directories. Root squash is acceptable only if ownership and permissions are correct; removing no_root_squash is not automatically fatal, but it can break pre-existing root-owned files. Test with a real write, rename, and delete as the service user, not just ls.

Database

All nodes must point to the same PostgreSQL primary with the same credentials and schema. Do not point Artifactory HA nodes at a read replica for writes, and do not use separate databases. Keep clocks synchronized and keep binarystore.xml identical. The join.key must match the cluster.

Immediate action

Treat the v3 remount as a change to validate, not a guess. Compare every node's mount and run a checksum read of a known artifact from each node. If v3 is required, verify locking and cache behavior under load before returning node 4.

Safe return to service

1. Keep node 4 out of the load balancer and stop Artifactory.
2. Mount the correct NFS export at /opt/jfrog/artifactory/var/data; confirm with findmnt and a write test as artifactory.
3. Preserve the local filestore. Compare its checksums with the shared filestore; copy only unique, verified blobs into the shared filestore using a supported maintenance procedure.
4. Remove or archive the local directory only after reconciliation, so the node cannot fall back to local disk.
5. Start the service, watch artifactory.log and replicator.log for cluster join and filestore errors, and read a known artifact from node 1 through node 4.
6. Add node 4 to the load balancer only after the health endpoint passes and a test download succeeds.

Prevent recurrence

Automate a pre-flight check that fails if the mount is not NFS, the path is not the expected export, or binarystore.xml differs. Make the load balancer health check hit the router health endpoint, not just the local process port. Alert on filestore latency and checksum errors.