Replication lag and a release bundle built from the wrong build name

Diagnosis

A 5-minute push schedule over a 1.2 TB repository with 2.8 million small files is likely overlapping itself. Replication is asynchronous and uses file-level locks; when the next run starts before the previous queue drains, Waiting for file lock and socket timeouts accumulate. Check the replication status, queue depth, replicator.log, WAN latency, and whether the target is applying checksums or cleanup concurrently. The Frankfurt repo being incomplete is a symptom, not proof that a single file is corrupt.

Options and trade-offs

• Increase the interval and run off-peak so runs do not overlap. This is the lowest-risk immediate change.
• Split libs-release-local by product or team into separate repositories and replicate them in parallel. This reduces lock contention and lets you prioritize payments.
• Use event-based replication if the version supports it, for lower RPO. It can increase load on the primary, so monitor.
• Pull replication at Frankfurt shifts initiation and catch-up control to DR, but it still consumes WAN and source reads. It does not fix an undersized pipe.

For a 1-hour DR goal, do not rely on a full-repo sweep. Measure one-hour delta volume and size the link and schedule to that.

Why it failed

The release bundle was created from build name payments-service, but CI published build-info as payments-api. Distribution resolves the exact build name and build number; a mismatch means no build-info is found, even though the artifacts exist. Also, jf rt upload followed by jf rt build-publish records artifacts but does not promote them to libs-release-local.

Correct flow

1. Upload artifacts with the same build name and number used by CI, then run jf rt build-publish payments-api <build-number>.
2. Promote with jf rt build-promote payments-api <build-number> libs-release-local --copy=true --status=Released so the artifacts and build-info are available in the release repo.
3. Verify the expected artifacts by AQL or a manifest check.
4. Create the release bundle from payments-api and the exact build number, sign it, and distribute it.

Distribution vs replication

Replication mirrors repository content asynchronously; it has no release signature, entitlement, or bundle version. Distribution delivers signed, immutable release bundles to edge nodes with a governed lifecycle. Do not use replication as a substitute for release governance.

Automation outline

1. CI publishes build-info and fails if jf rt build-publish returns non-zero.
2. Promotion job calls jf rt build-promote with --copy=true, then queries AQL for the expected artifact paths and checksums. Abort if any are missing.
3. Release job calls jf ds rbc with the exact build name and number, jf ds rbs to sign, and jf ds rbd to distribute to the edge group. Poll distribution status and fail on any edge error.
4. Monitoring job queries the replication status API under /api/replication and the replicator.log; alert if lag exceeds 60 minutes, queue depth grows for two consecutive checks, or a run is still active when the next is due.

Safety checks

Make each step idempotent and fail closed: verify build-info exists, checksums match, the bundle is signed, all target edges are online, and the DR lag is below threshold before promoting. Store tokens in the CI secret store, not in scripts. Log the build number, bundle version, digests, and replication timestamp for audit.