Release Integrity Failure: Three Binaries Named 4.8.0

Contain first, diagnose second, decide third. The immediate danger is that you cannot yet say which customers have which binary.

First 72 hours:

1. Freeze the artifact repository and the release/4.8 branch; stop all builds until the configuration is re-established. Preserve evidence - do not overwrite the three artifacts.
2. Establish the as-shipped set: for each of the three artifacts, record hash, build timestamp, build machine, source revision, and which distribution channel or customer segment received it. If the distribution record is incomplete, that is itself a finding.
3. Attempt to reproduce the customer's $0.02 difference by rebuilding from each candidate configuration: (a) release/4.8 with the hotfixes, (b) with library 2.2.9 versus 2.3.1, (c) with the modified obfuscation script. The goal is to attribute the behavior to a specific difference, not to guess.
4. Assess impact: is the rounding difference deterministic and bounded, or does it drift with volume? Financial reporting customers will care about a penny discrepancy more than about a crash, because it undermines reconciliation with their own systems.

Before deciding: know the affected population, the magnitude and reproducibility of the defect, and whether the two hotfix commits contain any other behavioral change. A recall of 3,400 customers is expensive and risky; a targeted patch with a clear advisory may be correct - but only once you can name exactly which binary each customer has.

All four CM functions failed, and they failed in a way that compounded: no single failure alone would have produced an unreliable as-built configuration.

Identification. Three different binaries share the name 4.8.0. A release identifier that does not uniquely identify a configuration is a definitional failure; the identity of the deliverable was never established.

Control. Two changes to the release branch were approved by email by the development lead, bypassing the change control board. The unauthorized changes were never merged back to trunk, so the release line and the mainline diverged silently.

Status accounting. The manifest says library 2.3.1; the binary contains 2.2.9. The record of what is in the build does not match the build. Any downstream consumer - auditor, support engineer, next release - is misinformed by the record itself.

Configuration audit. No functional or physical configuration audit was performed. A physical audit would have compared the delivered artifact against the manifest and caught the library mismatch; a functional audit would have confirmed the artifact satisfies the specified requirements from the tagged baseline.

On the checkbox: 'build verified' signed by the build engineer proves only that someone clicked it. It carries no evidence of reproducibility, no comparison of rebuild hash to delivered hash, and no independence - the builder verifying the builder is self-review. This is exactly the class of control that audit programs call a documentation artifact rather than a control, and it will fail the regulatory auditor's question about as-built configuration.

Corrective actions must make the as-built configuration independently verifiable; the measures must be capable of showing that, not just that procedures were rewritten.

Actions:

1. Reproducible builds. Add an independent rebuild step to the release checklist: rebuild from the tagged revision on a clean build agent and compare the SHA-256 hash of the artifact to the delivered binary. Any mismatch blocks release. Removing the uncommitted local script is necessary but not sufficient; the check must be structural.
2. Immutable, uniquely named artifacts. Make the repository write-once, so an artifact cannot be replaced; name artifacts with revision and hash rather than a marketing version alone. Lock dependency versions in a committed lock file and verify embedded library versions by scanning the binary at build time.
3. Automated change control. Enforce CCB approval through a tool gate on the release branch, so email approval cannot authorize a commit. Prohibit direct pushes.
4. Merge-back discipline. Require hotfixes to be merged to trunk within a defined window, with a report of unmerged commits on release branches.
5. Configuration audits. Perform a physical configuration audit (artifact versus manifest) and a functional configuration audit before every release, and record the evidence.

Effectiveness measures for the run-up to submission:

• 100% of builds independently rebuilt with matching hashes, with zero unexplained mismatches.
• Zero commits to release branches without a recorded CCB approval.
• Zero unmerged hotfix commits on any release branch older than the defined window.
• Audit of the previous three releases' as-built records to confirm the new process can answer the auditor's question for historical deliverables as well as future ones.