One Key, Four Years, Seven Years of Tape

Ranking, by likelihood-times-impact:

1. Key in the deployment manifest and in Git history. This is the top item because it converts every other control into decoration: anyone with repo read access, CI access, or a copy of the manifest has the decryption key. It is also the most likely to have already happened - repositories get cloned, forked, and exported.
2. Unencrypted backups on NAS and tape. Backups are the highest-value target in any ransomware or insider event and they have no protection at all. Seven years of tapes means the exposure window is effectively permanent.
3. One key across 1,400 tenants. No cryptographic isolation: one tenant's offboarding cannot be honoured, and one key compromise is a 1,400-tenant breach. It also makes the contractual 30-day shredding requirement impossible to meet.
4. No rotation in four years. Not because AES-256 ages badly, but because there is no rotation capability, no re-wrap path, and no evidence the key was ever handled under a controlled process.
5. Non-validated crypto module. Real, but a compliance gap rather than an exposure - it becomes the gate on the new contract, not the biggest risk.

Single highest-value change: get the key out of the code and into a KMS/HSM with access-controlled, audited, per-tenant key custody. That one move fixes items 1 and 3 and creates the mechanism for 4. Encrypting the backups is a close second and cheaper - do it in parallel, because a key-management program doesn't protect data that is sitting in plaintext on tape in the meantime.

Sequence to argue for: rotate the exposed key now (treat it as compromised, because it is in Git history), encrypt the backups and the NAS, then migrate to per-tenant keys via a re-encryption job, then retire the old key.

Hierarchy:

• Root KEK in an HSM/KMS with FIPS 140 validation, never exported, used only to wrap other keys.
• Per-tenant KEK (one per clinic), generated in and wrapped by the KMS, never present in application config or the repo.
• Per-tenant DEKs (or per-tenant-per-table) generated by the application or the KMS, wrapped by the tenant KEK, stored alongside the ciphertext as a wrapped blob. AES-256-GCM with a unique nonce per record; nonce reuse is the failure mode to design against.
• Encryption context bound to tenant ID so a ciphertext from tenant A cannot be decrypted under tenant B's key path even by mistake.

At offboarding (within 30 days):

1. Export obligation (if the contract has one) completed and verified.
2. Delete all wrapped DEK blobs for the tenant and schedule destruction of the tenant KEK in the KMS/HSM - including from key backups, which is the step everyone forgets. If the KMS keeps key backups with a retention period, that period is your shredding date; set it accordingly.
3. Record the destruction: key ID, tenant, timestamp, operator, and KMS audit-log extract.
4. Retain only the certificate of destruction, never the key.

Why deletion is sufficient: the ciphertext remains in the tables and in backups, but the only material that can unwrap it no longer exists anywhere. Without the KEK, the wrapped DEKs are random bytes; without a DEK, the columns are indistinguishable from random. This is why crypto-shredding works on immutable media - you are not required to find and erase every copy of the data, only every copy of the key. The reasoning depends on the key truly being destroyed: if the KEK exists in three KMS replicas, a tape catalogue, or a screenshot, the shredding did not happen. That is also why the evidence is the audit log, not the deletion request.

The corollary to argue in the design review: backups inherit the property. Tapes written before the tenant was offboarded become unreadable when the key is destroyed. That is an argument for per-tenant keys reaching back as far as retention goes - which is why the re-encryption of historical data matters, and why tapes written under the single shared key cannot be shredded at all.

You cannot re-encrypt what is in a vault you don't control, and recalling three years of LTO-8 for re-write is usually not fundable. So the decision is a risk decision, and it must be recorded as one.

Worth quantifying before deciding:

• Do the tapes actually contain the sensitive columns, or only the encrypted-and-now-plaintext-password-hashed form? Logical backups of an application that encrypts at the column level will contain ciphertext for those columns - the tape is not plaintext PII. Verify this by restoring one tape to an isolated environment and inspecting, rather than assuming in either direction.
• Who physically holds them, under what access control, and is the vault provider's chain of custody and personnel vetting documented?
• What does the customer contract and any regulator require for data at rest in archive?

Then the plan:

1. Going forward: tape encryption with KMS-managed keys, keys escrowed separately from the tapes, restore tested. Non-negotiable and cheap.
2. Existing tapes: if the inspection shows ciphertext for the sensitive columns, the residual risk is the key, not the data - which sharpens the urgency of rotating the exposed key and retiring it, and makes the tapes a lower priority than they first appear. If the inspection shows plaintext, escalate: pull the tapes from the vault and destroy the ones outside the legal retention obligation, and for those inside it, document a risk acceptance signed by the data owner with a destruction schedule.
3. Shrink the problem: apply retention minimisation to the seven-year policy. Seven years of monthly fulls is a business and legal decision, but the security team should force the question - most of those tapes exist because nobody asked. Cut what's defensible; every tape removed is a control you don't have to compensate for.
4. Test a restore from an encrypted tape before declaring the program done. An untested encrypted backup is a hypothesis.

The judgement call being tested: not 'encrypt everything retroactively', but 'determine what is actually on the media, then spend on the media that matters and destroy the rest'.