Code-Signing Key Migration and Timestamp Failure

The failure is a chain of independent verification conditions.

1. No timestamp. A code-signing certificate is valid only for a window. Verifiers accept a signature made during that window if there is trustworthy evidence of when it was made; an RFC 3161 timestamp token provides that evidence. Without a timestamp, the verifier evaluates the certificate at verification time. After 30 June the certificate is expired, so the bootloader rejects the image. The engineer disabled timestamping to work around an unreachable TSA, which removed the only proof that signing happened before expiry.
2. Pinned root and chain. The bootloader trusts a specific root public key. A new intermediate must chain to that pinned root. If the build used a chain not anchored in the pinned root, the device rejects it regardless of timestamps. The pilot rollout is consistent with a chain or timestamp problem rather than corruption, because devices entered recovery mode rather than failing to boot.
3. Clock. About 6% of devices report 1 January 1970 after RTC battery failure. If the verifier uses the device clock for certificate validity, a 1970 date is before the certificate's notBefore, so even a correctly timestamped image can fail. A secure verifier should use the timestamp, not the local clock, for signing-time validity, but that must be designed and tested.

The expired CRL adds a fourth failure mode if the bootloader checks revocation and fails closed. Recovery images must be signed and timestamped too, or recovery mode becomes a dead end.

Do not rotate the root in one step. Four thousand devices cannot be recalled, and a new root that old firmware does not trust will brick any device that receives it.

1. Root rollover by cross-signing. Create the new root and have it certified by the old root, or ship a firmware release that adds the new root to the trust store while retaining the old one. Sign that release with the old key and a valid timestamp. After telemetry shows the new root is present on the fleet, switch signing to the new chain and keep the old root trusted for verification during a defined overlap.
2. Key ceremony. Rewrite the ceremony procedure before generating the new key: dual control, split knowledge, witnessed and recorded, HSM backup and restore test, disaster recovery with a separate HSM, and a documented quorum. FIPS 140-2 Level 3 is acceptable for now, but plan the transition to FIPS 140-3 validated modules and confirm the module's certificate covers the intended use.
3. Algorithm choice. Move to SHA-256 or stronger, and RSA-3072 or ECDSA P-256 for the signing key. Keep the old public key available for verifying older firmware.
4. Build pipeline. Make timestamping mandatory and fail the build if no timestamp token is returned. Run two independent TSAs and verify the token against the signature before publishing. Never allow a flag that disables timestamping in production.
5. Rollout. Canary at 1%, then 5%, 25%, 100%, with telemetry on boot failures and recovery entries. Use A/B partitions and a signed rollback image. Test with the clock set to 1970 and to a date after certificate expiry.

TLS. The update client negotiating TLS 1.0 is a protocol and compliance problem; TLS 1.0 is deprecated and should not protect firmware distribution. Move the load balancer to TLS 1.2+ with AEAD cipher suites, and TLS 1.3 where the client supports it. For legacy devices, terminate TLS at a dedicated update endpoint that serves only signed manifests and images, require mutual authentication where possible, pin the update CA, and isolate that endpoint from the corporate network. Do not weaken the whole estate to accommodate old clients; give legacy devices a constrained path.

Crypto agility. Build an inventory of every algorithm and key length in the signing and update chain: image signature, manifest signature, TLS, HSM module, and any symmetric encryption. Define acceptable algorithms and deprecation dates, and make the verifier accept multiple algorithms during transitions. For firmware signing, plan a path to post-quantum signatures, for example LMS/XMSS hash-based signatures, which suit long-lived bootloaders, and hybrid key exchange for TLS. The migration should be testable in a lab with an old device, a clock-skewed device, and a device that has never seen the new root.