Nightly Pipeline Works When Run by Hand, Fails at 02:00

Divergence: a manual run executes with the credentials of whoever clicked Run - a currently valid user with the right policies. A scheduled run executes under the identity captured when the schedule was created, which here is the contractor's user. When that account was deactivated or its keys rotated, every scheduled execution lost its authorization path while every manual execution kept working. The error surfaces at step 1 because that is the first call needing Object Storage.

Target identity: run the pipeline under a resource principal. Create a dynamic group whose matching rule covers pipeline runs and job runs in the project compartment, grant it read on the transactions bucket, read on the model catalog, and the specific deploy permissions required. Recreate the schedule under a service-owned identity that the platform team controls, and document who owns it.

Remove the credential file as a dependency: code must not read ~/.oci/config on a job or pipeline. Use the resource principals signer inside the step code, and for the Autonomous Database use a Vault secret retrieved through the workload identity. Then re-test with a manual run started by a different user to prove the outcome no longer depends on who pressed the button.

Move the secret: store the database password (or wallet plus password) as a secret bundle in OCI Vault in a dedicated compartment. Reference it from pipeline step configuration by secret OCID, and retrieve it at runtime with the workload's resource principal. Nothing in notebook home directories, nothing in the git repo, nothing in step environment variables that end up in logs.

Access boundary: grant the pipeline's dynamic group read secret-bundles on that specific vault or secret, not on the tenancy and not to the human data science group. Keep secret administration (create, rotate, delete) with a separate group. That way a compromised notebook cannot read the production database credential just because it lives in the same project.

Rotation and cleanup: rotate the credential now, since it sat in ~/.oci/config on persistent block storage and the contractor's access ended last month - assume it is exposed. Delete the file and any notebook copies. Define a rotation cadence and an alarm on secret retrieval failures so a rotation does not silently break the 02:00 run.

Trade-off to state: referencing a secret OCID in pipeline config makes the pipeline non-portable across tenancies or regions. Accept that, and keep those OCIDs in the pipeline definition generated from source control rather than hand-edited in the console.

Three gaps, in order of severity:

1. The gate did not gate. Either the condition was evaluated on training AUC rather than validation AUC, or the deploy step ran unconditionally after the condition, or the deploy target was updated by a step outside the condition. Recompute the metric in the pipeline from a held-out, versioned snapshot and make the step fail the run when the threshold is missed - a skipped step that still lets later steps proceed is not a gate.
2. Nobody was told. Add notifications on pipeline run failure, on gate rejection, and on every deployment, addressed to a rota, not to an individual. Emit the deployed model's version and validation metric as a Monitoring metric with an alarm on regression.
3. The bad version stayed live. Keep the previous catalog version and deployment intact, automate rollback to the last known-good version when the post-deploy check fails or when no metric arrives within an hour, and test the rollback path rather than documenting it.

Also close the human loop: finance consumes at 06:00, so the run must finish and either deploy or explicitly report 'no deployment' well before that. Move the schedule earlier, or add a hard deadline step that alerts if the pipeline has not reached a terminal state by 04:30. The failure was not only a bad model, it was two days of silence with a downstream consumer.