Trigger Noise, Shared Secrets and Silent Failures Across Nine Services
Vela Media has 40 developers, 12 repositories in OCI DevOps code repositories, 9 build pipelines and one shared production deployment pipeline, all in the Frankfurt region. Three problems surfaced in one sprint review. First, every push to any branch of web-frontend starts a build, and a docs-only commit once triggered 9 build runs; feature-branch builds also push to the same OCIR repository and overwrite latest. Second, the production database password is typed into a build stage as a plain environment variable so the migration stage can reach the DB; it is rotated in OCI Vault every quarter, but the pipeline still carries the old value and fails after each rotation. Third, a build failed at 02:00 on a Friday and nobody noticed until customers reported a broken feature on Monday. Constraints: build runners have no internet egress, six named engineers must be able to approve production, and audit wants one year of evidence of who approved which deployment.
1. Redesign the triggers so builds start only when they should, without breaking feature-branch validation.
Read the worked answer
The symptom points at triggers attached too broadly. Nine build runs from one docs commit means either a project-level trigger or one shared trigger feeding multiple pipelines. Each pipeline needs its own trigger bound to its own repository and branch pattern.
Design:
- One trigger per repository, filtered to the branches that matter:
mainandrelease/*for pipelines that publish, plus a separate trigger for feature branches that runs tests only. Branch filtering is the reliable lever; file-path include/exclude filtering exists on push triggers, and where it is not granular enough, put a cheap first stage in the build spec that inspects changed paths and exits early or sets a skip output variable that gates later stages. That is far cheaper than running a full nine-stage pipeline for a README edit. - Path-filter the heavy pipelines (services, migrations, Dockerfiles) and let documentation changes hit a lightweight lint job at most.
- Separate build from publish. Feature-branch builds should run unit tests and stop; if they must publish, push to a branch-scoped repository path with
$BRANCH-$COMMIT_SHAtags, never to the repository prod resolves. - Keep pull-request validation triggers distinct from push triggers, so PR checks cannot deliver artifacts or trigger a deployment.
Verify the result by pushing a docs-only commit and confirming zero heavy builds, then a code commit on a feature branch and confirming tests run but no publish occurs.
2. How do the migration stage's credentials work given that the runners have no internet egress?
Read the worked answer
Separate the credential problem from the connectivity problem - they have different answers.
Credential: move the password into OCI Vault, declare a build pipeline parameter of type Vault secret, and reference it in the build or deployment stage as an injected environment variable. Grant use vaults and read secret-bundles to the DevOps pipeline's principal; do not grant the runner dynamic group anything on Vault, so the runner cannot read the secret on its own. Rotate by creating a new secret version: the next run resolves the new version by OCID, which is exactly what the console-typed variable cannot do.
Connectivity: no internet egress is irrelevant to database traffic - the migration only needs a private route to the database. If the migration runs on a private build runner whose subnet has a DRG or peering route to the database, place it there with a narrow NSG rule on the database port only. If the database is only reachable from the application subnet, do not widen the build subnet. Instead run the migration from the application tier - a Kubernetes Job in the OKE cluster that already holds the route, invoked by a deployment stage, or an instance in the app subnet via the DevOps agent. The pipeline orchestrates the migration; it should not become a tunnel.
Also split DB privileges: the migration user holds DDL rights, the application user does not.
3. Make failures visible and satisfy the approval and audit requirements for production.
Read the worked answer
Three separate mechanisms, often confused.
Notifications: create an OCI Notifications topic in Frankfurt, subscribe the on-call email and an HTTPS webhook to the paging tool, then attach the topic to the DevOps project's notification rules for build and deployment pipeline state changes - explicitly FAILED, optionally SUCCEEDED for prod only. A topic with no rule attached stays silent, which is the most common cause of the Friday-night blind spot. Route build failures to the owning team's topic and prod deployment failures to the on-call topic so the pager only fires for things that wake people up.
Approvals: an approval stage between the artifact promotion and the prod deploy stage, with an IAM group containing exactly the six engineers and permission to act on the project. Each approval is recorded per stage per pipeline run with approver identity and timestamp - that record is the audit evidence, not an email thread.
Audit retention: OCI Audit's default retention is far shorter than a year, so a one-year requirement means exporting audit records and pipeline run history to Object Storage with a Service Connector Hub and applying a retention or lifecycle rule there. Enable persistence of build and deployment logs to a logging bucket as well, and confirm the export runs before audit asks for the evidence - a retention rule cannot recover events that were never shipped.