Deployment Failure Integration with Duplicate Events

Use an internal relay or worker in the staging network that consumes the Kafka topic and calls the PagerDuty Events API over an approved egress path. Do not have the production deployment tool call PagerDuty directly, and do not keep the email cron path. Create a PagerDuty service per bounded domain or use event orchestration to route by service. Use a scoped integration key or OAuth-based app, stored in a secrets manager and rotated. Send deployment records through the Change Events API so responders can correlate failures with changes, and send failure triggers through the Events API with a stable dedup key. Use webhooks to post incident status to Slack. The relay must be asynchronous so a PagerDuty outage or rate limit does not block deployments. This pattern respects the no-public-egress rule, removes shared admin credentials, and gives an audit trail at the relay and in PagerDuty.

Use a deterministic dedup key derived from the deployment ID, service, and environment, not a random event ID. The first failure sends a trigger; subsequent retries for the same deployment send events with the same dedup key, which PagerDuty deduplicates rather than creating new incidents. When a rollback or successful redeploy happens, send a resolve event with the same key. Do not use acknowledge as a substitute for resolve, and do not create a new incident for each Kafka message. The relay should be idempotent and keep a short-lived record of processed message IDs. If the Events API returns a rate-limit response, queue and retry with backoff rather than dropping events. During the month-end burst, apply local rate limiting and prioritize customer-impacting production failures over non-production deployments. Post status changes to Slack from PagerDuty webhooks, not from the deployment tool, so the incident record remains the source of truth.

Test the relay against PagerDuty unavailability, rate limiting, malformed Kafka messages, and duplicate delivery. The relay should have a dead-letter queue, a circuit breaker, and monitoring for event acceptance latency. Confirm that a PagerDuty outage cannot block deployments and that queued events expire safely rather than paging hours later. Security testing should verify scoped credentials, rotation, least-privilege routing, audit logs for every event and configuration change, and no secrets in the deployment tool. Test that a compromised service key cannot page another team's service. Run the full path in staging with synthetic failures and confirm deduplication, resolution, Slack updates, and change correlation. Define rollback to the old path only as a temporary measure, since email is the source of the duplicate problem. Finally, set alerting on relay lag and rejected events so silence is visible rather than mistaken for health.