The Health Check That Passed Too Soon
An automated deployment script for orders-api restarts the service with systemctl restart orders-api, then polls http://127.0.0.1:8080/health with a 2-second timeout. It tries three times and sleeps 30 seconds between attempts. If any attempt returns HTTP 200, it marks the deployment successful; otherwise it fails the pipeline. In the last 200 CI runs, 36 failed the health check even though the service was serving traffic within four minutes. Separately, 5 runs were marked successful but the service returned 503 to real users for about 90 seconds because database migrations were still running; /health returns 200 as soon as the web process binds, while /ready returns 503 until migrations finish. The unit tests mock requests.get to return a 200 immediately, so they always pass. The pipeline has a 10-minute job timeout. Operators want fewer false failures without hiding real outages, and they want a test that would have caught the 503 window.
1. Explain both failure modes in terms of the polling design and the choice of endpoint.
Read the worked answer
The false failures come from too few attempts with too coarse a delay. Three attempts with 30-second sleeps give a roughly 90-second window, while the service legitimately needs up to four minutes. A 2-second request timeout can also fail under load even when the process is healthy, and a single timeout is treated like a hard failure.
The false successes come from treating /health as readiness. /health reports that the web process is alive; it does not report that migrations are complete. The script marks success on the first 200 and stops polling, so it never observes the 90-second user-facing 503 period on /ready. The unit test mocks an immediate 200, so it exercises neither the retry loop nor the 503-to-200 transition. It is testing the mock, not the deployment gate.
2. Redesign the readiness gate: endpoint, retry/backoff, deadline, and failure classification. Keep it within the pipeline timeout.
Read the worked answer
Poll /ready, not /health, with a total deadline shorter than the job timeout, for example eight minutes. Retry connection errors, timeouts, 500, and 503. Fail fast on 401/403/404 because those indicate a configuration or routing error that retries will not fix. Use a short interval with capped backoff rather than a fixed 30-second sleep.
import time, requests
def wait_until_ready(url, deadline_s=480, base=2, cap=10):
start = time.monotonic()
delay = base
last = None
while time.monotonic() - start < deadline_s:
try:
r = requests.get(url, timeout=5)
last = r.status_code
if r.status_code == 200:
return True
if r.status_code in (401, 403, 404):
raise RuntimeError(f'non-retryable readiness status {r.status_code}')
except (requests.ConnectionError, requests.Timeout) as exc:
last = repr(exc)
time.sleep(delay)
delay = min(delay * 2, cap)
raise TimeoutError(f'not ready after {deadline_s}s; last={last}')
On timeout, collect systemctl status, recent journal lines, and the last readiness response so the failure is diagnosable. Do not mark success on liveness alone.
3. Write a pytest test that would have caught the premature success. It must not sleep for 90 real seconds.
Read the worked answer
Inject the HTTP call and the clock, or monkeypatch requests.get and time.sleep/time.monotonic. Simulate the real sequence: 503 for several polls, then 200. Assert that success is returned only after the ready transition and that the loop does not return early.
def test_waits_for_ready_transition(monkeypatch):
statuses = iter([503, 503, 503, 200])
calls = []
class Resp:
def __init__(self, code): self.status_code = code
def fake_get(url, timeout):
code = next(statuses)
calls.append((url, code))
return Resp(code)
now = [0.0]
def fake_sleep(s): now[0] += s
def fake_monotonic(): return now[0]
monkeypatch.setattr(requests, 'get', fake_get)
monkeypatch.setattr(time, 'sleep', fake_sleep)
monkeypatch.setattr(time, 'monotonic', fake_monotonic)
assert wait_until_ready('http://127.0.0.1:8080/ready') is True
assert [c[1] for c in calls] == [503, 503, 503, 200]
assert calls[0][0].endswith('/ready')
Add a timeout test where statuses stay 503 and the fake clock advances past 480 seconds, asserting TimeoutError. Add a 404 test asserting immediate failure, so the gate does not retry a misconfigured path for eight minutes.