Checkout Service Group with Blind TCP Monitoring
An online ticket vendor runs a checkout service group with six servers behind one HTTP VIP. The service group uses the default round-robin method and a TCP monitor on port 8080 with interval 5 seconds, retries 3, and downtime 30 seconds. Two of the six servers are new canary nodes with half the CPU and memory of the stable nodes; one additional server is a cold standby that must receive traffic only if all six primary nodes fail. During a recent sale, one stable node lost its database connection and returned HTTP 500 for every checkout, but the ADC kept it in rotation because TCP 8080 still accepted connections. At the same time, the canary nodes received the same share of traffic as the stable nodes and their response times rose to 3 seconds, while stable nodes averaged 400 ms. A deployment also caused a canary node to be marked DOWN and then UP three times in 10 minutes because the Java app needed 45 seconds to start. The business wants canary traffic limited to 10% and no user-visible errors during future deployments.
1. Redesign the monitor. What type, send/receive strings, and timing would you use, and why is the existing monitor insufficient?
Read the worked answer
The TCP monitor only proves that something is listening on 8080. It cannot see that the application is returning HTTP 500 or that its database connection is dead. Replace it with an HTTP-ECV monitor that requests a health endpoint which performs a real dependency check, for example GET /health and expects a body such as STATUS=OK. Set interval 5 seconds, retries 3, downtime 30 seconds, and a response timeout short enough to catch a hung app but long enough to avoid false positives. Bind it to the service group so every member is checked. The receive string matters: a 200 response alone can still be a cached or generic page; the body should be produced only when the DB query succeeds. Keep a separate lightweight TCP monitor only if you need to distinguish port-down from app-down, but do not use it as the sole health signal.
2. Choose a load-balancing method and weighting. How do you cap canary traffic at about 10% and ensure the standby node receives no traffic while any primary is healthy?
Read the worked answer
Round robin ignores both capacity and current latency, so the canaries got an equal share and the standby was eligible. Use weighted least connection, or least response time once the HTTP-ECV monitor measures real app latency. With four stable nodes at weight 100 and two canaries at weight 22, canary share is 44/(400+44) = about 9.9%; adjust the weight to tune. Weights are a guardrail, not a substitute for health: if a canary fails the HTTP-ECV check it is removed regardless of weight. Put the cold standby in a separate service group or backup vserver and configure it as backup for the primary LB vserver; a backup is used only when no primary service is UP. Do not bind it as an equal member with a low weight, because low weight still receives traffic.
3. The canary app needs 45 seconds to start. How do you prevent monitor flapping during deployment without delaying detection of real failures?
Read the worked answer
Do not widen downtime to 60 seconds just to cover the 45-second startup; that would also delay detection of a real failure by a minute. Instead, take the node out of rotation before the deployment: disable the specific service or service-group member, deploy and wait for the app health endpoint to return the expected body, then enable it. If the platform supports graceful disable, use it to let existing connections drain. Keep the monitor tight (5-second interval, 3 retries, 30-second downtime) so a production failure is still detected in about 45 seconds. For canary deployments, automate the disable/enable around the deployment and verify with show service/show servicegroup that the member is disabled before restarting the app.