I85PREP
⌘ K
Alibaba SysOps Administrator study materials
Alibaba SysOps Administrator study resources

Rolling Deployment via ROS and Cloud Assistant Without Downtime

Alibaba SysOps Administrator · Put your knowledge to work in context.

All case studies

Rolling Deployment via ROS and Cloud Assistant Without Downtime

A team deploys a Java service to 12 ECS instances behind an SLB using a manual process: SSH to each host, stop the service, replace the JAR, start it, and verify. A release takes ~50 minutes and causes partial outages because the SLB continues to send traffic during restarts. They want to move to infrastructure as code with ROS and Cloud Assistant, keep the same instances, and achieve zero-downtime releases with automatic rollback. Health check on the SLB is HTTP /health with a 5s interval and 3 healthy / 2 unhealthy thresholds; the app takes ~40s to become healthy after start.

1. Design a zero-downtime release flow using Cloud Assistant and SLB, and explain the ordering constraints.

Read the worked answer

Flow per batch (e.g., 3 instances at a time, 4 batches):

  1. Deregister the batch from the SLB backend set (or set weight to 0) and wait for connection draining to complete so in-flight requests finish.
  2. Deploy via Cloud Assistant RunCommand (or a command invoked by ROS): stop service, back up the current JAR, replace it, start service.
  3. Wait for health: poll the local /health endpoint (and confirm the instance reports healthy to the SLB) for up to a timeout longer than the ~40s startup, e.g., 180s. Do not proceed until healthy.
  4. Re-register the batch in the SLB and verify health-check status is healthy before moving to the next batch.
  5. On failure at any step, roll back that batch (restore the backed-up JAR, restart, re-register) and halt the rollout.

Ordering constraints: never deploy to a batch that is still receiving traffic; never re-register before the app is healthy; keep at least (total − batch size) instances serving at all times. The SLB unhealthy threshold (2 × 5s = 10s) is shorter than app startup, so rely on explicit health polling rather than SLB detection alone.

2. Describe how you would express this in ROS and what safeguards and observability you would add.

Read the worked answer

ROS design:

  • Model the scaling group, SLB, and related resources in a template so the environment is reproducible; use update policies / rolling update settings on the scaling group where supported to control batch size and pause between batches.
  • Use Cloud Assistant commands invoked by ROS (e.g., via ALIYUN::ECS::RunCommand or an invocation resource) to perform the deploy steps on instances, with parameters for artifact URL and version.
  • Store the artifact in OSS and reference a versioned object; keep the previous version for rollback.
  • Use stack outputs and parameters for version and batch size so a release is a parameter change, not manual SSH.

Safeguards:

  • Pre-flight: verify artifact checksum and that the target instances are healthy before starting.
  • Concurrency limits and a failure threshold that triggers automatic rollback of the stack or the batch.
  • Timeouts on health polling; abort and roll back if exceeded.

Observability:

  • Cloud Assistant invocation records and exit codes shipped to Log Service; alert on non-zero exit codes.
  • CloudMonitor alerts on SLB unhealthy backend count and 5xx rate during the rollout.
  • Emit a release marker to logs so latency/error dashboards can be correlated with the deployment window.