Shared Uploads Across an Auto-Scaled Web Tier
PhotoNest runs eight web servers behind a load balancer, Auto Scale min 8 max 16, in KR1. Each server has a 2 TB Block Storage volume mounted at /var/www/uploads. Users upload about 1.5 TB per month; total uploads are 6 TB, and each volume is 75% full. After a refresh, users often see broken profile images; application logs show the request was served by a different instance. A nightly cron on each server copies its local uploads to Object Storage with aws s3 sync; restores are manual and take hours. The team wants to stop the missing-image incidents, keep the current application code (which uses local filesystem paths), and reduce backup risk. The app servers are spread across two availability zones. Budget is tight, and the team has no Kubernetes experience.
1. Diagnose the missing images and compare NAS and Object Storage for this workload. Which would you choose first, and why?
Read the worked answer
Each server writes to its own Block Storage volume. The load balancer has no session affinity, so a later request can hit a server that never received the file. Auto Scale makes it worse because new instances start with empty volumes. NAS is the lower-risk first move: it provides a shared NFS mount, so the existing code that writes to /var/www/uploads works unchanged. Object Storage is more durable and scales beyond a single NAS, but the app would need SDK or presigned-URL changes and cannot use ordinary file operations. Choose NAS first, but confirm it is reachable from both AZs; if NAS is AZ-scoped, either pin the web tier to one AZ or use one NAS per AZ with a routing strategy. Do not keep per-server cron sync as the primary consistency mechanism.
2. Design the migration with no downtime and a rollback path.
Read the worked answer
Create the NAS in the same VPC and confirm mount targets and security-group rules for NFS. Mount it on one canary web server and copy the existing 6 TB with rsync while the old volume remains the source of truth. Change the canary's upload path to the NAS mount, test upload, read, and failover across instances, then drain and roll the remaining servers one at a time behind the load balancer. Keep each old Block Storage volume for at least seven days as rollback. Do not delete local volumes until a full week of writes has been verified on NAS and a restore test passes. If the app caches file paths, clear caches during each rollout.
3. What backup and retention design would you use, and what cost trade-off matters?
Read the worked answer
Do not rely on eight independent cron syncs; they duplicate data and have no restore SLA. Use one scheduled job from a single host or a NAS snapshot to Object Storage, enable object versioning, and apply a lifecycle rule to move objects older than 90 days to Archive Storage only if the business accepts the archive retrieval time. Test a restore of 100 random files monthly and record the restore duration. Cost trade-off: standard Object Storage for the recent 90 days is fast but costs more; Archive Storage is cheaper but slow to retrieve, so it cannot be the only copy for active profile images. Keep seven days of old Block Storage volumes for rollback, then release them.