TechDocs OOM and the Intermittent 401s
Contoso runs Backstage 1.2x on EKS with three backend replicas behind an ALB. The backend image includes MkDocs and the TechDocs local builder; 90 of 1,200 catalog entities have docs. Pods have a 2 GiB memory limit and no CPU limit. During a catalog refresh, two or three TechDocs builds start at once; pods are OOMKilled, and the ALB returns 502s for two to four minutes. After each rolling deploy, some users are logged out or see 401 responses until they clear cookies; the behavior is worse when the ALB sends them to a different pod. The team stores app-config in a ConfigMap and secrets in Kubernetes Secrets, but the auth session key is not set explicitly. They want docs to stay fresh within 30 minutes of a merge, and they cannot give the backend pods more memory because the cluster is at its quota.
1. Diagnose the OOM pattern and design a TechDocs build and publish path that keeps docs fresh without building inside the backend pods.
Read the worked answer
The OOM pattern points to the local TechDocs builder running MkDocs inside the backend process or pod. A catalog refresh can enqueue several docs builds, and each build spikes memory; with a 2 GiB limit and no CPU limit, the kubelet kills the pod. Move builds out of the request path: set techdocs.builder: external, keep the publisher as S3 or another object store with bucket, region, and IAM configured, and have CI run techdocs-cli build plus techdocs-cli publish for changed docs on merge. The backend then only serves the published site, so pod memory stays flat. Trigger the CI job on path filters for docs/ and mkdocs.yml; a merge-triggered job meets the 30-minute freshness target. Keep the local builder for developer machines. Also set CPU requests and limits so a build cannot starve the event loop, and test a docs-only repository end to end.
2. Why do users see intermittent 401 responses after a rolling deploy, and what configuration and operational checks fix it?
Read the worked answer
With three replicas and no explicit session key, each process can generate its own signing secret. A cookie signed by pod A is rejected by pod B, producing intermittent 401s; sticky sessions only hide the problem until a pod rolls. Set the auth backend's session signing secret explicitly and inject the same value from one Kubernetes Secret into every replica. In current releases this is the auth.session.secret setting; older deployments may use backend.auth.keys, so confirm against your version. Verify the secret is present before startup, fail fast if it is missing, and do not log it. After changing it, expect existing sessions to be invalidated; roll all pods together and communicate a one-time sign-in. Check clock skew and ingress cookie settings only after the shared key is fixed. Sticky sessions are not a fix because they break during rolling deploys and defeat even load balancing.
3. The database allows 100 connections. Each backend pod opens up to 20. What happens if the deployment scales to five replicas, and how would you scale safely?
Read the worked answer
At five replicas with 20 connections each, the theoretical ceiling is 100, which equals the database limit before migrations, health checks, or admin sessions. You will see intermittent connection timeouts and failed catalog processing, not a clean failure. Lower the per-pod pool, for example to 10 or 12, add PgBouncer if you need many replicas, and monitor active connections and wait time. Do not scale the backend purely for catalog throughput; check whether the bottleneck is processing, TechDocs, or API latency. Also account for rolling updates overlapping old and new pods, which temporarily exceeds the steady-state count. Set connection acquisition timeouts so requests fail fast rather than hanging, and load-test at the target replica count before the next incident.