Forty Services Behind One Trusted Network
An organisation runs 40 microservices in a single shared VPC. All services connect to the database with one account that holds full privileges across every schema. Twelve services read third-party API keys from environment variables defined in a Helm values file committed to Git. The architecture document states that intra-VPC traffic is plaintext HTTP because the VPC is a trusted network. A new multi-tenant feature will let customers upload files that a worker pool processes and stores. The auditor asks for the trust boundary diagram; the existing one is a single box labelled "cloud" containing all 40 services. A recent incident involved a compromised container in a low-risk service that was contained only because the attacker did not attempt lateral movement.
1. Redraw the trust boundaries and identify which existing assumption carries the most risk.
Read the worked answer
The single "cloud" box is the problem: a diagram with no internal boundaries is a statement that every component trusts every other component, and the controls have been built to match.
Draw at minimum: internet to edge, edge to public services, public services to internal services, internal services to the database tier, the worker pool as its own boundary because it processes untrusted customer files, third-party API providers as an external boundary, and the build and secrets systems as their own.
The most dangerous assumption is that the shared VPC is a trusted network. It converts one compromised container into a platform-wide event. Combined with the single full-privilege database account, any service that is exploited can read and write every tenant's data. The recent incident is the proof: containment was luck, not control. Plaintext east-west traffic also means any host that can capture traffic on the segment sees credentials and payloads.
Rank by consequence rather than by how often the issue is discussed: shared database credentials first, because the blast radius is all data; flat network trust second, because it is the path to using those credentials; secrets in a Git-tracked values file third, because it is a direct disclosure channel with a long exposure window and rotation burden.
2. Design least privilege and secrets management for the platform, with a migration that does not block the multi-tenant feature.
Read the worked answer
Sequence the work so the new feature ships on the new model rather than inheriting the old one.
Database: create a role per service scoped to its own schema and required verbs, and per-tenant row-level security for the multi-tenant tables from day one. Migrate in waves, starting with services that have narrow access patterns; the monolith-style account is retired only when the last consumer moves. Measure progress by counting services still on the shared account, not by counting roles created.
Secrets: move the 12 services to a secrets manager with short-lived, dynamically issued credentials so rotation is automatic and a leaked value has a bounded useful life. Remove the values file from Git and rotate every key in it, since removing the file does not undo the exposure in history. Add pre-commit and CI scanning so the pattern cannot return.
Network: default-deny between services, allow only declared call paths, and enable mutual TLS for service-to-service traffic rather than relying on the VPC perimeter. Start with the worker pool, because it will handle untrusted input.
For the new feature specifically: build it with tenant-scoped credentials, per-tenant object storage prefixes, and mTLS from the start. Retrofitting later is far more expensive than refusing to add a 41st service to the old model.
3. Threat model the upload-to-worker path and state the design controls you would require before launch.
Read the worked answer
The path crosses from untrusted to trusted, so model the attacker as the uploader with full control of file bytes, filename, declared content type and size.
Threats: malicious file content exploiting a parser in the worker (the highest-impact case, since parsers are the classic memory-safety and deserialization surface); archive bombs and decompression loops exhausting CPU and disk; path traversal via filename or archive entry names; content-type spoofing so a script is stored and later served back to a browser; cross-tenant access by guessing storage keys; and resource exhaustion by a single tenant degrading the pool for everyone.
Required controls:
- Store uploads in quarantine, keyed by a server-generated identifier, never by the supplied filename. Strip paths and normalise names on any download.
- Validate by content inspection, not by the declared type or extension, and allowlist the formats actually needed.
- Process in a sandboxed worker with no database credentials beyond its own tenant-scoped role, with CPU, memory, wall-clock and output-size limits, and no outbound network access.
- Enforce per-tenant quotas and concurrency caps so one tenant cannot starve the pool.
- Serve stored files from a separate domain with a restrictive content-type and content-disposition, so stored content cannot execute in the application origin.
- Log every upload and processing result with tenant and correlation identifiers, and alert on parser crashes, which are often the first sign of a fuzzing attempt.