StreamLoop's Multi-Tenant Transcoding Bottleneck

The evidence points to the database and job-dispatch design, not to transcoder compute.

The database is at 92% CPU with 8,000 connections. p95 latency rises 9.5x (4 to 38 minutes) while the autoscaling group oscillates between 40 and 400 instances. That oscillation is a feedback loop: jobs pile up, the queue-depth alarm adds transcoders, each new transcoder opens connections and writes job-status updates to the same primary, contention worsens, jobs complete more slowly, the queue stays deep, and the group keeps scaling until a cooldown reverses it.

Serverless transcoders would remove instance management, but they would increase connection churn and per-job database writes unless the state model changes. The primary remains the bottleneck. Serverless also does not create tenant fairness, and it cannot drain a 2.5 million-message backlog faster than the downstream database can accept writes.

Fix the data path first: batch status writes, use a queue with visibility timeouts and a dead-letter path, cap concurrency per tenant, and move status/read traffic off the primary. Otherwise serverless simply moves the queue from one compute platform to another.

A pragmatic target has four parts.

1. Decouple ingestion from processing: object-storage events go to a queue; workers pull at a controlled rate. The queue absorbs bursts instead of the database.
2. Scale the metadata path: partition job metadata by tenant or time (for example, hash tenant_id), move hot status reads to a cache or read replica, and batch status updates into fewer writes.
3. Isolate large tenants: dedicated queues and concurrency caps, or a cell-based model in which a tenant is pinned to a shard or cell. This protects small tenants from the three tenants generating 60% of jobs.
4. Fix autoscaling: scale on backlog per worker and target completion time, with a maximum instance cap to prevent connection storms.

Tradeoffs and sequence: sharding adds cross-tenant query complexity and rebalancing work; caching risks stale status; per-tenant queues add operational overhead and idle capacity. Start with queueing, write batching, and read replicas because they are days-to-weeks changes with high leverage. Then add tenant partitioning, which is a months-long change. Reserve cell-based isolation for the top tenants only if SLA exposure or revenue justifies the cost. A big-bang serverless rewrite would delay the database fix and increase risk.

Define fairness as a per-tenant concurrency and enqueue-rate quota, not as equal CPU.

Policy example: each tenant gets a base of 20 concurrent jobs and a burst of 60; large tenants can buy reserved capacity. Enforce with weighted fair queuing or token buckets at the queue, and with per-tenant concurrency limits at the workers. This prevents one tenant's batch upload from consuming the shared queue.

Separate throughput from availability. Publish a throughput target such as '95% of jobs for tenants within quota complete within 10 minutes,' distinct from the 99.9% availability SLA. Availability alone does not cover queueing delay, and a tenant can be technically 'up' while waiting 38 minutes.

Commercialize isolation. Offer the three large tenants a premium tier with dedicated queues or cells and a higher price. That funds the isolation and aligns price with consumption. Measure per-tenant p95 latency and backlog age, and report them monthly.

Do not promise a per-tenant latency SLA yet. Until the database and queue are decoupled, StreamLoop cannot control contention on the shared primary. Publish quotas first, measure, then commit to a latency SLO once the architecture supports it.