Slow Bulk Ingestion via HTTP API With Duplicate Detection
A team ingests 250,000 assets nightly via the Assets HTTP API into /content/dam/imports/YYYY-MM-DD. Each night the job takes 6+ hours and sometimes times out. Logs show many 409 Conflict responses and repeated workflow executions. Assets are named by SKU; some SKUs are re-sent because the upstream system is not idempotent. The instance runs the default DAM Update Asset workflow, which includes metadata extraction and thumbnail generation. The window is 4 hours.
1. Diagnose the main causes of the long runtime and the 409s.
Read the worked answer
- 409s indicate the target node already exists. If the job re-sends SKUs, the API returns conflict for existing paths; a naive client may retry or log noisily. The upstream non-idempotency means duplicates are expected.
- Repeated workflow executions: if the client deletes and recreates, or if each retry triggers processing, the DAM Update Asset workflow runs multiple times per asset, multiplying metadata extraction and thumbnail cost.
- Throughput ceiling: 250k assets with full workflow (extraction + renditions) in a 4-hour window is aggressive; the default workflow is heavy. Timeouts suggest the batch is too large per request or the workflow queue saturates.
Confirm by sampling logs for 409 frequency, counting workflow instances per SKU, and measuring per-asset processing time.
2. Propose a redesigned ingestion approach that meets the 4-hour window and handles duplicates safely.
Read the worked answer
- Make the client idempotent: before upload, check existence (HEAD/GET on the asset path) and skip or update rather than blindly POST. Treat 409 as a benign "already exists" signal, not an error to retry.
- Batch and parallelize sensibly: chunk uploads (e.g., a few hundred per batch), run bounded concurrency, and separate upload from processing.
- Decouple processing: use a lighter workflow or a launcher that defers heavy steps; or process asynchronously and let the 4-hour window cover upload plus queued processing, monitoring queue depth.
- Reduce duplicate work: dedupe by SKU upstream (hash or manifest) so only new/changed assets are sent.
- Monitor: track 409 rate, workflow queue, and per-batch timing; alert if queue depth threatens the window.
This converts the job from "retry-heavy synchronous" to "idempotent, decoupled, observable."