The API Import That Silently Dropped 12% of Records
A B2B company pushes 250,000 contact records per day into Adobe Campaign via the SOAP API (xtk:session login, then write/update calls) from an external CRM. The integration batches 5,000 records per call. Recently the CRM team added a new field, jobTitle, and the daily sync began returning intermittent errors that the middleware logs as 'success' because it only checks the HTTP status. The Campaign instance shows about 220,000 records landing per day - a consistent ~12% shortfall. Records with very long free-text notes fields (over the field's declared length) are among the missing. The middleware has no retry logic and no reconciliation report. Compliance requires that every CRM contact be present in Campaign within 24 hours.
1. Explain the likely cause of the 12% shortfall and why the middleware believes the sync succeeded.
Read the worked answer
The shortfall is consistent with per-record failures inside a batch that returns an overall success at the transport layer. Adobe Campaign's write/update can reject individual records (e.g., a notes value exceeding the schema's declared string length, or a jobTitle that doesn't map to a declared field) while the API call itself completes. The middleware checks only the HTTP status, so a 200 with a partial-failure payload reads as 'success.'
Contributing factors:
- New field
jobTitle: if it wasn't added to the Campaign schema (or is spelled differently), writes referencing it fail validation for those records. - Over-length
notes: values longer than the schema's string length are rejected; those records are dropped silently from the batch's perspective. - No retry, no reconciliation: nothing surfaces the dropped records, so the 12% gap accumulates undetected.
The ~12% is the union of records failing on the new field and on over-length notes - consistent enough to look like a fixed loss rate.
2. Describe the schema, integration, and monitoring changes needed to guarantee 24-hour completeness.
Read the worked answer
Schema:
- Add
jobTitleto the recipient (or contact) schema with an appropriate type/length, and confirm the API field name matches what the CRM sends. Alignnoteslength to the real maximum (or truncate/validate upstream); document the limit in the integration contract.
Integration:
- Parse the API response body, not just HTTP status. Campaign returns per-record status; the middleware must inspect it and treat partial failures as failures.
- Add retry logic with backoff for transient errors, and route permanently failing records to a dead-letter queue with the error reason.
- Consider switching to a bulk import path (file-based import via workflow) for 250k/day rather than 5,000-record API calls - bulk import is more resilient and easier to reconcile, and is generally the recommended pattern at this volume.
Monitoring / completeness:
- Run a daily reconciliation: count of CRM records expected vs. count landed in Campaign, keyed on the CRM id; alert on any gap.
- Track dead-letter volume and surface it to ops.
- Add a workflow step that verifies the day's batch count against the CRM's expected count before marking the sync complete.
With response parsing, retry, dead-letter handling, and a reconciliation alert, the 24-hour requirement becomes verifiable rather than assumed.