Nightly Item Master Load: Concurrency, Timeouts and Idempotency

Three of these are platform limits and two are design errors, and they need different responses.

Limits:

• Account-level concurrency. Forty simultaneous RESTlet requests exceed what the account allows, producing SSS_REQUEST_LIMIT_EXCEEDED. Dropping to 4 confirmed this.
• Per-execution governance. A call that processes 5,000 items performs 5,000 record operations plus search paging. Even at low concurrency, the largest calls can exhaust the execution budget.
• Gateway and HTTP timeouts. Long-running calls hit the request timeout and return 504 regardless of governance.

Design errors:

• Partial commits treated as atomic. Each record.update commits on its own, so a timeout mid-call leaves price updated and description stale. Nothing rolls the earlier writes back.
• Retry without an idempotency key. The retry re-sends the whole batch, and the audit log receives a second row for every item, because nothing identifies the work as already done.

Concurrency is not the root cause of the partial writes and duplicates; it only made them frequent enough to notice. Lowering parallelism treated the symptom, which is why the window now overruns. The correct framing is: small units of work, safe to repeat, and parallelism supplied by the platform rather than by 40 hand-rolled threads.

Split by volume and latency requirement, not by convenience.

Nightly bulk: Map/Reduce.

• getInputData returns the item identifiers that need updating, from a saved search or a staged file.
• map handles a small number of items per key and performs the writes, so one bad item affects one key rather than a 5,000-record call.
• reduce collects failures into a durable error record, and summarize emits counts for monitoring.
• The platform supplies parallelism within limits, which is the point: it replaces 40 uncontrolled concurrent HTTPS calls with governed concurrency.
• For a pure data load with no per-record logic, an asynchronous CSV import submitted with N/task is a legitimate alternative; choose Map/Reduce when transformation or conditional logic per item is required.

Hourly delta: keep the RESTlet. Five hundred records is well within a single governed execution. Fewer moving parts, and the near-real-time requirement is met without running a map/reduce job every hour. Reduce parallelism to 2 to 4 and keep it.

Idempotency in both paths. Use record.upsert with the PLM external ID so a repeated send updates rather than duplicates. Give the audit record a deterministic key built from batch ID plus item external ID, so a retry cannot create a second row. Store batch ID with each audit entry so a rerun is queryable.

Failure handling. Return per-item status from each unit of work, persist failed item IDs, and retry only those IDs with exponential backoff and a retry cap. Never retry a whole batch after a partial failure.

Measure before promising.

• Run a timed pilot on a representative slice, for example 20,000 items, with the intended map concurrency, and measure throughput per minute including overheads. Extrapolate, then add headroom rather than assuming linear scaling, because item complexity, search paging, and contention with normal users all degrade it.
• Establish a hard cutoff: the job stops accepting new work at a set time and reports remaining items, so it degrades predictably instead of colliding with business hours.
• Confirm the schedule does not stack. If a run overruns, the next must not start; use a lock record or a task-status check before submission.

Monitoring once live:

• Items processed, failed, and skipped per run, written to a custom record or log, not just to script logs that expire.
• Run duration trend. A creeping increase is the early warning for a window overrun.
• Error taxonomy: governance errors, concurrency errors, and data validation errors counted separately, because each implies a different fix.
• Delta lag for the hourly RESTlet, so a backlog in the small path is visible before it becomes a nightly problem.
• A reconciliation count between PLM's sent total and NetSuite's updated total, which catches the silent partial writes that started this review.