6,000 Endpoints in a Six-Hour Window: Reworking an AXL Provisioning Run and Proving It Worked
Your team must provision 6,000 new 8845 endpoints, 6,000 lines, and 300 new directory numbers on a Cisco Unified CM 14 cluster (one publisher, three subscribers, 22,000 existing endpoints) inside a six-hour Sunday change window. A contractor wrote a Python script using AXL SOAP: 40 worker threads, one addPhone per device, then a separate addLine, then updatePhone to associate the line. In a dress rehearsal against 500 devices, the publisher hit 96% CPU, some requests returned HTTP 503, 37 phones were created without lines, and 14 directory numbers ended up shared across two devices after retries. The cluster has no third-party management tooling and no budget to buy any, and the security policy forbids direct SQL access. The service provider requires evidence that all 6,000 devices are registered and that no unintended DN conflicts exist before the window closes. You must make the run repeatable and prove success.
1. Redesign the provisioning approach. Explain why the current pattern fails and what you would change.
Read the worked answer
The failure is architectural, not a tuning problem. AXL is a SOAP interface served by the platform's web service tier, and it is not designed for 40 concurrent writers against the publisher. Once request queueing exceeds the service's capacity, it returns HTTP 503, which the script treats as a retryable transient; the retries then pile more load onto an already saturated service and produce the partial state you saw. The three-step add-add-update pattern compounds this, because it triples the request count and creates a window in which a phone exists without a line. Every write also touches the cluster's configuration cache.
Change three things. First, move the bulk of the work to BAT, which exists for exactly this and generates a job report; 6,000 devices in six hours is about 17 per minute, well within what BAT handles, and it keeps the load off the interactive API path. Second, if AXL must be used for any subset, reduce concurrency to a small number of workers (start at 4 and measure), treat 503 as backpressure rather than as a transient error, and apply exponential backoff with a cap plus a hard stop that aborts the run instead of hammering. Third, collapse the object graph: create lines first, then add each phone with its line associations in the same addPhone request, eliminating the update step and the line-less window entirely.
Run the whole rehearsal again at 500 devices and watch publisher CPU and AXL response times before you touch 6,000. If the rate is not comfortably below saturation at 500, it will not survive 6,000.
2. Make the run idempotent and recoverable, including fixing the 37 line-less phones and the 14 duplicated DNs.
Read the worked answer
Start with a single source of truth: a CSV containing device name, MAC, device pool, owner, and DN, generated once and treated as immutable for the run. Everything the script does is derived from it, and every action writes a journal entry to a local file with the object name, the operation, and the outcome. That journal is what makes a rerun safe.
Before each create, query for existence with getPhone keyed on device name and on MAC, and route to update instead of add. That is the idempotency layer: a rerun after a timeout converges to the desired state rather than creating a second object. Never auto-generate a device name inside the retry path, because that is how a retry creates a new object instead of repairing the first one. Set explicit request timeouts and treat a timeout as 'unknown', never as 'failed'.
For the existing damage: the 37 line-less phones are found by listing phones from the run's device name range and checking that each has the expected line count; repair them by adding the line association with a single update, not by deleting and recreating, because recreation loses the MAC registration history and can trip dependency checks. The 14 duplicated DNs are found by listing each DN in the run range and checking for more than one device; where a duplicate is unintended, remove the extra line from the device rather than deleting the DN, then confirm the remaining association matches the CSV. Do the repair as a separate, logged pass so the audit trail shows what happened, and dry-run it against the rehearsal cluster first. A repair script that has never been dry-run is a second outage.
3. Prove success within the window: what evidence do you collect for registration and DN uniqueness without database access?
Read the worked answer
Build the evidence chain from three independent sources so no single tool can hide a problem.
Inventory and DN uniqueness come from AXL itself. Use listPhone with pagination over the run's device name range, requesting only the tags you need, and compare the returned count and the per-device MAC and device pool against the CSV. Use listLine with pattern search over the DN range to detect any DN that appears on more than one device, and reconcile against the expected shared-line exceptions, of which there should be none for this run. Both queries are read-only and cheap; rate-limit them politely, because a verification script that saturates the API creates the incident it is trying to disprove.
Registration comes from real-time data, not from configuration. Use RTMT's registration summary or the Perfmon counter for registered hardware phones on the subscribers and compare against the expected 6,000 plus the pre-existing baseline. Registration counts are a live number, so record the reading with a timestamp and record it more than once across the window; a single snapshot at 04:00 can flatter a design that is still flapping.
Finally, verify a sample end to end: place calls from a spread of the new devices across device pools, confirm two-way audio, confirm the calling number presented, and confirm CDR records exist for those calls. Then attach the BAT or AXL job report, the inventory diff, the registration snapshots, and the sample call records to the change record. That package is what satisfies the service provider, and it is also what protects you when someone asks in three weeks why one phone never registered.