Channel and environment drift across four airline channels
Meridian Air publishes one watsonx Assistant to web chat, the mobile app, WhatsApp, and an IVR voice channel. The Draft knowledge base has 2,100 documents; Live has 2,040. A contractor edited the Live refund policy directly three weeks ago to say 48 hours; Draft contains the approved 24-hour policy. After a publish, web chat answers 24 hours, WhatsApp answers 48 hours, and the IVR reads markdown bullets and a URL character by character. WhatsApp splits long answers into three messages, and the voice channel times out after 12 seconds while knowledge search takes 9 seconds. The mobile app uses a custom response renderer that strips citations. The release manager wants to roll back, but there is no record of which documents changed in Live. The next peak travel period starts in 10 days.
1. Diagnose the environment and channel problems. What is the immediate rollback or recovery plan?
Read the worked answer
Two separate failures. First, environment and channel drift: the web chat channel was still bound to Draft after a partial migration, while WhatsApp and IVR were bound to Live; Live also had a direct contractor edit to 48 hours that was never promoted. The inconsistent 24 and 48 answers are therefore expected from different environments, not model randomness. Second, channel formatting: one response format is being sent to all channels. WhatsApp splits at its message limit, the IVR reads markdown and URLs, and the mobile renderer strips citations. The 12-second voice timeout against 9-second search is a latency budget failure. Immediate plan: freeze publishing and Live edits; snapshot both knowledge bases and diff document IDs and hashes; identify the contractor edit and restore the approved 24-hour policy to Live through emergency change; bind web chat to Live after verification, or temporarily route WhatsApp and IVR to a holding message with a link; do not blind-rollback without a record. For voice, shorten retrieval or add a 'let me look that up' prompt and cache the answer. Then communicate the corrected policy.
2. How should the team manage promotion, channel-specific responses, and knowledge-base versioning going forward?
Read the worked answer
Make Draft the single source of truth. No direct Live edits except break-glass with approval and logging. Lock Live roles; require release notes and a diff of documents, actions, and settings before publish. Version knowledge-base documents with effective_date and superseded_by, not filenames. Maintain a channel-environment binding register and include it in release checks. For channels, use response templates per channel: web full answer with citations; mobile keep citations in a details panel; WhatsApp concise under 300 characters, numbered segments, no markdown; voice use short sentences and SSML, no URLs, offer an SMS link. Do not let the knowledge answer format dictate the channel format. Add a promotion checklist: approved content, channel bindings, latency test, rollback owner. The contractor should work in Draft with review, not Live. Channel-specific rendering should be tested as a release gate, not fixed after complaints.
3. What tests would you require before the peak period?
Read the worked answer
Pre-peak test matrix: four channels by 20 utterances (refund, change, baggage, disruption) by Draft and Live. Check answer consistency for the refund policy, citation presence, WhatsApp segment count, IVR pronunciation and timeout, mobile citation display, and p95 latency under 8 seconds for voice and 5 seconds for text. Include a 2x peak load test and a search-latency spike. Run a canary on 5% of WhatsApp and IVR traffic with a rollback switch. Require signed content approval for the refund policy and a named on-call owner for the peak period. Freeze nonessential changes 48 hours before peak. Measure containment, repeat contacts, and complaints by channel, not just aggregate. The release should be blocked if the 24-hour policy answer is inconsistent across channels or if voice p95 exceeds the timeout budget.