Now Assist Readiness With Stale Knowledge and PII
A telecom company handles 18,000 incidents per month in ServiceNow. It owns Now Assist for ITSM and wants incident summarization and resolution-note generation for all 140 agents on day one. The knowledge base has 1,200 articles: 40% have not been reviewed in 18 months, 15% are duplicates, and only 55% have an owner. About 8% of incident short descriptions contain customer phone numbers or account IDs. Resolution notes are missing on 60% of closed incidents. The knowledge manager spends 80% of her time writing articles and has no review backlog process. CSAT is 78%, average handle time is 14 minutes, and 12 team leads review quality only monthly. Security has not approved any AI data flow, and the works council has asked for human oversight.
1. Would you approve day-one enablement for all agents? Assess readiness across data, governance, and operations.
Read the worked answer
No day-one, all-agent enablement. The data and governance conditions are not ready.
- Knowledge grounding: 40% stale, 15% duplicate, and 45% without an owner means retrieval can surface outdated or conflicting guidance. Now Assist outputs are only as good as the knowledge and records it grounds on.
- Resolution notes: 60% missing means the system has a weak corpus for note generation and summarization patterns. Do not treat missing notes as negative examples or synthesize them silently.
- PII: 8% of short descriptions contain phone numbers or account IDs. Enable redaction and DLP and define what may be sent to the AI service before any pilot.
- Approvals: Security and the works council have not approved the data flow or human-oversight model. Those are gates, not paperwork.
- Recommendation: Pilot with 10–20 agents and one or two incident categories. Use human review, suggestion-only mode, and stop criteria. Clean and assign ownership for the top knowledge articles first.
2. What governance controls and metrics would you set for the pilot?
Read the worked answer
Set a governance charter with named owners: AI product owner, knowledge owner, security and privacy, service desk operations, and works council liaison.
Controls:
- Data classification and redaction before inference; block fields that are not approved.
- Human-in-the-loop: agents must accept, edit, or reject generated summaries and notes; no auto-publish to the customer.
- Logging and audit of prompts and outputs, retention limits, and access controls.
- Hallucination and quality review on a sampled basis; a documented escalation path.
- Opt-out and feedback mechanism for agents.
- Change control for model or feature updates. Metrics:
- Acceptance and edit rate of summaries and notes.
- Average handle time and after-call work.
- CSAT and quality audit scores.
- Resolution-note completeness.
- Knowledge freshness and retrieval accuracy.
- PII incidents, with a target of zero. Set pre-agreed stop thresholds.
3. How would you use the 60% missing resolution notes as a signal without making the model learn bad habits?
Read the worked answer
Missing resolution notes are a data-quality gap, not a label to invent.
- Do not train on blank notes as if they meant “no resolution” or generate notes from incomplete records without review; that teaches the model to produce plausible but unsupported text.
- Build a gold set from incidents that have high-quality, reviewed notes. Use those for evaluation and, if appropriate, tuning.
- Separate the two use cases: summarization can work from the incident thread even when the final note is blank; note generation should be suggestion-only until quality improves.
- Change the closure process: structured resolution fields, a required minimum note, and team-lead review. That improves both the human record and the AI corpus.
- Track note completeness and edit rate by team; do not let the AI become a way to avoid documenting the resolution.
- Revisit enabling note generation broadly only after completeness and quality thresholds are met.