Deciding on a Knowledge Graph When Search Is Failing
Northline Manufacturing has 8,000 employees and six main content sources: SharePoint, a PLM system, ERP quality records, a legacy shared drive, email, and the service desk. A search pilot across these sources shows 55% of queries end in zero clicks or a refinement, and sessions average 4.2 queries. Engineers say search cannot find documents they know exist. A vendor proposes a $1.2 million knowledge graph with entity extraction and semantic search. A sample of search logs is dominated by project codes, customer names, internal acronyms, part numbers, and 'how do I...' questions. A content audit finds 38% of documents are more than five years old, 22% are duplicates, metadata completion is 41%, and permissions are inconsistent across repositories. The CIO wants a decision in 30 days. The KM lead must advise whether to buy the graph.
1. What is the most likely diagnosis, and why is a knowledge graph not the first fix?
Read the worked answer
The diagnosis is a search quality and content governance problem, not a missing semantic layer. The query mix includes exact identifiers such as project codes and part numbers, which lexical search and metadata should handle well. 'How do I' questions need curated answers or service-desk content, not a document index. The audit points to stale content, duplicates, weak metadata, and inconsistent permissions.
A knowledge graph improves entity relationships and semantic matching. It does not create missing titles, remove duplicates, assign owners, fix permissions, or retire obsolete documents. If the graph is built over poor content, it makes poor content easier to reach.
Sequence the work:
- Instrument queries by intent: identifier lookup, document lookup, how-to, person expertise, and policy.
- Fix the top 200 high-frequency intents with synonyms, acronym expansion, best bets, and curated answers.
- Assign content owners for high-value repositories and apply retention and duplicate cleanup.
- Normalize permissions before adding any semantic layer.
- Pilot entity extraction on one bounded domain, such as part-to-supplier-to-quality-issue, and measure task success before committing $1.2 million.
2. Design a 60-day sequence before any $1.2 million commitment. Include quick wins and evidence gates.
Read the worked answer
Days 1-15: build a query taxonomy from logs and interviews. Identify the top 200 intents and the repositories that should answer them. Publish a baseline for zero-click/refinement rate and time-to-answer.
Days 16-30: deliver quick wins. Add acronym and project-code synonyms, best bets for top intents, and a curated 'how do I' answer set owned by the service desk. Fix titles and metadata for the top 200 documents. Audit permissions on the two highest-value repositories.
Days 31-45: test content lifecycle controls. Assign owners, archive obsolete documents, and de-duplicate within high-value collections. Add a feedback control so users can flag wrong or stale results.
Days 46-60: run a bounded entity-extraction pilot on one domain with a defined document set. Measure precision, recall, time-to-answer, and user confidence against the baseline. Evidence gate: a material reduction in zero-click/refinement and improved task success. If the gate is not met, do not buy the graph; continue metadata and content remediation. The 30-day CIO decision should be a decision to fund the sequence, not a purchase order.
3. How would you evaluate the vendor's claim that semantic search will improve productivity? Name measures and a counter-metric.
Read the worked answer
Do not evaluate on a demo or on relevance ratings alone. Use a task-based benchmark drawn from real work: 20 to 30 tasks such as finding the approved quality record for a part, locating the current supplier corrective action, or identifying the engineer who resolved a similar defect.
Measures:
- Task success rate with and without the prototype.
- Time to answer and number of queries per task.
- User confidence and whether the user verified the answer at the source.
- Maintenance effort: hours per month to curate entities, synonyms, and relationships.
Counter-metric:
- False-positive and overconfident wrong answers. A semantic system can return a plausible but wrong document. Also track permission leaks or near-misses.
Include total cost of ownership: licensing, integration, ontology maintenance, and content remediation. The productivity claim is credible only if task success improves without increasing wrong-answer risk or maintenance burden.