RAG on a Rusty Data Foundation
Arcline Industrial makes heavy equipment and runs nine plants. Its maintenance knowledge base contains 380,000 documents accumulated over 22 years: scanned PDFs, revision-controlled manuals, service bulletins, and tribal notes in six languages. About 40% are duplicates or superseded, and only 12% have usable metadata. A ten-week GenAI pilot built an on-premises retrieval-augmented assistant for 300 technicians. In evaluation, 38% of answers were rated 'not useful,' and 22% cited an obsolete revision. The pilot had no golden question set; success was judged by demo applause. The CISO requires plant- and role-based access control, and data cannot leave the plant network. You have $600,000, one data engineer, two contractors, and six months. The executive sponsor wants a company-wide 'ChatGPT for maintenance' demo at the annual meeting in five months. The VP of operations wants to expand to three more plants in Q3.
1. Diagnose the failure modes-separate content, retrieval, and evaluation problems-and say which explain the obsolete citations.
Read the worked answer
Three failure modes are being conflated. Content: 40% duplicates and superseded revisions mean the corpus contains contradictory truth. Retrieval: scanned PDFs chunk poorly, and without metadata the retriever cannot filter by plant, equipment, revision, or effective date. Evaluation: demo applause is not a measurement, so nobody knows whether the system retrieves the right document.
The 22% obsolete-citation rate is primarily a content and retrieval problem, not a model problem: if the top-k passages include a 2019 bulletin and a 2023 revision, a fluent model will cite the wrong one. The 38% 'not useful' rate likely combines poor recall, scanned-text errors, and answers that are technically correct but unusable on the floor. Access control is a fourth gap: without plant and role metadata, you cannot enforce the CISO's requirement, and a demo that ignores it creates a security debt.
Diagnose with a stratified sample: take 100 real technician questions, hand-label the correct current document, and measure retrieval recall@5, citation accuracy, and answer usefulness separately. That tells you whether to invest in OCR, metadata, chunking, or the model. Do not retrain or swap models first; the evidence points to the data layer.
2. Build a six-month plan with sequencing, budget, and staffing; state what you will defer and why.
Read the worked answer
Six months, $600k, one data engineer, two contractors. Month 1: pick three maintenance domains at two plants, not nine. Build a 150-question golden set with technicians and a document taxonomy with revision authority. Month 2: OCR and dedupe the top 20,000 documents in those domains; create metadata fields-plant, equipment class, revision, effective date, supersedes, language, access group. Months 3–4: rebuild ingestion and retrieval with metadata filtering and ACL enforcement; require every answer to cite a current, effective revision and show the document date; add a confidence threshold and a 'no answer' path. Month 5: run a controlled evaluation with 50 technicians; measure accuracy, obsolete-citation rate, time-to-answer, and trust. Month 6: decide on expansion to two more plants.
Budget: about $200k for OCR/dedupe and content remediation, $150k for contractors, $100k for infrastructure and vector store, $100k for technician evaluation time, $50k contingency. Defer multilingual support, all nine plants, and the company-wide demo. The sponsor's five-month demo can be a scoped demo of the two-plant, three-domain assistant with evaluation results and limitations-not 'ChatGPT for maintenance.' Expansion to three more plants in Q3 is not supportable; each plant adds content remediation, not just infrastructure.
3. Define go/no-go metrics and how to handle the sponsor's demo request.
Read the worked answer
Go/no-go metrics: retrieval recall@5 on the golden set at least 90%; answer accuracy at least 85%; obsolete-citation rate below 2%; access-control test pass 100%; median time-to-answer under 30 seconds; technician-rated usefulness at least 4/5; and a documented 'no answer' behavior when confidence is low. Also track coverage-what percentage of real questions the corpus can support-because high accuracy on a narrow set can hide poor coverage.
The demo request should be handled by reframing, not refusing: tell the sponsor the annual meeting can show a verified assistant for two plants and three domains, with before-and-after metrics and a clear statement of what is not yet covered. That is more credible to the board than a nine-plant demo that cannot survive a technician's question. If the sponsor insists on the broader claim, escalate the risk in writing: obsolete citations in maintenance can create safety and warranty exposure. Offer a recorded, scripted demo only if it is labeled as such.
The decision to expand should be made after the Month 5 evaluation, by the sponsor plus operations and the CISO, using the metrics above. Make the sponsor's success metric 'verified answers,' not 'countries'.