Contoso Field Service Agent: Grounding with Unstructured Manuals
Contoso is deploying a Microsoft Copilot Studio agent to help 1,200 field technicians troubleshoot equipment. The knowledge base consists of 4,000 PDF manuals (average 80 pages each) and 12,000 support tickets exported from Dynamics 365. Technicians report that the agent frequently gives generic answers or cites the wrong manual section. The agent is configured with generative answers enabled and uses the default knowledge sources. The manuals are stored in a SharePoint library with versioning enabled, and tickets are in a Dataverse table. The project has a 6-week deadline and a strict requirement that answers must cite the exact document and page number. The IT team wants to avoid custom code if possible.
1. What is the most likely cause of the generic answers and incorrect citations, and what should you change in the agent configuration?
Read the worked answer
The most likely cause is that the agent is using default generative answers with automatic SharePoint indexing, which may not properly chunk and index large PDFs, leading to poor retrieval. Additionally, the Dataverse table may not be configured as a knowledge source with proper search indexing. The fix is to explicitly configure the SharePoint source as a knowledge source and enable 'Use generative answers' with citation. For Dataverse, create a knowledge source using the 'Dataverse' connector and ensure the table has search enabled. To improve citation accuracy, consider using the 'Generative answers' node with 'Allow the agent to answer using its own knowledge' disabled, and enable 'Show citations' in the agent settings. Since custom code is to be avoided, use the built-in knowledge sources and verify that the SharePoint library has the 'Allow items to be indexed' setting enabled. Also, ensure the agent's language model is set to a model that supports citations, such as GPT-4. Finally, test with a sample query and inspect the retrieved documents in the test pane to confirm the correct chunks are being used.
2. How would you ensure that answers cite the exact document and page number, given the constraints?
Read the worked answer
To cite exact page numbers, the PDFs must be indexed at the page level. SharePoint's native indexing may not provide page-level granularity. Since custom code should be avoided, consider using Azure AI Search with a built-in PDF parsing skill that extracts page numbers, but that requires custom code. Alternatively, use the 'SharePoint' knowledge source and rely on the citation format that includes the document name and a snippet; page numbers may not be available. To meet the requirement without custom code, you could pre-process the PDFs into individual page files (e.g., one PDF per page) and store them in SharePoint with a naming convention that includes the page number. Then the citation will show the file name, which includes the page number. This is a low-code workaround. Another option is to use the 'Document' knowledge source in Copilot Studio, which supports citations with page numbers if the document is indexed with the 'page number' metadata. Verify in the agent's test pane that citations include page numbers. If not, consider using Power Automate to split PDFs into pages and upload them to SharePoint, which is low-code. Document this approach and validate with a sample of 50 queries.