Bot Design: Handling Unstructured Data in Customer Onboarding
A financial services firm is automating customer onboarding for small business loans. The process involves extracting data from various documents: application forms (PDF), bank statements (PDF), tax returns (scanned images), and identity documents (images). The bot must validate the data, perform KYC checks, and enter it into the loan origination system. Currently, human agents take 45 minutes per application. The firm receives 500 applications per month. The RPA team proposes using a combination of OCR, NLP, and rule-based validation. However, the documents vary widely in format and quality. The team estimates that OCR accuracy will be 85% for bank statements, 70% for tax returns, and 95% for identity documents. The validation rules are complex and change frequently due to regulatory updates. The bot must also handle exceptions gracefully. The firm wants to achieve 80% straight-through processing and reduce handling time to 15 minutes per application.
1. Given the OCR accuracy rates, what is the overall expected straight-through processing rate if the bot requires all extracted data to be correct to proceed? Assume that for each document type, if OCR fails, the application goes to human review. Also assume that the documents are independent. Calculate the probability that all documents are correctly extracted. Is the 80% STP target achievable? If not, what strategies could improve it?
Read the worked answer
Calculation:
- Probability all correct = P(application) × P(bank) × P(tax) × P(identity).
- Assume application form OCR accuracy is not given; let's assume it's 95% (typical for structured forms). So P = 0.95 × 0.85 × 0.70 × 0.95 = 0.95 × 0.85 = 0.8075; × 0.70 = 0.56525; × 0.95 = 0.5369875 ≈ 53.7%.
- So STP rate ≈ 54%, far below 80%.
Strategies to improve:
- Use advanced OCR/AI models (e.g., deep learning) to improve accuracy, especially for tax returns.
- Implement confidence-based thresholds: if OCR confidence is high, auto-accept; if low, route to human. This can increase STP if confidence correlates with accuracy.
- Add validation rules that can correct common OCR errors (e.g., using checksums, cross-referencing fields).
- Use human-in-the-loop for low-confidence fields only, rather than whole application.
- Pre-process documents (deskew, enhance) to improve OCR.
- Consider using a document understanding AI that can handle unstructured formats.
- For tax returns, consider integrating with tax authority APIs to fetch data directly.
Judgment: The 80% target is not achievable with current accuracy. The bot design should include a confidence-based exception handling strategy. Also, the 15-minute handling time target may be met for STP cases but not for exceptions. The firm should invest in better OCR or adjust expectations.
2. The validation rules change frequently. How should the bot be designed to accommodate frequent rule changes without requiring redeployment? What governance process should be in place for rule updates?
Read the worked answer
Design:
- Externalize business rules into a rules engine or configuration file that can be updated without code changes.
- Use a decision table or DMN (Decision Model and Notation) approach.
- Implement a versioned rule repository with audit trails.
- Allow business users to modify rules via a controlled interface, with testing and approval workflows.
- Ensure the bot can fetch the latest rules at runtime from a central service.
Governance:
- Establish a change advisory board (CAB) for rule changes, including compliance and business representatives.
- Require impact analysis and regression testing before deployment.
- Maintain a sandbox environment for rule testing.
- Document all changes and approvals for audit.
- Schedule regular rule review cycles.
Judgment: Frequent rule changes are a sign that the process is dynamic. The bot should be designed for agility. However, too much flexibility can lead to errors. A balance is needed: business users can propose changes, but IT/compliance must approve. Also, consider using AI to learn rules from data, but that may be overkill. The key is to decouple rules from code.