The Admission Chatbot With 1,100 Logged Queries
A government senior secondary school in Barnala wants a website assistant so the office clerk is not on the phone all day answering admission queries. The clerk has exported three years of emails and WhatsApp messages: 1,100 queries in total. About 78% ask about admission dates, fees and required documents; 12% ask about scholarships and migration certificates; the remaining 10% are miscellaneous. Roughly 40% are typed in Punjabi (Gurmukhi) and the rest in English, often with spelling variants. Most messages came from one urban locality and almost none from the village feeder schools the school now wants to attract. The clerk cannot write code. A vendor offers a fixed rule-based menu bot ready in two weeks, or a machine-learning intent classifier in eight weeks that needs labelled data and monthly retraining. The school has ₹40,000 and one 4G connection.
1. Recommend which option to start with and justify the choice using the numbers and constraints given.
Read the worked answer
Start with the rule-based (keyword/intent) bot, designed so it can later be replaced.
Arithmetic of coverage: 78% + 12% = 90% of 1,100 queries, about 990 messages, fall into a bounded set of intents (dates, fees, documents, scholarships, migration). That set can be covered by roughly 30–40 intent groups with keyword lists and quick-reply menus. The remaining 10% - about 110 queries - should go straight to a fallback that logs the question and promises a clerk's reply.
Constraint check: two weeks fits before the admission rush; the clerk needs no code; ₹40,000 is ample for a hosted menu bot. The ML option needs labelled data that does not yet exist in clean form, eight weeks, and monthly retraining that nobody on staff can own - a model that decays silently is worse than a menu that says 'I don't know'.
Language trap: a rule bot must not do exact matching. Build Gurmukhi keyword lists plus common romanised spellings (e.g. fees/fiz/ਫੀਸ) and route unmatched Punjabi input to the clerk rather than guessing. Log every query from day one; those logs become the training set if the school later moves to ML.
2. Identify two data risks in the query log and describe how you would measure each before launch.
Read the worked answer
Risk 1 - Coverage bias. The log is dominated by one urban locality; village feeder-school questions (transport, hostel, fee concession, late admission) are almost absent. The bot will appear accurate while being silent on exactly the new audience. Measure: before launch, collect 100 real queries from feeder-school families by phone or visit, run them through the bot, and report the percentage answered correctly. If it is below 80%, the intent list is incomplete regardless of overall accuracy.
Risk 2 - Language and spelling imbalance. About 60% English, 40% Punjabi, with variants and code-mixing. A single accuracy figure hides a bot that works in English and fails in Gurmukhi. Measure: hold out 20% of logged queries, stratified by language × intent, and report a small table of accuracy and fallback rate per cell; set a floor such as 90% correct routing on the top five intents in both languages.
Two supporting checks: de-duplicate repeated emails before counting volume, and strip phone numbers from any export used for training or sharing with the vendor. Report fallback rate as a first-class metric - a high fallback is visible honesty, while a confident wrong answer about fees is a real harm.
3. State what should be logged during the first six months and the evidence that would justify upgrading to a machine-learning model.
Read the worked answer
Log, with consent: the query text, detected language, the intent or keyword matched, whether the fallback fired, the clerk's final answer, and time to resolution. Do not log unnecessary personal data; mask mobile numbers.
Monthly review, not annual:
- fallback rate overall and per language;
- share of queries outside the current intent list;
- any intent where fallback exceeds about 20% of its own volume;
- complaints or repeat questions on the same topic.
Upgrade trigger: move to ML when new or reworded intents exceed roughly 10% of monthly volume, or when a high-volume intent persistently falls back despite keyword work. At that point the six months of logs are already labelled data - every fallback resolved by the clerk is a training example - so the eight-week ML build becomes cheaper and testable against a real baseline.
Until then, spend on content quality and Punjabi keyword coverage, not on a model. The decision rule should be written down before launch so the upgrade is driven by evidence rather than by a vendor's sales call.