Sentiment on Romanised Bengali Reviews
An ed-tech startup has 3,200 free-text course reviews written by Bengali-speaking learners. Most write Bengali words in Roman script mixed with English, for example 'porikkha ta khub hard chilo, teacher bhalo chilen'. A student intern builds a bag-of-words plus logistic regression classifier to detect 'negative' reviews. On a held-out test set the model reports 88% accuracy. In that test set, 84% of reviews carry 4 or 5 stars and only 9% carry 1 or 2. The product manager wants to (i) auto-hide every review the model labels negative, and (ii) reuse the same model behind a chatbot that answers new learners' question 'is this course good?'. Reviews of the same course by the same learner are sometimes split across train and test. Nobody has checked what the model does with the review 'darun course, 3 din dhore video load hoyni', which means 'great course, the video did not load for 3 days'.
1. Assess the 88% accuracy figure and specify what you would measure instead before any decision is taken.
Read the worked answer
The test set is 84% positive, so predicting 'positive' for every review scores about 84%. The model's 88% is four points above a do-nothing baseline - it may be doing very little beyond recognising a handful of frequent words. Under this imbalance, accuracy also hides the fact that a model can be excellent on the positive class and near-useless on the negative class, which is the only class the product manager cares about.
Measure instead:
- The confusion matrix itself, then precision, recall and F1 for the negative class. A plausible outcome is recall around 40–50% on negatives even at 88% accuracy.
- PR-AUC for the negative class, plus the precision–recall curve so a threshold can be chosen deliberately.
- Macro-F1, so both classes count equally.
- Baseline comparison: majority class, and a simple keyword rule, both reported next to the model.
There is also a label problem: 'negative' is being inferred from star ratings, and 3-star reviews sit ambiguously in the middle. Define the target explicitly (for example, 1–2 stars = negative, 4–5 = positive, 3 stars excluded or handled separately) and state it in the report. Finally, fix the split: reviews of the same course by the same learner must be grouped into the same fold, or the test score is optimistic.
2. Describe the text preprocessing pipeline you would build for this data, and justify the choices that differ from standard English text cleaning.
Read the worked answer
Standard English recipes damage this data, so each step needs a reason.
- Normalise, do not erase. Lower-case, strip punctuation noise, and collapse repeated characters ('bhalooo'), but keep the words. Build a small normalisation dictionary for the spelling variants of common Romanised Bengali words, since the same word may appear three ways in one review.
- Keep negation words. 'na', 'ni', 'hoyni', 'chilo na' carry the entire sentiment of the sentence. Removing them as stopwords inverts the meaning of a review. Standard stopword lists are English-centric and should not be applied blindly.
- Do not English-stem. Porter/Snowball stemmers mangle Romanised Bengali morphology. Prefer word n-grams (1,2) and character n-grams (3–5), which capture 'bhalo chilo' versus 'bhalo chilo na' and absorb spelling variation without a stemmer.
- Use TF-IDF with sublinear_tf so a few repeated words do not dominate, cap max_features, and set class_weight='balanced' in the classifier.
- Handle code-switching explicitly by keeping English tokens; they are often the sentiment-bearing ones ('hard', 'worst', 'refund').
- Validate properly: group splits by learner/course, and build a small hand-labelled gold set with two annotators to measure agreement - if the annotators disagree with each other, the model's ceiling is already limited.
3. Evaluate the two proposed uses: auto-hiding negative reviews, and powering a 'is this course good?' chatbot with the same model.
Read the worked answer
Auto-hiding is the wrong action for this model. A bag-of-words classifier at this quality will produce false positives (legitimate complaints about video load failures hidden) and false negatives (sarcasm or abusive reviews left visible). The false positives are the worse failure: complaints about broken video are precisely the useful signal for the company, and hiding them creates a feedback loop - the training data becomes only what survived the filter, so the next model is trained on a sanitised world and gets worse at finding real problems. There is also a recourse problem: a learner whose review is hidden has no way to appeal.
A defensible alternative: use the score to prioritise reviews for a human moderator, show the aggregate rating unchanged, and route the top-scoring negatives to support so the video issue is actually fixed.
The chatbot is a different misuse. The model outputs one label for a whole review, so it cannot answer 'is this course good?' with any grounding - it has no course-level aggregate, no ability to handle a question it has not seen as a review, and it will confidently answer questions outside its training distribution. 'Darun course, 3 din dhore video load hoyni' is exactly the case it mishandles: the surface words are positive while the content is a complaint.
If a chatbot is wanted, build it as retrieval over FAQ and course pages with the review label as one weak signal, always show the underlying source, and hand off to a human when confidence is low.