Natural Language Processing
What this domain tests
Natural Language Processing tests text preprocessing, bag-of-words representation, sentiment analysis and chatbot or language-application concepts. Expect vocabulary and vector-counting questions plus interpretation of text pipelines.
Core rules and concepts
- Preprocessing: lowercase, tokenise, remove punctuation and stopwords, then stem or lemmatise. Stemming cuts word endings crudely; lemmatisation uses a dictionary form.
- Bag-of-words builds a vocabulary and counts word occurrences. It ignores word order and produces sparse vectors. TF-IDF downweights very common terms and boosts distinctive terms. N-grams capture some local order.
- Sentiment analysis can use a lexicon or a supervised classifier trained on labelled text. Negation, sarcasm and domain words are difficult.
- Chatbots may be rule-based, retrieval-based or generative. Intent classification maps a user utterance to an action.
Worked example
Sentence A = 'AI is fun'; sentence B = 'AI is useful'. Vocabulary = [ai, is, fun, useful]. A becomes [1, 1, 1, 0]; B becomes [1, 1, 0, 1]. Dot product = 2; each vector norm = sqrt(3); cosine similarity = 2/3 = 0.667. Reasoning: shared words create similarity, but the vectors ignore word order.
Traps that cost marks
Fit the vocabulary on training text only; using all text before splitting leaks information. Do not confuse stemming with lemmatisation. Bag-of-words loses order, so 'not good' and 'good' can look similar. Removing stopwords can destroy negation. Sparse vectors need appropriate models.