Natural Language Processing Application Development
What this domain covers
NLP questions test text preprocessing, representation learning, sequence models, Transformer/BERT fine-tuning, and Huawei NLP deployment options.
Core rules and concepts
- Preprocessing: normalization, tokenization, vocabulary construction, padding, and truncation to a maximum length.
- Subword tokenizers (BPE, WordPiece) handle out-of-vocabulary words by splitting them into frequent pieces.
- Embeddings map tokens to dense vectors; pretrained embeddings can be frozen or fine-tuned.
- RNN/LSTM process sequences step by step but suffer from vanishing gradients on long dependencies; attention addresses this.
- Transformer self-attention computes Q, K, V projections, scaled dot-product scores, softmax, and a weighted sum.
- BERT is pretrained with masked language modeling and next-sentence prediction, then fine-tuned with a task head; [CLS] is commonly used for sentence classification.
- Huawei Cloud provides NLP services for classification, NER, and sentiment; MindSpore NLP supports BERT-style training.
Worked example
A batch has sequence length 128 and hidden size 768 with 12 attention heads. Each head projects to 768/12 = 64 dimensions. Q, K, V are each 128×768 before splitting; per head, QKᵀ gives a 128×128 score matrix, softmax normalizes each row, and multiplication by V yields 128×64. Reasoning: the head split keeps total computation constant while letting each head attend to a different subspace.
Traps
Omitting the padding mask so pad tokens affect attention; applying softmax over the wrong axis; fitting the vocabulary on test data; exceeding BERT's 512-token limit without truncation; using [CLS] embeddings without fine-tuning the head.