Three Defective Quiz Items Before Grades Post
Ms. Pires is reviewing a 20-item quiz after the item analysis of 96 students. Three items behave badly.
Item 4: "A sopa ___ boa." Options: é / está / foi / era. The key is está, but 62% chose é, and several students argued in class that soup can simply be good.
Item 11: "O presente é ___ você." Options: por / para / de / com. The key is para, but 48% chose por, and the department's own answer-key notes that por can mean "on behalf of."
Item 17: "Qual é o nome correto desta fruta?" Options: mexerica / bergamota / tangerina. Students from different regions each chose a different answer and each was right in their family's usage.
The quiz counts for 15% of the quarter grade and must be returned Monday. The department has no written item-review procedure.
1. Diagnose each item against the basic requirements for a defensible multiple-choice item: a single defensible key, alignment with what was taught, and freedom from construct-irrelevant difficulty.
Read the worked answer
Item 4 violates the single-key requirement. A sopa é boa and A sopa está boa are both grammatical but mean different things: é boa characterizes the soup (a good soup, a good recipe), está boa evaluates this serving right now. With no context, the stem does not determine which meaning is intended, so high-ability students who notice the ambiguity choose the distractor and are penalized for being right. That is the classic signature: the distractor outperforms the key among stronger students, and the item's discrimination collapses.
Item 11 fails the same way, and the department's own key note concedes it. O presente é para você (you are the intended recipient) and o presente é por você (on your behalf, or because of you) are both well formed. The stem never specifies the intended meaning, so the item tests guessing rather than the por/para contrast.
Item 17 fails differently and more seriously: it has no defensible key at all. Mexerica, bergamota, and tangerina are regional names for the same fruit; all three are correct in their own variety, and no standard exists to rank them. The item also measures family dialect rather than course content, which is construct-irrelevant difficulty and, in a heritage-speaker class, is close to punishing students for their home variety. It should be discarded rather than revised, unless the objective is explicitly lexical variation - in which case the question must ask which region uses which term.
2. Rewrite items 4 and 11 so each has one defensible key. Explain what changed and why the rewrite works.
Read the worked answer
Item 4, revised: No restaurante, o cliente prova a sopa e diz ao garçom, sobre aquela porção específica: "Hoje ela ___ boa." Key: está. Distractors: é / era / será.
The rewrite supplies the meaning that selects estar: a temporary evaluation of a particular serving. Because the meaning is now stated, é is no longer defensible and can serve as a genuine distractor testing whether students know that a characteristic reading would require é. The item now measures the ser/estar contrast instead of the student's ability to guess the writer's intention.
Item 11, revised: Os colegas fizeram uma vaquinha e compraram o presente no lugar da turma inteira. O presente é ___ você. Key: por (on your behalf). Distractors: para / de / com.
Here the fix runs in the opposite direction: the context is written so that por is the only reading that matches it, and para becomes the attractive distractor for students who have memorized "para = destination/recipient" without checking meaning. Both rewrites follow the same rule: state the intended meaning in the stem, then let the grammar follow. An item that begins with a meaning is answerable; an item that begins with a blank and two grammatical options is a coin toss dressed as a grammar test.
3. What should Ms. Pires do about scoring this administration, and what item-review procedure should the department adopt?
Read the worked answer
For this administration: discard item 17, and either drop items 4 and 11 or credit both readings (é/está, para/por) as correct. Rescore out of the remaining 17 items and convert proportionally so the 15% quarter weight is unchanged. Return the quizzes with a short note explaining which items were flawed and why - this is not lost instructional time; it models for students that assessments are arguments that can be examined. Do not simply "curve" the whole quiz, which would leave the flawed items in the total and reward students who guessed.
Procedure going forward, four steps that would have caught all three:
- Meaning first. Every item stem must state the intended meaning or context; a blank plus two grammatical options is not an item.
- Blind second read. Another teacher takes the quiz and writes down the intended answer before seeing the key. If they say "it depends," the item is revised.
- Dialect screen. No item may turn a regional variant into a wrong answer unless the objective is explicitly lexical or phonological variation, and then the item must ask about distribution, not correctness.
- Post-administration item analysis. Flag any item whose distractors draw more responses than the key, and any item whose strongest students disproportionately miss it. Those flags trigger revision before the next administration, not after the grade is posted.