A 41% mean and the conclusion that 'they didn't know the content'
At the end of a Grade 9 biology unit on enzymes, 30 learners at CEFR B1 sit a 60-minute written test in the L2. The paper has 10 multiple-choice items with long, syntactically complex distractors, five short answers beginning 'Explain why…', and one essay. The mark scheme deducts one mark per spelling or grammar error. Results: the class mean is 41%. Three learners who gave accurate, detailed explanations during the practical and the class discussion scored below 50%. One learner answered two short-answer questions correctly in L1 and was given zero. The teacher's comment in the department meeting is: 'They simply didn't know the content - the practical flattered them.'
1. What is this test actually measuring? Identify at least three sources of construct-irrelevant difficulty.
Read the worked answer
The test is measuring L2 academic literacy and content simultaneously, and the two cannot be separated from the scores.
Sources of difficulty unrelated to enzyme knowledge:
- Reading load in the multiple-choice items. Long, complex distractors turn each item into a reading comprehension task; a learner who knows the content can still choose wrongly by misparsing a distractor.
- The 'explain why' genre under time pressure. Five extended causal explanations plus an essay demands discourse organisation and academic vocabulary that has not been separately taught or assessed.
- The spelling/grammar penalty. Deducting a mark per error makes the score partly a language-accuracy score. A learner with secure content knowledge and weak spelling is systematically marked down, and the resulting number cannot be interpreted as 'content known / not known'.
- The L1 response scored zero. This is the clearest evidence: the learner demonstrated the content and was recorded as not having it.
On the practical: it is not that the practical 'flattered' them - it is that the practical may have been the more valid measure of the content construct, because it let learners show enzyme knowledge through action and talk rather than through decontextualised L2 writing. The written test is the suspect instrument, not the practical.
2. Propose two changes to the assessment design and explain how each improves validity.
Read the worked answer
Change 1 - use a two-strand mark scheme: content marks and language marks reported separately. Each short answer is marked first for the scientific idea (does it identify the enzyme as a catalyst, the effect of temperature on the active site, and so on) and separately for communication. Why it improves validity: the two constructs stop contaminating each other. A learner scoring 6/8 for content and 2/4 for language is diagnosable and teachable; the current single score is not, and it is the content score that the teacher actually needs in order to decide whether to reteach.
Change 2 - reduce linguistic load while keeping cognitive demand. Shorten multiple-choice stems, make distractors parallel in length and structure, replace one 'explain why' with a diagram-plus-labelled-arrow task or a results table to complete, and give a writing frame for the remaining explanation question ('The enzyme ______ because the temperature ______, which means the active site ______'). Why it improves validity: the items now discriminate on enzyme knowledge rather than on reading speed and clause parsing. The thinking (explaining denaturation) is unchanged; only the language barrier to showing it is lowered.
A further option: add a short practical or performance task with a rubric, so the construct is sampled in more than one mode.
3. How should the teacher respond to the two L1 answers and to the spelling policy? Give a defensible position and state its limits.
Read the worked answer
On the L1 answers: in a test whose stated purpose is to find out what learners know about enzymes, a correct answer in L1 is evidence of content knowledge and should be credited for content, with the language recorded separately. Giving zero makes the score uninterpretable and is unfair to the learner.
On spelling: deduct only where the error makes the science ambiguous (e.g. enzyme written as another word, a unit or value altered) or where language is a declared criterion of the task. A blanket one-mark-per-error policy turns a biology test into a spelling test.
The limits of that position:
- If the qualification or national assessment requires L2 production, the learner will eventually be judged on it. Accepting L1 indefinitely would hide a real gap rather than address it.
- The defensible answer is therefore separate and teach, not remove: keep the L2 demand in the assessment, but (a) teach the explanation genre explicitly beforehand, (b) publish the language criteria in advance, and (c) report content and language separately so that a language weakness is visible as a language weakness rather than being misread as ignorance of enzymes.
Practical implementation: allow L1 or a bilingual glossary for a transition period; annotate the mark book with two columns (content / language); and use the class's actual scripts as the basis for the next lesson's language focus.