Ninety-Two Percent: A Materials Course Where the Exam Cannot See the Outcome
Engineering Materials (3 credits, 48 students) states three CLOs: (1) design an experiment to characterise a material's mechanical behaviour; (2) analyse experimental data and justify a material selection; (3) communicate findings in a technical report. Assessment is a final exam worth 70% (60 multiple-choice items plus 4 short-answer questions) and laboratory reports worth 30%, marked by a graduate assistant on a 0-10 scale for 'neatness, completeness, and effort'. Attainment is computed by counting students who score at least 60% on the final exam, at course level rather than per CLO. The report to the accreditation body claims 92% attainment for all three CLOs. At the site visit an evaluator asks for the evidence behind CLO 1.
1. Diagnose the alignment of this course. Which CLOs are actually evidenced by the current assessment, and what specifically invalidates the 92%?
Read the worked answer
Mapping the assessments to the cognitive demand of each CLO:
| CLO | Demand | What the assessment can elicit | Verdict |
|---|---|---|---|
| 1 Design an experiment | Create | MCQ cannot; lab reports have no design criterion | Not evidenced |
| 2 Analyse and justify | Analyse / Evaluate | 4 short-answer items may partially; lab reports not scored on reasoning | Weakly evidenced |
| 3 Communicate | Apply / Create | Lab reports score neatness, not communication of findings | Not evidenced |
Why the 92% is invalid:
- Construct underrepresentation. The instrument samples recall and routine calculation. 'Design' and 'justify' are performance constructs; they require constructed response or a product.
- The rubric does not reference the CLOs. 'Neatness, completeness, effort' measures compliance and presentation. Effort is not a property of the artefact and is highly susceptible to halo effects.
- Single marker, no calibration. No inter-rater reliability evidence, so the 30% component is unrepeatable.
- Aggregation at course level. Reporting one course number for three constructs makes it impossible to know which outcome failed. The 92% may be a true statement about exam performance and still be silent on all three CLOs.
So 92% is a measurement of something. It is not a measurement of CLO 1, 2, or 3. The evaluator's question should be answered with 'no evidence exists', because that is the truth.
2. Redesign the assessment and the attainment calculation. Be concrete about instruments, criteria, and the wording of the attainment statement.
Read the worked answer
Assessment redesign
- Keep MCQ for the knowledge base but cap it at ~20% of the grade. It is not the enemy; over-weighting it is.
- Lab reports scored on a 4-level analytic rubric with dimensions: experimental design validity, data analysis, material selection justification, technical communication. Each dimension maps to a CLO.
- Add a 2-page design memo: given a service condition, propose the experiment and defend the method. This is the only artefact that can evidence CLO 1 at the Create level.
- Calibrate: a 60-minute session where GA and instructor independently mark the same four scripts, compare, and rewrite ambiguous descriptors. Double-mark a 20% sample.
- Publish the rubric to students before submission. Criteria describe quality; they do not prescribe the answer.
Attainment calculation
- Compute per CLO, not per course.
- A student attains a CLO by scoring proficient (3 of 4) or above on the mapped rubric dimensions across at least two assessments.
- Report: n, instrument, criterion, and source.
Sample statement: 'CLO 1 (design an experiment): 34 of 48 students (71%) achieved proficient or above on the experimental-design dimension across the lab report and the design memo; program target is 75%.'
This is a finding the program can act on. 92% across all three was a finding nobody could act on.
3. The instructor objects that rubrics 'kill creativity' and that lab reports cannot be graded objectively. Respond as you would in the meeting.
Read the worked answer
Concede the real point, reject the false one.
- Concede: badly written rubrics do constrain. A rubric whose criteria reward format conformity - neatness, section order, page count - trains students to comply rather than to reason. That is a defect of this rubric, not of rubrics.
- Reject: the claim that criteria and creativity are opposed. A rubric states what quality looks like in reasoning (is the design valid? is the selection justified against the data?) and permits any number of correct answers. Criteria constrain the standard, not the solution.
- On objectivity: complete objectivity is not the goal and is not achievable in judging design. The goal is defensible, consistent judgement, achieved through explicit criteria, shared exemplars, and calibration. Analytic rubrics with separate dimensions also reduce halo effects, which is exactly what the current 0-10 single score maximises.
- Practical offer: write the criteria with the instructor, using two of last year's best and two of last year's weakest reports as anchors. This is how buy-in is produced, and it takes 45 minutes.
The unacceptable status quo is not subjectivity - it is unexamined subjectivity, with no evidence trail.