Fifty-Eight Percent for Three Years: A Loop That Never Closes
For three consecutive cycles a Mechanical Engineering program reports PO3 (design within realistic constraints) attainment at 58%, 61%, and 57% against a 70% target. Each cycle's continuous-improvement form reads, in substance, 'students are weak in design; instructors will provide more examples and encourage more tutorials.' No owner, date, or measure is recorded. The curriculum map assigns PO3 to four courses, but in two of them the CLOs contain no design verb and the assessment is a multiple-choice midterm. PO6 (ethics) is mapped to a single course whose syllabus devotes one week to it, examined by a 5-mark question. The accreditation visit is in eight months, and the program director proposes revising the target from 70% to 55% so the indicator turns green.
1. Diagnose why the improvement loop is not closing. Identify the distinct failures rather than giving one general complaint.
Read the worked answer
Four separable failures:
- The action is unfalsifiable. 'Provide more examples and encourage more tutorials' specifies no change to what is taught, how it is assessed, who does it, or by when. It cannot fail, therefore it cannot succeed. No owner and no date means no accountability and no way to know whether it happened.
- Root cause was never diagnosed. Is 58% a teaching problem, an instrument problem (the assessments do not elicit design), a curriculum problem (PO3 mapped to courses that do not teach design), or a target problem? The form assumes the first, every cycle, without testing.
- The mapping is aspirational, not actual. PO3 is mapped to four courses; two have no design CLO and a multiple-choice midterm. The map asserts coverage the curriculum does not deliver. This alone can explain a stable ~58%: the outcome is being measured somewhere it is barely taught.
- No re-measurement with a stable instrument. Cycles are compared without confirming that the same construct was measured the same way, so year-to-year variation is uninterpretable.
The diagnostic signal is the stability. 58, 61, 57 is a flat line. Flat lines point to a structural cause, not to fluctuating student quality - and no amount of tutorial encouragement moves a structural cause.
2. Evaluate the director's proposal to lower the target to 55%. Under what conditions, if any, is revising a target legitimate?
Read the worked answer
As proposed: not acceptable. It converts a red indicator to green without any change in student learning, which is precisely the behaviour accreditation review exists to detect. It also destroys the evidentiary value of three cycles of data by removing the finding rather than acting on it.
When revising a target is legitimate - all four conditions must hold:
- A valid baseline now exists. The old target was set before anyone could measure the construct properly. Once the instrument is sound, targets can be calibrated to a defensible baseline.
- The rationale is documented and external - cohort intake, program length, comparison with peer programs, or a change in instrument. 'We are not meeting it' is not a rationale.
- It is accompanied by an action plan, not substituted for one. A revised target with no intervention is the same non-answer with a nicer number.
- It is not applied in the same cycle as the finding it erases, and it is not applied retroactively. Targets set after the result is known are not targets.
The specific danger here: the flat 58/61/57 pattern is exactly the signal the data exist to produce. Lowering the bar suppresses the one useful thing three cycles have told the program. If the instrument turns out to be invalid, the right move is to fix the instrument and re-baseline - and to say so explicitly, with the old numbers retained and annotated.
3. Draft the improvement plan you would put in front of the review team, and specify the evidence that would demonstrate the loop actually closed. Address PO6 as well as PO3.
Read the worked answer
PO3 - five steps, each with an artefact:
- Audit the four mapped courses. Verify that each has a design CLO with a performance assessment. Republish the map so it reflects what is taught, not what was hoped for. (Artefact: revised curriculum map with assessment evidence per cell.)
- Diagnose the instrument. Blueprint existing PO3 assessments against the CLOs. If no assessment elicits design under constraints, the 58% is an instrument artefact and the fix is the assessment, not the teaching.
- Act with specificity. E.g. 'Introduce a constraint-based design brief into ME 3xx, weeks 5-9, rubric-scored on four dimensions; owner: course coordinator; completed by March; expected effect: PO3 +10 points.' Named change, named owner, named date.
- Re-measure with the same rubric and the same criterion. Same instrument is what makes the comparison meaningful.
- Record the decision. Finding → cause → action → result → next decision, in one page.
PO6 - the mapping is too thin to defend. One week and a 5-mark question cannot carry an ethics outcome at the level the framework expects. Either strengthen it (a case-based ethics assessment in a design or professional-practice course) or map it to a second course; a single 5-mark item is a coverage claim, not coverage.
Evidence the loop closed: before/after figures on an identical rubric with sample scripts retained; a calibration record for the markers; minutes showing the decision and its owner; and a next-cycle plan if the target is still unmet. Reviewers accept an unmet target with a documented, specific response. They do not accept a met target with no trail.