The Simulation That Looked Great and Taught Nothing

Satisfaction is a reaction measure, not a learning or behavior measure.

Why the disconnect is plausible:

• The scenario may have been too easy or too guided - if the facilitator cue'd heavily, learners felt successful without having to recognize work of breathing independently.
• It tested knowledge in a quiet lab, not recognition under load - the real unit has noise, interruptions, and 1:6 ratios. Transfer requires the simulation to resemble the transfer environment.
• Satisfaction measures how the learner felt, and feeling good is often inversely related to being challenged. A 4.8 may be a warning sign, not a triumph.
• No spaced practice - a single high-fidelity event decays; skills need retrieval at intervals.
• No objective outcome measure tied to the scenario - you measured the wrong thing at the wrong level.

Evaluation framework: use Kirkpatrick's four levels - reaction (satisfaction), learning (knowledge/skill check), behavior (chart audit, observed practice), results (patient outcomes). You only measured level 1 and assumed levels 2–4.

What this tells you: the program is not yet evidence-based; it is evidence-adjacent. The director's proposal ('run it more often') would scale an unmeasured intervention - more cost, same uncertainty.

Redesign around deliberate practice, realism, and spaced retrieval.

Scenario design:

• Make recognition the hard part. Build in subtle, evolving work of breathing (retractions, grunting, nasal flaring, head bobbing) that the learner must identify, not be told.
• Remove facilitator cueing during the scenario; debrief after.
• Add distractors - a ringing phone, a parent asking questions, a second patient - to mimic the unit.
• Standardize the debrief (e.g., PEARLS or plus/delta) with a focus on the recognition-to-escalation chain.

Dose and spacing:

• Not one-and-done. Use brief, spaced retrieval: a 10-minute in-situ scenario on the unit every 2–3 weeks, plus a short e-learning booster between. This is cheaper than the lab and tests transfer in the real environment.
• In-situ simulation solves your lab-booking constraint and increases realism.

Measurement (tie to Kirkpatrick levels 2–4):

• Level 2: objective structured check - did the learner identify work of breathing and state escalation criteria within a set time?
• Level 3: chart audit - time from first documented abnormal respiratory sign to escalation; observe real practice.
• Level 4: track respiratory arrests/rapid response escalations for this population.
• Pre/post with a comparison - cohort before vs. after, or a non-equivalent control unit if feasible.

Budget: shift spend from repeated high-fidelity lab runs to in-situ kits and booster content; the lab is your scarce resource - use it for the hardest skills only.

Make the case in the director's language: cost, risk, and defensibility.

Arguments:

1. Scaling an unmeasured intervention is a financial and safety risk. 'Run it more often' costs lab hours and staff time with no evidence of benefit - and if a patient is harmed, 'we did a lot of training' is not a defense; 'we measured and improved' is.
2. Satisfaction data will not survive scrutiny - accreditation, quality committees, and payers increasingly ask for outcome data. You are protecting the program, not burdening it.
3. The measurement can be nearly free - most of it is data you already collect (chart audit, rapid response logs, existing competency checks).

Minimum viable evaluation (low burden):

• One objective skills check at the end of each cohort (10 minutes, checklist).
• One chart audit metric - time from first abnormal respiratory sign to escalation, 20 charts pre and post.
• One run chart reviewed quarterly.
• One question on the post-course survey that actually predicts learning (e.g., 'I can list three signs of increasing work of breathing without help') rather than 'I enjoyed this.'

Compromise offer: run the redesigned program with the minimum metrics for one cohort, and agree in advance that if there is no signal by the next quarter, you will revisit the design together. That gives the director a decision point and gives you the data to defend the program.

Institutionalize the cycle: needs assessment → design → deliver → evaluate → revise.

1. Anchor to a real gap - start each cycle from a needs assessment (chart audit, incident reports, competency data), not from a topic someone likes.
2. Use a recognized framework - e.g., Kirkpatrick for evaluation levels and deliberate practice / spaced retrieval for design. Cite the evidence base so the program is defensible.
3. Build a faculty and a champion network - train 2–3 unit-based educators so the program doesn't depend on you alone.
4. Standardize the debrief and the checklist so results are comparable across cohorts and units.
5. Close the loop with staff - share what the data showed and what changed; this sustains engagement and psychological safety.
6. Integrate with quality and safety infrastructure - report to the nursing quality council, align metrics with unit dashboards, and connect to rapid response and sepsis work so education is part of a system, not a standalone event.
7. Budget realistically - protect in-situ time, booster content, and faculty development; the lab is for high-cost, high-value skills only.
8. Publish or present internally - a poster or unit report cements the program and models evidence-based practice for staff.

The CNS role here is translational: turning a popular activity into a measured intervention, and turning a measurement into a durable standard.