The $9,000 Laser and the 62% Claim
A dental hygiene practice is considering a $9,000 diode laser for treating peri-implant mucositis. A manufacturer-funded study reports: 24 patients with implants, split-mouth design (laser on one side, sham on the other), single operator, outcome of bleeding index at 4 weeks, with "a 62% reduction in bleeding versus control (p = 0.03)." The practice owner is impressed and asks you to evaluate. On reading the paper you find: 3 patients dropped out, all from the laser group; the outcome assessor knew which side received laser; no information on maintenance intervals or implant brands; the control side's bleeding index fell 15% from baseline. Patients received a mean of 4.1 laser sessions; the practice can afford 2. The manufacturer sells the laser and funded the study. The mean difference in bleeding index was 0.35 on a 0–3 scale.
1. Identify the main threats to validity in this study and state, for each, which direction it would push the estimated effect.
Read the worked answer
Differential attrition (3 dropouts, all laser side). If those patients left because the laser was uncomfortable or ineffective, the remaining laser group is selectively the responders. Direction: inflates the laser effect.
Unblinded outcome assessor. Bleeding index is a subjective score; an assessor who knows which side received laser tends to score treated sites more favorably, especially when the expected effect is known. Direction: inflates the laser effect.
Single operator, split-mouth. Split-mouth controls for patient-level confounders (plaque control, systemic factors) and is a genuine strength, but a single operator limits generalizability and raises the possibility of systematic differences in how the two sides were instrumented; contamination between sides over 4 weeks is also possible. Direction: unclear, but precision is overstated.
Surrogate, short outcome. Bleeding index at 4 weeks is not implant survival, bone level, or disease resolution. Direction: inflates apparent clinical importance.
Industry funding. Not proof of misconduct, but funding and authorship by the seller are associated with favorable results and with selective reporting of secondary outcomes.
Practitioner skill gap. Patients received 4.1 sessions; the practice can deliver 2, so even the reported effect is not the effect this practice would obtain.
2. Interpret p = 0.03, the 15% control-side improvement, and the 0.35 mean difference. Is this statistically and clinically significant?
Read the worked answer
p = 0.03 means that if there were truly no difference between laser and sham, data this extreme would arise about 3% of the time. It is a statement about compatibility with the null hypothesis, not about the probability that the laser works, and not about the size of the effect. It also says nothing about whether the result would replicate.
The 15% control-side drop is the most informative number in the paper. The sham side improved substantially. That is what you expect from regression to the mean (patients enrolled when inflamed), from the hygiene instruction and maintenance delivered to both sides, and possibly from Hawthorne effects. It means the true incremental effect of the laser is the difference between the two arms, not the 62% figure, which is calculated against the laser's own baseline and therefore flatters the device.
The 0.35 difference on a 0–3 bleeding scale is roughly 12% of the scale range - a small absolute shift, and possibly below the threshold a patient would notice. Report it with a confidence interval; if the interval spans a clinically trivial difference at one end, the decision is not clear-cut.
Statistical vs clinical: a p-value under 0.05 tells you the difference is unlikely to be zero, which with 24 patients can still be a small, unstable difference. Clinical significance requires asking whether patients bleed less in a way that changes symptoms, implant retention, or the number of maintenance visits.
3. Write the substance of what you tell the practice owner, and specify what a better study would need to show before purchase.
Read the worked answer
Message. The study is a signal, not proof. The direction is favorable and the split-mouth design is thoughtful, but differential dropout, an unblinded assessor, a surrogate outcome at 4 weeks, and manufacturer funding all push in the same direction - toward overstating the effect. The absolute difference is small, and the sham side improved 15% anyway, so a substantial share of the benefit may come from the hygiene care both sides received. At $9,000 plus per-visit time, and with only 2 sessions per patient available versus 4.1 in the trial, the practice would be buying an unproven increment.
Recommended sequence. Track bleeding indices on your own implant maintenance patients with a standardized, blinded scoring protocol for 6 months, and spend the money first on things with established benefit: maintenance interval adherence, implant-specific self-care instruction, and clinician calibration.
What a defensible study would require: randomized allocation with concealed sequence, blinded outcome assessment, intention-to-treat analysis with accounting for all dropouts, a follow-up of at least 6–12 months, patient-important outcomes alongside bleeding scores (implant loss, bone level, discomfort, visits avoided), a reported confidence interval and pre-registered primary outcome, and independent funding. Ask for the absolute risk difference and the number needed to treat; a vendor who cannot supply these has not shown a return on $9,000.