Content Safety Thresholds for a Children's Story Generator
A publisher plans a tool that generates short bedtime stories from a one-line prompt supplied by a parent. The audience is children aged 4 to 7. Prompts are free text, so a parent could type anything, and the model could also drift into frightening imagery on its own. The publisher's legal team requires that no story contain violence, self-harm, sexual content, or hateful language, and that scary elements such as monsters be mild. A trial of 500 prompts produced 12 outputs with content the editors considered too intense, all of which involved peril or threatening creatures rather than the four prohibited categories. The publisher wants to ship in six weeks and has no content moderation staff.
1. Design the moderation layers and explain what each layer catches that the others cannot.
Read the worked answer
Use three layers, because each covers a gap the others leave.
- Prompt screening before generation. Blocks or rewrites parent input that is itself prohibited. Catches deliberate misuse, which output filtering alone would only catch after an attempt. It cannot catch model drift.
- Output classification after generation. Scores the finished story against the prohibited categories and against a separate intensity judgement for peril and frightening creatures. This is the only layer that sees what the model actually wrote, so it is the backstop for drift. It cannot judge whether a monster is 'mild' using category filters alone, so pair it with an editorial rubric or a graded severity score.
- Structural constraints in the prompt and template. Fixed story length, a required gentle resolution, and an instruction to avoid peril. This reduces the frequency of borderline output so the output classifier has less to catch.
Key point: the 12 flagged outputs were not prohibited-category violations, so a filter tuned only to the four legal categories would have passed all 12. The intensity dimension needs its own check.
2. The publisher proposes setting every safety threshold to its strictest value to be safe. What is the downside, and what would you recommend instead?
Read the worked answer
Strictest-everywhere is a real trade-off, not a free win. At maximum strictness the output classifier will block or truncate many harmless stories: a friendly dragon, a character feeling sad, a mild thunderstorm. Parents see refusals or stories that stop mid-sentence, and the product looks broken. Because there is no moderation staff, every false block also becomes a support complaint nobody is staffed to handle.
Recommendation:
- Separate the two judgements. Keep the four legally prohibited categories at the strictest setting, since false blocks there are rare and the legal requirement is absolute. Set the peril and intensity threshold at a calibrated middle level, tuned on the 500-prompt trial plus a fresh sample, so mild monsters pass and genuine peril does not.
- Measure both error types. Track false blocks (harmless story rejected) and misses (intense story delivered) on a labeled sample. Tune the intensity threshold to the point where misses are near zero without collapsing acceptance.
- Add a graceful fallback. When a story is blocked, return a safe pre-written story rather than an error, so a block never reaches the child as a failure.
- Re-review monthly, since model behavior and prompt patterns drift.