Synthetic Test Data for a Clinical Scheduling App
MedSchedule is building a clinical appointment scheduler. The test team needs 5,000 patient records for performance and workflow testing but cannot use production data. An engineer proposes prompting a public LLM to generate realistic names, birth dates, diagnoses, insurance types, and appointment notes. Sample output includes a 6-year-old with a Medicare number, an appointment date before the patient's birth, and a note that closely resembles a real local news story. The team also plans to paste a few de-identified production records as style examples. The vendor's terms allow model training on inputs unless the enterprise tier is purchased, which is not yet approved. The release is in four weeks; the team has a data-privacy officer, a rule-based data generator, and one week for dataset validation.
1. Evaluate whether the LLM should generate all patient fields or only some of them.
Read the worked answer
The LLM is poorly suited to generating structured fields that must satisfy relational and clinical constraints. A 6-year-old with Medicare and an appointment before birth are validity failures that would cause false test results. Names, birth dates, diagnoses, insurance IDs, and appointment dates should come from a constrained rule-based generator with explicit age, coverage, and chronology rules, or from validated code sets.
The LLM may help with free-text appointment notes or complaint narratives if those outputs are validated for consistency, non-identifiability, and absence of copied text. Even then, notes must reference the structured record correctly. A hybrid approach is appropriate: deterministic generation for structured data, LLM assistance for limited free text, and automated constraint checks plus human sampling before use. Generating all fields with a public LLM creates unnecessary privacy and validity risk.
2. Identify the privacy and validity risks and recommend controls.
Read the worked answer
Privacy risks: pasting even de-identified production records can enable re-identification, especially with dates and rare diagnoses. The public vendor may train on inputs, creating onward disclosure risk. Generated notes may accidentally resemble real people or reproduce memorized text.
Validity risks: inconsistent age, insurance, diagnosis, and appointment chronology; biased or unrealistic distributions; non-reproducible datasets; free-text notes that do not match the structured record.
Controls: do not use production records as style examples. Use the enterprise no-training tier or a private endpoint with a data protection agreement. Generate structured fields with a seeded rule-based generator, and use the LLM only for free-text templates with synthetic seeds. Scan outputs for identifiers and known text overlap. Validate constraints automatically, review a sample clinically, document provenance and seed values, and keep the dataset reproducible. The data-privacy officer should approve the workflow before generation.
3. Define acceptance criteria for the 5,000-record dataset and state your recommendation.
Read the worked answer
Acceptance criteria should include: 100% referential integrity across patient, insurance, appointment, and note records; zero real identifiers or PII; 100% pass rate on chronology and coverage constraints; required coverage of edge cases such as pediatric, geriatric, multiple insurance types, and missing data; reproducible generation from a documented seed; no free-text note that copies a real source; and a clinically reviewed sample with no unexplained anomalies.
Recommendation: approve a constrained hybrid pilot, not public-LLM generation of all fields. Use rule-based generation for structured data, limited LLM assistance for notes behind an approved private endpoint, and automated validation plus human review. With one week for validation, the team should generate a small batch first, measure constraint failures, and expand only if the criteria pass. This protects privacy and produces data that can actually support meaningful performance and workflow tests.