The Dataset That Kept Growing
A multi-site cohort study of postoperative complications has been running for 3 years. Each site enters data into a shared spreadsheet stored on a departmental drive. Variable names differ slightly across sites (e.g., "readmit" vs "readmission" vs "re_adm"). Two research assistants have left, and no data dictionary exists. The statistician discovers that 8% of records have a discharge date earlier than the admission date, and that one site's complication counts are roughly half those of the others. The funder now requires a data management plan and a reproducibility statement before the final analysis. The team has 6 weeks before the analysis must begin.
1. What are the highest-priority data quality and management problems, and how should they be triaged?
Read the worked answer
The problems fall into four categories, and triage should follow risk to the final analysis.
1. Structural problems (fix first):
- No data dictionary and inconsistent variable names. Without harmonization, any merge or pooled analysis is unreliable. Build a data dictionary now, mapping each site's variables to a common schema, and document the mapping.
- Shared spreadsheet on a drive with no version control. Move to a controlled system with access logs, or at minimum freeze a read-only master and track changes.
2. Logical errors (fix second):
- Discharge before admission in 8% of records indicates entry or formatting errors (e.g., date-order conventions, typos). Quantify by site and variable, correct against source records where possible, and flag what cannot be verified.
3. Site-level anomaly (investigate in parallel):
- One site reporting half the complications of others could reflect a real difference in case mix, a different definition of complication, under-ascertainment, or data-entry practice. Do not assume error; compare case mix and definitions first.
4. Documentation (ongoing):
- The departed staff took institutional knowledge. Interview remaining staff, reconstruct decisions, and record them.
Six weeks is tight; the dictionary and logical-error audit are non-negotiable, and the site anomaly needs a documented explanation before pooling.
2. What should the reproducibility statement say, and what must be true before it can be written honestly?
Read the worked answer
A reproducibility statement should describe what another researcher would need to repeat or audit the analysis. It should cover:
- Data: what data exist, where they are held, access conditions, and why (e.g., patient confidentiality) they cannot be openly shared.
- Code: the analysis scripts, in what language, and where deposited (e.g., a repository with a DOI).
- Dictionary: the final data dictionary and variable mapping.
- Version: which dataset version and code version produced the reported results.
- Processing: how cleaning decisions (date corrections, exclusions, site harmonization) were made and recorded.
Before it can be written honestly, the team must:
- Freeze a master dataset and assign a version.
- Complete the data dictionary and variable harmonization.
- Resolve or document each logical error and the site anomaly.
- Write analysis code that runs from the frozen dataset to the reported tables, and have a second person run it independently.
A statement claiming reproducibility without these steps would be false. If some steps cannot be completed in 6 weeks, the statement should say so and describe the limitation, rather than overstate.