Reporting Clinical AI and Machine Learning Studies: TRIPOD+AI, CLAIM, and CONSORT-AI Checklist Guide
By Trace.Science Clinical AI & Biostatistics Working Group
Clinical AI and prediction model manuscripts must follow TRIPOD+AI (2024 update superseding TRIPOD 2015) or CLAIM (for medical imaging AI), reporting exact training/validation/test cohort patient-level splits (preventing data leakage), sample size justification (Riley et al. `pmsampsize`), both discrimination (AUROC with 95% CI) and calibration (calibration slope, intercept, and curves), net clinical benefit (Decision Curve Analysis), demographic subgroup fairness, and executable code/weights availability.
The 5 Common Fatal Flaws in Clinical AI Manuscripts
Reviewers at The Lancet Digital Health, JAMA Network, Radiology: AI, npj Digital Medicine, and JAIM routinely reject clinical machine-learning manuscripts for five preventable methodological omissions:
- 1. Reporting AUROC without Calibration: High discrimination (AUROC) does not prove risk estimates are accurate. Always include calibration plots (observed vs. predicted probability) and Decision Curve Analysis (`dcurves`).
- 2. Slice-Level or Encounter-Level Data Leakage: In medical imaging and longitudinal EHR studies, train/validation/test splits must be strictly partitioned at the patient level (and ideally evaluated on an external hospital site).
- 3. Missing Sample Size Justification: Use Riley et al.'s `pmsampsize` framework rather than arbitrary dataset convenience.
- 4. Unexamined Demographic & Scanner Subgroup Performance: Report performance stratified by age, sex, race/ethnicity, and imaging hardware vendor.
- 5. Opaque Preprocessing & Missing Code: Deposit preprocessing, model architecture, and evaluation scripts in GitHub/Zenodo.