Model Evaluation and Interpretation for Medical Diagnosis
A hospital uses Alteryx Machine Learning to build a model that predicts the likelihood of a patient having a disease based on lab results and demographics. The model will be used as a screening tool to prioritize patients for further testing. The dataset has 5,000 patients, with 10% positive cases. The model achieved an AUC of 0.85 on the test set. The hospital wants to understand which features are most important and how the model makes decisions, to ensure it aligns with medical knowledge and to gain trust from clinicians.
1. What metrics beyond AUC would you report to evaluate the model's clinical utility? Explain how you would choose an operating threshold.
Read the worked answer
AUC alone is not sufficient for clinical utility. We should report: sensitivity (recall) and specificity at various thresholds, positive predictive value (PPV) and negative predictive value (NPV), and the confusion matrix. Since the model is a screening tool, high sensitivity is crucial to not miss positive cases. We can plot the ROC curve and precision-recall curve. To choose a threshold, consider the clinical context: the cost of a false negative (missing a disease) is high, so we might choose a threshold that yields high sensitivity (e.g., >90%) while maintaining reasonable specificity. We can use the 'Threshold Analysis' in Alteryx Machine Learning to see trade-offs. Also, consider the prevalence: PPV depends on prevalence, so in a screening population with lower prevalence, PPV may be lower. We can use decision curve analysis to assess net benefit.
2. How would you interpret the model's feature importance and ensure it makes sense clinically? What tools in Alteryx Machine Learning can help?
Read the worked answer
Use global feature importance (e.g., permutation importance, SHAP values) to rank features. In Alteryx Machine Learning, the 'Feature Importance' chart and 'SHAP' summary plot can show which features drive predictions overall. Then, examine individual predictions with SHAP force plots or partial dependence plots to understand relationships. Clinicians should review the top features: do they align with known risk factors? If not, investigate data quality or potential leakage. For example, if 'patient ID' is important, that's suspicious. Also, check for interactions. Use the 'Partial Dependence Plots' to see how each feature affects the prediction on average. Validate with domain experts: if the model relies on unexpected features, it may indicate bias or confounding. Document the interpretation and share with clinicians for feedback.