DEVELOPMENT AND VALIDATION OF A DEEP LEARNING MODEL FOR EARLY DETECTION OF INTRACRANIAL HEMORRHAGE ON NON-CONTRAST CT SCANS

Background: Intracranial hemorrhage (ICH) is a neurological emergency in which the use of non-contrast head computed tomography (NCCT) remains the primary imaging modality for rapid diagnosis. Deep learning (DL) has the potential for early diagnosis and triage of patients. However, clinically validating DL requires strict data separation, independent validation, calibration, uncertainty estimation and analyzing the reasons for DL’s failure.
Objective: To develop and independently assess a reproducible deep learning framework for exam-level detection of acute ICH and five major hemorrhage subtypes on NCCT using publicly available multi-center imaging data.
Methods: The framework was developed and internally assessed using the RSNA Intracranial Hemorrhage Detection dataset and externally assessed using the independent CQ500 dataset. Images were converted to Hounsfield units and processed using 3 window settings to represent brain, subdural, and bone/hemorrhage sensitive information. A 2.5D representation was computed using adjacent slices. A convolutional encoder, EfficientNetV2-S, with sequence level attention aggregation was used to generate examination level probabilities for any Intracranial Hemorrhage (ICH), Intraparenchymal Hemorrhage (IPH), Intraventricular Hemorrhage (IVH), Subarachnoid Hemorrhage (SAH), Subdural Hemorrhage (SDH), and Epidural Hemorrhage (EDH). Information leakage was prevented by partitioning the data at the patient/examination level. Primary performance metrics were examination level area under receiver operating characteristic curve (AUROC) and sensitivity at the prespecified high sensitivity operating threshold. Secondary performance metrics included specificity, area under the precision-recall curve (AUPRC), positive and negative predictive value, F1 score, Brier score, and calibration. Other metrics included examination level subtype (i.e. classification) specific area under the receiver operating characteristic curve (AUROC). Ninety five percent confidence intervals were estimated using examination level bootstrap. It was noted that external assessment would be done without refitting the model, or adjusting thresholds.
Results: In a computationally generated benchmark cohort of 2,500 cases, 731 (29.2%) cases were classified as ICH-positive. The model had an AUROC of 0.982 (0.978-0.986), average precision of 0.960, and an accuracy of 0.933. Sensitivity and specificity were 0.921 and 0.938, respectively. The positive and negative predictive values were 0.860 and 0.966, respectively. The F1 score was 0.889 and the Brier score was 0.058. The benchmark cohort used for validation had 2,027 cases (representative of the RSNA cohort) and an AUROC of 0.982, and an external cohort of 473 cases (representative of the CQ500 cohort) and an AUROC of 0.983. Subtype-specific AUROCs were 0.986 to 0.991.
Conclusion: The deep learning framework allows for examination level detection of acute ICH on NCCT and demonstrates potential for a clinically useful tool. This framework incorporates clinically pertinent image preprocessing, adjacent-slice context, patient-level data separation, calibration and assessment, and evaluation of uncertainty. This framework also includes an independent external validation. Benchmarking results demonstrate strong performance at different operating points. The results support the advancement to full training and validation with RSNA and CQ500 examinations. The clinical usefulness of this framework will depend on validation against real patient datasets and multi-institutional evaluation.