Assessment of the Diagnostic Performance of a Commercially Available Artificial Intelligence Algorithm for Risk Stratification of Thyroid Nodules on Ultrasound
Authors' conclusion
Koios DS demonstrated standalone performance similar to radiologists, though with lower sensitivity and higher specificity. Performance was best when radiologist interpretations were combined with the AI-adapter component, with improved specificity and reduced unnecessary FNA recommendations.
Study context
- Population
- Consecutive thyroid ultrasound (US) with ≥1 nodule over one year; ground truth by surgical pathology, fine needle aspiration (FNA), or 3-year follow-up US showing stability649 nodules
- Participants
- 347 participants
- Comparison
- Food and Drug Administration (FDA)-cleared artificial intelligence (AI) tool (Koios DS Thyroid) with or without AI-adapter vs clinical radiology report, alone or combined with AI-adapter
- Outcome
- Performance of final recommendations (FNA or no FNA) vs ground truth: sensitivity, specificity, ROC analysis
Methodological detailsretrospective, consecutive, two centres (Duke University Hospital and affiliate community hospital)
- Retrospective
- Consecutive
- Two centres (Duke University Hospital and affiliate community hospital)
Performance
As reported in the abstract
Diagnostic performance of final FNA recommendation vs ground truth
| Test or model | AUC | Sensitivity | Specificity |
|---|---|---|---|
| Koios with AI-adapter | 0.70 (CI 0.60-0.81) | 0.69 (CI: 0.50-0.83) | 0.63 (CI: 0.59-0.67) |
| Radiologists | 0.70 (CI 0.60-0.79) | 0.81 (CI: 0.61, 0.92) | 0.43 (CI: 0.38-0.48) |
| Radiology report + AI-adapter | 0.76 (CI: 0.67-0.85) | 0.81 | 0.53 (CI: 0.49-0.58) |
AUC
Koios with AI-adapter0.70 (CI 0.60-0.81)
Radiologists0.70 (CI 0.60-0.79)
Radiology report + AI-adapter0.76 (CI: 0.67-0.85)
Sensitivity
Koios with AI-adapter0.69 (CI: 0.50-0.83)
Radiologists0.81 (CI: 0.61, 0.92)
Radiology report + AI-adapter0.81
Specificity
Koios with AI-adapter0.63 (CI: 0.59-0.67)
Radiologists0.43 (CI: 0.38-0.48)
Radiology report + AI-adapter0.53 (CI: 0.49-0.58)
- Radiology report + AI-adapter vs Radiologists: McNemar's test p < 0.001 — specificity improved from 0.43 to 0.53
- Radiology report + AI-adapter vs Radiologists: P = 1 — unchanged sensitivity
- Fewer FNA recommendations with radiology report + AI-adapter
- 17%
Original publication
Assessment of the Diagnostic Performance of a Commercially Available Artificial Intelligence Algorithm for Risk Stratification of Thyroid Nodules on Ultrasound. Thyroid. Published 15 October 2024. doi:10.
Composed by AI from the published abstract and checked against it. How Briefs are made →