Promise and pitfalls of artificial intelligence (AI) chatbots in complex decision-making for thyroid nodules and papillary thyroid cancer
Authors' conclusion
AI chatbots show evolving but inconsistent performance in complex thyroid management scenarios. While guideline concordance can be relatively high, substantial variability across models, limited temporal reproducibility, and poor alignment with clinical practice highlight the need for ongoing longitudinal evaluation before safe integration into clinical decision-making.
Study context
- Population
- 12 clinical vignettes on indeterminate thyroid nodules and low- and low-to-intermediate-risk papillary thyroid carcinoma (PTC)12 vignettes
- Participants
- 201 participants
- Exposure
- AI chatbots (ChatGPT, Gemini, Copilot, DeepSeek) vs endocrinologists' predominant answers and guideline-consistent options
- Endocrinologistsn = 201reference
- Outcome
- Agreement and concordance of chatbot outputs
Methodological detailsnationwide web-based survey, clinical vignettes, two time points, 11 months apart
- Nationwide web-based survey
- Clinical vignettes
- Two time points, 11 months apart
Main finding
As reported in the abstract
Alignment between chatbots and endocrinologists' predominant responses, at most
25%
Concordance with most guideline-consistent options, up to (depending on model and time point)
83%
10/12 scenarios
Observational data: associations, not effects.
Further findings
Events
| Outcome | ChatGPT | Gemini | Copilot |
|---|---|---|---|
| Temporal stability: scenarios with changed responses across 12 scenarios · 11 months apart | 4 | 7 | 5 |
Temporal stability: scenarios with changed responses across 12 scenarios · 11 months apart
- Some updates moved closer to, others further from, guideline-based answers
Inter-model agreement
- Inter-model agreement ranged from
- 33 to 67%
- Substantial variability among chatbots
Original publication
Composed by AI from the published abstract and checked against it. How Briefs are made →