A Preliminary Evaluation of the Diagnostic Performance of a Smartphone-Based Machine Learning-Assisted System for Evaluation of Clinical Activity Score in Digital Images of Thyroid-Associated Orbitopathy
Authors' conclusion
Our smartphone-based ML-assisted system shows reasonable accuracy in detecting active TAO, comparable with oculoplastic specialists and outperforming residents and general ophthalmologists.
Study context
- Population
- Patients with thyroid-associated orbitopathy (TAO); reference clinical activity score (CAS) by consensus of three oculoplastic specialists28 with active TAO
- Participants
- 100 patients
- Comparison
- Smartphone-based machine learning (ML)-assisted system vs ophthalmologists with varying clinical experience for detecting active TAO (CAS ≥3)
- Ophthalmology residentsn = 3
- General ophthalmologistsn = 3
- Oculoplastic specialistsn = 3
- Outcome
- Detection of active TAO (CAS ≥3): sensitivity, specificity and F1 score
Methodological detailspreliminary evaluation
- Preliminary evaluation
Performance
As reported in the abstract
Detection of active TAO (CAS ≥3) vs reference CAS
| Test or model | Sensitivity (top 3 smartphones) | Sensitivity | Specificity (top 3 smartphones) | Specificity | F1 score (top 3 smartphones) | F1 score |
|---|---|---|---|---|---|---|
| Smartphone-based system | 74.5% | — not stated in the abstract | 84.8% | — not stated in the abstract | 0.70 | — not stated in the abstract |
| Ophthalmology residents | — not stated in the abstract | 69.1% | — not stated in the abstract | 55.1% | — not stated in the abstract | 0.46 |
| General ophthalmologists | — not stated in the abstract | 61.9% | — not stated in the abstract | 79.6% | — not stated in the abstract | 0.55 |
| Oculoplastic specialists | — not stated in the abstract | 73.8% | — not stated in the abstract | 90.7% | — not stated in the abstract | 0.75 |
Sensitivity (top 3 smartphones)
Smartphone-based system74.5%
Ophthalmology residents— not stated in the abstract
General ophthalmologists— not stated in the abstract
Oculoplastic specialists— not stated in the abstract
Sensitivity
Smartphone-based system— not stated in the abstract
Ophthalmology residents69.1%
General ophthalmologists61.9%
Oculoplastic specialists73.8%
Specificity (top 3 smartphones)
Smartphone-based system84.8%
Ophthalmology residents— not stated in the abstract
General ophthalmologists— not stated in the abstract
Oculoplastic specialists— not stated in the abstract
Specificity
Smartphone-based system— not stated in the abstract
Ophthalmology residents55.1%
General ophthalmologists79.6%
Oculoplastic specialists90.7%
F1 score (top 3 smartphones)
Smartphone-based system0.70
Ophthalmology residents— not stated in the abstract
General ophthalmologists— not stated in the abstract
Oculoplastic specialists— not stated in the abstract
F1 score
Smartphone-based system— not stated in the abstract
Ophthalmology residents0.46
General ophthalmologists0.55
Oculoplastic specialists0.75
— not stated in the abstract
- Average sensitivity on images from all 6 smartphones
- 71.4%
- Average specificity on images from all 6 smartphones
- 81.6%
- Average F1 score on images from all 6 smartphones
- 0.66
- F1 score range by smartphone model
- 0.59-0.72
Further findings
| Outcome | Values |
|---|---|
| Predicted CAS within 1 point of reference CAS | Smartphone-based ML system 90.7% |
Original publication
A Preliminary Evaluation of the Diagnostic Performance of a Smartphone-Based Machine Learning-Assisted System for Evaluation of Clinical Activity Score in Digital Images of Thyroid-Associated Orbitopathy. Thyroid. Published 2 May 2024. doi:10.
Composed by AI from the published abstract and checked against it. How Briefs are made →