MON-391 The Large Language Model GPT-4 Compared to Guidelines on Thyroid Nodule Management Under Conditions of Clinical Uncertainty

Abstract

Disclosure: J.H. Flory: None. A. Petrov: None. M. Tuttle: None.

Background: Large language models (LLMs) may have a role in providing clinical decision support. If a clinical question has a clear right answer, modern LLMs often show impressive accuracy providing this answer. However, for clinical questions where there is genuine uncertainty (i.e., equipoise) as to the best course of action (for example, whether to biopsy a thyroid nodule of intermediate size and risk characteristics) LLM performance is not straightforward to characterize. Methods: The LLM GPT-4 was queried with 100 vignettes describing hypothetical patients with an incidentally discovered thyroid nodule. Patient characteristics (including demographics, comorbidities, and treatment preferences), nodule size, and nodule radiographic characteristics were varied at random. GPT-4 was asked to choose whether the nodules should be biopsied. Five different query formats (‘prompts’) were tested: a ‘simple’ prompt; a ‘less-aggressive’ prompt that requested to minimize unnecessary biopsies; an ‘ATA’ prompt that requested to follow American Thyroid Association guidelines, a ‘TI-RADS’ prompt that requested to follow the Thyroid Reporting and Data System guidelines, and a ‘TI-RADS detailed’ prompt that also provided a 225-word precis of the TI-RADS guidelines. Results: Overall rates of biopsy recommendation ranged from 67% for the ATA prompt to 33% for the less-aggressive prompt. Recommendations varied according to patient and nodule characteristics. For example, for nodules with TIRADS score 4-6 (‘intermediate risk’) and size < 1 cm, GPT-4 never recommended biopsy; for size 1-1.5 cm, the ‘TI-RADS detailed’ prompt recommended biopsy in 13% versus 20% for the ‘less-aggressive prompt’ and 33% for the other 3 prompts; for size 1.5-2.5 cm the biopsy recommendation rate ranged from 9% for the ‘less-aggressive’ prompt to 63% for the ‘TI-RADS detailed’ prompt; for size > 2.5 cm biopsy rates ranged from 31% for the ‘less aggressive’ prompt to 96% for the ‘TI-RADS detailed’ prompt. Recommendations were further personalized by patient characteristics. For example, patients who were randomly assigned severe comorbidities such as end stage heart failure were less likely to be recommended for biopsy (OR 0.21, 95% CI 0.05 - 0.91) Conclusions: GPT-4 provided thyroid nodule management recommendations that frequently varied from clinical guidelines, even when a brief precis of a relevant guideline was provided to the model as a reference. In at least some cases the variation may have reflected appropriate personalization of care (e.g., avoiding biopsy in patients with major comorbidities and limited life expectancy). These findings highlight the need for experts to make value judgments when interpreting and designing LLMs for clinical decision support. We recommend against reliance on LLM output until it is shown to align not just with clinical guidelines but also with patient and clinician preferences.

Presentation: Monday, July 14, 2025