Chat Generative Pre-trained Transformer (ChatGPT®) is a language model tool based on artificial intelligence that could analyze data from antimicrobial susceptibility tests in real time, especially in places where infectious diseases (ID) specialists are not available. Montiel-Romero, et al. (2025) evaluated the agreement between ChatGPT® and ID specialists regarding appropriate antibiotic prescription in simulated cases.
Using data from microbiological isolates recovered in our center, the authors fabricated 100 cases of patients with different infections. Each case included age, infectious syndrome, isolated organism and complete antibiogram. Considering a precise set of instructions, the cases were introduced into ChatGPT® and presented to five ID specialists. For each case, they asked, (1) “What is the most appropriate antibiotic that should be prescribed to the patient in the clinical case?” and (2) “According to the interpretation of the antibiogram, what is the most probable mechanism of resistance?”. They then calculated the agreement between ID specialists and ChatGPT®, as well as Cohen’s kappa coefficient.
Regarding the recommended antibiotic prescription, agreement between ID specialists and ChatGPT® was observed in 51/100 cases. The calculated kappa coefficient was 0.48. Agreement on antimicrobial resistance mechanisms was observed in 42/100 cases. The calculated kappa coefficient was 0.39. In a sub-analysis according to infectious syndromes and microorganisms, Agreement (range 25 – 80%) and kappa coefficients (range 0.21–0.79) varied.
The researchers conclude that they found poor agreement between ID specialists and ChatGPT® regarding the recommended antibiotic management in simulated clinical cases.
Reference: Montiel-Romero S, et al. Recommended antibiotic treatment agreement between infectious diseases specialists and ChatGPT®. BMC Infectious Diseases. Volume 25, article number 38 (2025).