AI versus clinicians: who diagnoses MS faster and better?

Gen Z will in all probability make increasing use of ChatGPT and other AI platforms for personalised healthcare guidance. In a study comparing its diagnostic accuracy with that of clinicians, ChatGPT performed well, but responses were not always generalisable to all users and bias may exist in select groups.

Many hospitals observe that the patients they see for the first time have already used online generative artificial intelligence (AI) platforms for medical information and advice, even though these platforms are not principally designed for this goal. This is especially true for the younger generation, the so-called Generation Z, who will predominantly be part of patients who will receive a diagnosis of MS in the next 10 years.

Ms Mahi Patel (UT Southwestern, TX, USA) and colleagues wanted to put ChatGPT to the test and see how useful it can be in the diagnosis of MS [1]. The study's objective was to determine whether patients with MS can be diagnosed earlier by ChatGPT than their clinical timeline and to assess whether accuracy differed based on age, sex, and race/ethnicity.

The study included 75 patients with MS between 18 and 59 years of age. Of these, 50 were Gen Z (i.e. ≤27 years) and had a mean age at first symptom of 20.6 years. The 25 non-Gen Z patients had a mean age at first symptom of 34.2 years. Their timeline was retrospectively identified and simulated using ChatGPT-3.5. A total of 386 digital simulations that represented the original cohort were generated. Thus, a total of 461 scripted conversations were available for analysis.

The researchers observed that the time to diagnosis was significantly longer in the clinic than with ChatGPT: 0.39 years versus 0.08 years (P<0.013). The diagnostic accuracy rate was 84.8% for a single diagnosis after the inclusion of MRI data. Some interesting differences were noted: men were 47.2% less likely to be correctly diagnosed than women (P=0.04) before the inclusion of MRI data; after the inclusion of MRI data, the diagnostic accuracy was 72.9% less in men than in women (P=0.004), and 69.8% less in White than in non-White individuals (P=0.003); and the odds of an accurate diagnosis was 3.8 times higher for Gen Z versus non-Gen Z (P=0.007).

Patients and physicians should be aware that generative AI platforms are not (yet) designed for personalised healthcare guidance, although an increase in their use is anticipated.

1. Patel M, et al. Generative artificial intelligence versus clinicians: who diagnoses multiple sclerosis faster and with greater accuracy? P600, ECTRIMS 2024, 18–20 September, Copenhagen, Denmark.

 

Copyright ©2024 Medicom Medical Publishers



Posted on 18 November 2024