Full text

Turn on search term navigation

Introduction

The term artificial intelligence (AI) was first coined during the Dartmouth Summer Workshop in 1956, where the technology was widely referred to as "thinking machines". In simple terms, AI can be defined as the capability of a machine to learn and recognize patterns and relationships from a sufficient number of representative examples, and to use this information effectively for decision-making based on unknown data [1]. Despite the extensive media attention given to AI, many healthcare professionals still perceive AI as a "black box," leading to both inflated expectations and unfounded fears [2].

With the advent of new AI technologies, the medical environment has undergone significant transformations. Patient diagnosis, based on radiological, pathological, endoscopic, ultrasonographic, and biochemical tests, has been enhanced with greater accuracy and reduced human workload. Furthermore, medical treatments during the perioperative period, which includes preoperative preparation, the surgical phase, and postoperative recovery, have witnessed significant improvements, resulting in more favorable surgical outcomes. AI technology has also played a crucial role in other areas, such as drug production, medical management, and medical education, steering these fields toward new approaches and solutions [3].

Functional neurosurgery, a complex and continuously evolving subspecialty, has been pivotal in treating neurological disorders that affect vital functions such as movement, sensory perception, and behavior. With the advent of AI in the field of medicine, new possibilities have emerged for enhancing diagnostic accuracy and therapeutic efficacy, especially in areas where clinical complexity challenges even the most experienced specialists. This study aims to explore the capabilities of various AI models - ChatGPT-3.5, ChatGPT-4, Perplexity, Gemini, and AtlasGPT - in formulating diagnoses and developing treatment plans for complex clinical cases in functional neurosurgery. The research addresses not only the accuracy of the AI models but also their ability to integrate clinical and radiological data in high-complexity contexts, a crucial competency in functional neurosurgery.

Materials and methods

Ten clinical cases related to functional neurosurgery were selected from the medical literature [4] to ensure clarity and reduce ambiguity, making the diagnosis and treatment as straightforward as possible. Each clinical case was presented to five AI systems: ChatGPT-3.5, ChatGPT-4, Perplexity, Gemini, and AtlasGPT. Before presenting each case, the following directive was given to the AI systems: "Imagine yourself as a functional neurosurgeon and propose the diagnosis and therapeutic approach for the patient. Subsequently, share your conclusions in a precise and specific manner, using scientific and medical terminology to detail the treatment and diagnosis." This directive was followed by the presentation of the case and two questions regarding the diagnosis and the most indicated treatment. It is noteworthy to say that the prompt did not limit AI’s answers to FDA-approved treatments. Radiological images were provided to the AI systems for cases that included such images, utilizing the upload option where available.

The responses from the AI systems were collected and then evaluated by a panel of seven functional neurosurgeons. Each response was scored on a scale from 0 to 10, assessing both the diagnosis and the treatment recommendation. The collected scores were subjected to statistical analysis. The analysis involved conducting a one-way ANOVA test to identify any statistically significant differences between the AI systems' performance. Median, quartile 1 (Q1), quartile 3 (Q3), and interquartile range (IQR) were used to statistically represent the scores of each AI. Further analysis entailed a post-hoc analysis using Tukey’s test to identify specific differences by comparing each AI system against the others. A p-value <0.05 and a 95% confidence interval (CI) were used as the criteria for significance. RStudio version 2023.12.1+402 "Ocean Storm"; the R packages ggplot2, dplyr, gridExtra, and grid were utilized for data visualization and analysis.

This study did not involve the use of human subjects or patient data, and it solely utilized clinical cases that are publicly available in the medical literature. Therefore, no institutional review board (IRB) or ethics committee approval was required.

Results

Baseline diagnosis and treatment: AI models' responses

Table 1 presents a detailed comparison of diagnoses and treatments for various neurosurgical functional disorders as determined by different AI models, including AtlasGPT, Perplexity, ChatGPT 3.5, and ChatGPT 4, alongside original clinical decisions. Overall, AI models frequently align with the original diagnoses, such as identifying Parkinson's disease in a case initially diagnosed as advanced Parkinsonism. Treatment recommendations from AI models showed some variations. For instance, while most models suggested deep brain stimulation (DBS) for Parkinson's disease, treatments for medically refractory epilepsy varied, with recommendations including responsive neurostimulation (RNS) and laser interstitial thermal therapy (LITT). The models consistently suggested treatments that aligned with or expanded upon the original treatment plans, indicating a level of agreement among AI models in their clinical decision-making capabilities.

Table 1

Baseline characteristics of the included cases and AI models' responses

CPSP: central post-stroke pain; DBS: deep brain stimulation; DREZ: dorsal root entry zone; LITT: laser interstitial thermal therapy; MR: magnetic resonance; MVD: microvascular decompression; RNS: responsive neurostimulation; SRS: stereotactic radiosurgery; STN: subthalamic nucleus; TN: trigeminal neuralgia

Case	Original diagnosis	Original treatment	AtlasGPT diagnosis	AtlasGPT treatment	Perplexity diagnosis	Perplexity treatment	ChatGPT-3.5 diagnosis	ChatGPT-3.5 treatment	ChatGPT-4 diagnosis	ChatGPT-4 treatment
1	Advanced parkinsonism	Bilateral STN DBS	Parkinson's disease	STN DBS	Parkinson's disease	DBS	Parkinson's disease	DBS	Parkinson's disease	DBS
2	Medically refractory epilepsy	Vagal nerve stimulator	Refractory epilepsy	RNS	Medically refractory epilepsy	LITT	Medically refractory epilepsy	LITT or RNS	Drug-resistant epilepsy	Minimally invasive surgery: LITT or SRS might be considered if a focal lesion becomes identifiable. RNS system
3	Temporal lobe epilepsy	Complete anterior temporal lobectomy	Right mesial temporal lobe epilepsy	Anteromedial temporal lobectomy	Mesial temporal lobe epilepsy	Anterior temporal lobectomy with amygdalohippocampectomy	Mesial temporal lobe epilepsy	Anterior temporal lobectomy or selective amygdalohippocampectomy	Temporal lobe epilepsy	Anterior temporal lobectomy or selective amygdalohippocampectomy
4	Essential tremor	DBS targeting the ventral intermediate STN bilaterally	Bilateral upper extremity action tremor	DBS and thalamotomy	Essential tremor with alcohol-responsive tremor subtype	DBS-focused ultrasound thalamotomy	Essential tremor	Targeting the ventral intermediate STN with DBS	Essential tremor	DBS: targeting the ventral intermediate STN
5	Type I TN	Initially, carbamazepine (Tegretol), in case of failure of medications, MVD via the retrosigmoidal approach	TN	Neurovascular compression, MVD	TN	MVD	TN secondary to vascular compression	MVD	TN	MVD
6	Intractable pain syndrome likely from deafferentation from his prior stroke	Multidisciplinary treatment prior to consideration for motor cortex stimulation	CPSP or thalamic pain syndrome	Motor cortex stimulation and DBS	CPSP, also known as thalamic pain syndrome	DBS	Post-stroke neuropathic pain	DBS	CPSP- thalamic pain syndrome	Motor cortex stimulation or DBS
7	Deafferentation pain of the right upper extremity	DREZ lesioning	Neuropathic pain refractory to conservative management	Neuromodulation: spinal cord stimulation or DREZ lesioning	Brachial plexus avulsion showing evidence of pseudomeningoceles from C5 to T1 on the right side and a flail arm	Consider surgical options such as pseudomeningocele repair, neuroma excision, or spinal cord stimulation	Neuropathic pain	Surgical Intervention: neurostimulation	Post-traumatic brachial plexus injury leading to a flail arm, associated with severe neuropathic pain	DREZ lesioning procedure to target the sensory pathways at the root entry zone
8	Medically refractory epilepsy that appears to be of extratemporal origin	Craniotomy and cortical grid placement, antiepileptic drug, and/or placement of vagal nerve stimulator should be considered	Focal epilepsy with secondary generalization	Resective surgery	Drug-resistant epilepsy	Surgical resection of the epileptogenic focus is a viable option	Refractory epilepsy	Surgical resection of the epileptogenic focus through a procedure such as a focal cortical resection or epilepsy surgery	Drug-resistant focal epilepsy	Resective surgery such as a lobectomy or lesionectomy or RNS or DBS
9	Hypothalamic hamartoma causing gelastic seizures	Open surgical resection, SRS, and MR-guided laser ablation	Hypothalamic hamartoma	MR-guided LITT or SRS	Hypothalamic hamartoma	Surgical resection; in cases where complete resection is not feasible due to the deep location of the tumor, SRS may be considered	Hypothalamic hamartoma	Microsurgical resection or disconnection of the hamartoma from surrounding structures	Hypothalamic hamartoma	Resection or disconnection procedures: endoscopic or open surgical approaches for resecting or disconnecting the hamartoma from surrounding structures, SRS
10	Classic TN	Open surgery, or needle-based procedures (radiofrequency ablation, glycerol rhizotomy, or balloon compression)	TN	Percutaneous treatments or SRS	TN	MVD	TN	MVD	TN	MVD, balloon compression, glycerol rhizotomy, or radiofrequency thermal rhizotomy, SRS

AI models' diagnosis evaluation by case

Comprehensive analysis of the diagnostic and treatment capabilities of AIs revealed varied results: In case 1, there was a significant statistical difference between the AI models (Figure 1). Post-hoc analysis showed AtlasGPT with statistically significant differences compared to Gemini (p=0.0067) and Perplexity (Figure 2), with a median score of 9 (Q1: 9; Q3: 10; IQR: 1). In cases 2 through 10, no statistically significant differences were found between the AI models (p>0.05 for all cases, Figures 3-11).

Figure 1