The rapid development of technologies related to artificial intelligence (AI) offers new opportunities for its implementation in clinical practice. The vast availability of diverse patient data, such as medical images [1] , text, and electronic health records [2] , combined with numerous studies evaluating the potential of AI in areas such as prevention [3] , screening and diagnosis [4] , disease progression prediction [5] , and clinical decision-making and treatment selection [6] , provides the foundation for developing AI solutions that can greatly enhance the diagnostic process.

There are studies evaluating the diagnostic capabilities of various AI models [7] , but important questions arise regarding their reliability, accuracy, and ability to interact with medical professionals and patients. This study focuses on three AI models—ChatGPT-4o, CodyMD, and Dr. Gupta—with the aim of assessing their diagnostic capabilities based on patient-reported symptoms. Key aspects such as diagnostic accuracy, level of interaction, and the provided differential diagnoses are evaluated. Understanding these factors is critically important, as AI has the potential to assist medical professionals in decision-making, but there are also risks of errors, especially in more complex cases. To implement AI effectively in medical practice, research must focus on evaluating the diagnostic accuracy of various AI models. This is crucial for their development, improvement, and training of Natural Language Processing (NLP) models. Such research is vital for advancing medical care and enhancing public health, as it enriches the datasets required for model training and provides key insights for optimizing and adapting algorithms to meet real clinical needs. Therefore, this type of research is particularly relevant and valuable.

This study aims to qualitatively evaluate the diagnostic capabilities of three different AI models—ChatGPT-4o, CodyMD, and Dr. Gupta—in terms of accuracy, user approach, level of detail, user interaction, and the number of differential diagnoses generated, based solely on patient-reported symptoms. ChatGPT-4o is one of the most widely used AI models, while CodyMD and Dr. Gupta are specifically designed for medical purposes. The qualitative evaluation of their diagnostic methods in this study provides valuable insights for both users and developers.

Based on the results of our previous research [7] on the diagnostic capabilities of ChatGPT-3.5, where the AI demonstrated over 70% diagnostic accuracy, we decided to extend our research and investigate the diagnostic capabilities of ChatGPT-4o as well as other active medical AIs. The methodological approach ( Figure 1 ) follows a structured process that includes selecting the AI models, choosing clinical cases, and analyzing the diagnostic results from the AI models.

The AI diagnoses ( Table 2 ) were compared with those published in the literature and evaluated based on the criteria outlined above. The results are presented in Table 3 . The table shows that out of ten clinical vignettes, ChatGPT made correct and accurate diagnoses in nine of them. In contrast, CodyMD and Dr. Gupta provided differential diagnoses, with the correct diagnosis being the first one about half the time. This can be viewed as a positive feature of these models, as they enable physicians to explore multiple potential diagnoses instead of being restricted to just one. By promoting a broader differential diagnosis, these models help prevent premature conclusions and foster a more thorough assessment, ultimately leading to more accurate and informed decision-making in patient care.

For case 3, CodyMD assigned a probability of 20% to the correct diagnosis, and for cases 5 and 6, –30%. Dr. Gupta provided potential differential diagnoses but did not assign any probability percentages. In case 3, the correct diagnosis was listed second, while for cases 4, 5, 7, and 8, Dr. Gupta did not provide a correct diagnosis. These two AIs are designed to provide medical information, but symptoms alone may not be sufficient for accurate diagnoses due to the numerous diseases with overlapping symptoms. As a result, they may require additional information about the patient’s physiological state, including blood tests, imaging studies (e.g., X-rays, MRI, or CT scans), and other diagnostic tools. These tools can be electrocardiograms (ECGs) to assess heart function, pulmonary function tests to evaluate lung capacity, biopsies for tissue analysis, genetic testing to identify hereditary diseases and urinalysis to detect abnormalities in kidney function.

ChatGPT-4o provides only one differential diagnosis based on the patient’s symptoms, using an informative approach without engaging in direct dialogue with the patient. It communicates in accessible language for the general public and provides a brief rationale for its response. While it suggests possible conditions, it typically emphasizes the need for further investigations, tests, and a physical examination by a medical professional. Although it mentions potential causes and provides context for the symptoms, it does not offer more than one differential diagnosis compared to other AI. It often focuses on a single condition, which can be limiting given the variety of presenting symptoms. The level of detail in its responses is moderate, providing plausible explanations in accessible language. However, it lacks personalized interaction, making it less engaging for the patient.

CodyMD, on the other hand, provides three differential diagnoses for all cases and assigns a percentage probability to each potential diagnosis. This AI uses an interactive approach, creating a conversational format that resembles a doctor-patient dialogue. It asks detailed questions to gather additional information about symptoms and pain levels, aiming to refine the diagnosis. CodyMD provides a list of three potential diagnoses and offers a detailed treatment plan for one of them. The level of detail is high, including comprehensive explanations, self-care tips, and potential lifestyle changes. CodyMD is very interactive, friendly, and polite, frequently using encouraging phrases such as “Thanks for the confirmation!” and “Thanks for sharing with me!” It maintains constant communication with the patient by asking questions like “What should I call you?” and “Would you like to see a treatment plan?” This fosters a collaborative atmosphere, and strengthens the physician-patient connection, and mimics a sense of empathy for the patient.

Dr. Gupta, like CodyMD, provides differential diagnoses in most cases (see Table 2 ) but does not state the percent likelihood for each. It begins by stating the most likely diagnosis and offers a brief explanation of the condition, along with potential causes of the symptoms. Dr. Gupta employs an educational approach, making it suitable for medical students during their training. The AI explains the disease and the causes of specific symptoms, emphasizing that a formal diagnosis can only be made by a medical professional through evaluation. It suggests appropriate diagnostic steps (such as physical exams and imaging) and recommends treatment options (including pain control, hormone therapy, and surgery). While it provides clear and precise explanations, it does not delve deeply into the specifics of the disease and lacks the interactivity and personalization found in CodyMD’s responses.

The results of the present study indicate that the three AIs investigated have the potential to assist healthcare professionals in diagnosing complex cases with multiple symptoms. They can also be used as virtual assistants for initial patient consultations, referring patients to appropriate tests or specialists based on their reported symptoms. In both cases, however, optimization and refinement are needed to improve their accuracy and reliability in clinical settings. The interactivity demonstrated by Cody is particularly useful for acquiring more accurate and complete information about the disease and its history. There is a need to advance more personalized AI decision-making based on patient-supplied symptoms and the analysis of genetic and clinical data to enable more accurate diagnoses and individualized treatment plans. Future studies will need to focus not only on refining the algorithms, but also on integrating these systems into real clinical scenarios to ensure their effectiveness and practical applicability.