November 3, 2026
A report published by the Israeli newspaper “Jerusalem Post” stated that a recent scientific study published in the journal “Nature Medicine” revealed the superiority of general artificial intelligence models, such as “ChatGPT”, “Gemini” and “Claude”, over specialized medical tools in answering medical questions, which raises new questions about the future of these technologies in the health sector.
The study showed that the general models outperformed specialized tools such as “OpenEvidence” and “UpToDate Expert AI,” through an evaluation that included 500 questions on medical knowledge that simulate American licensing exams, and 500 tasks to measure the agreement of answers with doctors’ judgments, in addition to 100 real questions asked by practitioners during their daily work.
In details of the results, the “Gemini” model recorded an accuracy of 97.4% in the “MedQA” medical knowledge test, followed by the “GPT” model with an accuracy of 94.2%, while the specialized medical tools recorded results of approximately 89%. Furthermore, the generic forms received higher ratings from clinicians regarding clinical accuracy, comprehensiveness, and clarity of answers.
The report attributed the superiority of general models to their ability to link multiple knowledge sources and analyze data with flexibility that goes beyond traditional medical references and guidelines, as well as benefiting from huge investments, rapid updates, and continuous testing that allow them to develop that are difficult for specialized tools to keep up with.
Despite these findings, the report stressed that specialized medical tools remain important, especially in applications related to hospital data and specific clinical tasks, with the possibility of their future integration with general models.
The report concluded that calling the system “medical artificial intelligence” does not necessarily mean its superiority in accuracy, stressing that the decisive criterion remains linked to actual performance, safety, and practical usefulness, while recalling that excellence in medical knowledge tests does not necessarily equate to proficiency in actual diagnosis and treatment, as the safety of clinical use and verification of answers remain basic considerations.
