ChatGPT Health, a widely used artificial intelligence (AI) tool that provides health guidance directly to the public — including advice on how to seek emergency medical care — may fail to properly direct users to emergency care in a significant number of serious cases, according to researchers at the Icahn School of Medicine at Mount Sinai.
The study, fast-tracked in the February 23, 2026 online issue Nature Medicine [https://doi.org/10.1038/s41591-026-04297-7]is the first independent safety assessment of the LLM-based tool since its launch in January 2026. It also identified serious concerns about the tool’s safeguards against suicidal ideation.
“LLMs have become patients’ first port of call for medical advice—but in 2026 they are less secure on the clinical fringes, where judgment separates missed emergencies from unnecessary alarm,” says Isaac S. Kohane, MD, PhD, Chair, Department of Biomedical Informatics at Harvard Medical School, who was not involved in the research.. “When millions of people use an AI system to decide if they need emergency care, the stakes are extremely high. Independent evaluation should be routine, not optional.”
Within weeks of its launch, ChatGPT Health’s maker, OpenAI, reported that about 40 million people use the tool daily to seek health information and guidance, including advice on whether to seek emergency or urgent care. At the same time, the researchers say, there was little independent evidence about how safe or reliable his advice actually was.
This gap prompted our study. We wanted to answer a very basic but critical question: if someone is experiencing a real medical emergency and reaches out to ChatGPT Health for help, will it clearly tell them to go to the emergency room?”
Ashwin Ramaswamy, MD, lead author, Instructor in Urology, Icahn School of Medicine, Mount Sinai
Regarding suicide risk alerts, ChatGPT Health was designed to direct users to the 988 Suicide and Crisis Lifeline in high-risk situations. However, the researchers found that these alerts appeared inconsistently, sometimes triggering lower-risk scenarios, while – alarmingly – failing to appear when users described specific plans to self-harm.
“This was a particularly surprising and disturbing finding,” says senior and co-corresponding author of the study Girish N. Nadkarni, MD, MPH, Barbara T. Murphy Chair of Windreich’s Department of Artificial Intelligence and Human Health, Director of the Hasso Plattner Institute for Digital Health, and Irene and Dr. Sinai, and Chief AI Officer of Mount Sinai Health System. “While we expected some variability, what we observed exceeded the inconsistency. The system’s alerts were inversely related to clinical risk, appearing more reliable for lower-risk scenarios than for cases where someone shared how they intended to harm themselves. In real life, when someone talks about exactly how they will harm themselves, that is a sign of more immediate, not less serious, risk.”
As part of the evaluation, the research team created 60 structured clinical scenarios covering 21 medical specialties. Cases ranged from minor conditions suitable for home care to true medical emergencies. Three independent physicians determined the correct level of urgency for each case using guidelines from 56 medical societies.
Each scenario was tested under 16 different contextual conditions, including variations in race, gender, social dynamics (such as someone minimizing symptoms), and barriers to care, such as lack of insurance or transportation. In total, the team conducted 960 interactions with ChatGPT Health and compared its recommendations to the consensus of doctors.
When testing 60 realistic patient scenarios developed by doctors, the researchers found that while the tool generally handled clear emergencies correctly, it underplayed more than half of the cases that doctors judged to require urgent care.
The researchers were also impressed by how the system failed in medical emergencies. The tool often proved to recognize dangerous findings in its own explanations, yet reassured the patient.
“ChatGPT Health has performed well in textbook emergencies like stroke or severe allergic reactions,” says Dr. Ramaswamy. “But it struggled in more nuanced situations where the risk is not immediately obvious, and these are often the situations where clinical judgment matters most. In an asthma scenario, for example, the system identified early warning signs of respiratory failure in its explanation, but recommended waiting rather than seeking emergency treatment.”
The study’s authors advise that for worsening or worrying symptoms, including chest pain, shortness of breath, severe allergic reactions or changes in mental status, people should seek medical attention directly rather than relying solely on the chatbot’s guidance. In cases involving thoughts of self-harm, people should contact 988 Suicide and Crisis Lifeline or go to an emergency department.
However, the researchers stress that the findings do not suggest that consumers should abandon AI health tools altogether.
“As a medical student in training at a time when AI health tools are already in the hands of millions, I see them as technologies that we must learn to carefully integrate into care, not substitutes for clinical judgment,” says Alvira Tyagi, a first-year medical student at the Icahn School of Medicine at Mount Sinai and second author of the study. “These systems are changing rapidly, so part of our training now must consider learning how to critically understand their results, identify where they fall short, and use them in ways that protect patients.”
The study evaluated the system at a single time point. Because AI models are updated frequently, performance can change over time, underscoring the need for independent evaluation, the researchers say.
“The start of medical education alongside tools evolving in real time makes it clear that today’s results are not static,” says Ms Tyagi. “This reality requires ongoing review to ensure that improvements in technology translate into safer care.”
The team plans to continue evaluating updates to ChatGPT Health and other consumer-facing AI tools, expanding future research into areas such as pediatric care, drug safety, and non-English language use.
The paper is titled “The performance of ChatGPT Health in a structured trial of triage recommendations.”
The authors of the study, as reported in the journal, are Ashwin Ramaswamy, MD, MPP. Alvira Tyagi, BA; Hannah Hugo, MD; Joy Jiang, PhD; Pushkala Jayaraman, PhD; Mateen Jangda, MSc; Alexis E. Te, MD; Steven A. Kaplan, MD; Joshua Lampert, MD; Robert Freeman, MSN, MS; Nicholas Gavin, MD, MBA; Ashutosh K. Tewari, MBBS, MCh; Ankit Sakhuja, MBBS MS; Bilal Naved, PhD; Alexander W. Charney, MD, PhD; Mahmoud Omar, MD; Michael A. Gorin, MD; Eyal Klang, MD; Girish N. Nadkarni, MD, MPH.
Source:
Journal Reference:
