→Claude Just Joined Your Meetings, But Only on Google Meet →An AI Vendor Breach Just Leaked Uber Driver IDs and Notion Chat Logs at the Same Time →A Safety Reminder Cut Harmful AI Medical Advice by a Third. It Still Got 1 in 10 Wrong. →A Power Bank Just Launched That Costs 75 Percent More to Not Catch Fire →Firefox's New Mini Window Is Genuinely Useful, With One Annoying Catch →How to Actually Check if Your Email or Password Was Leaked in a Data Breach →Google's New AI Model Reads DNA Cell by Cell, and the Numbers Are Real →Perplexity's Comet Browser Is Free Now: What You Gain and What You're Trusting It With →Dropbox Just Spent $350 Million on a Storage Startup to Escape Being Dropbox →AI Is Not Taking Your Job. It Might Just Be Freezing Your Raise →NotebookLM Is Gone, Gemini Notebook Is Here: What Actually Changed →Four AI Models, Five Accents, One Hierarchy: What a New Study Found →Claude Just Joined Your Meetings, But Only on Google Meet →An AI Vendor Breach Just Leaked Uber Driver IDs and Notion Chat Logs at the Same Time →A Safety Reminder Cut Harmful AI Medical Advice by a Third. It Still Got 1 in 10 Wrong. →A Power Bank Just Launched That Costs 75 Percent More to Not Catch Fire →Firefox's New Mini Window Is Genuinely Useful, With One Annoying Catch →How to Actually Check if Your Email or Password Was Leaked in a Data Breach →Google's New AI Model Reads DNA Cell by Cell, and the Numbers Are Real →Perplexity's Comet Browser Is Free Now: What You Gain and What You're Trusting It With →Dropbox Just Spent $350 Million on a Storage Startup to Escape Being Dropbox →AI Is Not Taking Your Job. It Might Just Be Freezing Your Raise →NotebookLM Is Gone, Gemini Notebook Is Here: What Actually Changed →Four AI Models, Five Accents, One Hierarchy: What a New Study Found
Panda Tech Bytes
All Posts
Latest AI Trends Guides Tools Tech Pulse Hardware & Gear Funding Watch Risk & Reality What It Means for You Security Watch On Your Desk
A Safety Reminder Cut Harmful AI Medical Advice by a Third. It Still Got 1 in 10 Wrong.

A Safety Reminder Cut Harmful AI Medical Advice by a Third. It Still Got 1 in 10 Wrong.

A Mount Sinai study ran 20 AI models through more than 10 million clinical scenarios and found that one short safety sentence cut harmful choices from 16.6% to 10.1%. That is real progress and still roughly 1.18 million unsafe answers.

Researchers at the Icahn School of Medicine at Mount Sinai wanted to answer a simple question: if you remind an AI model to be careful before asking it a risky medical question, does it actually behave more safely. The answer, published in Communications Medicine in September 2026, is yes, meaningfully. It is also a clear demonstration of how far these models still are from something you would want making decisions on its own.

The setup: 20 models, 10 million responses

The team, led by Dr. Mahmud Omar and Dr. Girish Nadkarni of Mount Sinai's Windreich Department of Artificial Intelligence and Human Health, tested 20 large language models against 501 variations of 50 clinical scenarios, plus 100 cases pulled from deidentified hospital discharge records. Each scenario was run 10 times with the order of answer choices shuffled, producing more than 10 million individual responses in total. That scale is unusually large for this kind of safety research, which is part of why the result is worth taking seriously rather than treating as a one-off anecdote.

The scenarios were not abstract trick questions. They mirrored the kind of pressure a real clinical setting creates, like a request framed as urgent, or an instruction that sounded like it came from a superior, pushing the model toward a shortcut that was not actually safe.

What "unsafe" looked like in practice

The researchers flagged specific categories of harmful choices, including:

Without any intervention, the models made a potentially harmful choice in 16.6% of responses. Across the full set of over 10 million outputs, that works out to roughly 1.18 million harmful clinical choices. That is the baseline these models operate from when nobody has told them to be cautious.

One sentence changed the outcome

The researchers then tested three short safety reminders, brief prompts inserted before the clinical question that told the model to prioritize patient safety and flag risky instructions rather than simply comply with them. The effect was immediate and fairly consistent: harmful choices dropped from 16.6% to 10.1%, and the improvement held across 19 of the 20 models tested.

That is a real reduction, not a marginal one. It means a huge share of the dangerous responses these models were generating could be prevented with something as simple as a sentence of instruction, no retraining, no new model version, no architecture change. For an industry that often reaches for expensive fixes, that is a notable finding on its own.

The part that should not get lost

Dr. Omar, the study's first author, put the limitation plainly: "A simple safety reminder reduced potentially harmful choices in most models we tested, which is encouraging. But it did not eliminate them." Even with the reminder in place, roughly 1 in 10 responses was still a potentially harmful clinical choice. Scaled back up to the study's full response count, that is still hundreds of thousands of unsafe answers, just with a smaller share than before.

The study also doesn't name which specific models it tested or which one failed to improve, which is a real gap in the publicly available coverage. That matters for anyone trying to judge whether a particular product they use is more or less exposed to this problem. Until that detail surfaces, the honest takeaway is about the category of models in general, not any single named AI assistant.

Why a prompt is not a fix

The researchers were careful to frame the safety reminder as one layer in a system, not a solution by itself. Their recommendation was that developers and health systems build automated safety testing directly into how clinical AI tools get evaluated, both before they are deployed and again every time the underlying model is updated. A model that passed a safety check in one version can regress in the next one if nobody is checking.

That framing matters beyond hospitals. The same basic lesson applies to anyone using a general AI chatbot for health questions, medication decisions, or anything with real consequences if the answer is wrong: a well-worded prompt can nudge a model toward more careful answers, but it is not a substitute for a professional checking the actual decision.

The takeaway

The interesting thing about this study is that it resists a tidy conclusion in either direction. A one-line safety reminder cutting harmful clinical choices by a third is a genuinely useful, cheap intervention that more AI products should be using today. But the fact that it still left 1 in 10 responses potentially harmful, in a sample of over 10 million, is the more important number for anyone tempted to treat an AI chatbot as a stand-in for a doctor. The gap between "better" and "safe" is exactly where the real risk still lives.