Why Generative AI Hallucinates
About hallucinations in Generative AI, why they happen, what you can do about it, and why it matters so much in healthcare.
Generative AI hallucinates. And sometimes it does it with such a great confidence.
The way it often manifests itself is coming up with those bizarre, made-up facts in the answers you get from the Generative AI model. Sometimes it is funny. Sometimes it is super annoying, because, if this is your niche area, you just know it is wrong. Either way, it is concerning. Especially when it comes to healthcare.
So, why does Generative AI go rogue sometimes and gives us nonsense? Let’s look at a few examples and then analyze the behavior behind that. Hold on to your seats.
Why it matters in healthcare
Let’s start from a few anecdotal examples of hallucinations that were observed, in the context of healthcare:
Hallucinating a symptom: when asked to summarize the patient’s complaints, a model added a symptom that was not there. In that example, the patient complained about abdominal pain and nausea, and the model added that the patient also experienced vomiting. Just like that, out of left field.
Random diagnosis that was never there: when asked to translate a clinical report to a simpler, more patient-friendly language, a model added to the simple version that the patient - who complained about abdominal pain and nausea - may be suffering from food poisoning. There was nothing about food poisoning in the original report, and the model was never asked to suggest a diagnosis. So.
Hallucinating laterality: when asking a model to summarize a medical encounter based on an audio recording of a physical encounter, where a patient complained about leg pain, a model added that it was the left leg, even though the patient never explicitly mention which leg they were talking about - the patient just pointed at it in the exam room.
Making up links: when asked about a certain clinical terminology that the model couldn’t find an answer for, the model made up an answer - a wrong answer - and even came up with links as evidence - links that either did not exist, or did not support the answer. Quite annoying.
And a weird one - making up a disease name: When asked to present questions to the patient about their medical history, a model asked whether the patient was suffering from a condition name that does not exist - as in, totally made up, not even a real word.
Hallucinations are an issue in a healthcare setting because they could have physical implications and even impact a patient life.
Not at all cool. That extra creativity could be a big deal when it comes to healthcare. We do not want the AI to add symptoms that were not reported. We don’t want AI to diagnose on its own. We don’t want AI to make up laterality.
Why? Obviously, because that could have physical implications and even impact a patient life.
Why this happens
Generative AI hallucinates because it doesn't really “know” anything —it predicts.
Generative AI models like GPT are trained on many diverse datasets. They learn patterns of how words typically follow one another. When you ask a question or give a prompt, the model generates text by statistically predicting what words are most likely to come next based on the training data.
Hallucinations occur when the model predicts something that makes sense but isn't backed up by facts. The model “fills gaps” with details that seem logical, to provide a coherent response. Problem is, sometimes those details may be made up.
One of the issues is over-reliance on pattern recognition without deeper understanding. The model works through statistical predictions, based on patterns it has seen in the past, not through true understanding of the data. To follow the above example, and put it in very simple terms, if the model has seen a pattern of abdominal pain and nausea often coming with vomiting, the model is more likely to add vomiting to the list of symptoms even if it was not reported by the patient.
Bias in models can come from the data they have been trained on, if the data itself is biased.
Replicating or amplifying biases is another reason. Bias in models can come from the data the model has been trained on, if the data itself is biased, for example - an overrepresented group, or data that is re-enforcing a stereotype. To follow a simple example again, if the model has only been exposed to data of people with pain in their left leg, it is more likely to infer laterality of the pain to be in the left leg, even if the patient has not explicitly said which leg hurts.
It is important to say that biases can also come from the user of the model, sometimes resulting from low quality prompting or misinterpreting the results. Learning to understand AI and how to work with it is sometimes referred to as the AI literacy.
Generative AI models are optimized to provide answers, no matter what.
Another problem is that those models are optimized to answer, no matter what. Tools like ChatGPT are designed to please the user by giving answers, even if they are not certain they have the right answer. Not providing an answer is just not seen by those tools as a good option. Combine that with their overly-confident tone, you end up with a model that provides a wrong answer with high over-confidence, aka hallucination.
As a side note - as I previously shared, that behavior often reminds me of an arrogant child that is eager to show-off its knowledge. Think Sheldon Cooper. Now wouldn’t that be a perfect voice for ChatGPT?
What you can do about it
A lot, actually. Here goes:
Better prompting: be clear and specific in your prompts to help the model provide grounded responses. Make sure to ground the model in credible sources.
OK to not know: allow the model to not know and not provide an answer if it cannot find an answer in high enough certainty.
Ask for evidence: instruct the model to provide sources of evidence for its answers.
Confidence level: Ask the model to provide the level of confidence of the result.
Human in the loop: oversight and validation of the results by a medical professional is key to leveraging Generative AI in healthcare more safely.
Fact-checking mechanisms: last but not least, cross-checking the AI output against the grounding data can be extremely helpful in detecting hallucinations in the results.
One more thing
Fact checking Generative AI results for hallucinations - and for omissions too, while we are at it - is critical when it comes to healthcare. This is why we included hallucinations and omissions detection in the Clinical Safeguards we have recently announced as Preview. We nickname that specific safeguard “HALOM” - which stands for Hallucinations/Omissions, but also means “dream” in Hebrew… classy.
And if you ask me, I predict that those fact-checking mechanisms are soon going to become mandatory.
Share your AI hallucinations
What AI hallucinations have you seen? Did you ever get AI answers that were completely off? Share your own AI hallucination stories and WTF moments in the comments section below!
About Verge of Singularity.
About me: Real person. Opinions are my own. Blog posts are not generated by AI.
See more here.
LinkedIn: https://www.linkedin.com/in/hadas-bitran/
X: @hadasbitran
Instagram: @hadasbitran
Recent posts:
#AIhalucinations
As part of my daily trial-and-error experiment with Gen AI, I asked ChatGPT the following question:
Who wrote the song "Shir Ahava Bedou’i" (Bedouin Love Song) by David Broza?
I received an answer detailing who wrote, composed, and performed the song. Initially, I didn’t realize the answer was incorrect since I wasn’t familiar with the identity of the actual writer. Once I discovered the error, I repeated the question and received a different set of incorrect answers. When I pointed out the mistake, I received a response saying, “I apologize, the correct answer is…”—but that, too, was incorrect.
To investigate further, I repeated the question approximately 10 more times and received 10 different incorrect responses. ChatGPT seemed to spiral into a "negative feedback loop."
For comparison, I switched to Claude AI and asked the same question. While I also received an incorrect answer there, the response to my correction was different. Claude replied with something like: “Please help me improve—can you tell me who wrote the song?” This indicated that it, too, would not provide the correct answer.
I then turned to Gemini by Google, where I received a response in a different style—this time correct. The reply was something like:
“There is some uncertainty regarding the identity of the writer, but most evidence points to [the correct name].”
It seemed to be statistically derived, yet accurate.
Note: Before composing this reflection, I asked ChatGPT the question again—and it still doesn’t provide the correct answer....
Example of a total hallucination - asked ChatGPT whether RadLex is included in UMLS and requested for evidence with links. It responded yes, it is, and even provided evidence with 2 links. But the truth is, RadLex is not part of UMLS. As for the evidence, the first link was real, but did not support the answer, and the second link didn't even exist. Relatively benign hallucination, but still annoying.