Clinical Safeguards for Generative AI

Why Agentic AI in healthcare needs domain-specific safeguards, and what many people don't understand about Generative AI.

Apr 12, 2025

Here's something many people don't understand about Generative AI:
You can build a beautiful proof-of-concept with a prompt. Few instructions, few examples, and boom, you think you've solved the problems of the universe.
It’s impressive. It’s convincing. And it’s so dangerously misleading. Because while your prototype might be gorgeous, it could give you the wrong impression that you’re just a sprint away from production.

But in healthcare, it is an illusion. You haven’t even started yet. Turning that prototype into a real, responsible, clinically solid tool is complicated. It’s a long road of hard work that requires clinical maturity, regulatory oversight, and specific safeguards - tailored for one of the most complex domains on the planet.

You can build a beautiful proof-of-concept with a prompt. But it is dangerously misleading.

Thing is, healthcare isn’t just another vertical. It’s a truly unique industry where mistakes have consequences that go way beyond financial business impact or end-users eye-rolling at you. Here, errors can harm people. Sometimes even slightly inaccurate responses could be critical. That means we can’t just bring over general-domain content safety tools or generic AI quality validation strategies and be done with it. We need solutions that understand the nuances of clinical workflows and the medical language, we need clinical validation performed by clinical experts, and we need domain-specific controls to apply AI responsibly.

Covering all those dimensions would likely require several posts. I will focus here on one dimension: domain-specific controls. And more specifically, clinical safeguards.

In healthcare, mistakes have consequences that go way beyond financial business impact or end-users eye-rolling at you.

So. Here goes.

What are Clinical Safeguards?

It was late 2022 when some of us were internally exposed, under NDA, to Generative AI. Like everyone else, we were amazed at first. It was jaw-dropping. But then, after the initial excitement, we started observing diverse weird behaviors, that, personally, gave me a serious heartburn.

We saw ungrounded answers even when grounding was in place. We saw inaccurate answers. Hallucinations were also starting to surface as a phenomena everyone was talking about - and the hallucinations were quite creative. In other words, Gen AI seemed to find many, many different ways to make up stuff. Couple of examples were described in my previous posts about why hallucinations happen and why Gen AI fails at clinical coding.

And so, seeing all those weird behaviors, we knew that’s not going to fly. We knew those things must be detected if we want to use Gen AI for real problems in healthcare. But we also knew that detecting them would not be straight-forward, and would require taking the clinical meaning and the overall clinical context into account.

Oh. And we knew we were not going to be satisfied with just asking Gen AI to verify its own output. In my native language we say don’t let the cat guard the cream - you get the analogy.

To address all of that, we created the Clinical Safeguards. Each one of them involves a set of sophisticated algorithms. They are deterministic, and they are based on healthcare-specific underlying models. We use them internally across the board in our products. They are used to detect hallucinations and omissions. To verify clinical codes are not made up. To validate outputs for certain clinical semantics. To prove claims with provenance - which means referring back the specific source. And so on.

In some cases, the Clinical Safeguards are used for evaluation of AI models against ground truth. Sometimes, they are used for further verification of Gen AI outputs. Gen AI models have also improved over time, so that combination helps. And while there’s more to be done, it is a good start.

Clinical Safeguards are used for evaluation of AI models against ground truth and for further verification of Generative AI outputs.

As a side note - there is a lot more to say about performance evaluation of AI models in this era - I will write more about that soon - so, subscribe to this blog to learn more.

The Clinical Safeguards is one toolset among many. They are part of our commitment to practicing responsible AI. We have announced the Clinical Safeguards as private preview a few months ago, and they are served as an API via our Agentic AI for healthcare service. Which also uses them internally, alongside other safeguarding mechanisms. We continue to improve those safeguards and introduce new ones.

And if I had to predict, I’d say that regulation would catch up eventually and would require applying those kinds of clinical safeguards when using AI in healthcare. Until that happens, self-regulation is emerging.

The Trustworthy & Responsible AI Network (TRAIN)

TRAIN is a consortium of leading healthcare institutions and technology partners committed to improving the safety, equity, and effectiveness of AI in healthcare. TRAIN was established in US in March 2024, and aims to transform responsible AI principles into real-world practices.

Founding members include institutions like Vanderbilt University Medical Center, Duke Health, Advocate Health, University of Texas Southwestern, and Northwestern Medicine, and Microsoft as the enabling technology partner.

TRAIN promotes improvement in AI systems that are used in clinical care by creating shared guidelines, evaluation tools, a registry of AI models, etc.

AI has incredible potential to improve healthcare, and initiatives like TRAIN help in applying it safely and responsibly.

In June 2024, TRAIN has expanded to Europe, including healthcare organizations across the EU and adding the European angle, such as multi-lingual needs and local regulatory aspects.

As a side note - I don’t think TRAIN Europe it is limited only to healthcare organizations in EU member countries. In other words, if you qualify for the Eurovision, then maybe you qualify for TRAIN EU too.

And as a side note to the side note - while I do like the Eurovision more than I am willing to admit, it is sad to see how political it has become. Much more than before.

Overall, TRAIN is a positive initiative, and an important one. AI has incredible potential to improve healthcare, and initiatives like this help to apply AI responsibly.

More on Responsible AI

The Microsoft's Responsible AI principles represent a comprehensive standard designed to ensure that AI technologies are developed and deployed in a safe, ethical, and inclusive way. It has several core principles: fairness, reliability & safety, privacy & security, inclusiveness, transparency, and accountability. These principles guide every stage of AI development and each one involves comprehensive processes. As an R&D team that builds AI products, we apply those principles to everything we do, every day.