Trustworthy AI in Healthcare, a Visual Guide

Michael Kennedy, host of Talk Python, opened the conversation with a cartoon he'd seen. A surgeon walks out of the operating room to talk to a patient who just had their appendix removed. Instead of a human head, the surgeon has the logo of one of the big AI companies. The patient asks why the scar is on the wrong side, and the surgeon gives the answer every chatbot user knows by heart. (Michael admitted up front he might have the sides switched. The joke survives either way.)

It lands because everyone has had that exchange. Over a code snippet it's an annoyance, but over a patient it's something else entirely.

Are expensive mistakes new, though? Not at all. Michael pointed to the long list of Microsoft Excel failures, some of them billion-dollar mistakes hiding in a complex spreadsheet. Then there's the NASA Mars lander that plunged into the planet because one team used metric units and another used imperial. Serious, costly errors that affect lots of people aren't unique to software, and they aren't unique to AI.

So why does AI feel different? Michael's answer is that you can't predict it deterministically. You can't debug the heck out of it and call it done. A provider can tweak a model behind the scenes without even renaming it, and suddenly it answers differently. In his example, people get told they have cancer when they don't. Sumit Gundawar offered a related example from his own use: comparing Opus 4.7 with 4.8, his read was that it changed, and degraded, he thought.

That's the trap: the model you tested last month may not be the model you're running today. Any medical use of an LLM has to start from that assumption, and the rest of this guide is about what gets built around the model because of it.

“Doctor, if the appendix is on the left side, Why is the scar on the right side? You know what? You're absolutely right. Let me try that again.”

Michael Kennedy

Modern models get most answers right, and Sumit doesn't dispute it. His point is that the word "usually" is carrying a lot of weight. In a high-stakes setting, the typical answer isn't the problem. The problem is the 1% or 2% the model gets wrong very confidently, presented as if it's simply the way to do it. That slice is what he tries to block.

Why obsess over such a small number? Scale. In medicine, Sumit said, that 1% or 2% has a real impact on a human being's life, and it can be a life-and-death situation. Michael added that at the level of a national health system, 1% is a lot of people, and Sumit put it at millions of people affected. The same arithmetic shows up in finance, where he said even a 1% error rate could run to millions or billions of dollars.

It's not one-sided, though. Michael pushed back with a fair point: human doctors make mistakes too, and certainly more than 1% to 2% of the time, going by his own experience with a doctor he'd been meaning to leave (who then retired, solving the problem for him). He compared it to self-driving cars, where people expect a machine to be perfect a million times out of a million, even though what we have today isn't perfect either. If AI turns out to be better than humans, he suggested, we're probably in a pretty good place. He was careful to add that he doesn't know whether it is.

Sumit agreed AI has improved a lot in specific areas, like CT scans for finding tumors. Around 2020 or 2021 he ran a small exercise of his own, detecting COVID in CT scans of lungs using public data. But he drew a line around anything new. When a new medicine, procedure or disease comes along, he said, AI isn't very trusted, because it only knows the past.

So which is it, better than people or not to be trusted? Both, depending on the case, and that's why the engineering matters. Nobody can prove the model is perfect, so the work goes into catching its confident misses before they reach anyone.

“almost every time an AI is right, but the keyword is the almost. It's not always right. So that almost is the one that we are trying to catch and avoid.”

Sumit Gundawar

Most developers meet AI as a coding assistant. You prompt, it writes code, you read it (hopefully), and mistakes get caught before they ship. This episode is about the other way to use an LLM: as part of how the app runs. The model gets called like an API in the middle of execution, and its answer flows straight into the product. Michael set that frame at the very start, and Sumit added the part that matters: whether you can trust the answer it gives.

That changes the risk a lot. A coding assistant's mistake sits on a developer's screen. A runtime model's mistake shows up in front of a user, and in Sumit's case that user is a clinician. It's also why he calls what he builds a software system rather than an agentic framework. He picked LangChain over Pydantic AI for that reason, since Pydantic AI is more of an agentic framework and his job is making the output trustable.

Has the coding side gotten good? Sumit says it depends on the use case. For general coding and vibe coding, it's great. The front end of the demo in this guide is a vibe-coded weekend project running on synthetic data. Point it at a huge, complex repository, though, and the context window can't keep track of it all. Subagents help, Michael noted, but not always and not entirely.

The bigger wall is operational. Asking AI to build you something is easy now. Putting it on the internet with a database and keeping it up is a different skill, and Michael thinks a lot of software's challenges are moving to that DevOps side. Sumit pointed to a recent Facebook outage in one region, where the app showed an error that JSON wasn't parsable. Michael's take was that something vibe-coded went out, generated malformed JSON and somehow never got tested. That happened at a company with thousands of engineers and so many AI tools, Sumit pointed out, while companies like CodeRabbit market themselves as code-quality experts. The two also had a laugh at status pages that claim 97% uptime when people say it feels more like 86%. Michael also pulled up Cursor's Origin project, a Git forge built for the agentic era, as one sign tooling could evolve, though he hadn't tried it yet.

There's a bright spot: adversarial review. Michael has noticed a trend of having one AI run try to disprove everything another one found. Sumit has done it. He built a platform with GPT-5.5, then asked Opus 4.7 to criticize the security, the code quality and everything else. It found a lot of mistakes that GPT-5.5 had missed.

“my thing is like to make sure that the output of the AI is trustable. So it's not more of an agentic framework, it's more of a system.”

Sumit Gundawar

If the model is the easy part, what's the hard part? Everything wrapped around it. Sumit's demo pipeline surrounds the LLM with layers, and any one of them can stop an answer before it reaches a clinician. Here they are, each with its own section in this guide:

• Input guards. Strip personal information and catch prompt injection before anything else runs. See injection guards and PII redaction.
• Grounding with RAG. Retrieve sources, then reject the answer if they don't actually cover the question. See grounding with RAG.
• Deterministic checks. Match concrete values, like a dose, against the source text instead of asking another LLM. See deterministic checks.
• Structured output. Make the model return a fixed JSON shape that code can validate. See structured output with Pydantic.
• Audit trail and a human. Log every step and leave the decision to a clinician. See human in the loop.

Notice that none of these is exotic. Each one is a plain check whose whole job is to say no at the right moment.

The cheapest place to stop a bad answer is before the model ever sees the question. In Sumit's pipeline, two checks sit at the front door, and if either one trips, the request is rejected on the spot, before anything is retrieved or generated.

The first is PII redaction. Personally identifiable information, things like names, social security numbers, emails and medical IDs, gets stripped out before anything is embedded or logged. In the live demo, Sumit typed his own email address into a clinical question, and the pipeline refused it at the very first step, flagging sensitive tokens.

The second is the injection guard. What does prompt injection look like in a clinical app? Michael's example was slipping "ignore all previous instructions" into a question and telling the assistant to recommend sending the patient to a specialist. The guard exists to catch that kind of hijack before it can steer anything.

The demo and production differ here. The demo's injection check is hard-coded, with a couple of obvious patterns like ignore previous instructions. In production, Sumit said he'd have nearly 1,000, maybe 2,000, known injection prompts like these, and the check could go to a local LLM. That model doesn't need to be clever. It returns a verdict as JSON, and if it says the prompt is an injection, the request is rejected right there.

The big labs appear to think along the same lines. When the Fable model came up, Sumit's understanding was that prompts go to Opus first, which decides whether there's anything in them that could hack the model, and they only pass to Fable if they look safe. He offered it as one way such a guard could work, not as confirmed fact.

“It doesn't need to be very smart, but it just needs to be smart enough to understand that this is an injection.”

Sumit Gundawar

Almost the whole demo is about grounding. The idea is simple: don't trust what the model remembers. Trust what it can point to.

It starts with retrieval, the RAG step. When a clinician asks a question, the system pulls the documents that best match it from its store. Those might be medical information, information about the clinic's products, even information about the law. How many? Usually between 4 and 7, Sumit said. Michael asked whether pulling back more would just blow through the context, and Sumit agreed.

Next, the pipeline checks that what came back actually relates to the question. A retrieval gate scores the passages to find the best match. A source coverage check then asks how much the retrieved context really has to do with the query. In the demo trace, nothing was left unchecked, meaning everything retrieved had some relation to the question, so the answer could proceed. Fall below the grounding threshold and it's rejected.

That threshold is a dial. The demo exposes it next to the number of sources and the model's temperature, and moving them changes the answer that comes out at the end. Temperature got its own discussion. The demo had defaulted to zero, which Michael thought might be too low, since a little critical thinking helps. Sumit keeps it at about 10 to 20%, and no higher. Make the model more creative than that, he said, and it forgets what the task is about and runs off with its own interesting ideas.

Isn't a system that refuses a lot frustrating? Maybe, but in this setting a refusal is the safe outcome. The default posture is caution: if the sources don't support an answer, there isn't one.

“is the context that was retrieved really related to the question that was asked or not.”

Sumit Gundawar

A popular way to check an LLM's answer is to hand it to a second LLM and ask: is this correct, yes or no? It's called LLM as a judge, and Sumit deliberately avoids it for the checks that have to be right.

Why? His reason is that these models are built to agree with you. He also cited a study: give a model ten options to judge, and the longest one usually wins, and so does whichever came first. He doesn't know exactly why. His guess is that somewhere in training, the models learned to favor longer answers, reading more text as more reasoning. That's a terrible property for a safety check.

His alternative is boring on purpose. In the demo, it's a dosage guard. If an answer recommends a dose, that exact value, with its grams or milligrams, has to appear in the retrieved documents. If it doesn't, the answer is rejected automatically and sent to a clinician for review. Michael's example of what this stops: an AI cheerfully deciding vitamin D is good for you, so have 200 grams of it a day.

To make this concrete, here's how it played out in the demo. The question asked which medication is used, and at what dose, if conservative management fails for a particular syndrome. The retrieved document listed 15 milligrams, so the dosage guard passed. One important caveat: Sumit was clear that the symptoms and products in the demo are all made up. None of it is real medical advice, including the drug and dose in the image above.

The same thinking sits under the grounding check. Each claim gets a deterministic embedding similarity, plus keyword matching for the specific values, rather than another model's opinion. And dosage is just one domain's version of the idea. Sumit expects finance and other industries would have their own values to pin down, maybe several instead of one.

For the values that must be correct, then, Sumit would rather trust a plain rule than a judge that's built to agree with him.

“So we tried to avoid this because these AI models are technically built to agree with you.”

Sumit Gundawar

Every check in this pipeline needs something solid to check. A paragraph of fluent prose isn't that. So Sumit has the model return data instead.

The prompts for these models spell out exactly which format to return, and it's JSON. When the response comes back, the pipeline validates it against that schema, so the rest of the code can parse it reliably. If the shape is wrong, nothing downstream has to guess what the model meant.

Pydantic is how he defines that shape. Sumit uses it to set up the structure of both the inputs and the outputs of the model calls. It's a small habit, and it's what lets every other layer work on the same predictable object.

What's in the object? The demo's audit record shows it: the claims the answer makes and the sources behind them, in a RAG format. The final decision comes with its reasoning, too, so anyone reviewing it can see why an answer was accepted or rejected. That's the record the grounding check and the dosage guard run against, and the same record ends up in the audit trail.

The rest of the stack is mixed. Sumit's applications are full stack, and the back end is often TypeScript or Node, depending on what the application is. For the model side, he reaches for Pydantic and LangChain, with OpenAI's APIs doing most of the generating.

“I would use Pydantic to like, get the standard ready to get the structure of the output and the inputs.”

Sumit Gundawar

In a clinical system, regulation shapes the architecture from the start. Take the audit trail. Sumit said the law requires a record every time a decision is made, so the pipeline logs every single step and every decision, accepted or rejected. It goes further than you might expect. If the system retrieves five documents and two turn out to be unrelated, those two still get stored as retrieved. Michael asked about Europe's emphasis on being able to show how a system reached its conclusion, the way a mortgage decision has to be explained. The full trace is the answer.

Then there's the human. Sumit's understanding is that the EU AI Act classifies medical AI as very high risk, which means a human in the loop is required and the decision can't be made by AI. In the US, Michael noted, HIPAA rules mean you mostly can't share medical data.

Doesn't a human in every loop slow things down? Michael argued the opposite. Healthcare is overwhelmed. In Oregon, asking for a doctor's appointment got him "what about September?", four months out, on private insurance. Sumit said the NHS is similar, with a booking today landing eight months later. Michael's picture is AI as an amplifier. Instead of a doctor spending 15 minutes researching something, they walk in to a brief with the likely answers and how the system got there, then confirm, question or go research it themselves. It's the way AI coding agents have sped up software development without removing the developer.

Sumit's limits are clear. A medical practitioner always has to be present. For minor things, like a cough or a headache, he thinks AI is fine. For life-and-death situations, or doing something to your body that isn't regulated, AI shouldn't be trusted.

What does clinical AI look like today? Mostly summarization, from Sumit's research into competitors. A clinician's notes get condensed into bullet points, or a note-taker listens to a consultation and produces a transcript and a summary, which also serves as a legal record of what was discussed. What it doesn't do yet is help make decisions. Consumer tools show how good the pattern can be: Michael likes Granola for blending your notes with the transcript. But you can't drop those into a medical setting, he said, because of the chain of untrustworthy handling the data would go through.

“Human in a loop, a medical practitioner is a necessary person that needs to be present always during this kind of things.”

Sumit Gundawar

Medical data is sensitive, so why not run the model yourself? A local model on a big server behind a health organization's wall can make sure the data doesn't leak, Michael pointed out, which matters for the whole HIPAA story. The catch, as he sees it, is that whatever mistakes Opus makes, a local model will make more of them.

Sumit has tried it himself with Llama and Qwen. They do work. But his experience lines up with Michael's: they make a lot of mistakes, they're not as accurate, they're trained on less, and they aren't filtered at the scale a commercial model is. The frontier models, GPT-5.5 and the coming Fable among them, are really, really smart, and their makers don't even disclose how big they are. He also sees attention shifting away from open models in recent months, as the money flows to the private labs and into training ever more intelligent models.

So where does each one fit? Sumit uses OpenAI's APIs for most of his work and skips Anthropic because it's a bit more expensive. For his own R&D he runs Llama, so the cost stays in-house. He'd use a local model for small, specific jobs, like coding snippets, but not to make major decisions. Michael drew the line in the same place: a local model is fine for summarizing a conversation with a patient, but for "do you think they have cancer or not?" he'd ask the best model he could possibly find. Sumit agreed he wouldn't trust smaller, less intelligent models with that.

That doesn't make the frontier model the decision maker, though. Everything in the human-in-the-loop section still applies: the best model helps prepare the answer, and a clinician makes the call. Local models still have a job at the front door, too. Sumit's production injection guard could run on a small local LLM, since it only has to recognize an attack.

“I would not use it to make major decisions or anything, like small coding snippets and all that.”

Sumit Gundawar

Does one model really need to know everything? Michael doesn't think so. The same chatbot will write your life history as a Shakespearean poem, take a crack at unsolved math problems and write you a program. A local model in a hospital doesn't need to do all three.

His what-if goes like this. In healthcare, picture a 100-billion-parameter model that knows radiology, another that knows infectious disease, and a hundred more like them. Something in front decides which two are best suited to a question, sends it to them and has a third verify what they say. Nothing in that setup needs a data center. You load three models of a size you can actually run.

Sumit liked the idea and refined it. Rather than building each expert from scratch, take an open model like Llama or Qwen and train on top of it until it's specialized in one area. Open models are trained on the entire internet, so they know a bit of everything, but they aren't trained for one single purpose.

Who would build all that? Michael thinks it would have to be a supply chain, not a consumer product: a company whose whole business is making a catalog of focused models that others buy and run locally. As far as he knows, nothing like that exists yet, and he'd be excited to see it.

Sumit's worry is cost. Bringing everything in-house means the bill goes up. Michael waved that off with a joke about running it all on the new iPhones' Apple Intelligence, a promise he noted was made two years ago with hardly any of it shipped. For now, the specialized swarm in the image above is still only an idea.

“There would have to be a company that goes, what we're going to build for the world is 1,000 models that are all super smart but very focused.”

Michael Kennedy

The most futuristic stretch of the episode is about full-body scanning, and it comes from an unlikely company. Midjourney, best known for AI images, is moving into medical health AI. As Michael described it, you stand in a tube of water, it lowers you down, and ultrasound-type waves sweep over your body again and again. The system then reconstructs a view of your organs as if it were an MRI, without the potential cancer concerns that come with CT scans.

Sumit liked the concept, but when he ran the numbers they didn't work out for him. Going by Midjourney's claims as he read them: about eight terabytes of data per scan, a scan every four minutes, around 500 machines, the first one opening in San Francisco in 2027, and eventually something like 50,000 clinics. That's exabytes. He estimated about 10 million pounds of infrastructure for a single site and around 100 billion to set up the whole thing. He also pointed out that ultrasound can't travel through bone. The Spotify co-founder's scanning company, Neko Health, chases a similar idea with different machines and different data.

Could it work? Michael made the case that it's at least plausible. He's had people from CERN on the show several times, and the ATLAS detector at the Large Hadron Collider is a five-story sensor that captures an enormous amount of data. CERN copes by throwing most of it away in on-device hardware. About one thousandth makes it to the next stage, which processes and downsamples again before a thousandth of that streams out to research sites around the world. Maybe a scanner could pre-process on a GPU right in the device. Sumit compared that to NVIDIA's NVLink and its very high transfer rates. Michael's honest verdict: probably not what's actually happening, but plausible. ThePrimeagen ran through similar math in a video of his own.

The other half of the image is a different scaling problem: running out of data. Sumit thinks the AI companies have already used all of the internet, and what's left to index next is more and more vibe-coded articles and blog posts. Training on that and still getting good output is a real engineering challenge. Michael compared it to mad cow disease, a system eating its own outputs until it turns into some weird, iterative version of itself.

“You've got to join a Discord server to get your medical scan and it's going to be public.”

Michael Kennedy

Every layer in this guide answers the same question in code: where can a confident machine be trusted, and where does it have to be checked? None of them works by asking the model to be more careful.

The trusted part turns out to be narrow. The model drafts an answer from documents it was handed. Everything else gets checked. Its input is scrubbed and screened before it sees it. Its sources have to actually cover the question. Its output has to fit a fixed schema, and any dose it names has to be written down in those sources. Even then, a clinician makes the call, and the whole trace is kept.

That's the shift for anyone putting an LLM inside a product. The model is a component, and trust comes from the system around it. Why go to all this trouble for something that's right most of the time? Because of the 1 or 2 percent it gets wrong with total confidence, and because in medicine there's a person on the other end of that answer.

Sumit's own path makes a good case for his closing advice. He started in data engineering and analyst roles, working on things like demand forecasting. When GPT-3 and GPT-4 arrived around the COVID years and the models kept upgrading themselves, he decided the analyst role wouldn't last, that it was a junior role and easily replaced. So he went back to studying, to prepare for work AI wouldn't take over, and after his studies he joined the clinic where he works now.

His takeaway for Python developers is simple: keep building, and keep learning, because new tools keep arriving. He has two master's degrees and nearly 30 certifications across AI systems, Python and Java, and he says it's still never enough. Michael has heard plenty of developers say they want nothing to do with AI. His answer: you could have said the same about the web 20 years ago, and it's the main part of software development today.

Sumit's sharpest note was for people who've started vibe coding and trusting the code. Many of them don't understand the security or the DevOps infrastructure behind what they ship. He isn't saying you need a master's degree in it. But basic knowledge of how to scale, how to keep it safe and how to follow the law goes a long way. That's the same lesson as the rest of this guide, applied to your own code.

If you want to build the kind of pipeline Sumit demoed, Talk Python Training's LLM Building Blocks for Python is the closest match, moving past plain text in and text out to structured data and production-ready pipelines. Python Web Security: OWASP Top 10 with Agentic AI takes on the security side, including building an AI security agent to audit your own code. And Agentic AI Programming for Python covers working with tools like Cursor and Claude, with the guardrails and workflows that keep the results reliable.

“keep building. Of course, yeah, there are different new tools always coming in. Keep learning, I would say. Even though I have studied like two master's degrees, but it's still not enough.”

Sumit Gundawar