What Is Agentic AI in Healthcare and How Do You Keep It Safe?

6 hours ago 12

Rommie Analytics

What Is Agentic AI in Healthcare and How Do You Keep It Safe?Sanjay Kumar, Chief Technology Officer at XiFin

Twenty-eight months. That’s how long it took AI to go from a punching bag to a result that surprised mathematicians.

In 2023, top AI scientists ridiculed large language models (LLMs), pointing out that a 2-billion-parameter neural network was ‘dumber than a $5 calculator’ because it couldn’t reliably multiply multi-digit numbers.

By July 2024, one earned a silver medal at the International Math Olympiad, though humans first had to translate the problems into a computing language the models could use, and it took the models up to three days to solve some questions. By July 2025, a model solved these problems using plain language in the same time limit as the students and reached gold-medal level. Then in May 2026, a general-purpose model overturned an 80-year-old open conjecture in geometry (Erdős problem 90, also known as the planar unit distance problem) from a single prompt.

Demonstrating this speed of learning—from failing arithmetic to a frontier conjecture in 28 months—is unfathomable for most things. And this level and speed of learning is already reaching the enterprise. Gartner projects that the average large enterprise ran fewer than 15 AI agents in 2025 and will run around 150,000 by 2028. At the same time, Gartner also finds that only about 13% of organizations believe they have the right agent governance in place today. Capability is moving much faster than our ability to govern it, and healthcare, with less room for error than most industries, feels that gap acutely.

When people say “AI,” they could mean any of the following five layers, each built on the one beneath it, and all still running in production today:

Expert systems: humans write the rules and the system applies them exactly. Machine learning: the system learns patterns from historical data, such as what rejected claims look like, and returns a score. Deep learning: multi-layered networks that read unstructured input, like the system that reads addresses on envelopes or unlocks your phone. Generative models: systems like ChatGPT, Claude, and Gemini that produce new text, code, and images. Agentic systems: a generative model given tools, memory, and a goal, which acts and then checks its own work.

Nothing was replaced along the way. The useful question is never “which AI?” but “which layer fits this problem?”

Who decides the path?

Traditional software, including most revenue cycle management (RCM) systems, is currently deterministic. Every path is drawn in advance. It is fast, repeatable, and auditable, but when it meets a situation nobody anticipated, it stops and throws an error.

Agentic AI works differently. You give it a destination, and it works out the route. When it hits an obstacle, it reasons around it, and when it truly can’t, it escalates to a person with its reasoning attached. The difference isn’t what the system can do. It’s who decides the path. In regulated work, where you want tight guardrails and little improvisation, you choose how soon it hands off to a human.

How an AI Agent is Like a Trip Planner….

Think of an agent the way you think about planning a family trip. The LLM is the reasoning core, which decides the next step. On its own, though, it can’t see, act, or remember, but it:

Sees through connectors, document parsers, and retrieval, so it can read the claim, the calendar, or the flight options. Acts through tools, so it can search, compare, and change something in the world. Remembers through externally supplied memory. Large language models are stateless, so we provide short-term and long-term memory ourselves. Loops. Each result feeds back in and changes what happens next. That return path is what turns a pipeline into an agent.

Why isn’t one agent enough?

If a single agent handled an entire denied provider claim, from eligibility verification through appeal, it would hit four walls. 

Runs out of room: Its context window fills and the earliest facts silently drop out.  Becomes a generalist: It knows a little about coverage and a little about coding, but not enough depth in either.  Reaches everything: It can read everything it needs to, including PHI, can query payors, and file appeals, with no way to scope it down. Leaves a thin trail: When an appeal comes back wrong, there are no steps to inspect.

I’m a football fan, so I think about it this way: you don’t want a team where everyone has the same skills. The answer isn’t a bigger agent, but a smaller scope per agent. A coding agent, an eligibility agent, and a denial management agent each hold one piece of the claim, while an orchestrator plans, routes, and re-plans. Every handoff is a contract, and every step is auditable.

Security must change

For twenty years, we secured software by validating input against a schema. A malformed claim gets rejected. This approach can identify when things are wrong, but it can’t instruct on resolution. New approaches demand a new discipline. Traditional controls were built for systems that execute, not systems that decide. An agent is different, because input becomes context, and a reasoning engine can be persuaded. That creates four failure modes. Two require an attacker:

Prompt injection: hidden instructions planted in a claim narrative. Jailbreaking: input framed so that bypassing a safeguard sounds reasonable.

And two don’t:

Confabulation: a real denial reason and a real diagnosis code joined by a policy citation that doesn’t exist. Unfamiliar input: the agent has no “I can’t handle this” reflex, so it makes a confident, best guess.

Agents are more than chatbots. They reason, act, and self-correct across multi-step work. Shifting from automation to orchestration is real and happening now.  The building blocks (perception, reasoning, memory, tools, and orchestration) must all be in place. Governance is a feature, not a constraint.  Start narrow, measure everything, and expand from there.


About Sanjay Kumar

Sanjay Kumar is CTO at XiFin, where he leads the company’s technology vision, engineering strategy, and innovation roadmap for its AI-powered XiFin Empower ecosystem. He focuses on scaling next-generation AI, agentic capabilities, and cloud-native infrastructure to support XiFin’s continued growth.

Read Entire Article