Tom: A key issue that every health system faces is, “How do we know what each patient needs?”
A patient can come through any channel: By phone, in person, through a digital platform, or online. We generally know something about them. As soon as we identify them, we know their medical record and history. Just off that, when a patient comes in—even when we don’t know why—we can predict the likelihood that they will need to be admitted. Or, as we’ve done in some of our hospital at home work, we can predict the likelihood that they will be eligible for care at home as their site of care.
The potential of how we understand what a patient needs is a huge lever. And at the end of the day, we can get better insight based on what we know about their past, their personality, their preferences, and their perceptions. For instance, my mom’s use of healthcare is very different from mine. She engages in a much different way than I do, and understanding that is part of defining her need.
We’ve done some really cool work with clients that use AI to identify and think about what patients actually need. Now, if you let AI do this on its own, using AI as a medical device, there is a whole set of approval recommendations and Food and Drug Administration (FDA) oversight.
But if you use AI to inform the creation of rules-based deterministic models that are overseen by a clinician, that’s really compelling: You can establish a deterministic spine that routes care based on what patients need.
The AI can ask the patient questions that explore whether other information should be added to the deterministic rule base. And what you actually get is something that looks like AI and feels like AI to the consumer—it feels empathetic, thoughtful, and customized.
But behind it, there’s a rules engine that the AI leans on and validates to make sure it does not send a patient off the rails by hallucinating or sending them in the wrong direction. No, the rules dictate this.
When health systems start to realize they can get a different level of granularity, they can have a high degree of confidence in knowing what patients need.
Think of the AI as a Plinko board of supply options. It’s a permutation of a care team member: Is it a super-super-specialized physician? Is it a more junior physician? Is it an advanced practice provider? Is it a nurse? Is it a tech? Is it a medical assistant? Is it just an administrative resource? Is it a bot that is going to be initiating an action? And then think about the setting it’s going to operate through.
So you can click on the care team member, you can click on a setting, and you can calculate a cost for that interaction. And now you’ve got a robust set of cost platforms and cost profiles with different clinical capabilities. You’re not going to expect the same level of capability from a bot over a chat interface as from a physician in an in-person interaction. They’re different clinical capabilities.
But AI triage based on a deterministic backbone can appropriately route patients, giving them options and saying, “This is what we recommend.” You’re getting better engagement with the patient. You’re giving them more timely information, and it’s likely at a lower cost. But again, that requires a fundamental shakeup of how we think about the economics of different sites of care.
Sara: I want to connect some points that you made. One is this deterministic backbone, which is critical for providing guardrails and constraints for AI. Connecting that to your trust point from earlier, every organization will have a different tolerance, a different risk posture, a different set of parameters that define and quantify trust in a system for them.
When exploring these new care models and operating models, those trust factors must be reflected in the solutions—whether they are clinician-facing or patient-facing.
In a past role, my organization at the time built a solution that allowed patients to ask questions to get a degree of screening and triaging. We just put guardrails around the large language models but did not include the deterministic backbone. And there were some instances, like two out of millions of conversations, where the AI—which was non-deterministic (meaning we couldn’t predict the output based on the inputs)—went outside of those guardrails.
I had an experience at a previous organization, for example, where we had built a solution where patients could come in and ask all sorts of questions and in a way do what you're describing—get a degree of screening and triaging as appropriate. But we learned we had put guardrails around the LLMs without the deterministic backbone.
One example was a patient who was traveling and came in and asked what time she should take her birth control in this new time zone. And, technically, that is a piece of clinical advice and is a boundary that shouldn’t have been violated. But the AI actually did answer. Based on our risk posture, we decided to eliminate all non-deterministic answers.
All of these things are really critical to factor into a trust platform because we actually wouldn’t have known that had happened if we didn’t have the underlying platform, the audit logs, and the mechanism to understand all interactions and to be able to go back and course correct. So that’s a really critical point. There’s a depth and nuance associated with AI, with how many different ways it could manifest. So you need a really strong foundation of trust.