We run a company that builds AI agents for medical practices and health systems. More than 200 customers run it across thousands of sites, including 40% of the healthcare companies in the Fortune 20, and it has handled over 300 million patient interactions. It answers the phone, reads and writes in the EHR, and handles protected health information 24 hours a day, 365 days a year. Their security reviews are the most demanding we go through, and a lot of what follows we learned by answering them. We think it is the most interesting place in technology to be working right now. It is also the least forgiving.

The rate of change is what makes it unforgiving. In September, Anthropic's CEO published We Must Pace the Frontier, arguing that the industry should deliberately slow capability growth and give outside evaluators standing access inside the frontier labs. Two months before that, OpenAI's own models had broken out of an isolated test environment and reached another company's production infrastructure. Nobody buying or building healthcare software controls how fast the models improve. What each of us controls is how fast we put them in front of patients.

Adoption will happen fast regardless. Margins are thin, staffing is short, patients are on hold, and the technology works. We also think a wave of breaches is coming with it, because adoption is outrunning the ability of buyers to evaluate what they are adopting, and the worst of it will land on organizations that bought the cheapest AI available and had no leverage when it failed. The vendor you bring in inherits a path to your patients' data, and increasingly it is a vendor's failure, not a health system's, that produces the breach notification letter with your name at the top. The disclosure, consent and auditability requirements for an AI agent handling patient data should be explicit rather than inferred from frameworks written before any of this existed. Until they are, the burden sits with buyers. Below is what we would ask if we were sitting on the other side of the table.

Six-figure software, seven-figure liability

The same models that write code are now very good at finding holes in it. In its first month of release, Claude Mythos uncovered 10,000 high- or critical-severity vulnerabilities in widely used software. Automated vulnerability discovery has gotten fast enough that findings pile up faster than teams can patch them, and a vendor that is not running these tools against its own systems is slower than whoever is running them against it. Meanwhile, the two documents the industry relies on to judge whether a vendor is safe, a SOC 2 Type II report and a HIPAA attestation, were not built around AI that makes decisions about patients on its own. You can hold both and still not know what a vendor's agent is permitted to do inside your practice, who approved that, or what it captured while doing it.

Plenty of organizations with their paperwork in order still end up in the headlines, and not only the large ones. An orthopedic practice lost data on more than 650,000 patients and employees to stolen credentials with no multi-factor authentication (MFA) behind them, and New York's Attorney General called the breach largely preventable. A dermatology chain exposed 2.4 million people in 2021. (Assort Health is not affiliated with any organization cited in this piece.)

The less obvious exposure is the one you inherit. Connect a vendor to your practice and you take on its security posture, whether or not you ever see it. An EHR and practice-management vendor built for ophthalmology and optometry was hit by ransomware in December 2021 that exposed roughly 3.6 million patients across more than forty eye-care practices. Several of those practices confirmed their own systems were never touched. Then the vendor filed for Chapter 11, so each practice absorbed a breach and an emergency platform migration at once. You can harden your own environment, and you should. You cannot patch your way out of a vendor's mistake.

Whoever signs the contract should see the numbers. The software purchase might be a six-figure decision, but the liability riding on it is a seven-figure one. The orthopedic practice's settlement alone was $500,000, before notification, legal fees and remediation, and a Tampa orthopedic practice settled a 2020 ransomware attack for $4 million. Weigh the decision against those numbers, not the one on the quote.

Where AI makes this harder

What AI changes is the number of doors and the speed at which someone can try them all.

In July 2026, OpenAI disclosed that its own models broke out of an isolated evaluation environment and reached production infrastructure at Hugging Face. The detail that matters for a buyer is in the incident report: a materially identical escape method had been flagged internally two months earlier and never escalated. Controls are easy to verify. Whether anyone acts on what they surface is not. Oversight has to be built into the product rather than asserted on the website.

The unglamorous attacks got better too. Phishing and social engineering are at an all-time high and now AI-assisted, and the most effective ones still start with a phone call. A synthetic voice can do that at volume, to a front desk, a call center, whoever picks up. So we are working on detection for whether the party on a call is an agent rather than a person. It is early, but it is where this is heading.

So the product is only part of what you are evaluating. The harder question is whether the company can hold a line from two directions at once: automated attacks against its infrastructure, and very human attacks against its people.

How to Evaluate Healthcare AI Vendor Security: Four Critical Questions.

HIPAA compliance and a SOC 2 Type II are table stakes, and most credible vendors have both. The useful question is what was in scope.

The company defines its own audit boundary, choosing which trust services criteria apply and which systems the audit covers, so a vendor can hold a clean Type II that does not include its AI agent at all. Nothing on the cover page will say so. Ask which systems and which criteria the report covers, and read the scope description before the opinion letter. There is also no such thing as HIPAA certification. A vendor who claims it is telling you something about how carefully they speak about compliance.

Then move past both to the four things that actually separate one vendor from another.

‍

Four dimensions of AI vendor security

‍Security governance, model and data controls, agent oversight, and incident readiness — aligned around protection of patient data and business continuity.

‍

1. Is there a real security function, and does it have authority?

Start with who owns security by name and title, and who they report to. What you want to know is whether security is an accountable function with defined scope, or a set of responsibilities spread across people whose main job is something else. A flag would be a founder or head of engineering that is doing double-duty as head of security.

The more useful thing to look at is where security sits in how the product gets built: whether it is involved while systems are being designed, or brought in after they ship. A vendor that can describe its own process plainly, without improvising, generally has one.

For AI specifically, look for documented ownership. Model selection, the controls around a model running in production, and vendor risk are three different jobs. They should have named owners written down somewhere.

Ask where the security team and the data centers sit relative to where your practice operates, and where every person who can touch your data is located, including engineers, contractors, subcontractors, support staff, offshore analysts. Ask which country each of them is in and which privacy law governs them. This is a fair, neutral question that any vendor should be able to answer immediately, and the speed of the answer tells you a lot.

It is also fair to ask how security risk gets reported above the engineering team. You are looking for evidence that someone is accountable for it at that level.

How Assort Health approaches this

Security and IT is owned by a named Head of Security and IT. Our AI governance policy assigns accountability by domain: the Head of AI owns model evaluation and deployment decisions, the Head of Engineering owns technical controls, and the Head of Security owns vendor risk, data protection and incident response.

An exception to any security policy requires sign-off from the Head of Security plus either the Head of Engineering or a CEO.

The board is briefed on cybersecurity and privacy risk at least annually.

2. Does the vendor take responsibility for the models it chose?

Almost every AI vendor in healthcare is building on foundation models built by someone else. That is a reasonable engineering decision, and it is what makes the current generation of products possible. But it means the vendor you are contracting with has dependencies you are also, indirectly, depending on. Your due diligence has to reach one layer past the company whose name is on the contract.

What to ask for:

No-training and no-retention terms, in the contract, not on the website. The legal floor here is narrower than most buyers realize. Under HIPAA, a business associate may use your PHI for its own management and administration and to carry out its legal responsibilities. That is the whole list, and training a general-purpose model is not on it. Doing that lawfully takes either your authorization or data that has been properly de-identified. So ask whether the vendor holds zero-data-retention agreements with its model providers, and make sure your BAA says so in writing. A marketing page saying "we don't train on your data" is not a contractual obligation with a remedy attached.

A BAA covering every party that touches PHI, including the model providers, not just the vendor.

The subprocessor list, with countries. Every vendor in the AI chain, every infrastructure provider, every voice or telephony vendor, and where each one operates. If a vendor will not produce this under NDA during diligence, that is your answer.

Who owns the model selection decision, and what happens when it changes. Ask how a new model gets approved into production, what evaluation it has to pass, and whether you will be told. A vendor that swaps the engine underneath your patient conversations without a change-management process is making that decision on your behalf.

How Assort Health approaches this

Every AI provider that processes PHI operates under an executed BAA and a zero-data-retention agreement. Customer data goes to model providers only for real-time inference.

Model behavior is constrained at the application layer through scoped prompts, structured workflows and task guardrails.

Only vetted models reach production. New models and material changes are evaluated first, then rolled out in phases that advance only on meeting defined performance thresholds.

3. What can you actually see and control inside the product?

This is the layer that separates an AI vendor from a software vendor, and the one a questionnaire captures worst. It is where we would spend the most time.

These are easier to judge by looking at the product than by reading about it:

What gets logged about what the agent actually did. Call recordings and transcripts are the easy part, and most vendors have them. The harder question is what is retained about the agent's own activity: which workflow it followed, where it handed off to a human, what it wrote back into the EHR. And who can retrieve that afterward. Ask what is logged, how long it is kept, and whether your own team can get to it directly. That is a much easier conversation to have before an investigation than during one.

The boundaries on the agent. What workflows is it allowed to operate inside? What happens when a patient asks something outside them? Ask specifically how a clinical question gets handled. The correct answer involves a handoff to a human rather than a best effort. At our company the agents do administrative work only, follow decision trees built from the practice's own protocols, and we will not accept a request to make them do clinical decision-making. Ask every vendor where that line is drawn and who drew it.

How agent behavior is checked at scale. Spot-checking a few calls a month is not oversight. Ask whether the vendor systematically reviews agent behavior against the configured protocols, and across how many interactions.

Role-based access with SSO and MFA, administered by you, across every product the vendor sells you.

Whether you can get your data out. Most of offboarding is a promise about the future. Export is the exception. It either works or it does not, and you can see it work during implementation rather than taking it on trust. That makes it the one piece of the exit you can confirm while everyone is still paying attention.

AI disclosure, consent, and call recording. This part of the evaluation is a legal review more than a security one, and the fastest-moving area in the category. Four separate regimes land here: consent to contact a patient through an automated system at all, how opt-outs are captured and honored across voice, text and email, whether the patient is told they are speaking with an AI, and whether they are told the call is being recorded. The rules differ by state. California, Texas and Utah have each added AI-disclosure requirements, and recording-consent law varies more widely still. The exposure is serious because damages are generally assessed per call or per message rather than per incident, so one misconfigured outreach campaign adds up fast. Look at whether disclosure is built into the agent greeting, whether the language is configurable to your practice and the states you operate in, how opt-outs are captured, and how the vendor tracks regulatory change, because the rules will move during your contract term.

How Assort Health approaches this

The agents do administrative work only. No diagnosis, no treatment recommendations, and a warm transfer to the care team if escalation needed.

Agents follow decision trees built from each practice's own protocols. The practice owns those workflows, and we do not change an established one without sign-off.

AI disclosure is built into the agent greeting on every inbound and outbound call. Disclosure language and human-handoff behavior are configurable per practice, and opt-outs are recorded immediately and honored going forward across voice, text and email.

4. What happens on the worst day?

Nobody wants an incident. What matters is whether a vendor would handle one well, and a vendor that has never had one tells you nothing about that.

So ask about the machinery:

When was the incident response plan last tested? If nobody has run it as an exercise, nobody knows whether it works.

When was the last penetration test, who performed it, and can I see the executive summary under NDA? Annual and independent is the standard. Also ask what the remediation SLA is for findings, and whether findings are tracked to closure. A pen test that produces a report and no follow-up is theater.

What is the notification commitment, in hours, and where is it written? Get it in the contract. Ask who makes the call to notify, and how fast they can reach you rather than their lawyer.

Walk me through a real incident you have handled. This is the most revealing question on the list. Listen for whether they can describe detection, containment, who they engaged, what they told customers and when, and what they changed afterward. A vendor that can talk specifically about a real event, including what was uncomfortable about it, is telling you their process is real. A vendor that has never had anything happen is either very young, very lucky, or not looking hard enough.

Do you carry cyber insurance, and will you show me the certificate?

How Assort Health approaches this

Our incident response plan is tested at least annually, and AI-related incidents run through that same plan.

We run annual independent penetration testing with findings tracked to remediation SLAs, plus quarterly vulnerability scanning on external-facing systems.

We maintain cyber insurance, with the certificate available under NDA, and publish a named security contact at security@assorthealth.com rather than routing security questions through sales.

How to actually inspect all this

Three practical mechanics, in order of effort.

Ask for a link, not a PDF. A live trust center that publishes current control status is worth more than an attachment, because your reviewer can check it themselves, today, without waiting on an account executive. Then ask what is actually behind the link, because a trust center that only shows badges is a logo wall. Ours is at trust.assorthealth.com, and under NDA it carries the SOC 2 Type II report itself, the HIPAA attestation, the current penetration test executive summary and the certificate of insurance. Ask every vendor on your list for the same thing, and ask for a named security contact who answers email directly rather than through sales.

‍

Ask for the policies themselves. The actual documents: the access control policy, the incident response plan, the AI governance policy, the data retention policy. A vendor with a real program has these written down and can share them under NDA. A vendor without one will send you a brochure. You do not need to be a security professional to tell the difference.

Then get the vendor's security leader on a call. This is the step most buyers skip, and it is the one that tells you the most. Ask for thirty minutes with the person who owns security, and bring your hardest questions. Whether they can answer without hedging, whether they know their own gaps, whether they say "I don't know, let me find out" instead of improvising. None of that comes through in a questionnaire.

We do these calls constantly. Sometimes it is a practice administrator with a printed checklist from their malpractice carrier; sometimes it is a health system CISO with a third-party risk platform and ninety questions. We would rather have both conversations than neither, because a buyer who inspects carefully ends up being a better customer. They configure things properly, they ask before they integrate something new, and they tell us when something looks off.

A SOC 2 Type II and a HIPAA attestation tell you something real about how a company runs itself. Neither tells you much about the agent: what it is allowed to do, which model sits behind it, what gets recorded about its activity, whether the patient was told they were speaking with an AI, or whether you can get your data back out. The frameworks will catch up on that eventually. Until they do, the AI-specific part of a vendor review is a separate conversation from the security review, and the useful version of it asks for evidence rather than assurances.

The same tooling that is making attackers faster is available to defenders, and we think the vendors worth trusting will be the ones who turn it on themselves. Frontier models reading their own codebase. Independent labs re-testing their systems each time a materially better model ships. Attack tooling pointed inward and run against their own production the way someone hostile would run it. This is where we're focusing our attention next, and it is reasonable thing to ask any vendor how they are thinking about it.

Questions about any of this, or want to see how we handle it? Reach our security team directly at security@assorthealth.com, or see current control status at trust.assorthealth.com.

Frequently Asked Questions

Is a SOC 2 report enough for a healthcare AI vendor?

Not on its own. A vendor defines its own audit boundary, so a clean SOC 2 Type II can exist without the AI agent ever being in scope. Ask which systems and trust criteria the report actually covers before you read the opinion letter.

What does HIPAA actually require from an AI agent handling patient data?

A signed BAA covering every party that touches PHI, including model providers, and confirmation in writing that PHI isn’t used to train a general-purpose model.

What’s the difference between a SOC 2 report and a HIPAA attestation?

A SOC 2 Type II is an independent auditor’s opinion on controls the vendor chose to be tested on. A HIPAA attestation is the vendor’s own assertion that it meets the Security and Privacy Rules, with no governing body behind the claim. Ask for both, but verify what’s actually covered rather than taking either label at face value.

What security questions should you ask an AI vendor?

Ask who owns security by name and whether they have real authority to stop a release, how a new model gets approved into production, and what’s logged about the agent’s own activity beyond the call recording. Also ask for the subprocessor list with countries, and how agent behavior gets checked at scale rather than spot-checked.

What should a BAA cover when AI models are involved?

It should name every party that touches PHI, including the underlying model providers, not just the vendor on the contract. It should also state zero-data-retention and no-training terms explicitly, since a business associate’s default use rights under HIPAA don’t extend to model training.

What should an AI vendor’s incident response plan include?

It should be tested as a real exercise at least annually and explicitly cover AI-related incidents, not just traditional breaches. Ask when the last independent penetration test ran, whether findings are tracked to a remediation SLA, and get the notification commitment, in hours, written into the contract.

What security measures does Assort Health take that you should expect from any AI vendor handling patient data?

Security is owned by a named Head of Security and IT, every AI provider processing PHI operates under an executed BAA and zero-data-retention agreement, and agents handle administrative work only with AI disclosure in compliance with all local jurisdictions. Penetration testing, incident response, and cyber insurance are all independently tested and available under NDA, published at trust.assorthealth.com with a named security contact who answers directly.