Sanj Chandran
Vancouver--:--

We ask about one AI. Never the other.

Patients are asked before an AI note-taker joins the consultation. The AI they used the night before never comes up.

4 min read HealthcareAIHealthcareRegulation
A line running from The doctor's AI, marked asked about, to The patient's AI, marked never asked about.

There's a new little ritual at the start of a lot of consultations.

"Do you mind if I use an AI note-taker today?"

In my experience most people say yes, and honestly, that's a good thing. The clinician gets to look at the patient instead of a screen, and the patient knows what's going on.

But I keep noticing what doesn't get asked. Plenty of patients have already talked to an AI before they walk in. Maybe late the night before, typing their symptoms into a chatbot. That AI never gets a mention. It just quietly shapes the conversation.

So now there are two AIs in one consultation. One gets a consent question. The other doesn't come up at all.

The AI everyone asks about

AI note-takers, or ambient scribes, listen to the consultation and draft the notes. The NHS is increasingly using them, and the rules around them are getting clearer.

In July the MHRA published guidance on when a scribe counts as a medical device. The short version:

  • Transcribing or summarising alone doesn't make it a medical device.
  • Suggesting diagnoses or treatments, or acting on its own (like ordering tests without a clinician confirming), does.
  • The notes are a draft for the clinician to check and correct, and hallucination is named as a known risk.

NHS England's own guidance expects organisations to be open with patients about scribes and give them a chance to object. In the US, Rhode Island and Louisiana have now put disclosure into law.

So the doctor's AI comes with guidance, transparency and a question at the start. Fair enough. That's how it should be.

The AI nobody asks about

Now the other one.

In February, Oxford researchers published a study in Nature Medicine that I keep coming back to. They gave 1,298 people in the UK medical scenarios written by doctors. Some used a chatbot, the rest used whatever they'd normally use.

Tested on their own, the chatbots named a relevant condition about 95% of the time. When real people used them, that dropped below 35%. On deciding what to do next, the chatbot group did no better than everyone else.

Why? People often didn't give the chatbot the information it needed, and the answers mixed good and bad suggestions with nothing to say which was which. In one case, two similar descriptions of a bleed on the brain got opposite advice: lie down in a dark room, or get emergency care.

That's not a reason to panic about chatbots. It's a reason to know when one has been involved. Whatever came out of that late-night chat can shape which symptoms get mentioned, which get left out, and the theory the patient walks in with.

Then the note-taker writes all of it down, faithfully.

A hand-drawn infographic titled We ask about one AI. Never the other. A patient and a clinician sit across a table. The clinician asks whether an AI note-taker is okay, and the patient says sure. On the table, the note-taker is marked asked, while the patient's phone, showing a chatbot chat about a three-day headache, is marked never asked. Below the table is the line Responsible, left blank.

Who's responsible for the bit in between?

The scribe gets checked on whether it captured what was said. Nobody's really checking what shaped what was said.

I haven't found guidance that covers the patient's side, and I get why. It happens outside the clinic, on someone's own phone. But the effect lands inside the clinic, in the record, with a clinician's name on it.

HSSIB, the patient safety investigator, has started looking at AI scribes in hospitals, including how accountability for their safety is understood. My question sits one step earlier. If two AIs touched this consultation, who owns what happened in between?

Maybe it's one question

I don't think the answer is telling patients to stop. They won't, and that's not really the point.

Maybe it's just one more question, asked without judgement: "Did you look anything up before coming in?"

It takes seconds. It tells the clinician what theory might already be in the room. And it treats the patient's AI the same way we already treat the clinician's: openly.

This is also the side we're exploring in our R&D, starting with online weight-loss consultations, where what the patient sends in is close to the whole picture. Many providers do this properly. But the input is where things can quietly slip, for all of us.

So I'm curious. If you're a clinician: you ask patients about your AI. Do you ever ask about theirs?

Sources

Not medical advice. Talk to a clinician who knows your history.

Ask about this post

Draft

P201SANJ 201VANCOUVER--- -- --- --:--:--
SANJ·VISION