This is my exact concern, patients will feel compelled to say they consent even if they do not, whether it is because of politeness or deferring to authority
When Rebecca Green arrived for her first appointment with a urologist, she was asked whether she consented to having the appointment transcribed by artificial intelligence (AI).
She said yes without hesitation.
“I don’t want to be in her bad books or be that annoying patient who takes up more of her time,” she said.
Clinicians should be well educated on how this technology works but they aren’t. They’re given basic legal liability nagging (which, tbf, absolutely says to review the output). For my own doctors almost all of them use this and I refuse consent every time they can’t answer very basic questions: are recordings encrypted end to end, or at all? Are recordings destroyed after generation, or at all? Are the recordings processed by the emr vendor (typically epic or oracle) or a third party? Who is the third party? Are generated notes used for reinforcement learning and/or model refinement? Etc. so then it’s quite easy to say no; I will not consent to a technology that you don’t understand the basic security of because you don’t “do computers”
The penalties for violating this should be severe and supervision is even more necessary. I know for a fact when you use these tools you must attest that you reviewed the output for accuracy with every single note submission, because the people selling the models don’t want to touch that liability at all. What does this matter though if enforcement never occurs? The government is severely lacking at managing this kind of behavior. The only hope is insurers going at it, not out of altruism but because it gives them a reason to revoke payment for services
Exactly. I can really imagine these tools van be helpful and save time, but ffs don’t just blindly trust everything like that. If you brought in a new coworker to transcribe for you you’d also first just see if it’s OK, right?
And yeah, not sure if we can ever fully trust such an AI (actually maybe humans as well?) so reviewing and supervision will be necessary.
If only someone could’ve seen this coming!!! /$
I’ve seen enough errors made by humans that I always check my record after an appointment.
She wonders why she isn’t getting the familiarity she wants from her doctor, and feels that the dr didn’t review the notes well enough. Well that’s probably because the doctor sees dozens of patients a week and doesn’t have time / isn’t paid to do notes. Medical errors aren’t new, they’re very common. Picking a few cases of AI errors is meaningless - stats on error levels with and without AI would be more useful.
From what I got talking to friends who work in the field, the problem starts at this point
doesn’t have time / isn’t paid to do notes
Insurance and bureaucratic pressures have increased the number and size of notes that doctors are expected to create. Then, instead of hiring scribes or compensating doctors appropriately for taking notes, they are expected to meet those ever increasing requirements with the limited time that they have.
We already had audio processing “AI”, and doctors have been using them for years. That’s actually one area where the recent advances around open models have really helped, because now it’s cheaper and easier to run those systems locally and doctors (or scribes) can document faster than typing.
IMO it should be the administrative body that should be apologizing for this one.
The part I disagree with is this
Picking a few cases of AI errors is meaningless - stats on error levels with and without AI would be more useful.
This would be fair if we were comparing typed notes to speach-to-text notes. However when talking about these LLM systems, the type of error that a doctor makes is different from the types of errors an LLM makes, and so the scale of the damage can be much higher. A doctor isn’t going to hallucinate new conditions, or make up that a patient is being abused (or abusing a child) out of nowhere. I don’t have a source on the later, but it’s something I remember from a research article that was shared here in the past. That kind of thing can have severe consequences on a person because doctors are often relied on as experts to determine how someone should be treated by other systems (civil/legal etc).
Also, human errors are generally predictable and so we can create some safeguards against them. For example, writing 5mg instead of 5.0 mg in hand written charts, double checking numbers during shift changes, etc. It’s another thing to try and account for unpredictable hallucinations.
