A Lack of Regulation
Another concern, Dr. Yazdany said, is that AI tools are typically not federally regulated, because the companies that create them say they are not intended to be used for direct clinical decision making.
“This is giving tech companies free reign to deploy unregulated software in our clinics, tools that do impact us and our patients,” she said.
She said that one of her patients, who had severe lupus nephritis, was taking mycophenolate mofetil and tacrolimus. She started having serious side effects from the tacrolimus, so Dr. Yazdany told the patient to stop taking it and switch to an available alternative. But the AI-powered scribe that automatically produces notes from the clinical encounter misinterpreted what was said. It documented that tacrolimus was to be continued.
“What if she had read the note?” Dr. Yazdany said. “If something bad had happened, the liability would be entirely on me. The AI-scribe company says their device is not for medical decision making. They’re entirely shielded. That should make every one of us pause.”
Both Dr. Curtis and Dr. Yazdany called for guardrails to ensure that AI is improving healthcare rather than hurting it. Before clinicians put AI to use, they should review the evidence supporting it, Dr. Yazdany said. “Review the data just like you would any clinical trial.”
Thomas R. Collins is a freelance medical writer based in Florida.
References
- Tai-Seale M, Baxter SL, Vaida F, et al. AI-generated draft replies integrated into health records and physicians’ electronic communication. JAMA Netw Open. 2024 Apr 1;7(4):e246565.
- Kalweit M, Burden AM, Boedecker J, et al. Patient groups in rheumatoid arthritis identified by deep learning respond differently to biologic or targeted synthetic DMARDs. PLoS Comput Biol. 2023 Jun 2;19(6):e1011073.
- Obermeyer Z, Powers B, Vogeli C, et al. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019 Oct 25;366(6464):447–453.
- Seyyed-Kalantari L, Zhang H, McDermott MBA, et al. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nat Med. 2021 Dec;27(12):2176–2182.
- Budzyń K, Romańczyk M, Kitala D, et al. Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: A multicentre, observational study. Lancet Gastroenterol Hepatol. 2025 Oct;10(10):896–903.
