Doctors turn to imperfect AI to spend more quality time with patients

Amid pandemic burnout and increased demand for documentation, doctors are recording patient conversations and giving note-writing work to an AI-based tool, even if that risks privacy or medical errors.


The DAX system uses an app installed on a doctor’s personal or work-issued mobile device to record audio from visits with patients.

Illustration: Christopher T. Fong/Protocol

“Pajama time.” To most of us, that term connotes bedtime for kids, or cozying up on the couch. To doctors, pajama time means homework. In fact, it’s a common phrase describing the nighttime ritual of finishing up clinical notes about the patients they saw earlier that day.

As demands for notes and data to chronicle patient interactions from hospital administration and insurance industry payers have increased, the amount of time physicians spend on the computer has squeezed their already tight schedules. A 2017 study published in Annals of Family Medicine found that primary care physicians spend nearly six hours a day interacting with their electronic health records systems during and after clinic hours.

Amid pandemic burnout, the stress is enough for doctors to hand over the work of writing clinical notes to an AI-based tool, even if it could create patient data privacy risks and medical errors.

“Nobody went into medicine to write notes or fill out insurance authorization forms,” said J. Scott Smitherman, chief medical information officer at Providence Health, a national not-for-profit health system. But he said medical payers now require so much documentation related to patients that “it’s almost an arms race.”

It’s no wonder some physicians he works with have been eager to test software that, with a simple press of a button on their phones during a patient visit, automatically writes the clinical notes they are responsible for entering into the hospital group’s electronic health records system.

The software, called Dragon Ambient eXperience — or DAX — requires that a patient’s visit with their doctor be recorded, whether in real life or virtually. It’s made by Nuance, recently acquired by Microsoft, and is available today through integrations with two of the top electronic health records software providers, EPIC and Cerner. DAX is also plugged into Microsoft’s virtual meeting software, Teams, for use in telehealth settings.

Oracle, a Microsoft competitor in the health care cloud and AI arena, is in the process of acquiring Cerner, making for some potentially strange bedfellows.

Around 100 Providence primary care physicians in California, Montana, Oregon and Washington have tested the Nuance technology — only in in-person settings — for the past year through the EPIC health records platform they use, said Smitherman. “It’s taking the burden of the documenting off the clinician,” he said.

An imperfect assistant

The DAX system uses an app installed on a doctor’s personal or work-issued mobile device to record audio from visits with patients. Nuance incorporates AI such as speech recognition and natural-language processing, or NLP, to transcribe that patient appointment audio recording then transform it into a formatted clinical note including information about the patient’s symptoms, their chief complaints that brought them to visit their physician and their treatment plan, such as suggested procedures or prescribed medications. Some physicians also access the DAX system through hardware installed in their offices.

Several other hospitals and health care groups also use DAX through direct partnerships with Nuance, including Augusta University Health, Boston Children’s Hospital, University of Michigan Health West, Monument Health and WellSpan Health, according to Nuance. SouthEast Alaska Regional uses the DAX system through its relationship with Cerner, according to Nuance and Cerner.

But the technology is far from perfect. Right now, it is not fully integrated with either EPIC’s or Cerner’s patient-records-management platforms. That means the automatically generated clinical notes have to be pasted manually into the proper places in those records systems.

And there are more significant problems.

Nobody went into medicine to write notes or fill out insurance authorization forms.

“The NLP that creates that note, it isn't perfect,” said Kenneth Harper, vice president and general manager for DAX at Nuance. “It's such a hard task, it will make mistakes. Sometimes it might omit clinical facts; sometimes it may even hallucinate something.” Hallucination is a term commonly used to reflect when an NLP model conjures up information.

The system also has trouble parsing conversations in which multiple languages or a variety of voices — from people like nurse practitioners or family members — are involved. Writing the note in the format preferred by each individual doctor can trip it up, too.

Plus, doctors don’t always vocalize every important piece of information they are detecting about a patient, like if their heart sounds good.

“We don't really want the technology to infer something, and if it wasn't spoken, that can definitely lead to a medical error,” Harper said. “So there are some behavioral things we have to work with providers on."

But even without AI, doctors sometimes wait until the end of the day to write their clinical notes manually, when their memories might be fuzzy. “That inherently is a high-error-rate process,” said Will Manidis, CEO and co-founder of ScienceIO, which builds large-scale machine-learning systems that read medical documents and has provided technology to automatically generate clinical notes for customers. “I don’t think it’s inherently worse to have that automatically encoded than to have a physician do it,” Manidis said.

“The major concerns I would have here is I’m not sure the computer would be smart enough to know what is important [enough] to pull out into the note,” said Dr. Shravani Durbhakula, a pain physician and anesthesiologist at the Johns Hopkins School of Medicine, who said the hospital does not use tools like DAX to automate clinical notes. “You could miss critical information."

For instance, she said the system might reflect the fact that a patient experienced endocrine dysfunction, but only code it as the result of a testosterone deficiency rather than an opioid-induced deficiency, a piece of information critical to understanding the person’s condition. As that information gets passed into health care records and from specialist to specialist, “that could continue to get missed in other portions of the evaluation process,” she said.

AI reviewers behind the curtain

For now, until DAX improves, the process has a safety net: real-life human reviewers. After the system generates a clinical note — but before it is uploaded into the health record platform — reviewers in the U.S. and overseas, some of whom are Nuance employees and some contractors, step in to edit. They have access to the notes themselves and the audio of the conversation. Harper emphasized that the people editing the notes work in “a highly secure environment” that does not allow the data or the clinical notes to leave their work interface.

Other NLP systems also vet the outputs of their models through a human review process. Though the company does not use a time-consuming and expensive human review process all the time, Deepgram, which builds voice AI products using speech recognition and deep learning, does use it selectively to help improve language models used for things like bank customer service or closed captioning, said Scott Stephenson, the company’s CEO and co-founder. He said Deepgram might select a small portion of outputs the company was not confident were accurate, have those assessed by a person, then feed the information back into the system to keep training the model.

Patient privacy and malpractice lawsuit worries

But there’s an obvious patient privacy factor sometimes glossed over when discussing these systems. Physicians using DAX are supposed to notify or ask their patients about recording their health care visit and obtain verbal consent, though it is unclear whether everyone using DAX does that.

“When we’ve talked to physicians, it wasn’t an issue that came up,” said Boyd Stewart, vice president of Payer/Provider Collaboration at health care industry research firm KLAS. “There are patients from time to time that say no,” he said.

A simulation of Nuance's Dax technology capturing a conversation between a doctor and patient.A simulation of Nuance's Dax technology capturing a conversation between a doctor and patient.Photo: Nuance

Smitherman said Providence vetted the consent process with legal teams in states where it has used DAX. He said doctors have not expressed concern about patient data privacy. They have worried about lawsuits, though.

“‘Could this be used against me in a malpractice case’ is the concern that comes up most commonly,” Smitherman said.

Harper said once a patient meeting is recorded, it does not remain on the doctor’s mobile device; rather, it is encrypted and sent to the cloud using a secure streaming process. “All patient data is fully encrypted and running in HIPAA-compliant environments,” a Nuance spokesperson said. DAX is built on top of Microsoft’s Azure cloud.

As for its partner Cerner, the company does not send or store audio recordings used to generate clinical notes, although it does store transcripts of those recordings for up to 10 days, after which time they are destroyed, said Stephanie Greenwood, Cerner’s media relations manager.

Sometimes it might omit clinical facts, sometimes it may even hallucinate something.

But there’s another way the patient data and clinical notes are used by Nuance: to continue teaching Nuance’s system to improve. The company feeds the patient data in its original form into its models, and depending on its contractual agreements with customers, de-identifies or deletes the data after a contracted period of time.

“That gives us a data source of: What did the note get right from the conversation, and what did the note miss or get wrong? And we can go back and keep re-teaching the summarization model to get smarter based on those changes,” Harper said.

For Microsoft and Oracle, a conduit to real-world clinical data

Providence pays around $8,000 to $10,000 per year, per doctor, to license the use of DAX, Smitherman said.

But for EPIC, Cerner, Microsoft and Oracle, a service like DAX isn’t just about revenue. It carves out a new route toward capturing valuable real-world clinical data.

When Cerner bought Kantar Health for $375 million in April 2021, Cerner said the acquisition would enhance its pool of real-world clinical evidence data — the sort of information used by pharmaceutical clients such as Pfizer and Novartis.

“Real-world data is essentially data that’s coming from electronic medical records and commercial claims, and when you combine that data together in an anonymized and compliant way, you have an enormous asset for understanding the real-world utility of a device or a drug or a molecule, as well as the health-economic aspects of that,” said Todd Greenwood, senior director of Outcomes at ZS Associates, a tech consultancy.

Technologies like DAX also open a window on actual doctor-patient conversational interactions, Greenwood added. For instance, he said, “What these kinds of much-more-ethnographic approaches allow is for us to understand how often there is a directive conversation versus a collaborative conversation” between doctor and patient.

Data from patient interaction transcripts and clinical notes could be used for analysis companies might pass on to their own customers, helping pharmaceutical manufacturers learn how often their drugs are mentioned by name at the doctor’s office, for instance.

Their approaches to serving the health care market do not directly mirror one another, but Microsoft’s and Oracle’s goals are strikingly similar. When they have discussed their respective acquisitions, both have emphasized the use of speech recognition and NLP to detach overworked medical professionals from their computer keyboards.

“We will make Cerner’s systems much easier to learn and use by making Oracle’s hands-free Voice Digital Assistant the primary interface to Cerner’s clinical systems. This will allow medical professionals to spend less time typing on computer keyboards and more time caring for patients,” said Mike Sicilia, EVP of Vertical Industries at Oracle, in a December press release where the proposed Cerner acquisition was announced.

Microsoft touts the combination of its cloud services with Nuance’s speech recognition technology, which it refers to as “ambient” rather than “hands-free.”

For now, Harper said Microsoft’s acquisition of Nuance doesn’t change much about the company’s relationship with Cerner. In fact, he said the combination of Microsoft’s computing resources and neural networks is helping improve its technology for customers like Cerner.

“The idea of using [Microsoft’s] scale, using their compute to further advance what our technology can do within these enterprise and health care verticals, that's where we think the most synergy is going to come about. What it means is for DAX, getting higher accuracy faster, getting to automation faster, feeding into Cerner high-quality information from the conversation.”

In the end, if systems like DAX improve, the promise of less “pajama time” could compel more physicians and hospitals to find ways to access them. Preliminary research by KLAS evaluating the effect that DAX is having on doctors’ attitudes toward their oft-maligned electronic health record platforms indicates interest could grow, despite the risks. Compared to KLAS survey participants who do not use DAX, those who do use the system said the burden of digital documentation was reduced as a result.

“They’re getting more time with the patients — more connection, more meaningful time,” said Stewart.

But Durbhakula is skeptical. “Until we get the inputs right and the systems are smart enough to encode those inputs accurately, then we’re going to have issues with bias and inaccuracies and those could negatively impact clinical outcomes,” she said.

This story was updated to correct the name of Providence Health.


Judge Zia Faruqui is trying to teach you crypto, one ‘SNL’ reference at a time

His decisions on major cryptocurrency cases have quoted "The Big Lebowski," "SNL," and "Dr. Strangelove." That’s because he wants you — yes, you — to read them.

The ways Zia Faruqui (right) has weighed on cases that have come before him can give lawyers clues as to what legal frameworks will pass muster.

Photo: Carolyn Van Houten/The Washington Post via Getty Images

“Cryptocurrency and related software analytics tools are ‘The wave of the future, Dude. One hundred percent electronic.’”

That’s not a quote from "The Big Lebowski" — at least, not directly. It’s a quote from a Washington, D.C., district court memorandum opinion on the role cryptocurrency analytics tools can play in government investigations. The author is Magistrate Judge Zia Faruqui.

Keep ReadingShow less
Veronica Irwin

Veronica Irwin (@vronirwin) is a San Francisco-based reporter at Protocol covering fintech. Previously she was at the San Francisco Examiner, covering tech from a hyper-local angle. Before that, her byline was featured in SF Weekly, The Nation, Techworker, Ms. Magazine and The Frisc.

The financial technology transformation is driving competition, creating consumer choice, and shaping the future of finance. Hear from seven fintech leaders who are reshaping the future of finance, and join the inaugural Financial Technology Association Fintech Summit to learn more.

Keep ReadingShow less
The Financial Technology Association (FTA) represents industry leaders shaping the future of finance. We champion the power of technology-centered financial services and advocate for the modernization of financial regulation to support inclusion and responsible innovation.

AWS CEO: The cloud isn’t just about technology

As AWS preps for its annual re:Invent conference, Adam Selipsky talks product strategy, support for hybrid environments, and the value of the cloud in uncertain economic times.

Photo: Noah Berger/Getty Images for Amazon Web Services

AWS is gearing up for re:Invent, its annual cloud computing conference where announcements this year are expected to focus on its end-to-end data strategy and delivering new industry-specific services.

It will be the second re:Invent with CEO Adam Selipsky as leader of the industry’s largest cloud provider after his return last year to AWS from data visualization company Tableau Software.

Keep ReadingShow less
Donna Goodison

Donna Goodison (@dgoodison) is Protocol's senior reporter focusing on enterprise infrastructure technology, from the 'Big 3' cloud computing providers to data centers. She previously covered the public cloud at CRN after 15 years as a business reporter for the Boston Herald. Based in Massachusetts, she also has worked as a Boston Globe freelancer, business reporter at the Boston Business Journal and real estate reporter at Banker & Tradesman after toiling at weekly newspapers.

Image: Protocol

We launched Protocol in February 2020 to cover the evolving power center of tech. It is with deep sadness that just under three years later, we are winding down the publication.

As of today, we will not publish any more stories. All of our newsletters, apart from our flagship, Source Code, will no longer be sent. Source Code will be published and sent for the next few weeks, but it will also close down in December.

Keep ReadingShow less
Bennett Richardson

Bennett Richardson ( @bennettrich) is the president of Protocol. Prior to joining Protocol in 2019, Bennett was executive director of global strategic partnerships at POLITICO, where he led strategic growth efforts including POLITICO's European expansion in Brussels and POLITICO's creative agency POLITICO Focus during his six years with the company. Prior to POLITICO, Bennett was co-founder and CMO of Hinge, the mobile dating company recently acquired by Match Group. Bennett began his career in digital and social brand marketing working with major brands across tech, energy, and health care at leading marketing and communications agencies including Edelman and GMMB. Bennett is originally from Portland, Maine, and received his bachelor's degree from Colgate University.


Why large enterprises struggle to find suitable platforms for MLops

As companies expand their use of AI beyond running just a few machine learning models, and as larger enterprises go from deploying hundreds of models to thousands and even millions of models, ML practitioners say that they have yet to find what they need from prepackaged MLops systems.

As companies expand their use of AI beyond running just a few machine learning models, ML practitioners say that they have yet to find what they need from prepackaged MLops systems.

Photo: artpartner-images via Getty Images

On any given day, Lily AI runs hundreds of machine learning models using computer vision and natural language processing that are customized for its retail and ecommerce clients to make website product recommendations, forecast demand, and plan merchandising. But this spring when the company was in the market for a machine learning operations platform to manage its expanding model roster, it wasn’t easy to find a suitable off-the-shelf system that could handle such a large number of models in deployment while also meeting other criteria.

Some MLops platforms are not well-suited for maintaining even more than 10 machine learning models when it comes to keeping track of data, navigating their user interfaces, or reporting capabilities, Matthew Nokleby, machine learning manager for Lily AI’s product intelligence team, told Protocol earlier this year. “The duct tape starts to show,” he said.

Keep ReadingShow less
Kate Kaye

Kate Kaye is an award-winning multimedia reporter digging deep and telling print, digital and audio stories. She covers AI and data for Protocol. Her reporting on AI and tech ethics issues has been published in OneZero, Fast Company, MIT Technology Review, CityLab, Ad Age and Digiday and heard on NPR. Kate is the creator of RedTailMedia.org and is the author of "Campaign '08: A Turning Point for Digital Media," a book about how the 2008 presidential campaigns used digital media and data.

Latest Stories