Doctors turn to imperfect AI to spend more quality time with patients

Amid pandemic burnout and increased demand for documentation, doctors are recording patient conversations and giving note-writing work to an AI-based tool, even if that risks privacy or medical errors.


The DAX system uses an app installed on a doctor’s personal or work-issued mobile device to record audio from visits with patients.

Illustration: Christopher T. Fong/Protocol

“Pajama time.” To most of us, that term connotes bedtime for kids, or cozying up on the couch. To doctors, pajama time means homework. In fact, it’s a common phrase describing the nighttime ritual of finishing up clinical notes about the patients they saw earlier that day.

As demands for notes and data to chronicle patient interactions from hospital administration and insurance industry payers have increased, the amount of time physicians spend on the computer has squeezed their already tight schedules. A 2017 study published in Annals of Family Medicine found that primary care physicians spend nearly six hours a day interacting with their electronic health records systems during and after clinic hours.

Amid pandemic burnout, the stress is enough for doctors to hand over the work of writing clinical notes to an AI-based tool, even if it could create patient data privacy risks and medical errors.

“Nobody went into medicine to write notes or fill out insurance authorization forms,” said J. Scott Smitherman, chief medical information officer at Providence Health, a national not-for-profit health system. But he said medical payers now require so much documentation related to patients that “it’s almost an arms race.”

It’s no wonder some physicians he works with have been eager to test software that, with a simple press of a button on their phones during a patient visit, automatically writes the clinical notes they are responsible for entering into the hospital group’s electronic health records system.

The software, called Dragon Ambient eXperience — or DAX — requires that a patient’s visit with their doctor be recorded, whether in real life or virtually. It’s made by Nuance, recently acquired by Microsoft, and is available today through integrations with two of the top electronic health records software providers, EPIC and Cerner. DAX is also plugged into Microsoft’s virtual meeting software, Teams, for use in telehealth settings.

Oracle, a Microsoft competitor in the health care cloud and AI arena, is in the process of acquiring Cerner, making for some potentially strange bedfellows.

Around 100 Providence primary care physicians in California, Montana, Oregon and Washington have tested the Nuance technology — only in in-person settings — for the past year through the EPIC health records platform they use, said Smitherman. “It’s taking the burden of the documenting off the clinician,” he said.

An imperfect assistant

The DAX system uses an app installed on a doctor’s personal or work-issued mobile device to record audio from visits with patients. Nuance incorporates AI such as speech recognition and natural-language processing, or NLP, to transcribe that patient appointment audio recording then transform it into a formatted clinical note including information about the patient’s symptoms, their chief complaints that brought them to visit their physician and their treatment plan, such as suggested procedures or prescribed medications. Some physicians also access the DAX system through hardware installed in their offices.

Several other hospitals and health care groups also use DAX through direct partnerships with Nuance, including Augusta University Health, Boston Children’s Hospital, University of Michigan Health West, Monument Health and WellSpan Health, according to Nuance. SouthEast Alaska Regional uses the DAX system through its relationship with Cerner, according to Nuance and Cerner.

But the technology is far from perfect. Right now, it is not fully integrated with either EPIC’s or Cerner’s patient-records-management platforms. That means the automatically generated clinical notes have to be pasted manually into the proper places in those records systems.

And there are more significant problems.

Nobody went into medicine to write notes or fill out insurance authorization forms.

“The NLP that creates that note, it isn't perfect,” said Kenneth Harper, vice president and general manager for DAX at Nuance. “It's such a hard task, it will make mistakes. Sometimes it might omit clinical facts; sometimes it may even hallucinate something.” Hallucination is a term commonly used to reflect when an NLP model conjures up information.

The system also has trouble parsing conversations in which multiple languages or a variety of voices — from people like nurse practitioners or family members — are involved. Writing the note in the format preferred by each individual doctor can trip it up, too.

Plus, doctors don’t always vocalize every important piece of information they are detecting about a patient, like if their heart sounds good.

“We don't really want the technology to infer something, and if it wasn't spoken, that can definitely lead to a medical error,” Harper said. “So there are some behavioral things we have to work with providers on."

But even without AI, doctors sometimes wait until the end of the day to write their clinical notes manually, when their memories might be fuzzy. “That inherently is a high-error-rate process,” said Will Manidis, CEO and co-founder of ScienceIO, which builds large-scale machine-learning systems that read medical documents and has provided technology to automatically generate clinical notes for customers. “I don’t think it’s inherently worse to have that automatically encoded than to have a physician do it,” Manidis said.

“The major concerns I would have here is I’m not sure the computer would be smart enough to know what is important [enough] to pull out into the note,” said Dr. Shravani Durbhakula, a pain physician and anesthesiologist at the Johns Hopkins School of Medicine, who said the hospital does not use tools like DAX to automate clinical notes. “You could miss critical information."

For instance, she said the system might reflect the fact that a patient experienced endocrine dysfunction, but only code it as the result of a testosterone deficiency rather than an opioid-induced deficiency, a piece of information critical to understanding the person’s condition. As that information gets passed into health care records and from specialist to specialist, “that could continue to get missed in other portions of the evaluation process,” she said.

AI reviewers behind the curtain

For now, until DAX improves, the process has a safety net: real-life human reviewers. After the system generates a clinical note — but before it is uploaded into the health record platform — reviewers in the U.S. and overseas, some of whom are Nuance employees and some contractors, step in to edit. They have access to the notes themselves and the audio of the conversation. Harper emphasized that the people editing the notes work in “a highly secure environment” that does not allow the data or the clinical notes to leave their work interface.

Other NLP systems also vet the outputs of their models through a human review process. Though the company does not use a time-consuming and expensive human review process all the time, Deepgram, which builds voice AI products using speech recognition and deep learning, does use it selectively to help improve language models used for things like bank customer service or closed captioning, said Scott Stephenson, the company’s CEO and co-founder. He said Deepgram might select a small portion of outputs the company was not confident were accurate, have those assessed by a person, then feed the information back into the system to keep training the model.

Patient privacy and malpractice lawsuit worries

But there’s an obvious patient privacy factor sometimes glossed over when discussing these systems. Physicians using DAX are supposed to notify or ask their patients about recording their health care visit and obtain verbal consent, though it is unclear whether everyone using DAX does that.

“When we’ve talked to physicians, it wasn’t an issue that came up,” said Boyd Stewart, vice president of Payer/Provider Collaboration at health care industry research firm KLAS. “There are patients from time to time that say no,” he said.

A simulation of Nuance's Dax technology capturing a conversation between a doctor and patient. A simulation of Nuance's Dax technology capturing a conversation between a doctor and patient.Photo: Nuance

Smitherman said Providence vetted the consent process with legal teams in states where it has used DAX. He said doctors have not expressed concern about patient data privacy. They have worried about lawsuits, though.

“‘Could this be used against me in a malpractice case’ is the concern that comes up most commonly,” Smitherman said.

Harper said once a patient meeting is recorded, it does not remain on the doctor’s mobile device; rather, it is encrypted and sent to the cloud using a secure streaming process. “All patient data is fully encrypted and running in HIPAA-compliant environments,” a Nuance spokesperson said. DAX is built on top of Microsoft’s Azure cloud.

As for its partner Cerner, the company does not send or store audio recordings used to generate clinical notes, although it does store transcripts of those recordings for up to 10 days, after which time they are destroyed, said Stephanie Greenwood, Cerner’s media relations manager.

Sometimes it might omit clinical facts, sometimes it may even hallucinate something.

But there’s another way the patient data and clinical notes are used by Nuance: to continue teaching Nuance’s system to improve. The company feeds the patient data in its original form into its models, and depending on its contractual agreements with customers, de-identifies or deletes the data after a contracted period of time.

“That gives us a data source of: What did the note get right from the conversation, and what did the note miss or get wrong? And we can go back and keep re-teaching the summarization model to get smarter based on those changes,” Harper said.

For Microsoft and Oracle, a conduit to real-world clinical data

Providence pays around $8,000 to $10,000 per year, per doctor, to license the use of DAX, Smitherman said.

But for EPIC, Cerner, Microsoft and Oracle, a service like DAX isn’t just about revenue. It carves out a new route toward capturing valuable real-world clinical data.

When Cerner bought Kantar Health for $375 million in April 2021, Cerner said the acquisition would enhance its pool of real-world clinical evidence data — the sort of information used by pharmaceutical clients such as Pfizer and Novartis.

“Real-world data is essentially data that’s coming from electronic medical records and commercial claims, and when you combine that data together in an anonymized and compliant way, you have an enormous asset for understanding the real-world utility of a device or a drug or a molecule, as well as the health-economic aspects of that,” said Todd Greenwood, senior director of Outcomes at ZS Associates, a tech consultancy.

Technologies like DAX also open a window on actual doctor-patient conversational interactions, Greenwood added. For instance, he said, “What these kinds of much-more-ethnographic approaches allow is for us to understand how often there is a directive conversation versus a collaborative conversation” between doctor and patient.

Data from patient interaction transcripts and clinical notes could be used for analysis companies might pass on to their own customers, helping pharmaceutical manufacturers learn how often their drugs are mentioned by name at the doctor’s office, for instance.

Their approaches to serving the health care market do not directly mirror one another, but Microsoft’s and Oracle’s goals are strikingly similar. When they have discussed their respective acquisitions, both have emphasized the use of speech recognition and NLP to detach overworked medical professionals from their computer keyboards.

“We will make Cerner’s systems much easier to learn and use by making Oracle’s hands-free Voice Digital Assistant the primary interface to Cerner’s clinical systems. This will allow medical professionals to spend less time typing on computer keyboards and more time caring for patients,” said Mike Sicilia, EVP of Vertical Industries at Oracle, in a December press release where the proposed Cerner acquisition was announced.

Microsoft touts the combination of its cloud services with Nuance’s speech recognition technology, which it refers to as “ambient” rather than “hands-free.”

For now, Harper said Microsoft’s acquisition of Nuance doesn’t change much about the company’s relationship with Cerner. In fact, he said the combination of Microsoft’s computing resources and neural networks is helping improve its technology for customers like Cerner.

“The idea of using [Microsoft’s] scale, using their compute to further advance what our technology can do within these enterprise and health care verticals, that's where we think the most synergy is going to come about. What it means is for DAX, getting higher accuracy faster, getting to automation faster, feeding into Cerner high-quality information from the conversation.”

In the end, if systems like DAX improve, the promise of less “pajama time” could compel more physicians and hospitals to find ways to access them. Preliminary research by KLAS evaluating the effect that DAX is having on doctors’ attitudes toward their oft-maligned electronic health record platforms indicates interest could grow, despite the risks. Compared to KLAS survey participants who do not use DAX, those who do use the system said the burden of digital documentation was reduced as a result.

“They’re getting more time with the patients — more connection, more meaningful time,” said Stewart.

But Durbhakula is skeptical. “Until we get the inputs right and the systems are smart enough to encode those inputs accurately, then we’re going to have issues with bias and inaccuracies and those could negatively impact clinical outcomes,” she said.

This story was updated to correct the name of Providence Health.

Sponsored Content

How cybercrime is going small time

Blockbuster hacks are no longer the norm – causing problems for companies trying to track down small-scale crime

Cybercrime is often thought of on a relatively large scale. Massive breaches lead to painful financial losses, bankrupting companies and causing untold embarrassment, splashed across the front pages of news websites worldwide. That’s unsurprising: cyber events typically cost businesses around $200,000, according to cybersecurity firm the Cyentia Institute. One in 10 of those victims suffer losses of more than $20 million, with some reaching $100 million or more.

That’s big money – but there’s plenty of loot out there for cybercriminals willing to aim lower. In 2021, the Internet Crime Complaint Center (IC3) received 847,376 complaints – reports by cybercrime victims – totaling losses of $6.9 billion. Averaged out, each victim lost $8,143.

Keep Reading Show less
Chris Stokel-Walker

Chris Stokel-Walker is a freelance technology and culture journalist and author of "YouTubers: How YouTube Shook Up TV and Created a New Generation of Stars." His work has been published in The New York Times, The Guardian and Wired.


SEC cyber reporting regs may be stuck. CISA is poised to do better.

CISA’s initiative to regulate critical infrastructure on incident reporting is just beginning. The focus on industry engagement by CISA and its director, Jen Easterly, could be about to pay off.

CISA director Jen Easterly is focusing on cyber industry engagement.

Photo: Kevin Dietsch/Getty Images

As the chief information security officer of a large, publicly traded tech company, Drew Simonis has been keeping a close eye on the SEC's proposed rules to require reporting of major cyberattacks.

Simonis, who works at Juniper Networks, has some serious concerns shared by many executives in U.S. private industry. Some of the proposed cyber incident reporting rules seem like they'd be counterproductive to the goal of creating transparency, and would likely just increase confusion for corporate shareholders, he said. Overall, by requiring public disclosure of major cyber incidents within four business days, the approach seems to lack a basic understanding of the "fluid nature of security events," Simonis said.

Keep Reading Show less
Kyle Alspach

Kyle Alspach ( @KyleAlspach) is a senior reporter at Protocol, focused on cybersecurity. He has covered the tech industry since 2010 for outlets including VentureBeat, CRN and the Boston Globe. He lives in Portland, Oregon, and can be reached at kalspach@protocol.com.


EA mobile chief Jeff Karp on EA’s live service future

Electronic Arts is faring better than its rivals. The company’s mobile chief knows why.

FIFA Mobile, a new version of which launched in January, just had its best-ever quarter.

Photo: Electronic Arts

Electronic Arts, the sports game publisher that spent years neglecting the mobile gaming market, couldn’t have picked a better time to jump in the deep end.

Last year, EA spent close to $4 billion acquiring its way to a stronger position in mobile. This year, it launched a new iteration of its popular FIFA franchise for smartphones and released a mobile spinoff of its hugely successful Apex Legends battle royale. The company's competitors have also followed suit with even more eye-popping acquisitions, including Take-Two Interactive’s purchase of Zynga for nearly $13 billion back in January and Microsoft’s record $69 billion acquisition of Activision Blizzard, which includes Candy Crush studio King, soon after.

Keep Reading Show less
Nick Statt

Nick Statt is Protocol's video game reporter. Prior to joining Protocol, he was news editor at The Verge covering the gaming industry, mobile apps and antitrust out of San Francisco, in addition to managing coverage of Silicon Valley tech giants and startups. He now resides in Rochester, New York, home of the garbage plate and, completely coincidentally, the World Video Game Hall of Fame. He can be reached at nstatt@protocol.com.


NYC's bungled monkeypox vaccine rollout has a familiar ring

The failures raise questions about the due diligence undertaken by officials in awarding contracts.

The tech failures are part of a broader mishandling of the monkeypox outbreak at all levels of government, which is causing public health experts to fear that the virus could already be out of hand.

Photo: Kobi Wolf/Bloomberg via Getty Images

In 2016, New York's state Attorney General Eric Schneiderman reached a settlement with a company known as MedRite Urgent Care, after Yelp caught the company paying for fake positive reviews. At the time, the attorney general's office accused MedRite of "misrepresentation and deceptive acts," for which the urgent care provider agreed to pay a $100,000 fine.

And yet, just six years later, in the midst of its fast-growing monkeypox outbreak, New York City chose MedRite to operate its monkeypox vaccine scheduler. The first day of the rollout, MedRite's system crashed, leaving New Yorkers scrambling to get access to a vaccine that is already in limited supply.

Keep Reading Show less
Kwasi Gyamfi Asiedu

Kwasi (kway-see) is a fellow at Protocol with an interest in tech policy and climate. Previously, he covered global religion news at the Associated Press in New York. Before that, he was a freelance journalist based out of Accra, Ghana, covering social justice, health, and environment stories. His reporting has been published in The New York Times, Quartz, CNN, The Guardian, and Public Radio International. He can be reached at kasiedu@protocol.com.

Latest Stories