Doctors turn to imperfect AI to spend more quality time with patients

Amid pandemic burnout and increased demand for documentation, doctors are recording patient conversations and giving note-writing work to an AI-based tool, even if that risks privacy or medical errors.


The DAX system uses an app installed on a doctor’s personal or work-issued mobile device to record audio from visits with patients.

Illustration: Christopher T. Fong/Protocol

“Pajama time.” To most of us, that term connotes bedtime for kids, or cozying up on the couch. To doctors, pajama time means homework. In fact, it’s a common phrase describing the nighttime ritual of finishing up clinical notes about the patients they saw earlier that day.

As demands for notes and data to chronicle patient interactions from hospital administration and insurance industry payers have increased, the amount of time physicians spend on the computer has squeezed their already tight schedules. A 2017 study published in Annals of Family Medicine found that primary care physicians spend nearly six hours a day interacting with their electronic health records systems during and after clinic hours.

Amid pandemic burnout, the stress is enough for doctors to hand over the work of writing clinical notes to an AI-based tool, even if it could create patient data privacy risks and medical errors.

“Nobody went into medicine to write notes or fill out insurance authorization forms,” said J. Scott Smitherman, chief medical information officer at Providence Health, a national not-for-profit health system. But he said medical payers now require so much documentation related to patients that “it’s almost an arms race.”

It’s no wonder some physicians he works with have been eager to test software that, with a simple press of a button on their phones during a patient visit, automatically writes the clinical notes they are responsible for entering into the hospital group’s electronic health records system.

The software, called Dragon Ambient eXperience — or DAX — requires that a patient’s visit with their doctor be recorded, whether in real life or virtually. It’s made by Nuance, recently acquired by Microsoft, and is available today through integrations with two of the top electronic health records software providers, EPIC and Cerner. DAX is also plugged into Microsoft’s virtual meeting software, Teams, for use in telehealth settings.

Oracle, a Microsoft competitor in the health care cloud and AI arena, is in the process of acquiring Cerner, making for some potentially strange bedfellows.

Around 100 Providence primary care physicians in California, Montana, Oregon and Washington have tested the Nuance technology — only in in-person settings — for the past year through the EPIC health records platform they use, said Smitherman. “It’s taking the burden of the documenting off the clinician,” he said.

An imperfect assistant

The DAX system uses an app installed on a doctor’s personal or work-issued mobile device to record audio from visits with patients. Nuance incorporates AI such as speech recognition and natural-language processing, or NLP, to transcribe that patient appointment audio recording then transform it into a formatted clinical note including information about the patient’s symptoms, their chief complaints that brought them to visit their physician and their treatment plan, such as suggested procedures or prescribed medications. Some physicians also access the DAX system through hardware installed in their offices.

Several other hospitals and health care groups also use DAX through direct partnerships with Nuance, including Augusta University Health, Boston Children’s Hospital, University of Michigan Health West, Monument Health and WellSpan Health, according to Nuance. SouthEast Alaska Regional uses the DAX system through its relationship with Cerner, according to Nuance and Cerner.

But the technology is far from perfect. Right now, it is not fully integrated with either EPIC’s or Cerner’s patient-records-management platforms. That means the automatically generated clinical notes have to be pasted manually into the proper places in those records systems.

And there are more significant problems.

Nobody went into medicine to write notes or fill out insurance authorization forms.

“The NLP that creates that note, it isn't perfect,” said Kenneth Harper, vice president and general manager for DAX at Nuance. “It's such a hard task, it will make mistakes. Sometimes it might omit clinical facts; sometimes it may even hallucinate something.” Hallucination is a term commonly used to reflect when an NLP model conjures up information.

The system also has trouble parsing conversations in which multiple languages or a variety of voices — from people like nurse practitioners or family members — are involved. Writing the note in the format preferred by each individual doctor can trip it up, too.

Plus, doctors don’t always vocalize every important piece of information they are detecting about a patient, like if their heart sounds good.

“We don't really want the technology to infer something, and if it wasn't spoken, that can definitely lead to a medical error,” Harper said. “So there are some behavioral things we have to work with providers on."

But even without AI, doctors sometimes wait until the end of the day to write their clinical notes manually, when their memories might be fuzzy. “That inherently is a high-error-rate process,” said Will Manidis, CEO and co-founder of ScienceIO, which builds large-scale machine-learning systems that read medical documents and has provided technology to automatically generate clinical notes for customers. “I don’t think it’s inherently worse to have that automatically encoded than to have a physician do it,” Manidis said.

“The major concerns I would have here is I’m not sure the computer would be smart enough to know what is important [enough] to pull out into the note,” said Dr. Shravani Durbhakula, a pain physician and anesthesiologist at the Johns Hopkins School of Medicine, who said the hospital does not use tools like DAX to automate clinical notes. “You could miss critical information."

For instance, she said the system might reflect the fact that a patient experienced endocrine dysfunction, but only code it as the result of a testosterone deficiency rather than an opioid-induced deficiency, a piece of information critical to understanding the person’s condition. As that information gets passed into health care records and from specialist to specialist, “that could continue to get missed in other portions of the evaluation process,” she said.

AI reviewers behind the curtain

For now, until DAX improves, the process has a safety net: real-life human reviewers. After the system generates a clinical note — but before it is uploaded into the health record platform — reviewers in the U.S. and overseas, some of whom are Nuance employees and some contractors, step in to edit. They have access to the notes themselves and the audio of the conversation. Harper emphasized that the people editing the notes work in “a highly secure environment” that does not allow the data or the clinical notes to leave their work interface.

Other NLP systems also vet the outputs of their models through a human review process. Though the company does not use a time-consuming and expensive human review process all the time, Deepgram, which builds voice AI products using speech recognition and deep learning, does use it selectively to help improve language models used for things like bank customer service or closed captioning, said Scott Stephenson, the company’s CEO and co-founder. He said Deepgram might select a small portion of outputs the company was not confident were accurate, have those assessed by a person, then feed the information back into the system to keep training the model.

Patient privacy and malpractice lawsuit worries

But there’s an obvious patient privacy factor sometimes glossed over when discussing these systems. Physicians using DAX are supposed to notify or ask their patients about recording their health care visit and obtain verbal consent, though it is unclear whether everyone using DAX does that.

“When we’ve talked to physicians, it wasn’t an issue that came up,” said Boyd Stewart, vice president of Payer/Provider Collaboration at health care industry research firm KLAS. “There are patients from time to time that say no,” he said.

A simulation of Nuance's Dax technology capturing a conversation between a doctor and patient. A simulation of Nuance's Dax technology capturing a conversation between a doctor and patient.Photo: Nuance

Smitherman said Providence vetted the consent process with legal teams in states where it has used DAX. He said doctors have not expressed concern about patient data privacy. They have worried about lawsuits, though.

“‘Could this be used against me in a malpractice case’ is the concern that comes up most commonly,” Smitherman said.

Harper said once a patient meeting is recorded, it does not remain on the doctor’s mobile device; rather, it is encrypted and sent to the cloud using a secure streaming process. “All patient data is fully encrypted and running in HIPAA-compliant environments,” a Nuance spokesperson said. DAX is built on top of Microsoft’s Azure cloud.

As for its partner Cerner, the company does not send or store audio recordings used to generate clinical notes, although it does store transcripts of those recordings for up to 10 days, after which time they are destroyed, said Stephanie Greenwood, Cerner’s media relations manager.

Sometimes it might omit clinical facts, sometimes it may even hallucinate something.

But there’s another way the patient data and clinical notes are used by Nuance: to continue teaching Nuance’s system to improve. The company feeds the patient data in its original form into its models, and depending on its contractual agreements with customers, de-identifies or deletes the data after a contracted period of time.

“That gives us a data source of: What did the note get right from the conversation, and what did the note miss or get wrong? And we can go back and keep re-teaching the summarization model to get smarter based on those changes,” Harper said.

For Microsoft and Oracle, a conduit to real-world clinical data

Providence pays around $8,000 to $10,000 per year, per doctor, to license the use of DAX, Smitherman said.

But for EPIC, Cerner, Microsoft and Oracle, a service like DAX isn’t just about revenue. It carves out a new route toward capturing valuable real-world clinical data.

When Cerner bought Kantar Health for $375 million in April 2021, Cerner said the acquisition would enhance its pool of real-world clinical evidence data — the sort of information used by pharmaceutical clients such as Pfizer and Novartis.

“Real-world data is essentially data that’s coming from electronic medical records and commercial claims, and when you combine that data together in an anonymized and compliant way, you have an enormous asset for understanding the real-world utility of a device or a drug or a molecule, as well as the health-economic aspects of that,” said Todd Greenwood, senior director of Outcomes at ZS Associates, a tech consultancy.

Technologies like DAX also open a window on actual doctor-patient conversational interactions, Greenwood added. For instance, he said, “What these kinds of much-more-ethnographic approaches allow is for us to understand how often there is a directive conversation versus a collaborative conversation” between doctor and patient.

Data from patient interaction transcripts and clinical notes could be used for analysis companies might pass on to their own customers, helping pharmaceutical manufacturers learn how often their drugs are mentioned by name at the doctor’s office, for instance.

Their approaches to serving the health care market do not directly mirror one another, but Microsoft’s and Oracle’s goals are strikingly similar. When they have discussed their respective acquisitions, both have emphasized the use of speech recognition and NLP to detach overworked medical professionals from their computer keyboards.

“We will make Cerner’s systems much easier to learn and use by making Oracle’s hands-free Voice Digital Assistant the primary interface to Cerner’s clinical systems. This will allow medical professionals to spend less time typing on computer keyboards and more time caring for patients,” said Mike Sicilia, EVP of Vertical Industries at Oracle, in a December press release where the proposed Cerner acquisition was announced.

Microsoft touts the combination of its cloud services with Nuance’s speech recognition technology, which it refers to as “ambient” rather than “hands-free.”

For now, Harper said Microsoft’s acquisition of Nuance doesn’t change much about the company’s relationship with Cerner. In fact, he said the combination of Microsoft’s computing resources and neural networks is helping improve its technology for customers like Cerner.

“The idea of using [Microsoft’s] scale, using their compute to further advance what our technology can do within these enterprise and health care verticals, that's where we think the most synergy is going to come about. What it means is for DAX, getting higher accuracy faster, getting to automation faster, feeding into Cerner high-quality information from the conversation.”

In the end, if systems like DAX improve, the promise of less “pajama time” could compel more physicians and hospitals to find ways to access them. Preliminary research by KLAS evaluating the effect that DAX is having on doctors’ attitudes toward their oft-maligned electronic health record platforms indicates interest could grow, despite the risks. Compared to KLAS survey participants who do not use DAX, those who do use the system said the burden of digital documentation was reduced as a result.

“They’re getting more time with the patients — more connection, more meaningful time,” said Stewart.

But Durbhakula is skeptical. “Until we get the inputs right and the systems are smart enough to encode those inputs accurately, then we’re going to have issues with bias and inaccuracies and those could negatively impact clinical outcomes,” she said.

This story was updated to correct the name of Providence Health.


To clear the FTC, Microsoft’s Activision deal might require compromise

The FTC is in the process of reviewing the biggest-ever gaming acquisition. Here’s how it could change the Xbox business.

Will the Microsoft acquisition of Activision get through the FTC?

Image: Microsoft; Protocol

Microsoft’s planned acquisition of Activision Blizzard is the largest-ever deal in the video game market by a mile. With a sale price of $68.7 billion, the deal is nearly 450% larger than Grand Theft Auto publisher Take-Two Interactive’s acquisition of Zynga in January, the next-largest game acquisition ever recorded.

The eye-popping price underlines the scale and scope of Microsoft’s ambitions for its gaming business: If the deal is approved, Microsoft would own — alongside its current major properties, such as Halo and Minecraft — Warcraft, Overwatch and Call of Duty, to name just a few. In turn, the deal has invited a rare level of scrutiny and attention from lawmakers and policy professionals now turning their sights on an industry that’s flown under the regulatory radar for the last several decades of its existence.

Keep Reading Show less
Nick Statt

Nick Statt is Protocol's video game reporter. Prior to joining Protocol, he was news editor at The Verge covering the gaming industry, mobile apps and antitrust out of San Francisco, in addition to managing coverage of Silicon Valley tech giants and startups. He now resides in Rochester, New York, home of the garbage plate and, completely coincidentally, the World Video Game Hall of Fame. He can be reached at nstatt@protocol.com.

Sponsored Content

Why the digital transformation of industries is creating a more sustainable future

Qualcomm’s chief sustainability officer Angela Baker on how companies can view going “digital” as a way not only toward growth, as laid out in a recent report, but also toward establishing and meeting environmental, social and governance goals.

Three letters dominate business practice at present: ESG, or environmental, social and governance goals. The number of mentions of the environment in financial earnings has doubled in the last five years, according to GlobalData: 600,000 companies mentioned the term in their annual or quarterly results last year.

But meeting those ESG goals can be a challenge — one that businesses can’t and shouldn’t take lightly. Ahead of an exclusive fireside chat at Davos, Angela Baker, chief sustainability officer at Qualcomm, sat down with Protocol to speak about how best to achieve those targets and how Qualcomm thinks about its own sustainability strategy, net zero commitment, other ESG targets and more.

Keep Reading Show less
Chris Stokel-Walker

Chris Stokel-Walker is a freelance technology and culture journalist and author of "YouTubers: How YouTube Shook Up TV and Created a New Generation of Stars." His work has been published in The New York Times, The Guardian and Wired.


Okta CEO: 'We should have done a better job' with the Lapsus$ breach

In an interview with Protocol, Okta CEO Todd McKinnon said the cybersecurity firm could’ve done a lot of things better after the Lapsus$ breach of a third-party support provider earlier this year.

From talking to hundreds of customers, “I've had a good sense of the sentiment and the frustrations,” McKinnon said.

Photo: David Paul Morris via Getty Images

Okta co-founder and CEO Todd McKinnon agrees with you: Disclosing a breach that impacts customer data should not take months.

“If that happens in January, customers can't be finding out about it in March,” McKinnon said in an interview with Protocol.

Keep Reading Show less
Kyle Alspach

Kyle Alspach ( @KyleAlspach) is a senior reporter at Protocol, focused on cybersecurity. He has covered the tech industry since 2010 for outlets including VentureBeat, CRN and the Boston Globe. He lives in Portland, Oregon, and can be reached at kalspach@procotol.com.


Ethereum's co-founder thinks the blockchain can fix social media

But before the blockchain can fix social media, someone has to fix the blockchain. Frank McCourt, who’s put serious money behind his vision of a decentralized social media future, thinks Gavin Wood may be the key.

Gavin Wood, co-founder of Ethereum and creator of Polkadot, is helping Frank McCourt's decentralized social media initiative.

Photo: Jason Crowley

Frank McCourt, the billionaire mogul who is donating $100 million to help build decentralized alternatives to the social media giants, has picked a partner to make the blockchain work at Facebook scale: Ethereum co-founder Gavin Wood.

McCourt’s Project Liberty will work with the Web3 Foundation’s Polkadot project, it said Tuesday. Wood launched Polkadot in 2020 after leaving Ethereum. Project Liberty has a technical proposal to allow users to retain their data on a blockchain as they move among future social media services. Wood’s involvement is to give the idea a shot at actually working at the size and speed of a popular social network.

Keep Reading Show less
Ben Brody

Ben Brody (@ BenBrodyDC) is a senior reporter at Protocol focusing on how Congress, courts and agencies affect the online world we live in. He formerly covered tech policy and lobbying (including antitrust, Section 230 and privacy) at Bloomberg News, where he previously reported on the influence industry, government ethics and the 2016 presidential election. Before that, Ben covered business news at CNNMoney and AdAge, and all manner of stories in and around New York. He still loves appearing on the New York news radio he grew up with.


Gensler: Bitcoin may be a commodity

The SEC has been vague about crypto. But Gensler said bitcoin is a commodity, “maybe.” It’s the clearest glimpse of his views on digital assets yet.

“Bitcoin — maybe that’s a commodity token. That has a big market value, but that goes over there,” Gensler said, referring to another regulator, the CFTC.

Photoillustration: Al Drago/Bloomberg via Getty Images; Protocol

SEC Chair Gary Gensler has long argued that many cryptocurrencies are subject to regulation as securities.

But he recently clarified that this view wouldn’t apply to the best-known cryptocurrency, bitcoin.

Keep Reading Show less
Benjamin Pimentel

Benjamin Pimentel ( @benpimentel) covers crypto and fintech from San Francisco. He has reported on many of the biggest tech stories over the past 20 years for the San Francisco Chronicle, Dow Jones MarketWatch and Business Insider, from the dot-com crash, the rise of cloud computing, social networking and AI to the impact of the Great Recession and the COVID crisis on Silicon Valley and beyond. He can be reached at bpimentel@protocol.com or via Google Voice at (925) 307-9342.

Latest Stories