Enterprise

Doctors turn to imperfect AI to spend more quality time with patients

Amid pandemic burnout and increased demand for documentation, doctors are recording patient conversations and giving note-writing work to an AI-based tool, even if that risks privacy or medical errors.

Caduceus

The DAX system uses an app installed on a doctor’s personal or work-issued mobile device to record audio from visits with patients.

Illustration: Christopher T. Fong/Protocol

“Pajama time.” To most of us, that term connotes bedtime for kids, or cozying up on the couch. To doctors, pajama time means homework. In fact, it’s a common phrase describing the nighttime ritual of finishing up clinical notes about the patients they saw earlier that day.

As demands for notes and data to chronicle patient interactions from hospital administration and insurance industry payers have increased, the amount of time physicians spend on the computer has squeezed their already tight schedules. A 2017 study published in Annals of Family Medicine found that primary care physicians spend nearly six hours a day interacting with their electronic health records systems during and after clinic hours.

Amid pandemic burnout, the stress is enough for doctors to hand over the work of writing clinical notes to an AI-based tool, even if it could create patient data privacy risks and medical errors.

“Nobody went into medicine to write notes or fill out insurance authorization forms,” said J. Scott Smitherman, chief medical information officer at Providence Health, a national not-for-profit health system. But he said medical payers now require so much documentation related to patients that “it’s almost an arms race.”

It’s no wonder some physicians he works with have been eager to test software that, with a simple press of a button on their phones during a patient visit, automatically writes the clinical notes they are responsible for entering into the hospital group’s electronic health records system.

The software, called Dragon Ambient eXperience — or DAX — requires that a patient’s visit with their doctor be recorded, whether in real life or virtually. It’s made by Nuance, recently acquired by Microsoft, and is available today through integrations with two of the top electronic health records software providers, EPIC and Cerner. DAX is also plugged into Microsoft’s virtual meeting software, Teams, for use in telehealth settings.

Oracle, a Microsoft competitor in the health care cloud and AI arena, is in the process of acquiring Cerner, making for some potentially strange bedfellows.

Around 100 Providence primary care physicians in California, Montana, Oregon and Washington have tested the Nuance technology — only in in-person settings — for the past year through the EPIC health records platform they use, said Smitherman. “It’s taking the burden of the documenting off the clinician,” he said.

An imperfect assistant

The DAX system uses an app installed on a doctor’s personal or work-issued mobile device to record audio from visits with patients. Nuance incorporates AI such as speech recognition and natural-language processing, or NLP, to transcribe that patient appointment audio recording then transform it into a formatted clinical note including information about the patient’s symptoms, their chief complaints that brought them to visit their physician and their treatment plan, such as suggested procedures or prescribed medications. Some physicians also access the DAX system through hardware installed in their offices.

Several other hospitals and health care groups also use DAX through direct partnerships with Nuance, including Augusta University Health, Boston Children’s Hospital, University of Michigan Health West, Monument Health and WellSpan Health, according to Nuance. SouthEast Alaska Regional uses the DAX system through its relationship with Cerner, according to Nuance and Cerner.

But the technology is far from perfect. Right now, it is not fully integrated with either EPIC’s or Cerner’s patient-records-management platforms. That means the automatically generated clinical notes have to be pasted manually into the proper places in those records systems.

And there are more significant problems.

Nobody went into medicine to write notes or fill out insurance authorization forms.

“The NLP that creates that note, it isn't perfect,” said Kenneth Harper, vice president and general manager for DAX at Nuance. “It's such a hard task, it will make mistakes. Sometimes it might omit clinical facts; sometimes it may even hallucinate something.” Hallucination is a term commonly used to reflect when an NLP model conjures up information.

The system also has trouble parsing conversations in which multiple languages or a variety of voices — from people like nurse practitioners or family members — are involved. Writing the note in the format preferred by each individual doctor can trip it up, too.

Plus, doctors don’t always vocalize every important piece of information they are detecting about a patient, like if their heart sounds good.

“We don't really want the technology to infer something, and if it wasn't spoken, that can definitely lead to a medical error,” Harper said. “So there are some behavioral things we have to work with providers on."

But even without AI, doctors sometimes wait until the end of the day to write their clinical notes manually, when their memories might be fuzzy. “That inherently is a high-error-rate process,” said Will Manidis, CEO and co-founder of ScienceIO, which builds large-scale machine-learning systems that read medical documents and has provided technology to automatically generate clinical notes for customers. “I don’t think it’s inherently worse to have that automatically encoded than to have a physician do it,” Manidis said.

“The major concerns I would have here is I’m not sure the computer would be smart enough to know what is important [enough] to pull out into the note,” said Dr. Shravani Durbhakula, a pain physician and anesthesiologist at the Johns Hopkins School of Medicine, who said the hospital does not use tools like DAX to automate clinical notes. “You could miss critical information."

For instance, she said the system might reflect the fact that a patient experienced endocrine dysfunction, but only code it as the result of a testosterone deficiency rather than an opioid-induced deficiency, a piece of information critical to understanding the person’s condition. As that information gets passed into health care records and from specialist to specialist, “that could continue to get missed in other portions of the evaluation process,” she said.

AI reviewers behind the curtain

For now, until DAX improves, the process has a safety net: real-life human reviewers. After the system generates a clinical note — but before it is uploaded into the health record platform — reviewers in the U.S. and overseas, some of whom are Nuance employees and some contractors, step in to edit. They have access to the notes themselves and the audio of the conversation. Harper emphasized that the people editing the notes work in “a highly secure environment” that does not allow the data or the clinical notes to leave their work interface.

Other NLP systems also vet the outputs of their models through a human review process. Though the company does not use a time-consuming and expensive human review process all the time, Deepgram, which builds voice AI products using speech recognition and deep learning, does use it selectively to help improve language models used for things like bank customer service or closed captioning, said Scott Stephenson, the company’s CEO and co-founder. He said Deepgram might select a small portion of outputs the company was not confident were accurate, have those assessed by a person, then feed the information back into the system to keep training the model.

Patient privacy and malpractice lawsuit worries

But there’s an obvious patient privacy factor sometimes glossed over when discussing these systems. Physicians using DAX are supposed to notify or ask their patients about recording their health care visit and obtain verbal consent, though it is unclear whether everyone using DAX does that.

“When we’ve talked to physicians, it wasn’t an issue that came up,” said Boyd Stewart, vice president of Payer/Provider Collaboration at health care industry research firm KLAS. “There are patients from time to time that say no,” he said.

A simulation of Nuance's Dax technology capturing a conversation between a doctor and patient. A simulation of Nuance's Dax technology capturing a conversation between a doctor and patient.Photo: Nuance

Smitherman said Providence vetted the consent process with legal teams in states where it has used DAX. He said doctors have not expressed concern about patient data privacy. They have worried about lawsuits, though.

“‘Could this be used against me in a malpractice case’ is the concern that comes up most commonly,” Smitherman said.

Harper said once a patient meeting is recorded, it does not remain on the doctor’s mobile device; rather, it is encrypted and sent to the cloud using a secure streaming process. “All patient data is fully encrypted and running in HIPAA-compliant environments,” a Nuance spokesperson said. DAX is built on top of Microsoft’s Azure cloud.

As for its partner Cerner, the company does not send or store audio recordings used to generate clinical notes, although it does store transcripts of those recordings for up to 10 days, after which time they are destroyed, said Stephanie Greenwood, Cerner’s media relations manager.

Sometimes it might omit clinical facts, sometimes it may even hallucinate something.

But there’s another way the patient data and clinical notes are used by Nuance: to continue teaching Nuance’s system to improve. The company feeds the patient data in its original form into its models, and depending on its contractual agreements with customers, de-identifies or deletes the data after a contracted period of time.

“That gives us a data source of: What did the note get right from the conversation, and what did the note miss or get wrong? And we can go back and keep re-teaching the summarization model to get smarter based on those changes,” Harper said.

For Microsoft and Oracle, a conduit to real-world clinical data

Providence pays around $8,000 to $10,000 per year, per doctor, to license the use of DAX, Smitherman said.

But for EPIC, Cerner, Microsoft and Oracle, a service like DAX isn’t just about revenue. It carves out a new route toward capturing valuable real-world clinical data.

When Cerner bought Kantar Health for $375 million in April 2021, Cerner said the acquisition would enhance its pool of real-world clinical evidence data — the sort of information used by pharmaceutical clients such as Pfizer and Novartis.

“Real-world data is essentially data that’s coming from electronic medical records and commercial claims, and when you combine that data together in an anonymized and compliant way, you have an enormous asset for understanding the real-world utility of a device or a drug or a molecule, as well as the health-economic aspects of that,” said Todd Greenwood, senior director of Outcomes at ZS Associates, a tech consultancy.

Technologies like DAX also open a window on actual doctor-patient conversational interactions, Greenwood added. For instance, he said, “What these kinds of much-more-ethnographic approaches allow is for us to understand how often there is a directive conversation versus a collaborative conversation” between doctor and patient.

Data from patient interaction transcripts and clinical notes could be used for analysis companies might pass on to their own customers, helping pharmaceutical manufacturers learn how often their drugs are mentioned by name at the doctor’s office, for instance.

Their approaches to serving the health care market do not directly mirror one another, but Microsoft’s and Oracle’s goals are strikingly similar. When they have discussed their respective acquisitions, both have emphasized the use of speech recognition and NLP to detach overworked medical professionals from their computer keyboards.

“We will make Cerner’s systems much easier to learn and use by making Oracle’s hands-free Voice Digital Assistant the primary interface to Cerner’s clinical systems. This will allow medical professionals to spend less time typing on computer keyboards and more time caring for patients,” said Mike Sicilia, EVP of Vertical Industries at Oracle, in a December press release where the proposed Cerner acquisition was announced.

Microsoft touts the combination of its cloud services with Nuance’s speech recognition technology, which it refers to as “ambient” rather than “hands-free.”

For now, Harper said Microsoft’s acquisition of Nuance doesn’t change much about the company’s relationship with Cerner. In fact, he said the combination of Microsoft’s computing resources and neural networks is helping improve its technology for customers like Cerner.

“The idea of using [Microsoft’s] scale, using their compute to further advance what our technology can do within these enterprise and health care verticals, that's where we think the most synergy is going to come about. What it means is for DAX, getting higher accuracy faster, getting to automation faster, feeding into Cerner high-quality information from the conversation.”

In the end, if systems like DAX improve, the promise of less “pajama time” could compel more physicians and hospitals to find ways to access them. Preliminary research by KLAS evaluating the effect that DAX is having on doctors’ attitudes toward their oft-maligned electronic health record platforms indicates interest could grow, despite the risks. Compared to KLAS survey participants who do not use DAX, those who do use the system said the burden of digital documentation was reduced as a result.

“They’re getting more time with the patients — more connection, more meaningful time,” said Stewart.

But Durbhakula is skeptical. “Until we get the inputs right and the systems are smart enough to encode those inputs accurately, then we’re going to have issues with bias and inaccuracies and those could negatively impact clinical outcomes,” she said.

This story was updated to correct the name of Providence Health.

Policy

Musk’s texts reveal what tech’s most powerful people really want

From Jack Dorsey to Joe Rogan, Musk’s texts are chock-full of überpowerful people, bending a knee to Twitter’s once and (still maybe?) future king.

“Maybe Oprah would be interested in joining the Twitter board if my bid succeeds,” one text reads.

Photo illustration: Patrick Pleul/picture alliance via Getty Images; Protocol

Elon Musk’s text inbox is a rarefied space. It’s a place where tech’s wealthiest casually commit to spending billions of dollars with little more than a thumbs-up emoji and trade tips on how to rewrite the rules for how hundreds of millions of people around the world communicate.

Now, Musk’s ongoing legal battle with Twitter is giving the rest of us a fleeting glimpse into that world. The collection of Musk’s private texts that was made public this week is chock-full of tech power brokers. While the messages are meant to reveal something about Musk’s motivations — and they do — they also say a lot about how things get done and deals get made among some of the most powerful people in the world.

Keep Reading Show less
Issie Lapowsky

Issie Lapowsky ( @issielapowsky) is Protocol's chief correspondent, covering the intersection of technology, politics, and national affairs. She also oversees Protocol's fellowship program. Previously, she was a senior writer at Wired, where she covered the 2016 election and the Facebook beat in its aftermath. Prior to that, Issie worked as a staff writer for Inc. magazine, writing about small business and entrepreneurship. She has also worked as an on-air contributor for CBS News and taught a graduate-level course at New York University's Center for Publishing on how tech giants have affected publishing.

Sponsored Content

Great products are built on strong patents

Experts say robust intellectual property protection is essential to ensure the long-term R&D required to innovate and maintain America's technology leadership.

Every great tech product that you rely on each day, from the smartphone in your pocket to your music streaming service and navigational system in the car, shares one important thing: part of its innovative design is protected by intellectual property (IP) laws.

From 5G to artificial intelligence, IP protection offers a powerful incentive for researchers to create ground-breaking products, and governmental leaders say its protection is an essential part of maintaining US technology leadership. To quote Secretary of Commerce Gina Raimondo: "intellectual property protection is vital for American innovation and entrepreneurship.”

Keep Reading Show less
James Daly
James Daly has a deep knowledge of creating brand voice identity, including understanding various audiences and targeting messaging accordingly. He enjoys commissioning, editing, writing, and business development, particularly in launching new ventures and building passionate audiences. Daly has led teams large and small to multiple awards and quantifiable success through a strategy built on teamwork, passion, fact-checking, intelligence, analytics, and audience growth while meeting budget goals and production deadlines in fast-paced environments. Daly is the Editorial Director of 2030 Media and a contributor at Wired.
Fintech

Circle’s CEO: This is not the time to ‘go crazy’

Jeremy Allaire is leading the stablecoin powerhouse in a time of heightened regulation.

“It’s a complex environment. So every CEO and every board has to be a little bit cautious, because there’s a lot of uncertainty,” Circle CEO Jeremy Allaire told Protocol at Converge22.

Photo: Circle

Sitting solo on a San Francisco stage, Circle CEO Jeremy Allaire asked tennis superstar Serena Williams what it’s like to face “unrelenting skepticism.”

“What do you do when someone says you can’t do this?” Allaire asked the athlete turned VC, who was beaming into Circle’s Converge22 convention by video.

Keep Reading Show less
Benjamin Pimentel

Benjamin Pimentel ( @benpimentel) covers crypto and fintech from San Francisco. He has reported on many of the biggest tech stories over the past 20 years for the San Francisco Chronicle, Dow Jones MarketWatch and Business Insider, from the dot-com crash, the rise of cloud computing, social networking and AI to the impact of the Great Recession and the COVID crisis on Silicon Valley and beyond. He can be reached at bpimentel@protocol.com or via Google Voice at (925) 307-9342.

Enterprise

Is Salesforce still a growth company? Investors are skeptical

Salesforce is betting that customer data platform Genie and new Slack features can push the company to $50 billion in revenue by 2026. But investors are skeptical about the company’s ability to deliver.

Photo: Marlena Sloss/Bloomberg via Getty Images

Salesforce has long been enterprise tech’s golden child. The company said everything customers wanted to hear and did everything investors wanted to see: It produced robust, consistent growth from groundbreaking products combined with an aggressive M&A strategy and a cherished culture, all operating under the helm of a bombastic, but respected, CEO and team of well-coiffed executives.

Dreamforce is the embodiment of that success. Every year, alongside frustrating San Francisco residents, the over-the-top celebration serves as a battle cry to the enterprise software industry, reminding everyone that Marc Benioff’s mighty fiefdom is poised to expand even deeper into your corporate IT stack.

Keep Reading Show less
Joe Williams

Joe Williams is a writer-at-large at Protocol. He previously covered enterprise software for Protocol, Bloomberg and Business Insider. Joe can be reached at JoeWilliams@Protocol.com. To share information confidentially, he can also be contacted on a non-work device via Signal (+1-309-265-6120) or JPW53189@protonmail.com.

Policy

The US and EU are splitting on tech policy. That’s putting the web at risk.

A conversation with Cédric O, the former French minister of state for digital.

“With the difficulty of the U.S. in finding political agreement or political basis to legislate more, we are facing a risk of decoupling in the long term between the EU and the U.S.”

Photo: David Paul Morris/Bloomberg via Getty Images

Cédric O, France’s former minister of state for digital, has been an advocate of Europe’s approach to tech and at the forefront of the continent’s relations with U.S. giants. Protocol caught up with O last week at a conference in New York focusing on social media’s negative effects on society and the possibilities of blockchain-based protocols for alternative networks.

O said watching the U.S. lag in tech policy — even as some states pass their own measures and federal bills gain momentum — has made him worry about the EU and U.S. decoupling. While not as drastic as a disentangling of economic fortunes between the West and China, such a divergence, as O describes it, could still make it functionally impossible for companies to serve users on both sides of the Atlantic with the same product.

Keep Reading Show less
Ben Brody

Ben Brody (@ BenBrodyDC) is a senior reporter at Protocol focusing on how Congress, courts and agencies affect the online world we live in. He formerly covered tech policy and lobbying (including antitrust, Section 230 and privacy) at Bloomberg News, where he previously reported on the influence industry, government ethics and the 2016 presidential election. Before that, Ben covered business news at CNNMoney and AdAge, and all manner of stories in and around New York. He still loves appearing on the New York news radio he grew up with.

Latest Stories
Bulletins