Deepdub uses AI to dub movies in the voice of famous actors

Fresh out of stealth, the startup is using artificial intelligence to automate the localization process for global streaming.

Deepdub AI dubbing demo

Deepdub promises AI dubbing at the click of a button.

Image: Deepdub

The streaming wars aren't just about domestic viewers anymore: Netflix, Disney, HBO Max and the like increasingly compete around the world. Around a third of Disney+ subscribers, for instance, are based in India, and the company is looking to further grow its international audience in Europe and Latin America.

Tel Aviv-based startup Deepdub wants to help streaming services accelerate this kind of international rollout by using artificial intelligence for their localization needs. Deepdub, which came out of stealth on Wednesday, has built technology that can translate a voice track to a different language, all while staying true to the voice of the talent. This makes it possible to have someone like Morgan Freeman narrate a movie in French, Italian or Russian without losing what makes Freeman's voice special and recognizable.

Deepdub was founded in late 2019, and has been in conversations with a number of potential Hollywood clients, according to Deepdub CMO Oz Krakowski. "People's jaws are dropping to the floor," he said, adding that industry insiders especially value the ability to adjust the voice of an AI-generated dubbing track. "We call it voice design," he explained.

Deepdub has published an impressive demo video on YouTube. Krakowski recently gave me some more context on how the company's technology works:

  • Deepdub currently can dub videos into six different languages, including French, German and Spanish. The company aims to double that number in the next few months, and ultimately wants to be able to localize videos into 60 languages.
  • The technology is designed to fully automate the dubbing process, but Krakowski admitted that higher-end projects still require a bit of manual intervention for things like local idioms and brands. For instance, the convenience store chain 7-Eleven doesn't exist in many countries, so employees may have to help the system find an adequate replacement.
  • However, this kind of intervention only has to happen once, and it benefits all future projects. "The system keeps learning," Krakowski said. "The amount of manual work decreases over time."
  • Deepdub can work with a variety of source materials. This includes complete video files with mixed-down audio tracks from which the system is able to remove and replace individual voices. That could especially come in handy for streaming services with a lot of library content. "You have streaming services with thousands of videos that are stuck on a single language," Krakowski said.

Deepdub is not the only company looking to use AI for dubbing and voice localization. Toronto-based Resemble has built tech to clone your voice in different languages, which is already being used for chatbots and automated call centers. London-based Synthesia also targets Hollywood, albeit with a bit of a different approach: It changes not only the audio track, but actually uses deepfake image manipulation to match an actor's mouth movement to the new voice. Krakowski pointed out that such an approach is still challenged by different camera angles as well as onscreen motion, but he acknowledged that the two technologies may ultimately converge. "It's not a question of if, just of when," he said.

In the meantime, AI dubbing may be able to help Hollywood with some more pressing problems. The pandemic forced streaming services to limit dubbing for a number of markets. For instance, Netflix has been telling viewers in recent months that it had to delay localization for some of its titles. "We're prioritizing voice actors' and dubbing studio safety," the company informed viewers within its app. Artificial intelligence could help to bridge those gaps, even if studios ultimately decide to rely on real voice actors for the final version.

A version of this article will appear in this week's Next Up newsletter. Sign up here to receive it in your inbox every Thursday.

Can we bring malls into the metaverse?

Soon you might buy digital sneakers to wear on your digital date in a digital world.

Combined with the hype around digital goods and cryptocurrency, companies and futurists are starting to imagine what shopping in the metaverse might look like.

Photo illustration: Mark Abramson/Bloomberg via Getty Images; The Fabricant; Protocol

Click banner image for more Shopping Week coverage

Before the internet, the mall was the spot for watching movies, hanging out, listening to music, finding love — and an embodiment of all-American consumerism. "The shopping center was Amazon, it was Facebook, it was Tinder, it was Spotify, it was Netflix," said retail futurist Doug Stephens. "It was the gathering point in the community."

Keep Reading Show less
Lizzy Lawrence

Lizzy Lawrence ( @LizzyLaw_) is a reporter at Protocol, covering tools and productivity in the workplace. She's a recent graduate of the University of Michigan, where she studied sociology and international studies. She served as editor in chief of The Michigan Daily, her school's independent newspaper. She's based in D.C., and can be reached at llawrence@protocol.com.

The Bureau of Labor Statistics indicates that by 2026, the shortage of engineers in the U.S. will exceed 1.2 million, while 545,000 software developers will have left the market by that time. Meanwhile, business is becoming increasingly more digital-first, and teams need the tools in place to keep distributed teams aligned and able to respond quickly to changing business needs. That means businesses need to build powerful workplace applications without relying on developers.

In fact, according to Gartner, by 2025, 70% of new applications developed by enterprises will use low-code or no-code technologies and, by 2023, there will be at least four times as many active citizen developers as professional developers at large enterprises. We're on the cusp of a big shift in how businesses operate and how organization wide innovation happens.

Keep Reading Show less
Andrew Ofstad
As Airtable’s co-founder, Andrew spearheads Airtable’s long-term product bets and represents the voice of the customer in major product decisions. After co-founding the company, he helped scale Airtable’s original product and engineering teams. He previously led the redesign of Google's flagship Maps product, and before that was a product manager for Android.

Can we bring malls into the metaverse?

Soon you might buy digital sneakers to wear on your digital date in a digital world.

Combined with the hype around digital goods and cryptocurrency, companies and futurists are starting to imagine what shopping in the metaverse might look like.

Photo illustration: Mark Abramson/Bloomberg via Getty Images; The Fabricant; Protocol

Click banner image for more Shopping Week coverage

Before the internet, the mall was the spot for watching movies, hanging out, listening to music, finding love — and an embodiment of all-American consumerism. "The shopping center was Amazon, it was Facebook, it was Tinder, it was Spotify, it was Netflix," said retail futurist Doug Stephens. "It was the gathering point in the community."

Keep Reading Show less
Lizzy Lawrence

Lizzy Lawrence ( @LizzyLaw_) is a reporter at Protocol, covering tools and productivity in the workplace. She's a recent graduate of the University of Michigan, where she studied sociology and international studies. She served as editor in chief of The Michigan Daily, her school's independent newspaper. She's based in D.C., and can be reached at llawrence@protocol.com.

It’s still too hard to give crypto this holiday

Here's how the industry Scrooged itself out of a big opportunity.

If only giving bitcoin were this simple.

Photo illustration: gpointstudio/Getty Images Plus; Protocol

Click banner image for more Shopping Week coverage

Crypto took huge steps toward the mainstream this year. Bitcoin soared in value, Coinbase went public and VCs poured even more money into the industry. In case consumers didn't get the message, they'll surely notice when the Staples Center turns into Crypto.com Arena next month and FTX airs its first Super Bowl ad in February.

Keep Reading Show less
Benjamin Pimentel

Benjamin Pimentel ( @benpimentel) covers fintech from San Francisco. He has reported on many of the biggest tech stories over the past 20 years for the San Francisco Chronicle, Dow Jones MarketWatch and Business Insider, from the dot-com crash, the rise of cloud computing, social networking and AI to the impact of the Great Recession and the COVID crisis on Silicon Valley and beyond. He can be reached at bpimentel@protocol.com or via Signal at (510)731-8429.

It’s time to rethink Black Friday

The pandemic didn't end Black Friday, but it'll never look the same again.

We can expect Black Friday to stick around but lose relevance as retailers effectively dilute its meaning and purpose.

Illustration: Christopher T. Fong/Protocol

Click banner image for more Shopping Week coverage

"I'm selling meditation, so I shouldn't be stressed," said Charlie Rousset, the co-founder of sleep and relaxation gadget-maker Morphée. But even deep breathing can't help Rousset feel less on edge this Black Friday.

Keep Reading Show less
Janko Roettgers

Janko Roettgers (@jank0) is a senior reporter at Protocol, reporting on the shifting power dynamics between tech, media, and entertainment, including the impact of new technologies. Previously, Janko was Variety's first-ever technology writer in San Francisco, where he covered big tech and emerging technologies. He has reported for Gigaom, Frankfurter Rundschau, Berliner Zeitung, and ORF, among others. He has written three books on consumer cord-cutting and online music and co-edited an anthology on internet subcultures. He lives with his family in Oakland.

Latest Stories