next-upnext upauthorJanko RoettgersNoneDo you know what's coming next up in the world of tech and entertainment? Get Janko Roettgers' newsletter every Thursday.9147dfd6b1
×

Get access to Protocol

I’ve already subscribed

Will be used in accordance with our Privacy Policy

Deepdub uses AI to dub movies in the voice of famous actors

Fresh out of stealth, the startup is using artificial intelligence to automate the localization process for global streaming.

Deepdub AI dubbing demo

Deepdub promises AI dubbing at the click of a button.

Image: Deepdub

The streaming wars aren't just about domestic viewers anymore: Netflix, Disney, HBO Max and the like increasingly compete around the world. Around a third of Disney+ subscribers, for instance, are based in India, and the company is looking to further grow its international audience in Europe and Latin America.

Tel Aviv-based startup Deepdub wants to help streaming services accelerate this kind of international rollout by using artificial intelligence for their localization needs. Deepdub, which came out of stealth on Wednesday, has built technology that can translate a voice track to a different language, all while staying true to the voice of the talent. This makes it possible to have someone like Morgan Freeman narrate a movie in French, Italian or Russian without losing what makes Freeman's voice special and recognizable.

Deepdub was founded in late 2019, and has been in conversations with a number of potential Hollywood clients, according to Deepdub CMO Oz Krakowski. "People's jaws are dropping to the floor," he said, adding that industry insiders especially value the ability to adjust the voice of an AI-generated dubbing track. "We call it voice design," he explained.

Deepdub has published an impressive demo video on YouTube. Krakowski recently gave me some more context on how the company's technology works:

  • Deepdub currently can dub videos into six different languages, including French, German and Spanish. The company aims to double that number in the next few months, and ultimately wants to be able to localize videos into 60 languages.
  • The technology is designed to fully automate the dubbing process, but Krakowski admitted that higher-end projects still require a bit of manual intervention for things like local idioms and brands. For instance, the convenience store chain 7-Eleven doesn't exist in many countries, so employees may have to help the system find an adequate replacement.
  • However, this kind of intervention only has to happen once, and it benefits all future projects. "The system keeps learning," Krakowski said. "The amount of manual work decreases over time."
  • Deepdub can work with a variety of source materials. This includes complete video files with mixed-down audio tracks from which the system is able to remove and replace individual voices. That could especially come in handy for streaming services with a lot of library content. "You have streaming services with thousands of videos that are stuck on a single language," Krakowski said.

Deepdub is not the only company looking to use AI for dubbing and voice localization. Toronto-based Resemble has built tech to clone your voice in different languages, which is already being used for chatbots and automated call centers. London-based Synthesia also targets Hollywood, albeit with a bit of a different approach: It changes not only the audio track, but actually uses deepfake image manipulation to match an actor's mouth movement to the new voice. Krakowski pointed out that such an approach is still challenged by different camera angles as well as onscreen motion, but he acknowledged that the two technologies may ultimately converge. "It's not a question of if, just of when," he said.

In the meantime, AI dubbing may be able to help Hollywood with some more pressing problems. The pandemic forced streaming services to limit dubbing for a number of markets. For instance, Netflix has been telling viewers in recent months that it had to delay localization for some of its titles. "We're prioritizing voice actors' and dubbing studio safety," the company informed viewers within its app. Artificial intelligence could help to bridge those gaps, even if studios ultimately decide to rely on real voice actors for the final version.

A version of this article will appear in this week's Next Up newsletter. Sign up here to receive it in your inbox every Thursday.

Politics

What tech policy could look like in Biden’s first 100 days

More antitrust laws and bridging the digital divide should be top of mind for the incoming administration.

Antitrust enforcement is one of the big lessons going into the Biden administration.
Photo: Alex Edelman/Getty Images

Although it is too soon to tell with certainty how President-elect Joe Biden will address the questions surrounding tech policy, it is clear that his inaugural transition on Wednesday will affect the world of tech.

Protocol reporters Issie Lapowsky and Emily Birnbaum, virtually met up with panelists Tuesday to discuss what tech policy and regulation could look like in Biden's first 100 days in office — as well as the next four years.

Keep Reading Show less
Penelope Blackwell
Penelope Blackwell is a reporting fellow at Protocol covering ed-tech, where she reports on the decisions leading up toward the advances of remote learning. Previously, she interned at The Baltimore Sun covering emerging news and produced content for Carnegie-Knight’s News21 documenting hate and bias incidents in the U.S. She is also a recent graduate of Columbia University’s Graduate School of Journalism and Morgan State University.
Power

What TV remotes tell us about power struggles in streaming

TV remote controls are a major battlefield in the TV wars, which are fought one branded button at a time.

LG's 2021 smart TV remote control features a total of three buttons for voice control.

Image: LG

Don't touch that dial: As TV manufacturers are unveiling their 2021 models at this year's virtual CES, they're also giving us a first look at the remote controls that will be shipping with those big, shiny and smart TV sets.

There were a few surprises. LG's remotes come with built-in NFC to transfer videos from mobile devices to the TV, and Samsung's remotes incorporate solar cells that are meant to reduce battery waste. The new crop of 2021 TV remotes also perfectly encapsulates the conflicts and power struggles in the TV industry, from streaming services vying for attention to voice assistant platforms' fierce competition.

Keep Reading Show less
Janko Roettgers

Janko Roettgers (@jank0) is a senior reporter at Protocol, reporting on the shifting power dynamics between tech, media, and entertainment, including the impact of new technologies. Previously, Janko was Variety's first-ever technology writer in San Francisco, where he covered big tech and emerging technologies. He has reported for Gigaom, Frankfurter Rundschau, Berliner Zeitung, and ORF, among others. He has written three books on consumer cord-cutting and online music and co-edited an anthology on internet subcultures. He lives with his family in Oakland.

Power

Roku is becoming the most powerful company in streaming

A growing user base will give it even more power in content negotiations.

Roku's emerging as one of the streaming war's biggest winners.

Photo: Luke Sharrett/Getty Images

Roku's bet on smart TVs is paying off: Seven years after the company first began licensing its operating system to TV manufacturers, it has become a market leader in North America. Roku and its hardware partners sold more smart TVs in the U.S. in 2020 than competitors like Samsung, LG and Vizio, according to data from the NPD Group released by Roku on Friday.

Roku TVs had a 38% market share in the U.S. and a 31% market share in Canada, according to NPD's data. Roku also announced earlier this week that it had ended 2020 with 51.2 million active accounts, adding around 14 million accounts over the past 12 months. Altogether, consumers streamed 58.7 billion hours of entertainment through their Roku devices in 2020, according to a news release.

Keep Reading Show less
Janko Roettgers

Janko Roettgers (@jank0) is a senior reporter at Protocol, reporting on the shifting power dynamics between tech, media, and entertainment, including the impact of new technologies. Previously, Janko was Variety's first-ever technology writer in San Francisco, where he covered big tech and emerging technologies. He has reported for Gigaom, Frankfurter Rundschau, Berliner Zeitung, and ORF, among others. He has written three books on consumer cord-cutting and online music and co-edited an anthology on internet subcultures. He lives with his family in Oakland.

Protocol | Enterprise

Databricks plans to take on Snowflake and Google and score a huge IPO

Even against intensifying competition, Databricks hopes to be a hit when it heads to the public markets this year.

Ali Ghodsi is the CEO of Databricks.

Photo: Databricks

Enterprise software had a huge 2020 on Wall Street as companies such as Snowflake and C3.ai went public with blockbuster initial offerings. Databricks CEO Ali Ghodsi is hoping to ride the same wave in 2021.

The public debut of the data analytics startup, valued at $6.2 billion, is among the most-watched IPOs for the year. And for good reason: It competes in a similar space as the much-hyped Snowflake, helping customers find the data to power the algorithms that help with everything from picking which products to order to which candidates to bring in for job interviews. While Databricks has been tight-lipped on its specific plans, including which bankers it is tapping to help navigate the often arduous process, it is taking steps internally to prepare.

Keep Reading Show less
Joe Williams

Joe Williams is a senior reporter at Protocol covering enterprise software, including industry giants like Salesforce, Microsoft, IBM and Oracle. He previously covered emerging technology for Business Insider. Joe can be reached at JWilliams@Protocol.com. To share information confidentially, he can also be contacted on a non-work device via Signal (+1-309-265-6120) or JPW53189@protonmail.com.

Protocol | Enterprise

How Salesforce, despite big setbacks, had a banner 2020

Amid the chaos of major layoffs and top executive departures, Salesforce announced a key acquisition and managed to report blockbuster earnings.

Marc Benioff is the CEO of Salesforce.

Photo: Kimberly White/Getty Images

On Aug. 27, Salesforce announced it would lay off around 1,000 employees.

The news came as a shock to many. At the beginning of the pandemic, CEO Marc Benioff committed to making no "significant" layoffs for 90 days. (The 1,000 job losses occurred 155 days after that pledge was made.) But any blowback to the announcement appears to have been brushed aside by some of the company's top leaders.

Keep Reading Show less
Joe Williams

Joe Williams is a senior reporter at Protocol covering enterprise software, including industry giants like Salesforce, Microsoft, IBM and Oracle. He previously covered emerging technology for Business Insider. Joe can be reached at JWilliams@Protocol.com. To share information confidentially, he can also be contacted on a non-work device via Signal (+1-309-265-6120) or JPW53189@protonmail.com.

Latest Stories