AI dubbing is coming for feature films

The Big Story

This Netflix-licensed movie is getting dubbed by AI

Israel-based AI dubbing startup Deepdub is going longform: The company has struck a deal with New York's MiLa Media to localize the studio's feature film "Every Time I Die," which currently streams in English on Netflix, for the Americas. Deepdub will use its AI voice synthesis technology to dub the film in Latin American Spanish and Brazilian Portuguese, and the resulting versions will feature voices that sound just like those of the original actors.

"This means that we need to accurately capture the voice style and attributes of the original actors' voice and properly adapt them and modify as needed to match the target language," explained Deepdub CMO Oz Krakowski. "In this case, Latin American Spanish and Brazilian Portuguese require different adaptations."

• Deepdub came out of stealth in December, and has been offering this kind of AI dubbing for commercials and other projects into six target languages.
• The feature film project still requires a fair amount of hand holding, with Krakowski estimating that the process may take up to eight weeks. "We incorporated advanced manual and automatic quality check procedures," he told me via email. "The manual process includes a dubbing director and a language expert that go over the final result. Their comments are fed back into the machine in order to improve the results in the current project and improve the automatic process in future projects."
• Even with that level of involvement, AI is still a lot cheaper than manual dubbing, which can help level the playing field. "This tool gives indie producers the chance to compete, and even beat, big studios in reaching a wider audience," said MiLa Media CEO Ohad Ashkenazi.

Ashkenazi told me that he plans to approach Netflix about distributing the dubbed versions of "Every Time I Die" as well. "At the same time, with more cinemas opening towards the summer, MiLa's distribution team is negotiating with several big distributors in the Latin American market," he said.

This is the first time Deepdub has tackled an entire feature film, and in fact may be a first for the entire industry. And even if Netflix decides to pass on the dubbed versions, it's just a matter of time until you get to see an AI-dubbed version of a film or TV show on your favorite streaming service. If the technology works as intended, you won't even notice a difference.

Deepdub is looking to tackle additional projects of this size in the coming months, while also growing its geographic reach; Krakowski told me the company is looking to add additional target languages in the second half of the year.

Overheard

• "While a rising tide will lift boats, it's going to lift the best boats, not necessarily all the boats in the market." —Pluto TV CEO Tom Ryan, throwing shade at the growing number of ad-supported video services during a FAST TV Summit fireside chat.
• "My new Oculus Avatar looks like someone who files a lot of complaints with his neighborhood association." —Software developer Joe Simpson's response to Facebook's new VR avatars.

A MESSAGE FROM INTEL AND MICROSOFT AZURE

Financial fraud isn't waning, and as it increases, security tools designed to protect the data in use need to get stronger to combat more complex fraud. Confidential computing is going to play a big role in the future of financial services.

Learn more

Watch Out

Netflix launches 'Play Something' button worldwide

To help consumers overwhelmed with choice, Netflix is adding a "Play Something" button to its TV interface this week. Pressing it automatically launches a new show or movie based on the service's existing personalized recommendations. And if it's not the right title for the moment, consumers can click to play something else.

Netflix began testing the "Play Something" button last summer. These trials included both A/B tests with millions of consumers around the globe as well as more traditional focus groups, according to Netflix Product Innovation Director Cameron Johnson. The result is a button that is featured prominently in three different places: the profile splash page, the sidebar menu and on the home screen itself, where it is strategically placed in the 10th row of content, as a kind of hint for people who just keep scrolling down. "That is an indication that you are struggling to choose," Johnson said.

• In early tests, Netflix was using the button to launch the last show someone had been binging on. Ultimately, Johnson's team decided to focus on new content instead. "It will always play a title that you've never watched before as the first title," he said. Aside from skipping known shows for the first result, Netflix's new button relies on the same algorithms that are also being used to curate a subscriber's home page.
• Johnson painted the new button as a solution to those moments when you don't want to have to choose, but it could also be seen as an answer to a challenge Netflix has been dealing with ever since it began emphasizing its own shows over licensed content: Without access to known franchises from studios like Disney, Netflix is increasingly creating its own cinematic universes, often based on previously unknown stories and characters.
• The company arguably has done this reasonably well, but getting people to take a chance on these new franchises is always challenging — even more so if the shows are from around the world, featuring actors unknown anywhere but in their home countries. A "Play Something" button can help introduce these new stories to global audiences by nudging viewers to take a chance.
• With a more leanback-friendly discovery mode, Netflix can also appeal to recent cord cutters who are looking to replace more passive cable viewing. Part of the new "Play Something" feature is an option to switch to "something else" by pressing just one button, which Johnson likened to channel surfing.

However, the company decided against a true channel-based, lean-back experience, which has been adopted by streaming services like Pluto TV and Samsung TV+ to offer cable defectors a familiar interface. "We didn't want to replicate internet TV," Johnson said. "We want to invent the future."

Netflix is now looking at ways to bring "Play Something" to mobile, where the company plans to test the new feature within the first half of this year. For that implementation, Netflix may look to social apps like Instagram and TikTok for inspiration, with Johnson praising their take on video as successful watch-first experiences. "That is something that we should all aspire to," he said.

A version of this story first appeared on Protocol.com.

Fast Forward

• Spotify is undercutting Apple for paid podcasts. The streaming service wants to win over podcasters by taking a much smaller cut.
• On Protocol: Cloning Bitmoji stickers helped Facebook build better VR avatars. Facebook is starting to make these avatars available in a small number of VR apps.
• More Protocol: Roku is accusing Google of anticompetitive behavior, and one of the points of contention is support of the AV1 video codec.
• SiriusXM is buying "99% Invisible." The satellite radio giant wants to double down on original podcasts.
• Another Protocol story: As streaming surges globally, Roku is falling behind abroad. We got some exclusive Conviva data on Roku usage around the globe.
• TV sales during pandemic broke records. And guess what? Virtually all of those TVs come with streaming apps preinstalled.
• Campfire launches AR headset with Meta roots. A new AR startup with a headset that looks vaguely familiar, pitched to an enterprise crowd.
• An in-depth review of Spotify's Car Thing. Spotify's first hardware device turns out to be a bit of a one-trick pony.
• CES 2022 is happening. But with most of the world not yet vaccinated, will anyone actually want to go?

Auf Wiedersehen

We were promised giant holograms: Hollywood's vision of our collective future often disappoints, especially if you are among the people building that very future. That's especially true for AR interfaces, which tend to be completely overwhelming and borderline useless. For instance, AR developers have long criticized Marvel's "Iron Man" as an example for bad tech, and rightly so. I mean, seriously: Can you imagine getting bombarded with schematics and progress bars and frantically blinking lights while trying not to crash-land your flying suit?

However, Niantic UX designer Alexandria Heston took to Twitter this week to show the things that "Iron Man" got right about AR interfaces, from gamification to multi-modal interaction. The whole thread, including some thoughtful responses, is a great read. And now I want to watch "Iron Man" again. For science!