March 18, 2021
Good morning, and welcome to Protocol Next Up. This week's edition is about Facebook's work on a wristband to control your AR glasses and Amazon's patent of AI voice-dubbing technology.
And a quick reminder: Our live event on the state of VR in 2021 is happening today! Tune in at 10 a.m. PT for insights from Baobab Studios CEO Maureen Fan, HP global head of virtual reality for location-based entertainment Joanna Popper and Survios co-founder and president Nathan Burba.
(Was this email forwarded to you? Sign up here to get Next Up every week.)
The future of computing may be on your wrist: Facebook is working on wristbands capable of measuring electrical signals sent from the brain to the hand to control its future AR glasses products.
Researchers from the company's AR/VR arm, also known as the Facebook Reality Labs, stressed this week that it may take years before any of it makes it into a finished product — time that they want to use to get feedback on this work. "We want to be transparent about what we are working on," said Sean Keller, FRL's research science director. "We want to open up an important discussion with the public about how to build these technologies responsibly."
There simply isn't a good existing interaction model for AR glasses. Hand tracking would technically work, but it's not something you may want to do on the subway. "That's a little weird if I am in public," Facebook's VP of consumer hardware, Andrew Bosworth, told Protocol. The same is true for voice commands. "There's lots of places that I want to use my face computer that I don't want to use my voice," he said.
Another reason has to do with the ability to tap into a lot of data. "You have more neurons of your brain dedicated to controlling your wrist than any other parts of your body," said FRL's neuromotor interfaces director, Thomas Reardon.
"What we're talking about right now is a research device," Bosworth said. "The eventual consumer product that you want to have is a wristband that is comfortable for all-day wear, that hopefully doesn't look too different from the types of watches that people are already comfortable with."
That may still be a few years down the road, said Facebook CTO Mike Schroepfer. "We don't have a timetable today."
"It's moving fast, that's the good news," Reardon said.
Building a wristband is only one part of the puzzle in Facebook's quest to reinvent interaction for AR glasses. The other is a whole new type of AR assistant. Read more about that in my interview with Andrew Bosworth.
"The irrational spending by most streaming services is 🤯. This is not sustainable. But does the street understand that?" —Tubi CEO Farhad Massoudi, tweeting about the billions of dollars newcomers like Peacock and Discovery+ are spending on content.
"If you want to really compete with the depth and breadth of content that it takes to compete on a global platform in a global sense, yeah, you need more consolidation." —Former Disney exec Kevin Mayer, discussing whether media companies like Discovery, WarnerMedia and ViacomCBS can be successful on their own in a streaming-first world.
Section 230 of the Communications Decency Act is the most-discussed and least-understood law governing the modern internet. This event will delve into the future of Section 230 and how to change the law without compromising the internet as we know it. Join Protocol's Emily Birnbaum and Issie Lapowsky in conversation with Senator Mark Warner. This event is presented by Internet Association.
Amazon Prime Video content could soon be available in many more languages, if a recently published patent is any indication. Granted last month, the patent titled "Automatic voice dubbing for media content localization" outlines ways to use AI for automatic dubbing, complete with computer-generated voices that sound like those of the original actors.
Dubbing is important for companies like Amazon, as the patent itself mentions: "Having video content with localized voices, rather than only having localized text subtitles, significantly impacts the global reach of streaming video service providers."
But it's also expensive and can slow down localization efforts considerably. "Content studios spend significant amounts of time and money every year to generate localized voices for video media content," the patent notes. "Dubbing is an especially challenging step in the media content localization process for the movie industry, since the process of dialogue translation relies on experts in the targeted localization language."
Neural networks could help automate this process and produce dubbing tracks that aren't just reflective of the local language, but actually sound like the original talent. Amazon envisions using other movies starring the same actors as training material to get the sound right, and may use multiple deep learning techniques in concert to fine-tune the results.
There is some precedent for this approach:
The patent awarded in February makes a lot more specific references to the world of streaming, and predicts that this technology could result in the dubbing of a movie taking "hours, rather than weeks/months."
Now for the caveats: Companies regularly patent all kinds of things, and many of those patents don't result in actual products. And even if Amazon did decide to productize AI dubbing, it's not clear that Amazon Prime Video would be the primary use case. It's equally possible that the company would make the technology available to media companies that already use Amazon's cloud services. That would also allow Amazon to test and refine the technology before it is being used on its own crown jewels.
Then again, the patent does prominently mention the Tom Cruise movie "The Last Samurai" as an example, explaining how AI could make the Japanese dub sound like it was spoken by Cruise himself. "The Last Samurai" happens to be a movie that's actually available for rent on Amazon Prime Video — albeit, at least for now, with traditional dubbing.
Google's Arts & Culture group has released a project called AR Synth that lets you make music with classic synthesizers and drum machines in augmented reality. It's fun, it's free and it finally gives me a chance to pretend I'm a member of Kraftwerk. I got the accent down already; now all I need is a red button-down shirt.
Thanks for reading — see you next week!