The Big Story

Wearables for dropping beats, not pounds

Forget step counting and heart tracking: Swiss consumer hardware startup Mictic has developed a wearable device to make music and play with 3D soundscapes. Add AR or VR to the mix, and Mictic may just be a key to unlocking the immersive and interactive audio experiences in the future.

At its core, Mictic is a wristband that pairs with a phone. It packs a bunch of sensors, including an accelerometer, gyroscope and magnetometer, which are used to trigger sounds with less than 20 milliseconds latency. Users strap one band on each wrist, with an option to add a pair to their ankles. Then it's time to move and make the magic happen.

• Mictic CEO Mershad Javan demoed the product over Zoom for me this week, which included him playing air guitar and virtual cello. He also triggered sounds of a hip-hop track, launching beats and scratches by moving his hands. To get a sense of how it works, check out this video.
• Javan was quick to point out that this is not the same as actually playing a cello. "It's not meant to replace an instrument right now," he told me. "It's definitely a consumer entertainment product."
• After starting with single instruments, Mictic added more complex soundscapes to the product (think drum and bass or hip-hop tracks). The team even developed a few audio games, letting users trigger lightsaber sounds and more.
• Mictic also plans to let consumers edit and share their own sounds and 3D soundscape maps.

Mictic initially planned to ship the product in time for the upcoming holidays, but COVID-19 forced the company to adjust its timeline. Now, the startup expects the bands to be commercially available by next June, with a target price of around $100 per pair. Mictic is looking to open an office in Los Angeles in the coming weeks to strike up partnerships with musicians and influencers.

Mictic looks fun, but wait for the future. Audio will be key to the first generation of consumer AR glasses, and music games are already hugely popular on VR headsets. Adding wrist-worn sensors to the mix could help unlock a whole new range of audio-visual experiences. Imagine playing with AR audio objects that are being visualized in real-time in your living room.

• Mictic isn't the only company experimenting with new music toys. Another startup I've been tracking, ODD, has developed a kind of musical bouncy ball.
• Mictic CTO Andreas Gautschi said that a lot of this new creativity has to do with Arduinos and other low-cost computing platforms being readily available to tinkerers everywhere. "They are enabling the general public to make prototypes," he said.
• And some of these fun little inventions may ultimately turn into the next big thing. Javan said that the startup already got approached by a major watchmaker looking to license its technology.

Overheard

"Twitter acquired Vine for $30 million (TikTok currently valued at $180 billion) and Periscope for $75 million (Twitch acquired for $1 bn, probably worth $10+ [bn] now). Talk about a clown car driving into a gold mine." —Twitter user Quantian reacting to the news that Twitter is shutting down its livestreaming app Periscope.

"Forgot Periscope was still a separate product." —LightShed analyst Rich Greenfield reacting to the same story.

A MESSAGE FROM AMAZON FIRE TV

The lockdowns this year have transformed our homes into offices, schools, concert halls, movie theaters and gyms. Our homes are working harder for us, but so is our technology. The device that is working the hardest is perhaps the TV—becoming our lifeline to a far more virtual world.

Learn more.

Watch Out

AI is coming to dubbing

Tel Aviv-based startup Deepdub wants to help streaming services gain new subscribers around the globe with the help of artificial intelligence. Deepdub, which came out of stealth on Wednesday, has built technology that can translate a voice track to a different language, all while staying true to the voice of the talent. This makes it possible to have someone like Morgan Freeman narrate a movie in French, Italian or Russian without losing what makes Freeman's voice special and recognizable.

Deepdub has been in conversations with a number of potential Hollywood clients, according to Deepdub CMO Oz Krakowski. "People's jaws are dropping to the floor," he said, adding that industry insiders especially value the ability to adjust the voice of an AI-generated dubbing track. "We call it voice design," he explained.

Deepdub has published an impressive demo video on YouTube. Krakowski recently gave me some more context on how the company's technology works:

• Deepdub currently can dub videos into six different languages, including French, German and Spanish. The company aims to double that number in the next few months, and ultimately wants to be able to localize videos into 60 languages.
• The technology is designed to fully automate the dubbing process, but Krakowski admitted that higher-end projects still require a bit of manual intervention for things like local idioms and brands. For instance, the convenience store chain 7-Eleven doesn't exist in many countries, so employees may have to help the system find an adequate replacement.
• However, this kind of intervention only has to happen once, and it benefits all future projects. "The system keeps learning," Krakowski said. "The amount of manual work decreases over time."
• Deepdub can work with a variety of source materials. This includes complete video files with mixed-down audio tracks from which the system is able to remove and replace individual voices. That could especially come in handy for streaming services with a lot of library content. "You have streaming services with thousands of videos that are stuck on a single language," Krakowski said.

Deepdub isn't the only company looking to use AI for dubbing and voice localization.

• Toronto-based Resemble has built tech to clone your voice in different languages, which is already being used for chatbots and automated call centers.
• London-based Synthesia also targets Hollywood, albeit with a bit of a different approach: It changes not only the audio track, but actually uses deepfake image manipulation to match an actor's mouth movement to the new voice. Krakowski pointed out that such an approach is still challenged by different camera angles as well as onscreen motion, but he acknowledged that the two technologies may ultimately converge. "It's not a question of if, just of when," he said.

AI dubbing may help Hollywood with some more pressing problems in the short term. The pandemic forced streaming services to limit dubbing for a number of markets. For instance, Netflix has been telling viewers in recent months that it had to delay localization for some of its titles. "We're prioritizing voice actors' and dubbing studio safety," the company informed viewers within its app. Artificial intelligence could help to bridge those gaps, even if studios ultimately decide to rely on real voice actors for the final version.

A version of this story previously appeared on Protocol.

Fast Forward

• On Protocol: Disney+ reached 86.8 million paying subscribers. The streaming service will raise pricing for U.S. subscribers by $1 in March.
• Also on Protocol: How a tiny startup fixed the future of TV. This story about a streaming services aggregator is (I'm so sorry) real good.
• Roblox acquired avatar startup Loom.ai. The acquisition could help Roblox make much more realistic avatars for its take on the metaverse.
• Fire TV now has 50 million monthly active users, as Amazon continues to sell a ton of streaming sticks.
• Chromecast and Android TV are getting the Apple TV+ app, as Apple continues to spread its software rather than release new TV hardware.
• Spatial launched on mobile, meaning the VR and AR teleconferencing service now works without headsets.
• BBC is launching a new streaming service next year, which will offer documentaries and more to North American audiences.
• One last Protocol story: HBO Max is finally launching on Roku, starting today.

Auf Wiedersehen

I mentioned Lucas Rizzotto's amazing exploits here before, but his latest YouTube video is definitely worth watching as well: To re-create the magic of Harry Potter's Marauder's Map, Rizzotto rented a small castle in Northern England (Tiny Hogwarts, as he called it), mapped the grounds with a drone and developed an AR app capable of overlaying his friends' real-time locations atop a piece of parchment. I love this project for a number of reasons, including that it's using technology that is readily available to everyone. Go check it out, and please follow Rizzotto on YouTube. He's up to no good in the best kinds of ways.

A MESSAGE FROM AMAZON FIRE TV

The lockdowns this year have transformed our homes into offices, schools, concert halls, movie theaters and gyms. Our homes are working harder for us, but so is our technology. The device that is working the hardest is perhaps the TV—becoming our lifeline to a far more virtual world.

Learn more.