next-upnext upauthorJanko RoettgersNoneDo you know what's coming next up in the world of tech and entertainment? Get Janko Roettgers' newsletter every Thursday.9147dfd6b1
×

Get access to Protocol

Will be used in accordance with our Privacy Policy

I’m already a subscriber
People

Adobe wants to use AI to make you a better dancer

The company's new tech demo could improve everyone's TikTok videos — and hints at the potential for AI to democratize video editing and visual effects.

Adobe wants to use AI to make you a better dancer

The technology can also be used to take clips and adjust movements to a different song, or take multiple clips from different sources and adjust all of them to the piece of music.

Image: Adobe

Can't dance? You're not alone.

"Syncing up music and dancing can be hard," said Adobe Research Scientist Jimei Yang during a recent interview. Not only can holding the beat be challenging for some people, but using consumer-grade recording equipment can also introduce additional delays that make the result look off-beat. "It isn't that trivial," Yang said.

That's why Yang's team at Adobe developed technology that can tweak an existing video and make dance moves match the rhythm of the music. The researchers are set to demonstrate their work, dubbed "Project On The Beat," at Adobe's annual Max conference Wednesday, but Yang gave Protocol an exclusive preview last week.

This is how Adobe's dance improvement algorithms work in a nutshell:

  • First, the project uses Adobe's Sensei AI tech for skeletal tracking to identify the motions of the dancer.
  • Then, it breaks those motions down to find the strongest movements — think claps and stomps — which Adobe's researchers call "motion beats."
  • The video's music is segmented based on these key poses, and the motion beats are compared to the actual beats of the song.
  • Finally, the video is adjusted segment by segment to match the motion beats to the music. "We just need to retime the video, basically warp the video," Yang said.

The technology can also be used to take clips and adjust movements to a different song, or take multiple clips from different sources and adjust all of them to the piece of music. This even works with videos that only show a person's upper body.

Like much of the technology shown during these Adobe Max research demos, Yang's work isn't part of any Adobe product yet. However, that could change. "We are actively working with the product team," Yang said. One could imagine it as an addition to a future version of Adobe Premiere Rush, the company's mobile video-editing app for influencers and other social media users.

Beyond any such specific use cases, the demo shows the potential for AI to automate and ultimately democratize video editing. Until now, video editors would have to manually segment and adjust each part of a video to synchronize music and movement with desktop video-editing software. By letting AI do the work, the same can be done faster, and on mobile devices, literally putting this kind of video editing into many more hands. "AI is really changing the video editing business, especially for amateurs," Yang said.

A version of this story will appear in our entertainment newsletter, Next Up. Sign up here to get it in your inbox every Thursday.

Protocol | Fintech

Plaid’s COO is riding fintech’s choppy waves

He's a striking presence on the beach. If he navigates Plaid's data challenges, Eric Sager will loom large in the financial world as well.

Plaid COO Eric Sager is an avid surfer.

Photo: Plaid

Eric Sager is an avid surfer. It's a fitting passion for the No. 2 executive at Plaid, a startup that's riding fintech's rough waters — including a rogue wave on the horizon that could cause a wipeout.

As Plaid's chief operating officer, Sager has been helping the startup navigate that choppiness, from an abandoned merger with Visa to a harsh critique by the CEO of a top Wall Street bank.

Keep Reading Show less
Benjamin Pimentel

Benjamin Pimentel ( @benpimentel) covers fintech from San Francisco. He has reported on many of the biggest tech stories over the past 20 years for the San Francisco Chronicle, Dow Jones MarketWatch and Business Insider, from the dot-com crash, the rise of cloud computing, social networking and AI to the impact of the Great Recession and the COVID crisis on Silicon Valley and beyond. He can be reached at bpimentel@protocol.com or via Signal at (510)731-8429.

Sponsored Content

The future of computing at the edge: an interview with Intel’s Tom Lantzsch

An interview with Tom Lantzsch, SVP and GM, Internet of Things Group at Intel

An interview with Tom Lantzsch

Senior Vice President and General Manager of the Internet of Things Group (IoT) at Intel Corporation

Edge computing had been on the rise in the last 18 months – and accelerated amid the need for new applications to solve challenges created by the Covid-19 pandemic. Tom Lantzsch, Senior Vice President and General Manager of the Internet of Things Group (IoT) at Intel Corp., thinks there are more innovations to come – and wants technology leaders to think equally about data and the algorithms as critical differentiators.

In his role at Intel, Lantzsch leads the worldwide group of solutions architects across IoT market segments, including retail, banking, hospitality, education, industrial, transportation, smart cities and healthcare. And he's seen first-hand how artificial intelligence run at the edge can have a big impact on customers' success.

Protocol sat down with Lantzsch to talk about the challenges faced by companies seeking to move from the cloud to the edge; some of the surprising ways that Intel has found to help customers and the next big breakthrough in this space.

What are the biggest trends you are seeing with edge computing and IoT?

A few years ago, there was a notion that the edge was going to be a simplistic model, where we were going to have everything connected up into the cloud and all the compute was going to happen in the cloud. At Intel, we had a bit of a contrarian view. We thought much of the interesting compute was going to happen closer to where data was created. And we believed, at that time, that camera technology was going to be the driving force – that just the sheer amount of content that was created would be overwhelming to ship to the cloud – so we'd have to do compute at the edge. A few years later – that hypothesis is in action and we're seeing edge compute happen in a big way.

Keep Reading Show less
Saul Hudson
Saul Hudson has a deep knowledge of creating brand voice identity, especially in understanding and targeting messages in cutting-edge technologies. He enjoys commissioning, editing, writing, and business development, in helping companies to build passionate audiences and accelerate their growth. Hudson has reported from more than 30 countries, from war zones to boardrooms to presidential palaces. He has led multinational, multi-lingual teams and managed operations for hundreds of journalists. Hudson is a Managing Partner at Angle42, a strategic communications consultancy.
People

Google vows to do better on DEI and firings. Timnit Gebru is not impressed.

Google AI lead Jeff Dean said Google had concluded its investigation into Timnit Gebru's dismissal in an email to employees Friday.

Google has ended its investigation into the dismissal of prominent AI ethicist Timnit Gebru.

Photo: John Nacion/Getty Images

Google has concluded its investigation into the firing of prominent AI ethics researcher Timnit Gebru, and it announced some changes to its hiring, firing and research policies in an email from AI leader Jeff Dean to employees Friday.

While Dean did not share the results of the investigation into the circumstances surrounding Gebru's dismissal, he said that the company would enact new policies to "review employee exits that are sensitive in nature." His email, which was obtained by Protocol, said the company will also begin linking performance reviews for vice presidents and above, in part regarding diversity and inclusion goals, and it will report DEI goals and progress to the Alphabet board of directors in quarterly reviews.

Keep Reading Show less
Anna Kramer

Anna Kramer is a reporter at Protocol (@ anna_c_kramer), where she helps write and produce Source Code, Protocol's daily newsletter. Prior to joining the team, she covered tech and small business for the San Francisco Chronicle and privacy for Bloomberg Law. She is a recent graduate of Brown University, where she studied International Relations and Arabic and wrote her senior thesis about surveillance tools and technological development in the Middle East.

Transforming 2021

Blockchain, QR codes and your phone: the race to build vaccine passports

Digital verification systems could give people the freedom to work and travel. Here's how they could actually happen.

One day, you might not need to carry that physical passport around, either.

Photo: CommonPass

There will come a time, hopefully in the near future, when you'll feel comfortable getting on a plane again. You might even stop at the lounge at the airport, head to the regional office when you land and maybe even see a concert that evening. This seemingly distant reality will depend upon vaccine rollouts continuing on schedule, an open-sourced digital verification system and, amazingly, the blockchain.

Several countries around the world have begun to prepare for what comes after vaccinations. Swaths of the population will be vaccinated before others, but that hasn't stopped industries decimated by the pandemic from pioneering ways to get some people back to work and play. One of the most promising efforts is the idea of a "vaccine passport," which would allow individuals to show proof that they've been vaccinated against COVID-19 in a way that could be verified by businesses to allow them to travel, work or relax in public without a great fear of spreading the virus.

Keep Reading Show less
Mike Murphy

Mike Murphy ( @mcwm) is the director of special projects at Protocol, focusing on the industries being rapidly upended by technology and the companies disrupting incumbents. Previously, Mike was the technology editor at Quartz, where he frequently wrote on robotics, artificial intelligence, and consumer electronics.

People

20 years of orange cones: The history of VLC

How a student project became one of the world's most popular open-source apps, powering much of modern-day media, without ever creating a huge windfall for its developers.

VLC is 20 years old and has been downloaded more than 3.5 billion times.

Photo: Getty Images

When students at the École Centrale Paris lobbied for a campus network upgrade in the '90s, they weren't really thinking about the future of media. All they wanted was to play Duke Nukem 3D. But the public-private partnership that made the first-person shooter work on their campus network also laid the groundwork for the birth of the popular media player VLC.

Open sourced 20 years ago this month, VLC has since been downloaded more than 3.5 billion times, making it one of the most popular free software projects to date. Software developed for VLC is being used to power some of the world's largest streaming services. Despite all of this, VLC has largely remained a labor of love, with developers saying no to deals worth tens of millions of dollars.

Keep Reading Show less
Janko Roettgers

Janko Roettgers (@jank0) is a senior reporter at Protocol, reporting on the shifting power dynamics between tech, media, and entertainment, including the impact of new technologies. Previously, Janko was Variety's first-ever technology writer in San Francisco, where he covered big tech and emerging technologies. He has reported for Gigaom, Frankfurter Rundschau, Berliner Zeitung, and ORF, among others. He has written three books on consumer cord-cutting and online music and co-edited an anthology on internet subcultures. He lives with his family in Oakland.

Latest Stories