Google is rethinking search because TikTok and podcasts are taking over the internet

Multimedia search is a core focus for Google going forward. So is understanding an increasingly multimedia internet.

Phones with different search screens

Google's emphasis on visual search is about the search bar ... but also about the internet.

Photo: Google

One of Google's favorite statistics is that every day, roughly 15% of Google queries are for things that have never before been typed into the search box. And even at Google's impossible scale, the number never seems to go down. "Part of it, I have to admit, is that people find new and creative ways of misspelling words," Pandu Nayak, a Google fellow and the company's VP of search, told me earlier this month. But there are two other reasons, he said: The world changes all the time, and people's curiosity "is quite infinite in its complexity."

Google's challenge on the web is to find ever-better ways to collect and sort information. Crawling web pages is the easy part, relatively speaking. Understanding what's authoritative vaccine guidance and what's dangerous misinformation, or whether you typed "spaghetti" looking for definitions or recipes? That's all much more complicated. Nayak rattled off numbers like the 3,600 changes made to the search system last year, or the 60,000-plus experiments run internally. It's a lot of work, but Google's better at it than most.

But there's a core change happening on the internet that threatens Google in a serious, potentially existential way. An increasingly large amount of the web is not web pages full of text and hyperlinks. It's images, video and audio. TikTok and Instagram, podcasts and videos: Those platforms are just as much "the internet" as the Wikipedias and publisher sites Google has long relied on. And for the company that has spent two decades dedicated to organizing the world's information, that presents a problem.

At Google's Search On event on Wednesday, Google executives showed off some fancy new features, like a camera feature that can take a picture of a shirt and find socks with the same pattern. Or a way to take a photo of your broken bike chain and get search results for how to fix it. It's all part of Google Lens, the visual-first search system the company has been building for several years. Google has long talked about wanting to take search beyond the text box, to make it easier for people to input information and get answers. Context is crucial to that, too.

But just as important, and just as difficult, is understanding the information on the other side. It's technically possible to search TikTok and Instagram through Google, but the results are pretty primitive and mostly based on hashtags and video descriptions. Google is reportedly working on deals with ByteDance and Facebook to bring more content with better metadata into Google's search results, but that, too, is only half the battle.

Even on YouTube — itself the world's second-largest search engine, and obviously a Google-owned company — Google's search relies on metadata and automatically generated transcripts to figure out what's going on in a video. Introducing chapter markers made the system better, but only because creators gave Google hints about where to look. Its search crawlers don't understand what's on the screen in any meaningful way.

When he introduced Google's new Multitask Unified Model system (or MUM, as it's known) at Google I/O in May, Nayak hinted that things might be about to change. "MUM is multimodal," he wrote in a blog post, "so it understands information across text and images and, in the future, can expand to more modalities like video and audio." He echoed the sentiment in our conversation. "You can give [MUM] inputs that are both text and images, as a sequence of tokens," he said. "It only thinks about tokens … and it essentially learns the relationships between image tokens and word tokens, and I think we'll see a number of interesting examples coming out of that." He said that's not coming immediately, but "in the maybe not-too-distant future."

If Google can unlock a truly visual search engine in both directions — visual queries, visual data, visual output — it can be much better equipped to be to the future what, well, Google was to the past. More than two decades ago, the company took a disparate set of content and put it at users' fingertips. Now the content has changed, but the need hasn't.

The other upside for Google? Shopping. Practically every corner of the internet is embracing shopping as a way to make money, both for creators and for the platforms. For Google, the potential is massive: It could allow users to click on any product in any video or image anywhere on the internet, from the gadget in the foreground to the lamp in the background to the shoes on creators' feet, and be taken to a store to buy that thing. MUM could help Google build the world's biggest catalog, with Google as a happy fulfillment and payment service.

Companies around the industry, from Spotify to Pinterest to Apple to practically every other platform and service that deals in audiovisual content, are trying to figure out how to better understand and index the content in their systems. Google, as the trillion-dollar tech giant predicated on understanding and indexing all content everywhere, is in a high-stakes race to do it better.


The Roe decision could change how advertisers use location data

Over the years, the digital ad industry has been resistant to restricting use of location data. But that may be changing.

Over the years, the digital ad industry has been resistant to restrictions on the use of location data. But that may be changing.

Illustration: Christopher T. Fong/Protocol

When the Supreme Court overturned Roe v. Wade on Friday, the likelihood for location data to be used against people suddenly shifted from a mostly hypothetical scenario to a realistic threat. Although location data has a variety of purposes — from helping municipalities assess how people move around cities to giving reliable driving directions — it’s the voracious appetite of digital advertisers for location information that has fueled the creation and growth of a sector selling data showing who visited specific points on the map, when, what places they came from and where they went afterwards.

Over the years, the digital ad industry has been resistant to restrictions on the use of location data. But that may be changing. The overturning of Roe not only puts the wide availability of location data for advertising in the spotlight, it could serve as a turning point compelling the digital ad industry to take action to limit data associated with sensitive places before the government does.

Keep Reading Show less
Kate Kaye

Kate Kaye is an award-winning multimedia reporter digging deep and telling print, digital and audio stories. She covers AI and data for Protocol. Her reporting on AI and tech ethics issues has been published in OneZero, Fast Company, MIT Technology Review, CityLab, Ad Age and Digiday and heard on NPR. Kate is the creator of RedTailMedia.org and is the author of "Campaign '08: A Turning Point for Digital Media," a book about how the 2008 presidential campaigns used digital media and data.

Businesses are evolving, with current events and competition serving as the catalysts for technology adoption. Events from the pandemic to the ongoing war in Ukraine have exposed the fragility of global supply chains. The topic of sustainability is now on every board room agenda. Industries from manufacturing to retail and everything in between are exploring the latest innovations like process automation, machine learning and AI to identify potential safeguards against future disruption. But according to a recent survey from Boston Consulting Group, while 80% of companies are adopting digital solutions to navigate existing business challenges or opportunities like the ones mentioned, only about 30% successfully digitally transform their business.

For the last 50 years, SAP has worked closely with our customers to solve some of the world’s most intricate problems. We have also seen, and have been a part of, rapid accelerations in technology in response. Across industries, certain paths have emerged to help businesses manage the unexpected challenges over the last few years.

Keep Reading Show less
DJ Paoni

DJ Paoni is the President of SAP North America and is responsible for the strategy, day-to-day operations, and overall customer success in the United States and Canada. Dedicated to helping customers become best-run businesses, DJ has established himself as a trusted advisor who places a high priority on their success. He works with many of SAP North America's 155,000 customers and helps them adopt business and technology best practices across 25 different industries.


Russian cyberattacks against the US may still be coming, experts say

In response to strong sanctions and military aid to Ukraine, Russia was expected to launch disruptive cyberattacks against the West but never did. But a cyberescalation from Russia still remains possible, as soon as later this year, according to experts.

"I fear this is a 'calm before the storm' situation," said Chester Wisniewski, principal research scientist at Sophos.

Illustration: Nanzeeba Ibnat/iStock/Getty Images Plus

In the four months since its invasion of Ukraine, Russia hasn't intensified its usual pattern of cyberattacks against the U.S. and Western Europe in response to sanctions and Ukrainian military aid, as many expected. But that doesn't mean the risk of escalation with the West is gone, numerous experts told Protocol.

In other words, don't lower your shields just yet.

Keep Reading Show less
Kyle Alspach

Kyle Alspach ( @KyleAlspach) is a senior reporter at Protocol, focused on cybersecurity. He has covered the tech industry since 2010 for outlets including VentureBeat, CRN and the Boston Globe. He lives in Portland, Oregon, and can be reached at kalspach@protocol.com.


Affirm CEO: 'Buy now, pay later' becomes more attractive in a slump

With consumers grappling with rising rates and prices, the question of whether they’ll still buy now and pay later is open. Max Levchin thinks Affirm knows the answer.

Affirm CEO Max Levchin spoke with Protocol about "buy now, pay later."

Photo: John Lamparski/Getty Images

Shortly after Affirm went public last year, CEO Max Levchin told Protocol that he saw “an ocean of opportunities” for the “buy now, pay later” pioneer. Wall Street agreed.

Affirm’s stock soared in its trading debut as the company blazed a trail for a fast-growing alternative to the credit cards that Levchin says consumers are increasingly rejecting.

Keep Reading Show less
Benjamin Pimentel

Benjamin Pimentel ( @benpimentel) covers crypto and fintech from San Francisco. He has reported on many of the biggest tech stories over the past 20 years for the San Francisco Chronicle, Dow Jones MarketWatch and Business Insider, from the dot-com crash, the rise of cloud computing, social networking and AI to the impact of the Great Recession and the COVID crisis on Silicon Valley and beyond. He can be reached at bpimentel@protocol.com or via Google Voice at (925) 307-9342.


The post-layoff playbook: How to avoid 'survivor's guilt'

Taking care of your laid-off employees is important. But how can you restore trust with the employees who make it through?

Employees who survive layoffs are charged with holding the company together. Whether or not managers listen to their concerns can make or break a company’s culture.

Photo: Justin Pumfrey/The Image Bank/Getty Images

Jennifer Burke was on her way to Hawaii for her daughter’s wedding when Zillow followed through on its long-anticipated layoff. She asked her manager to break the news to her by message in the car. You’re one of the safe ones, her manager responded.

“I felt relieved, of course,” Burke said. “I felt apprehensive. I felt sympathy for my co-workers that I knew were going to be laid off.”

Keep Reading Show less
Lizzy Lawrence

Lizzy Lawrence ( @LizzyLaw_) is a reporter at Protocol, covering tools and productivity in the workplace. She's a recent graduate of the University of Michigan, where she studied sociology and international studies. She served as editor in chief of The Michigan Daily, her school's independent newspaper. She's based in D.C., and can be reached at llawrence@protocol.com.

Latest Stories