Google is rethinking search because TikTok and podcasts are taking over the internet

Multimedia search is a core focus for Google going forward. So is understanding an increasingly multimedia internet.

Phones with different search screens

Google's emphasis on visual search is about the search bar ... but also about the internet.

Photo: Google

One of Google's favorite statistics is that every day, roughly 15% of Google queries are for things that have never before been typed into the search box. And even at Google's impossible scale, the number never seems to go down. "Part of it, I have to admit, is that people find new and creative ways of misspelling words," Pandu Nayak, a Google fellow and the company's VP of search, told me earlier this month. But there are two other reasons, he said: The world changes all the time, and people's curiosity "is quite infinite in its complexity."

Google's challenge on the web is to find ever-better ways to collect and sort information. Crawling web pages is the easy part, relatively speaking. Understanding what's authoritative vaccine guidance and what's dangerous misinformation, or whether you typed "spaghetti" looking for definitions or recipes? That's all much more complicated. Nayak rattled off numbers like the 3,600 changes made to the search system last year, or the 60,000-plus experiments run internally. It's a lot of work, but Google's better at it than most.

But there's a core change happening on the internet that threatens Google in a serious, potentially existential way. An increasingly large amount of the web is not web pages full of text and hyperlinks. It's images, video and audio. TikTok and Instagram, podcasts and videos: Those platforms are just as much "the internet" as the Wikipedias and publisher sites Google has long relied on. And for the company that has spent two decades dedicated to organizing the world's information, that presents a problem.

At Google's Search On event on Wednesday, Google executives showed off some fancy new features, like a camera feature that can take a picture of a shirt and find socks with the same pattern. Or a way to take a photo of your broken bike chain and get search results for how to fix it. It's all part of Google Lens, the visual-first search system the company has been building for several years. Google has long talked about wanting to take search beyond the text box, to make it easier for people to input information and get answers. Context is crucial to that, too.

But just as important, and just as difficult, is understanding the information on the other side. It's technically possible to search TikTok and Instagram through Google, but the results are pretty primitive and mostly based on hashtags and video descriptions. Google is reportedly working on deals with ByteDance and Facebook to bring more content with better metadata into Google's search results, but that, too, is only half the battle.

Even on YouTube — itself the world's second-largest search engine, and obviously a Google-owned company — Google's search relies on metadata and automatically generated transcripts to figure out what's going on in a video. Introducing chapter markers made the system better, but only because creators gave Google hints about where to look. Its search crawlers don't understand what's on the screen in any meaningful way.

When he introduced Google's new Multitask Unified Model system (or MUM, as it's known) at Google I/O in May, Nayak hinted that things might be about to change. "MUM is multimodal," he wrote in a blog post, "so it understands information across text and images and, in the future, can expand to more modalities like video and audio." He echoed the sentiment in our conversation. "You can give [MUM] inputs that are both text and images, as a sequence of tokens," he said. "It only thinks about tokens … and it essentially learns the relationships between image tokens and word tokens, and I think we'll see a number of interesting examples coming out of that." He said that's not coming immediately, but "in the maybe not-too-distant future."

If Google can unlock a truly visual search engine in both directions — visual queries, visual data, visual output — it can be much better equipped to be to the future what, well, Google was to the past. More than two decades ago, the company took a disparate set of content and put it at users' fingertips. Now the content has changed, but the need hasn't.

The other upside for Google? Shopping. Practically every corner of the internet is embracing shopping as a way to make money, both for creators and for the platforms. For Google, the potential is massive: It could allow users to click on any product in any video or image anywhere on the internet, from the gadget in the foreground to the lamp in the background to the shoes on creators' feet, and be taken to a store to buy that thing. MUM could help Google build the world's biggest catalog, with Google as a happy fulfillment and payment service.

Companies around the industry, from Spotify to Pinterest to Apple to practically every other platform and service that deals in audiovisual content, are trying to figure out how to better understand and index the content in their systems. Google, as the trillion-dollar tech giant predicated on understanding and indexing all content everywhere, is in a high-stakes race to do it better.

Theranos trial reveals DeVos family invested $100 million

The family committed "on the spot" to double its investment, an investment adviser said. Meanwhile, the jury lost another two members, with two alternates left.

Betsy DeVos' family invested $100 million in Theranos, an investment adviser said.

Photo: Alex Wong/Getty Images

Lisa Peterson, a wealth manager for the DeVos family, testified in Elizabeth Holmes's criminal fraud trial Tuesday, as prosecutors continued to highlight allegations about how the Theranos CEO courted investors in the once-high-flying blood-testing startup.

An email presented by the defense revealed that the family committed to doubling their investment in Theranos to $100 million "on the spot" during a 2014 visit to company headquarters.

Keep Reading Show less
Michelle Ma
Michelle Ma (@himichellema) is a reporter at Protocol, where she writes about management, leadership and workplace issues in tech. Previously, she was a news editor of live journalism and special coverage for The Wall Street Journal. Prior to that, she worked as a staff writer at Wirecutter. She can be reached at mma@protocol.com.

If you've ever tried to pick up a new fitness routine like running, chances are you may have fallen into the "motivation vs. habit" trap once or twice. You go for a run when the sun is shining, only to quickly fall off the wagon when the weather turns sour.

Similarly, for many businesses, 2020 acted as the storm cloud that disrupted their plans for innovation. With leaders busy grappling with the pandemic, innovation frequently got pushed to the backburner. In fact, according to McKinsey, the majority of organizations shifted their focus mainly to maintaining business continuity throughout the pandemic.

Keep Reading Show less
Gaurav Kataria
Group Product Manager, Trello at Atlassian
Protocol | Enterprise

Google Cloud helped design Intel’s newest data center chip

Mount Evans is Intel's first IPU data center chip, and Google Cloud, which played a role in its development, will be the first customer.

Intel CEO Pat Gelsinger has a new data center chip.

Photo: Pau Barrena/Bloomberg

When Intel announced that it had turned to technology developed by longtime rival Arm for a new infrastructure processing unit called Mount Evans, it said the technology was co-developed by a cloud-service provider that it wouldn't name: until now.

Google Cloud is that design partner, and it has committed to deploying the technology inside its cloud data centers, Intel plans to announce Wednesday at its Innovation event.

Keep Reading Show less
Max A. Cherney

Max A. Cherney is a Technology Reporter at Protocol covering the semiconductor industry. He has worked for Barron's magazine as a Technology Reporter, and its sister site MarketWatch. He is based in San Francisco.

Protocol | Workplace

Lessons from Facebook’s civil rights audit, a year later

Before the Facebook Papers, Facebook's audit made the case for transparency.

A new report released Wednesday lays out how companies can successfully conduct their own civil rights audit.

Photo: Kirill Kudryavtsev/AFP via Getty Images

Before Frances Haugen, before the Facebook Papers, before The Wall Street Journal's Facebook Files, Facebook had a chance to correct some of its algorithmic bias issues through an internal "civil rights audit" that concluded last year. According to people who contributed to the audit at the time, the company's response fell short.

That audit was conducted by Laura W. Murphy, a former director at the ACLU who has experience running similar audits for companies like Airbnb and Starbucks.

Keep Reading Show less
Michelle Ma
Michelle Ma (@himichellema) is a reporter at Protocol, where she writes about management, leadership and workplace issues in tech. Previously, she was a news editor of live journalism and special coverage for The Wall Street Journal. Prior to that, she worked as a staff writer at Wirecutter. She can be reached at mma@protocol.com.

The case for flying cars — and why they’re coming sooner than you think

Kitty Hawk's Sebastian Thrun on why he believes in the avian future of transportation. And why he'd prefer you not call them "flying cars."

Kitty Hawk's Heaviside might be flying over your house sometime in the next few years.

Photo: Kitty Hawk

Sebastian Thrun was one of the early pioneers of the self-driving car, and spent years working at Google and elsewhere to make autonomous vehicles a reality. Then he ditched the industry entirely and went for something even bigger: flying cars.

Except, wait, don't call them flying cars. Thrun, now the CEO of Kitty Hawk, calls them "electric vertical take-off and landing aircrafts," or eVTOLs for short. (It's not quite as catchy.) But whatever the name, Thrun is betting that they'll be transformative. No more dealing with existing infrastructure and outdated systems, no more worrying about the human driver next to you. He imagines a fully autonomous, fully safe, much more environmentally-friendly skyway system that doesn't have to worry about terrestrial matters at all. And he's convinced that's all coming much faster than you might think.

Keep Reading Show less
David Pierce

David Pierce ( @pierce) is Protocol's editorial director. Prior to joining Protocol, he was a columnist at The Wall Street Journal, a senior writer with Wired, and deputy editor at The Verge. He owns all the phones.

Latest Stories