Entertainment

Apple is a massive force in AR. It’s also been holding the technology back.

Apple has yet to add support for web-based AR to the iPhone. That’s been a big hurdle for the entire industry, according to insiders.

Phone with question mark

Apple has yet to add WebXR support to Safari on iOS.

Photo illustration: Robson Hatsukami Morgan/Unsplash; Protocol

“AR is going to change the way we use technology forever.” For the past five years, Tim Cook has used every possible opportunity to tell the world how incredibly excited he is about AR, and how AR glasses could be the next iPhone. “I’m AR fan, No. 1,” Cook recently told an interviewer. “I think it’s that big.”

By all accounts, Cook has put his money where his mouth is. The company reportedly has a team of 1,000 engineers working on AR glasses, with plans to release a still-unannounced mixed-reality device as early as this year. Apple was also first to introduce a dedicated AR developer framework for mobile apps in 2017, leading Cook to proclaim that “AR is going to change everything.”

Cook’s enthusiasm, and the company’s massive investments into its own AR hardware, stand in stark contrast to a notable lack of commitment on another front: web-based AR. While ostensibly part of an industrywide effort to standardize AR for modern browsers, Apple has yet to add support for web-based AR to the iPhone’s Safari browser — an omission that has severely hampered adoption of augmented reality, according to industry insiders.

“Apple has been a drag on innovation in WebAR,” said immersive computing specialist Christopher Lepkowski.

An Apple spokesperson declined to comment when contacted for this story.

The promise and perils of mobile AR

Lepkowski is a technical director and partner at Pretty Big Monster, a Los Angeles-based marketing agency that has produced AR experiences for brands like Netflix, Warner Bros. Discovery and American Airlines. With consumer AR glasses still years away, those types of AR marketing campaigns nowadays all focus on phones. “There is no more ubiquitous device in the world to reach consumers than the mobile phone,” Lepkowski said.

In the early days of phone-based AR, some brands experimented with dedicated apps to distribute their experiences. However, few if any consumers were willing to download yet another app just to be able to to see a movie character come alive in their living room.

“Every time you make someone click, you potentially lose 50% of your audience,” said Pretty Big Monster managing partner Jason Steinberg. The drop-off is exponentially higher when you force your audience to download and install an app. “For 95% of our clients, that's a nonstarter,” Steinberg said.

The good news is that AR can work outside of native apps, thanks to the WebXR standard developed by a broad coalition of browser- and device-makers including Google, Meta, Samsung, Mozilla and Magic Leap. WebXR makes it possible to run AR experiences in a browser, allowing people to launch immersive experiences directly from a website.

Apple has been a drag on innovation in WebAR.

Google’s Chrome browser for Android began supporting WebXR in beta in 2018; Samsung and Opera added support to their respective mobile browsers in 2020. Apple, on the other hand, has yet to add WebXR support to Safari on iOS.

At the same time, there is no avenue for other browser-makers to bring WebXR to the iPhone, as Apple forces them to use Safari’s WebKit rendering engine to build iOS versions of their browsers. Under the hood, the iOS version of Chrome is using Safari code — which means that Chrome on iOS isn’t able to support WebXR. This arrangement means WebXR is effectively out of reach on the iPhone.

Consequently, that means that in the U.S., it’s not an option for the majority of smartphone users. “They have blocked off a huge portion of our consumers from the most successful way to view AR experiences,” Steinberg said. “It's a shame that all these great experiences could be two clicks away.”

As a result of Apple’s lack of commitment, the web is playing a relatively minor role in AR these days. In a recent survey, only 15% of respondents said they had used a web-based AR experience, while more than 50% had used AR in an app like Snapchat or Instagram.

“We would have seen a lot more everyday usage of AR on the web [if it wasn’t] for this reluctance of theirs to step in, to support at all,” Lepkowski said.

Workarounds can be costly and inadequate

Without official support from Apple, a few companies have been filling the void with workarounds to run AR on the iPhone’s Safari browser.

The most popular solution comes from 8th Wall, an AR startup that was recently acquired by Niantic. 8th Wall’s solution is essentially a full-blown AR engine that enables key features like SLAM, a technique for understanding and tracking 3D real-world environments, without making use of Apple’s ARKit. “They do a great job,” acknowledges Lepkowski. “It's a great product.”

However, by effectively piggybacking on other browser APIs, 8th Wall’s technology isn’t able to take advantage of the deep hardware integration that Apple is making available to apps via ARKit. “8th Wall is amazing for what it does,” said Georgia Tech professor and WebXR pioneer Blair MacIntyre. But without access to the sensor data that ARKit can tap into, a browser-based workaround can never work as well as app-based AR, he cautioned. “And it's incredibly taxing on the battery and power,” MacIntyre said.

Using 8th Wall’s browser-based AR engine also comes with recurring monthly licensing costs. That may not be a huge deal for brands looking to launch massive marketing campaigns, but it’s enough to give some companies a pause, said Steinberg. “That is very problematic for many of our clients,” he said. “They're just not prepared to pay an ongoing fee.”

And while big companies may be able to justify paying a fee to run their AR experience on iPhones, that’s simply not an option for many educators, artists and other independent developers looking to experiment with mobile AR.

Why did we build this entire open web if we weren't going to use it?

For many of the same developers, building full-fledged AR apps for the iPhone is equally not an option. During a recent school year, one of MacIntyre’s students at Georgia Tech was looking to address the growing issue of sexually transmitted diseases among middle schoolers in a way that the generation understands best: with an AR game designed to teach them about safe sex. “It was a cute, little fun game,” said MacIntyre. Distributing it, however, was challenging. “There's zero chance that [a game like this] could ever make it into any app store,” MacIntyre said.

Even for less-controversial AR content that wouldn’t get flagged by moderators, building native apps is often too high of a barrier. “In order to ship something in an app store, it has to be polished and substantial,” MacIntyre said. “Doing that is time-consuming, [and requires] lots of money.”

The web, on the other hand, has a long history of experimentation, of people trying out ideas in public. Everyone’s favorite brain-teaser Wordle, for instance, started as a personal pet project accessed best by clicking a link through a search engine. Many believe the same could be true for web-based AR, if only Apple were to support it on iPhones.

Or as Lepkowski put it: “Why did we build this entire open web if we weren't going to use it?”

A seat at the table, but no word on its plans

Why Apple hasn’t implemented WebXR is a mystery to many industry insiders. The company has for years been part of the World Wide Web Consortium’s Immersive Web Group, which has been hashing out the WebXR standard, and Apple’s head of WebKit Engineering Maciej Stachowiak has gone on the record saying the company is “enthusiastic about WebXR.” In September 2020, an Apple employee who has been taking part in the Immersive Web Group even revealed that WebXR code had found its way into WebKit.

Those statements were celebrated by developers, and the following silence on the subject has led to increasingly desperate pleas on Apple’s forums. “Everyone and their mother is waiting eagerly for this,” wrote one developer last year. “Very interested in the progress of this issue as clients ‘can't believe’ that WebXR is not yet supported on iOS,” added another. “Please try to complete this … before 2050,” wrote one commenter, while another on a different thread concluded: “Safari is the new Internet Explorer.”

MacIntyre is among those who didn’t think it would take Apple this long. In 2017, he was working as a principal research scientist of Mozilla’s Mixed Reality team, looking for ways to experiment with web-based AR on the iPhone. Absent official support from Apple, his group ended up building a WebXR Viewer iOS app capable of rendering web-based AR experiences. The idea at the time was to sunset the project as soon as Apple added WebXR support to iOS. Five years later, the WebXR Viewer app remains available for download on the App Store.

Some have speculated that Apple has a general animosity to the open web, and that manifests in its unwillingness to support anything that could help consumers bypass the company’s lucrative App Store. MacIntyre doesn’t buy into these theories. Instead, he pointed to other web standards, including WebGL and WebRTC. Apple was slow to adopt both, only to one day unceremoniously flip the switch.

“It wouldn't surprise me if they have [WebXR] working [internally],” he mused. “But if they're working on a [headset] as everyone assumes they are, then they're probably waiting for that.”

As Apple gets closer to releasing its own headset, there are signs it is also warming to WebXR. Recent beta versions of iOS have included code snippets hinting at WebXR support, and company representatives began more actively participating in the Immersive Web Group. However, with Apple’s roadmap closely guarded, many industry insiders are still unsure what Apple’s ultimate goals are.

If they're working on a [headset] as everyone assumes they are, then they're probably waiting for that.

Case in point: The company recently proposed a new HTML element to display 3D content in browsers. Apple representatives used a meeting of the W3C Immersive Web Group in San Francisco last month to push for this new element to become part of the WebXR spec, and publicly available meeting notes indicate the company regards it as a way to better display 3D objects embedded in 2D websites when viewed with AR glasses.

At the meeting, representatives from other companies seemed to be at a loss about why Apple was focusing on this relatively narrow use case, as opposed to just implementing the entire WebXR spec. “People [are] frustrated by gaps of mobile AR today: Closing those is a priority,” said Microsoft HoloLens principal program manager Alex Turner, according to the meeting minutes.

Still, Apple actively participating in these discussions is a change for the company. A W3C Immersive Web Group member, who spoke to Protocol on condition of anonymity, said the company had largely remained on the sidelines in the past, with representatives not participating in discussions or giving any indications for Apple’s plans to support WebXR. That person credited Apple’s culture of secrecy for this. "It's the way they do business," the source said.

Apple’s support would make web-based AR real

While AR insiders have been aware of the WebXR issue for some time, there are some signs Apple could face regulatory pressure over its restrictions on third-party developers. The company has only been able to effectively bar WebXR from the iPhone because of its tight control of the phone’s browser engine.

That type of lockdown, a hallmark of Apple’s closed ecosystem, has come under increasing scrutiny from regulators both in the U.S. and overseas. In a recent draft of the Digital Markets Act, European regulators specifically called out restricting browser engine choices as anticompetitive. “When gatekeepers operate and impose browser engines, they are in a position to determine the functionality and standards that will apply not only to their own web browsers, but also to competing web browsers and, in turn, to web software applications,” the draft reads, according to The Register.

Whether by choice or under pressure, if Apple does change its heart on supporting WebXR, industry insiders predict a massive shift for the entire industry, with a new wave of innovation even before consumer-grade AR glasses are market-ready. “It would be a new dawn for AR,” said Steinberg. “You'd see a lot of investment going into this area.”

“It would be a big deal,” agreed MacIntyre. Web-based AR is simply not getting enough traction without support on iOS devices, he argued, which would make Apple stepping up to the table a game-changer.

“It just makes it real,” MacIntyre said.

Workplace

The tools that make you pay for not getting stuff done

Some tools let you put your money on the line for productivity. Should you bite?

Commitment contracts are popular in a niche corner of the internet, and the tools have built up loyal followings of people who find the extra motivation effective.

Photoillustration: Anna Shvets/Pexels; Protocol

Danny Reeves, CEO and co-founder of Beeminder, is used to defending his product.

“When people first hear about it, they’re kind of appalled,” Reeves said. “Making money off of people’s failure is how they view it.”

Keep Reading Show less
Lizzy Lawrence

Lizzy Lawrence ( @LizzyLaw_) is a reporter at Protocol, covering tools and productivity in the workplace. She's a recent graduate of the University of Michigan, where she studied sociology and international studies. She served as editor in chief of The Michigan Daily, her school's independent newspaper. She's based in D.C., and can be reached at llawrence@protocol.com.

Sponsored Content

Foursquare data story: leveraging location data for site selection

We take a closer look at points of interest and foot traffic patterns to demonstrate how location data can be leveraged to inform better site selecti­on strategies.

Imagine: You’re the leader of a real estate team at a restaurant brand looking to open a new location in Manhattan. You have two options you’re evaluating: one site in SoHo, and another site in the Flatiron neighborhood. Which do you choose?

Keep Reading Show less

Elon Musk has bots on his mind.

Photo: Christian Marquardt/Getty Images

Elon Musk says he needs proof that less than 5% of Twitter's users are bots — or the deal isn't going ahead.

Keep Reading Show less
Jamie Condliffe

Jamie Condliffe ( @jme_c) is the executive editor at Protocol, based in London. Prior to joining Protocol in 2019, he worked on the business desk at The New York Times, where he edited the DealBook newsletter and wrote Bits, the weekly tech newsletter. He has previously worked at MIT Technology Review, Gizmodo, and New Scientist, and has held lectureships at the University of Oxford and Imperial College London. He also holds a doctorate in engineering from the University of Oxford.

Policy

Nobody will help Big Tech prevent online terrorism but itself

There’s no will in Congress or the C-suites of social media giants for a new approach, but smaller platforms would have room to step up — if they decided to.

Timothy Kujawski of Buffalo lights candles at a makeshift memorial as people gather at the scene of a mass shooting at Tops Friendly Market at Jefferson Avenue and Riley Street on Sunday, May 15, 2022 in Buffalo, NY. The fatal shooting of 10 people at a grocery store in a historically Black neighborhood of Buffalo by a young white gunman is being investigated as a hate crime and an act of racially motivated violent extremism, according to federal officials.

Photo: Kent Nishimura / Los Angeles Times via Getty Images

The shooting in Buffalo, New York, that killed 10 people over the weekend has put the spotlight back on social media companies. Some of the attack was livestreamed, beginning on Amazon-owned Twitch, and the alleged shooter appears to have written about how his racist motivations arose from misinformation on smaller or fringe sites including 4chan.

In response, policymakers are directing their anger at tech platforms, with New York Governor Kathy Hochul calling for the companies to be “more vigilant in monitoring” and for “a legal responsibility to ensure that such hate cannot populate these sites.”

Keep Reading Show less
Ben Brody

Ben Brody (@ BenBrodyDC) is a senior reporter at Protocol focusing on how Congress, courts and agencies affect the online world we live in. He formerly covered tech policy and lobbying (including antitrust, Section 230 and privacy) at Bloomberg News, where he previously reported on the influence industry, government ethics and the 2016 presidential election. Before that, Ben covered business news at CNNMoney and AdAge, and all manner of stories in and around New York. He still loves appearing on the New York news radio he grew up with.

We're answering all your questions about the crypto crash.

Photo: Chris Liverani/Unsplash

People started talking about another crypto winter in January, when falling prices had wiped out $1 trillion in value from November’s peak. Prices rallied back in March, restoring some of the losses. Then crypto fell hard again, with bitcoin down more than 60% from its all-time high and other cryptocurrencies harder hit. The market’s message was clear: Crypto winter was no longer coming. It’s here.

If you’ve got questions about the crypto crash, the Protocol Fintech team has answers.

Keep Reading Show less
Latest Stories
Bulletins