Entertainment

Apple is a massive force in AR. It’s also been holding the technology back.

Apple has yet to add support for web-based AR to the iPhone. That’s been a big hurdle for the entire industry, according to insiders.

Phone with question mark

Apple has yet to add WebXR support to Safari on iOS.

Photo illustration: Robson Hatsukami Morgan/Unsplash; Protocol

“AR is going to change the way we use technology forever.” For the past five years, Tim Cook has used every possible opportunity to tell the world how incredibly excited he is about AR, and how AR glasses could be the next iPhone. “I’m AR fan, No. 1,” Cook recently told an interviewer. “I think it’s that big.”

By all accounts, Cook has put his money where his mouth is. The company reportedly has a team of 1,000 engineers working on AR glasses, with plans to release a still-unannounced mixed-reality device as early as this year. Apple was also first to introduce a dedicated AR developer framework for mobile apps in 2017, leading Cook to proclaim that “AR is going to change everything.”

Cook’s enthusiasm, and the company’s massive investments into its own AR hardware, stand in stark contrast to a notable lack of commitment on another front: web-based AR. While ostensibly part of an industrywide effort to standardize AR for modern browsers, Apple has yet to add support for web-based AR to the iPhone’s Safari browser — an omission that has severely hampered adoption of augmented reality, according to industry insiders.

“Apple has been a drag on innovation in WebAR,” said immersive computing specialist Christopher Lepkowski.

An Apple spokesperson declined to comment when contacted for this story.

The promise and perils of mobile AR

Lepkowski is a technical director and partner at Pretty Big Monster, a Los Angeles-based marketing agency that has produced AR experiences for brands like Netflix, Warner Bros. Discovery and American Airlines. With consumer AR glasses still years away, those types of AR marketing campaigns nowadays all focus on phones. “There is no more ubiquitous device in the world to reach consumers than the mobile phone,” Lepkowski said.

In the early days of phone-based AR, some brands experimented with dedicated apps to distribute their experiences. However, few if any consumers were willing to download yet another app just to be able to to see a movie character come alive in their living room.

“Every time you make someone click, you potentially lose 50% of your audience,” said Pretty Big Monster managing partner Jason Steinberg. The drop-off is exponentially higher when you force your audience to download and install an app. “For 95% of our clients, that's a nonstarter,” Steinberg said.

The good news is that AR can work outside of native apps, thanks to the WebXR standard developed by a broad coalition of browser- and device-makers including Google, Meta, Samsung, Mozilla and Magic Leap. WebXR makes it possible to run AR experiences in a browser, allowing people to launch immersive experiences directly from a website.

Apple has been a drag on innovation in WebAR.

Google’s Chrome browser for Android began supporting WebXR in beta in 2018; Samsung and Opera added support to their respective mobile browsers in 2020. Apple, on the other hand, has yet to add WebXR support to Safari on iOS.

At the same time, there is no avenue for other browser-makers to bring WebXR to the iPhone, as Apple forces them to use Safari’s WebKit rendering engine to build iOS versions of their browsers. Under the hood, the iOS version of Chrome is using Safari code — which means that Chrome on iOS isn’t able to support WebXR. This arrangement means WebXR is effectively out of reach on the iPhone.

Consequently, that means that in the U.S., it’s not an option for the majority of smartphone users. “They have blocked off a huge portion of our consumers from the most successful way to view AR experiences,” Steinberg said. “It's a shame that all these great experiences could be two clicks away.”

As a result of Apple’s lack of commitment, the web is playing a relatively minor role in AR these days. In a recent survey, only 15% of respondents said they had used a web-based AR experience, while more than 50% had used AR in an app like Snapchat or Instagram.

“We would have seen a lot more everyday usage of AR on the web [if it wasn’t] for this reluctance of theirs to step in, to support at all,” Lepkowski said.

Workarounds can be costly and inadequate

Without official support from Apple, a few companies have been filling the void with workarounds to run AR on the iPhone’s Safari browser.

The most popular solution comes from 8th Wall, an AR startup that was recently acquired by Niantic. 8th Wall’s solution is essentially a full-blown AR engine that enables key features like SLAM, a technique for understanding and tracking 3D real-world environments, without making use of Apple’s ARKit. “They do a great job,” acknowledges Lepkowski. “It's a great product.”

However, by effectively piggybacking on other browser APIs, 8th Wall’s technology isn’t able to take advantage of the deep hardware integration that Apple is making available to apps via ARKit. “8th Wall is amazing for what it does,” said Georgia Tech professor and WebXR pioneer Blair MacIntyre. But without access to the sensor data that ARKit can tap into, a browser-based workaround can never work as well as app-based AR, he cautioned. “And it's incredibly taxing on the battery and power,” MacIntyre said.

Using 8th Wall’s browser-based AR engine also comes with recurring monthly licensing costs. That may not be a huge deal for brands looking to launch massive marketing campaigns, but it’s enough to give some companies a pause, said Steinberg. “That is very problematic for many of our clients,” he said. “They're just not prepared to pay an ongoing fee.”

And while big companies may be able to justify paying a fee to run their AR experience on iPhones, that’s simply not an option for many educators, artists and other independent developers looking to experiment with mobile AR.

Why did we build this entire open web if we weren't going to use it?

For many of the same developers, building full-fledged AR apps for the iPhone is equally not an option. During a recent school year, one of MacIntyre’s students at Georgia Tech was looking to address the growing issue of sexually transmitted diseases among middle schoolers in a way that the generation understands best: with an AR game designed to teach them about safe sex. “It was a cute, little fun game,” said MacIntyre. Distributing it, however, was challenging. “There's zero chance that [a game like this] could ever make it into any app store,” MacIntyre said.

Even for less-controversial AR content that wouldn’t get flagged by moderators, building native apps is often too high of a barrier. “In order to ship something in an app store, it has to be polished and substantial,” MacIntyre said. “Doing that is time-consuming, [and requires] lots of money.”

The web, on the other hand, has a long history of experimentation, of people trying out ideas in public. Everyone’s favorite brain-teaser Wordle, for instance, started as a personal pet project accessed best by clicking a link through a search engine. Many believe the same could be true for web-based AR, if only Apple were to support it on iPhones.

Or as Lepkowski put it: “Why did we build this entire open web if we weren't going to use it?”

A seat at the table, but no word on its plans

Why Apple hasn’t implemented WebXR is a mystery to many industry insiders. The company has for years been part of the World Wide Web Consortium’s Immersive Web Group, which has been hashing out the WebXR standard, and Apple’s head of WebKit Engineering Maciej Stachowiak has gone on the record saying the company is “enthusiastic about WebXR.” In September 2020, an Apple employee who has been taking part in the Immersive Web Group even revealed that WebXR code had found its way into WebKit.

Those statements were celebrated by developers, and the following silence on the subject has led to increasingly desperate pleas on Apple’s forums. “Everyone and their mother is waiting eagerly for this,” wrote one developer last year. “Very interested in the progress of this issue as clients ‘can't believe’ that WebXR is not yet supported on iOS,” added another. “Please try to complete this … before 2050,” wrote one commenter, while another on a different thread concluded: “Safari is the new Internet Explorer.”

MacIntyre is among those who didn’t think it would take Apple this long. In 2017, he was working as a principal research scientist of Mozilla’s Mixed Reality team, looking for ways to experiment with web-based AR on the iPhone. Absent official support from Apple, his group ended up building a WebXR Viewer iOS app capable of rendering web-based AR experiences. The idea at the time was to sunset the project as soon as Apple added WebXR support to iOS. Five years later, the WebXR Viewer app remains available for download on the App Store.

Some have speculated that Apple has a general animosity to the open web, and that manifests in its unwillingness to support anything that could help consumers bypass the company’s lucrative App Store. MacIntyre doesn’t buy into these theories. Instead, he pointed to other web standards, including WebGL and WebRTC. Apple was slow to adopt both, only to one day unceremoniously flip the switch.

“It wouldn't surprise me if they have [WebXR] working [internally],” he mused. “But if they're working on a [headset] as everyone assumes they are, then they're probably waiting for that.”

As Apple gets closer to releasing its own headset, there are signs it is also warming to WebXR. Recent beta versions of iOS have included code snippets hinting at WebXR support, and company representatives began more actively participating in the Immersive Web Group. However, with Apple’s roadmap closely guarded, many industry insiders are still unsure what Apple’s ultimate goals are.

If they're working on a [headset] as everyone assumes they are, then they're probably waiting for that.

Case in point: The company recently proposed a new HTML element to display 3D content in browsers. Apple representatives used a meeting of the W3C Immersive Web Group in San Francisco last month to push for this new element to become part of the WebXR spec, and publicly available meeting notes indicate the company regards it as a way to better display 3D objects embedded in 2D websites when viewed with AR glasses.

At the meeting, representatives from other companies seemed to be at a loss about why Apple was focusing on this relatively narrow use case, as opposed to just implementing the entire WebXR spec. “People [are] frustrated by gaps of mobile AR today: Closing those is a priority,” said Microsoft HoloLens principal program manager Alex Turner, according to the meeting minutes.

Still, Apple actively participating in these discussions is a change for the company. A W3C Immersive Web Group member, who spoke to Protocol on condition of anonymity, said the company had largely remained on the sidelines in the past, with representatives not participating in discussions or giving any indications for Apple’s plans to support WebXR. That person credited Apple’s culture of secrecy for this. "It's the way they do business," the source said.

Apple’s support would make web-based AR real

While AR insiders have been aware of the WebXR issue for some time, there are some signs Apple could face regulatory pressure over its restrictions on third-party developers. The company has only been able to effectively bar WebXR from the iPhone because of its tight control of the phone’s browser engine.

That type of lockdown, a hallmark of Apple’s closed ecosystem, has come under increasing scrutiny from regulators both in the U.S. and overseas. In a recent draft of the Digital Markets Act, European regulators specifically called out restricting browser engine choices as anticompetitive. “When gatekeepers operate and impose browser engines, they are in a position to determine the functionality and standards that will apply not only to their own web browsers, but also to competing web browsers and, in turn, to web software applications,” the draft reads, according to The Register.

Whether by choice or under pressure, if Apple does change its heart on supporting WebXR, industry insiders predict a massive shift for the entire industry, with a new wave of innovation even before consumer-grade AR glasses are market-ready. “It would be a new dawn for AR,” said Steinberg. “You'd see a lot of investment going into this area.”

“It would be a big deal,” agreed MacIntyre. Web-based AR is simply not getting enough traction without support on iOS devices, he argued, which would make Apple stepping up to the table a game-changer.

“It just makes it real,” MacIntyre said.

Elon Musk's influence over Twitter was clear at its annual meeting

Even though executives tried not to talk about Musk's deal to buy the company, they couldn't help but address his agenda.

Elon Musk loomed over Twitter's annual shareholder meeting.

Photoillustration: Getty Images; Unsplash; Protocol

In his opening remarks at Twitter's annual shareholder meeting on Wednesday, CEO Parag Agrawal said he wouldn't discuss the pending acquisition bid from Elon Musk, which wasn't on the agenda. That didn’t matter much: Musk’s fingerprints were all over the event, even overshadowing the expected if still-emotional news that Jack Dorsey would step away from Twitter’s board at the meeting's conclusion.

Keep Reading Show less
Hirsh Chitkara

Hirsh Chitkara ( @HirshChitkara) is a reporter at Protocol focused on the intersection of politics, technology and society. Before joining Protocol, he helped write a daily newsletter at Insider that covered all things Big Tech. He's based in New York and can be reached at hchitkara@protocol.com.

Sponsored Content

Why the digital transformation of industries is creating a more sustainable future

Qualcomm’s chief sustainability officer Angela Baker on how companies can view going “digital” as a way not only toward growth, as laid out in a recent report, but also toward establishing and meeting environmental, social and governance goals.

Three letters dominate business practice at present: ESG, or environmental, social and governance goals. The number of mentions of the environment in financial earnings has doubled in the last five years, according to GlobalData: 600,000 companies mentioned the term in their annual or quarterly results last year.

But meeting those ESG goals can be a challenge — one that businesses can’t and shouldn’t take lightly. Ahead of an exclusive fireside chat at Davos, Angela Baker, chief sustainability officer at Qualcomm, sat down with Protocol to speak about how best to achieve those targets and how Qualcomm thinks about its own sustainability strategy, net zero commitment, other ESG targets and more.

Keep Reading Show less
Chris Stokel-Walker

Chris Stokel-Walker is a freelance technology and culture journalist and author of "YouTubers: How YouTube Shook Up TV and Created a New Generation of Stars." His work has been published in The New York Times, The Guardian and Wired.

Workplace

Netflix’s layoffs reveal a larger diversity challenge in tech

Netflix just laid off 150 full-time employees and a number of agency contractors. Many of them were the company’s most marginalized employees.

It quickly became clear that many of the laid-off contractors possessed marginalized identities.

Illustration: Christopher T. Fong/Protocol

After Netflix’s first round of layoffs, there was a brief period of relief for the contractors who ran Netflix’s audience-oriented social media channels, like Strong Black Lead, Most and Con Todo. But the calm didn’t last.

Last week, Netflix laid off 150 full-time employees and a number of agency contractors. The customary #opentowork posts flooded LinkedIn, many coming from impacted members of Netflix’s talent and recruiting teams. A number of laid-off social media contractors also took to Twitter to share the news. It quickly became clear that similar to the layoffs at Tudum, Netflix’s entertainment site, many of the affected contractors possessed marginalized identities. The channels they ran focused on Black, LGBTQ+, Latinx and Asian audiences, among others.

Keep Reading Show less
Lizzy Lawrence

Lizzy Lawrence ( @LizzyLaw_) is a reporter at Protocol, covering tools and productivity in the workplace. She's a recent graduate of the University of Michigan, where she studied sociology and international studies. She served as editor in chief of The Michigan Daily, her school's independent newspaper. She's based in D.C., and can be reached at llawrence@protocol.com.

Fintech

Crypto doesn’t have to be red or blue

Sens. Cynthia Lummis and Kirsten Gillibrand are backing bipartisan legislation that establishes regulatory clarity for cryptocurrencies. This is the right way to approach a foundational technology.

"Crypto doesn’t neatly fall along party lines because, as a foundational technology, it is — or should be — inherently nonpartisan," says Diogo Mónica, co-founder and president of Anchorage Digital.

Photo: Anchorage Digital

Diogo Mónica is president and co-founder of Anchorage Digital.

When I moved from Portugal to the United States to work at Square, it was hard to wrap my head around the two-party system that dominates American politics. As I saw at home, democracies, by their very nature, can be messy. But as an outsider looking in, I can’t help but worry that the ever-widening gap between America’s two major parties looms over crypto’s future.

Keep Reading Show less
Diogo Mónica
Diogo Mónica is the co-founder and president of Anchorage Digital, the premier digital asset platform for institutions. He holds a Ph.D. in computer science from the Technical University of Lisbon, and has worked in software security for over 15 years. As an early employee at Square, he helped build security architecture that now moves $100 billion annually. At Docker, he helped secure core infrastructure used in global banks, governments and the three largest cloud providers.
Fintech

What downturn? A16z raises $4.5 billion for latest crypto fund

The new fund is more than double the $2.2 billion fund the VC firm raised just last June.

A16z general partner Arianna Simpson said that despite the precipitous drop in crypto prices in recent months, the firm is looking to stay active in the market and isn’t worried about short-term price changes.

Photo: Andreessen Horowitz

Andreessen Horowitz has raised $4.5 billion for two crypto venture funds. They’re the industry’s largest ever and represent an outsized bet on the future of Web3 startups, even with the industry in the midst of a steep market downturn.

The pool of money is technically two separate funds: a $1.5 billion fund for seed deals and a $3 billion fund for broader venture deals. That’s more than other megafunds recently raised by competitors such as Paradigm and Haun Ventures.

Keep Reading Show less
Tomio Geron

Tomio Geron ( @tomiogeron) is a San Francisco-based reporter covering fintech. He was previously a reporter and editor at The Wall Street Journal, covering venture capital and startups. Before that, he worked as a staff writer at Forbes, covering social media and venture capital, and also edited the Midas List of top tech investors. He has also worked at newspapers covering crime, courts, health and other topics. He can be reached at tgeron@protocol.com or tgeron@protonmail.com.

Latest Stories
Bulletins