Entertainment

Apple is a massive force in AR. It’s also been holding the technology back.

Apple has yet to add support for web-based AR to the iPhone. That’s been a big hurdle for the entire industry, according to insiders.

Phone with question mark

Apple has yet to add WebXR support to Safari on iOS.

Photo illustration: Robson Hatsukami Morgan/Unsplash; Protocol

“AR is going to change the way we use technology forever.” For the past five years, Tim Cook has used every possible opportunity to tell the world how incredibly excited he is about AR, and how AR glasses could be the next iPhone. “I’m AR fan, No. 1,” Cook recently told an interviewer. “I think it’s that big.”

By all accounts, Cook has put his money where his mouth is. The company reportedly has a team of 1,000 engineers working on AR glasses, with plans to release a still-unannounced mixed-reality device as early as this year. Apple was also first to introduce a dedicated AR developer framework for mobile apps in 2017, leading Cook to proclaim that “AR is going to change everything.”

Cook’s enthusiasm, and the company’s massive investments into its own AR hardware, stand in stark contrast to a notable lack of commitment on another front: web-based AR. While ostensibly part of an industrywide effort to standardize AR for modern browsers, Apple has yet to add support for web-based AR to the iPhone’s Safari browser — an omission that has severely hampered adoption of augmented reality, according to industry insiders.

“Apple has been a drag on innovation in WebAR,” said immersive computing specialist Christopher Lepkowski.

An Apple spokesperson declined to comment when contacted for this story.

The promise and perils of mobile AR

Lepkowski is a technical director and partner at Pretty Big Monster, a Los Angeles-based marketing agency that has produced AR experiences for brands like Netflix, Warner Bros. Discovery and American Airlines. With consumer AR glasses still years away, those types of AR marketing campaigns nowadays all focus on phones. “There is no more ubiquitous device in the world to reach consumers than the mobile phone,” Lepkowski said.

In the early days of phone-based AR, some brands experimented with dedicated apps to distribute their experiences. However, few if any consumers were willing to download yet another app just to be able to to see a movie character come alive in their living room.

“Every time you make someone click, you potentially lose 50% of your audience,” said Pretty Big Monster managing partner Jason Steinberg. The drop-off is exponentially higher when you force your audience to download and install an app. “For 95% of our clients, that's a nonstarter,” Steinberg said.

The good news is that AR can work outside of native apps, thanks to the WebXR standard developed by a broad coalition of browser- and device-makers including Google, Meta, Samsung, Mozilla and Magic Leap. WebXR makes it possible to run AR experiences in a browser, allowing people to launch immersive experiences directly from a website.

Apple has been a drag on innovation in WebAR.

Google’s Chrome browser for Android began supporting WebXR in beta in 2018; Samsung and Opera added support to their respective mobile browsers in 2020. Apple, on the other hand, has yet to add WebXR support to Safari on iOS.

At the same time, there is no avenue for other browser-makers to bring WebXR to the iPhone, as Apple forces them to use Safari’s WebKit rendering engine to build iOS versions of their browsers. Under the hood, the iOS version of Chrome is using Safari code — which means that Chrome on iOS isn’t able to support WebXR. This arrangement means WebXR is effectively out of reach on the iPhone.

Consequently, that means that in the U.S., it’s not an option for the majority of smartphone users. “They have blocked off a huge portion of our consumers from the most successful way to view AR experiences,” Steinberg said. “It's a shame that all these great experiences could be two clicks away.”

As a result of Apple’s lack of commitment, the web is playing a relatively minor role in AR these days. In a recent survey, only 15% of respondents said they had used a web-based AR experience, while more than 50% had used AR in an app like Snapchat or Instagram.

“We would have seen a lot more everyday usage of AR on the web [if it wasn’t] for this reluctance of theirs to step in, to support at all,” Lepkowski said.

Workarounds can be costly and inadequate

Without official support from Apple, a few companies have been filling the void with workarounds to run AR on the iPhone’s Safari browser.

The most popular solution comes from 8th Wall, an AR startup that was recently acquired by Niantic. 8th Wall’s solution is essentially a full-blown AR engine that enables key features like SLAM, a technique for understanding and tracking 3D real-world environments, without making use of Apple’s ARKit. “They do a great job,” acknowledges Lepkowski. “It's a great product.”

However, by effectively piggybacking on other browser APIs, 8th Wall’s technology isn’t able to take advantage of the deep hardware integration that Apple is making available to apps via ARKit. “8th Wall is amazing for what it does,” said Georgia Tech professor and WebXR pioneer Blair MacIntyre. But without access to the sensor data that ARKit can tap into, a browser-based workaround can never work as well as app-based AR, he cautioned. “And it's incredibly taxing on the battery and power,” MacIntyre said.

Using 8th Wall’s browser-based AR engine also comes with recurring monthly licensing costs. That may not be a huge deal for brands looking to launch massive marketing campaigns, but it’s enough to give some companies a pause, said Steinberg. “That is very problematic for many of our clients,” he said. “They're just not prepared to pay an ongoing fee.”

And while big companies may be able to justify paying a fee to run their AR experience on iPhones, that’s simply not an option for many educators, artists and other independent developers looking to experiment with mobile AR.

Why did we build this entire open web if we weren't going to use it?

For many of the same developers, building full-fledged AR apps for the iPhone is equally not an option. During a recent school year, one of MacIntyre’s students at Georgia Tech was looking to address the growing issue of sexually transmitted diseases among middle schoolers in a way that the generation understands best: with an AR game designed to teach them about safe sex. “It was a cute, little fun game,” said MacIntyre. Distributing it, however, was challenging. “There's zero chance that [a game like this] could ever make it into any app store,” MacIntyre said.

Even for less-controversial AR content that wouldn’t get flagged by moderators, building native apps is often too high of a barrier. “In order to ship something in an app store, it has to be polished and substantial,” MacIntyre said. “Doing that is time-consuming, [and requires] lots of money.”

The web, on the other hand, has a long history of experimentation, of people trying out ideas in public. Everyone’s favorite brain-teaser Wordle, for instance, started as a personal pet project accessed best by clicking a link through a search engine. Many believe the same could be true for web-based AR, if only Apple were to support it on iPhones.

Or as Lepkowski put it: “Why did we build this entire open web if we weren't going to use it?”

A seat at the table, but no word on its plans

Why Apple hasn’t implemented WebXR is a mystery to many industry insiders. The company has for years been part of the World Wide Web Consortium’s Immersive Web Group, which has been hashing out the WebXR standard, and Apple’s head of WebKit Engineering Maciej Stachowiak has gone on the record saying the company is “enthusiastic about WebXR.” In September 2020, an Apple employee who has been taking part in the Immersive Web Group even revealed that WebXR code had found its way into WebKit.

Those statements were celebrated by developers, and the following silence on the subject has led to increasingly desperate pleas on Apple’s forums. “Everyone and their mother is waiting eagerly for this,” wrote one developer last year. “Very interested in the progress of this issue as clients ‘can't believe’ that WebXR is not yet supported on iOS,” added another. “Please try to complete this … before 2050,” wrote one commenter, while another on a different thread concluded: “Safari is the new Internet Explorer.”

MacIntyre is among those who didn’t think it would take Apple this long. In 2017, he was working as a principal research scientist of Mozilla’s Mixed Reality team, looking for ways to experiment with web-based AR on the iPhone. Absent official support from Apple, his group ended up building a WebXR Viewer iOS app capable of rendering web-based AR experiences. The idea at the time was to sunset the project as soon as Apple added WebXR support to iOS. Five years later, the WebXR Viewer app remains available for download on the App Store.

Some have speculated that Apple has a general animosity to the open web, and that manifests in its unwillingness to support anything that could help consumers bypass the company’s lucrative App Store. MacIntyre doesn’t buy into these theories. Instead, he pointed to other web standards, including WebGL and WebRTC. Apple was slow to adopt both, only to one day unceremoniously flip the switch.

“It wouldn't surprise me if they have [WebXR] working [internally],” he mused. “But if they're working on a [headset] as everyone assumes they are, then they're probably waiting for that.”

As Apple gets closer to releasing its own headset, there are signs it is also warming to WebXR. Recent beta versions of iOS have included code snippets hinting at WebXR support, and company representatives began more actively participating in the Immersive Web Group. However, with Apple’s roadmap closely guarded, many industry insiders are still unsure what Apple’s ultimate goals are.

If they're working on a [headset] as everyone assumes they are, then they're probably waiting for that.

Case in point: The company recently proposed a new HTML element to display 3D content in browsers. Apple representatives used a meeting of the W3C Immersive Web Group in San Francisco last month to push for this new element to become part of the WebXR spec, and publicly available meeting notes indicate the company regards it as a way to better display 3D objects embedded in 2D websites when viewed with AR glasses.

At the meeting, representatives from other companies seemed to be at a loss about why Apple was focusing on this relatively narrow use case, as opposed to just implementing the entire WebXR spec. “People [are] frustrated by gaps of mobile AR today: Closing those is a priority,” said Microsoft HoloLens principal program manager Alex Turner, according to the meeting minutes.

Still, Apple actively participating in these discussions is a change for the company. A W3C Immersive Web Group member, who spoke to Protocol on condition of anonymity, said the company had largely remained on the sidelines in the past, with representatives not participating in discussions or giving any indications for Apple’s plans to support WebXR. That person credited Apple’s culture of secrecy for this. "It's the way they do business," the source said.

Apple’s support would make web-based AR real

While AR insiders have been aware of the WebXR issue for some time, there are some signs Apple could face regulatory pressure over its restrictions on third-party developers. The company has only been able to effectively bar WebXR from the iPhone because of its tight control of the phone’s browser engine.

That type of lockdown, a hallmark of Apple’s closed ecosystem, has come under increasing scrutiny from regulators both in the U.S. and overseas. In a recent draft of the Digital Markets Act, European regulators specifically called out restricting browser engine choices as anticompetitive. “When gatekeepers operate and impose browser engines, they are in a position to determine the functionality and standards that will apply not only to their own web browsers, but also to competing web browsers and, in turn, to web software applications,” the draft reads, according to The Register.

Whether by choice or under pressure, if Apple does change its heart on supporting WebXR, industry insiders predict a massive shift for the entire industry, with a new wave of innovation even before consumer-grade AR glasses are market-ready. “It would be a new dawn for AR,” said Steinberg. “You'd see a lot of investment going into this area.”

“It would be a big deal,” agreed MacIntyre. Web-based AR is simply not getting enough traction without support on iOS devices, he argued, which would make Apple stepping up to the table a game-changer.

“It just makes it real,” MacIntyre said.

Fintech

Judge Zia Faruqui is trying to teach you crypto, one ‘SNL’ reference at a time

His decisions on major cryptocurrency cases have quoted "The Big Lebowski," "SNL," and "Dr. Strangelove." That’s because he wants you — yes, you — to read them.

The ways Zia Faruqui (right) has weighed on cases that have come before him can give lawyers clues as to what legal frameworks will pass muster.

Photo: Carolyn Van Houten/The Washington Post via Getty Images

“Cryptocurrency and related software analytics tools are ‘The wave of the future, Dude. One hundred percent electronic.’”

That’s not a quote from "The Big Lebowski" — at least, not directly. It’s a quote from a Washington, D.C., district court memorandum opinion on the role cryptocurrency analytics tools can play in government investigations. The author is Magistrate Judge Zia Faruqui.

Keep ReadingShow less
Veronica Irwin

Veronica Irwin (@vronirwin) is a San Francisco-based reporter at Protocol covering fintech. Previously she was at the San Francisco Examiner, covering tech from a hyper-local angle. Before that, her byline was featured in SF Weekly, The Nation, Techworker, Ms. Magazine and The Frisc.

The financial technology transformation is driving competition, creating consumer choice, and shaping the future of finance. Hear from seven fintech leaders who are reshaping the future of finance, and join the inaugural Financial Technology Association Fintech Summit to learn more.

Keep ReadingShow less
FTA
The Financial Technology Association (FTA) represents industry leaders shaping the future of finance. We champion the power of technology-centered financial services and advocate for the modernization of financial regulation to support inclusion and responsible innovation.
Enterprise

AWS CEO: The cloud isn’t just about technology

As AWS preps for its annual re:Invent conference, Adam Selipsky talks product strategy, support for hybrid environments, and the value of the cloud in uncertain economic times.

Photo: Noah Berger/Getty Images for Amazon Web Services

AWS is gearing up for re:Invent, its annual cloud computing conference where announcements this year are expected to focus on its end-to-end data strategy and delivering new industry-specific services.

It will be the second re:Invent with CEO Adam Selipsky as leader of the industry’s largest cloud provider after his return last year to AWS from data visualization company Tableau Software.

Keep ReadingShow less
Donna Goodison

Donna Goodison (@dgoodison) is Protocol's senior reporter focusing on enterprise infrastructure technology, from the 'Big 3' cloud computing providers to data centers. She previously covered the public cloud at CRN after 15 years as a business reporter for the Boston Herald. Based in Massachusetts, she also has worked as a Boston Globe freelancer, business reporter at the Boston Business Journal and real estate reporter at Banker & Tradesman after toiling at weekly newspapers.

Image: Protocol

We launched Protocol in February 2020 to cover the evolving power center of tech. It is with deep sadness that just under three years later, we are winding down the publication.

As of today, we will not publish any more stories. All of our newsletters, apart from our flagship, Source Code, will no longer be sent. Source Code will be published and sent for the next few weeks, but it will also close down in December.

Keep ReadingShow less
Bennett Richardson

Bennett Richardson ( @bennettrich) is the president of Protocol. Prior to joining Protocol in 2019, Bennett was executive director of global strategic partnerships at POLITICO, where he led strategic growth efforts including POLITICO's European expansion in Brussels and POLITICO's creative agency POLITICO Focus during his six years with the company. Prior to POLITICO, Bennett was co-founder and CMO of Hinge, the mobile dating company recently acquired by Match Group. Bennett began his career in digital and social brand marketing working with major brands across tech, energy, and health care at leading marketing and communications agencies including Edelman and GMMB. Bennett is originally from Portland, Maine, and received his bachelor's degree from Colgate University.

Enterprise

Why large enterprises struggle to find suitable platforms for MLops

As companies expand their use of AI beyond running just a few machine learning models, and as larger enterprises go from deploying hundreds of models to thousands and even millions of models, ML practitioners say that they have yet to find what they need from prepackaged MLops systems.

As companies expand their use of AI beyond running just a few machine learning models, ML practitioners say that they have yet to find what they need from prepackaged MLops systems.

Photo: artpartner-images via Getty Images

On any given day, Lily AI runs hundreds of machine learning models using computer vision and natural language processing that are customized for its retail and ecommerce clients to make website product recommendations, forecast demand, and plan merchandising. But this spring when the company was in the market for a machine learning operations platform to manage its expanding model roster, it wasn’t easy to find a suitable off-the-shelf system that could handle such a large number of models in deployment while also meeting other criteria.

Some MLops platforms are not well-suited for maintaining even more than 10 machine learning models when it comes to keeping track of data, navigating their user interfaces, or reporting capabilities, Matthew Nokleby, machine learning manager for Lily AI’s product intelligence team, told Protocol earlier this year. “The duct tape starts to show,” he said.

Keep ReadingShow less
Kate Kaye

Kate Kaye is an award-winning multimedia reporter digging deep and telling print, digital and audio stories. She covers AI and data for Protocol. Her reporting on AI and tech ethics issues has been published in OneZero, Fast Company, MIT Technology Review, CityLab, Ad Age and Digiday and heard on NPR. Kate is the creator of RedTailMedia.org and is the author of "Campaign '08: A Turning Point for Digital Media," a book about how the 2008 presidential campaigns used digital media and data.

Latest Stories
Bulletins