Policy

Here’s what happened when Facebook stopped protecting users — on purpose

Internal documents reveal the impact of withholding certain integrity protections from a subset of Facebook users.

Facebook Papers: an illustration of the Facebook logo cracking

Frances Haugen said Facebook withholds certain protections from a subset of users to see how they'll react.

Image: Protocol

In her testimony before Congress last month, whistleblower Frances Haugen told lawmakers Facebook has conducted experiments where it withholds certain protections from a subset of users to see how they'll react. Facebook refers to this experimental group internally, she said, as "integrity holdouts."

"These are people who don't get protections from integrity systems to see what happens to them," Haugen said. "And those people who deal with a more toxic, painful version of Facebook use Facebook less."

Internal documents reveal a more complex story. According to one internal report from April 2019, Facebook has studied the impact of removing some protections against problematic content like clickbait and untrustworthy news for some users, but the results, at least in that report, were decidedly mixed.

The report showed that during one March 2019 test, when the company rolled back some protections from millions of users' News Feeds, their exposure to some of the worst forms of harmful content, like graphic violence, barely changed. As the report's author wrote, referring to the company's news feed protections at the time, "We are likely having little (if any) impact on violence."

The report also suggested that far from using Facebook less, integrity holdouts actually commented more and had more sessions on the app. "Given that Integrity is changing a ranking algorithm that is optimized for engagement, it is not surprising that integrity has some negative engagement impact," the report read.

A spokesperson for Haugen said that other documents she collected showed that retention is stronger among regular users than holdouts, but those findings were not included in this report.

The report was included in disclosures made to the Securities and Exchange Commission and provided to Congress in redacted form by Frances Haugen's legal counsel. A consortium of news organizations, including Protocol, has reviewed the redacted versions received by Congress. They offer a glimpse at how Facebook has analyzed the efficacy of its user protections — and weighed them against their impact on other company priorities like growth.

"Testing product and safety features is an important part of improving our platform and something that's standard in tech and many other industries," Facebook spokesperson Drew Pusateri told Protocol, noting that the holdout affected about 2% of Facebook users. "It helps us build the tools to reduce the prevalence of hate speech and other types of problematic content on our platform."

'High-harm spaces'

Facebook conducts holdout experiments for a range of business goals, not just its integrity work. Holdouts are effectively control groups that Facebook can compare to its larger pool of users. As former Facebook data scientist Sophie Zhang recently told Protocol, Facebook has also studied the impact of withholding ads from users. "The company wants to know the very long-term impacts of advertising on retention and usage for Facebook," Zhang said. "The argument was usually that we need to know what the impact of this is. We need to know if people like it or not. But this is also motivated by wanting to know the impact for growth."

By early 2019, it appears, the company had begun applying this approach to integrity protections. The report, published in late April 2019, detailed the initial findings from an experiment that tinkered with certain integrity protections in News Feed. Some of the findings were encouraging: The report showed, for instance, that regular users got substantially less clickbait and ad-farm content than the holdouts did, something the author notes is "not surprising," given that the company was demoting clickbait and ad farms "quite a bit."

The report showed that regular users' exposure to what the company considers "low quality news" was down about 18% compared to the holdouts. The company also found it was boosting content from major publishers more when integrity protections were in place. Holdouts, by contrast, were more likely to click the dislike button and report the posts they saw, and they were also more likely to see content from public pages than regular users were.

But the main takeaway from the report, the author wrote, was that the News Feed protections that were in place at the time weren't having an equally significant effect on more severe types of harm, like graphic violence. "I believe strongly that this needs to change," the author wrote.

During the experiment, the company continued to demote hate speech and graphic violence in at-risk countries, the report said. But for holdouts who weren't in at-risk countries, those demotions didn't exist. And yet, the report found no impact on regular users' exposure to violence compared to the holdouts.

"11% of users see content that has been marked as disturbing every day; 16% of users see content that is likely to be bullying; 39% of users see hateful content (i.e. borderline hate); 32% of users see borderline 3+ nudity content," the author wrote. "These are significant proportions of [daily active users] and we have effectively no ranking interventions in place to mitigate this." The author added, however, that those particular numbers "should be taken with a grain of salt," as measuring bad experiences on the platform was still a work in progress.

The report also made no secret of the negative impact of News Feed integrity protections on engagement. "By definition, Integrity is going to cause some engagement regression," the author wrote, noting that there are "tradeoffs between Integrity and Engagement."

Integrity efforts, the report found, were a blow to the company's "meaningful social interactions" metric, which emphasizes interactions between friends over public-facing content. One reason for that, the author proposed, was that holdouts commented more than regular users did. While regular users were more likely to like posts on Facebook compared to holdouts, the author wrote, it was "not enough to make up for the decline in comments." The report also showed that content views and time spent on the app were down slightly among regular users compared to holdouts.

The report's limitations

It would be easy to construe the findings from this report as a total contradiction of Haugen's claims and a condemnation of integrity work's impact on the worst types of content in general. But that would be a misread, said Sahar Massachi, a former member of Facebook's integrity team and co-founder of the new Integrity Institute think tank. It's important to note, he said, that this document appears to be only looking at integrity protections that existed in News Feed rankings at the time, and doesn't take into account other integrity interventions that other teams at Facebook might have been working on.

It also only looks at the integrity interventions that the News Feed team had already deployed, not the full range of possible interventions that may have been proposed but were shot down. "Their view on what 'integrity' covers is likely scoped to whichever team they're on," Massachi said of the report's author. "I read this as: Integrity interventions that were allowed to ship — in the scoped set that this person considered — didn't affect views of that kind of content."

The report itself isn't clear on exactly what protections were being withheld from the holdouts, but a comment posted along with the document suggests that the experiment affected protections related to clickbait, ad farms, engagement bait and news trustworthiness, among other things. Given that fact, it shouldn't be all that surprising that exposure to graphic violence wasn't impacted by the experiment.

But what the report is calling attention to is the fact that, at the time at least, Facebook's integrity protections for News Feed weren't designed so they would capture more severe harms. The company had only begun demoting what it called "borderline" content that nearly violated its policies a few months before the report was published, and the rollout of those demotions was slow.

"This document says: We should expand the definitions more," said one current Facebook employee who has worked on News Feed ranking and reviewed the report. And according to that employee, the message stuck. "This person's argument was successful in that the program was expanded in various dimensions."

The employee said, however, that some of those expansions were rolled back before Facebook published a public list of content it demotes.

"The story of integrity is you try to do the good thing and you go to the execs, and they shoot you down, and you come back with something more conservative, and you realize you didn't do anything, so you try again," the employee said. "What you're seeing [in this document] is that middle part."

Facebook wouldn't comment on whether the company changed its demotions before publishing its list, but Pusateri said the demotions included on that list are still in place today.

Both Zhang and Massachi — as well as Facebook's own public relations team — cautioned Protocol not to cast Facebook's decision to withhold these protections at all as a scandal in and of itself. Measuring the effectiveness of these interventions, they said, is critical to strengthening them. As Massachi put it: "In the vaccine trials, some people have to get the placebo."

[Editor's note: Below, OCQ stands for "objective content quality," which refers to clickbait and ad-farm content. A high OCQ score means likely clickbait.]


A MESSAGE FROM QUALCOMM

www.protocol.com

Just as the power of the PC fueled the early leaps of the tech revolution and the accessibility of the web built on that, the smartphone and 5G networking technology will reshape our world with blazingly fast connected devices. Leading that charge is 5G, the high-speed next generation of mobile wireless connectivity that will connect virtually everyone and everything, including machines, objects and devices.

LEARN MORE

Fintech

Affirm CEO: 'Buy now, pay later' becomes more attractive in a slump

With consumers grappling with rising rates and prices, the question of whether they’ll still buy now and pay later is open. Max Levchin thinks Affirm knows the answer.

Affirm CEO Max Levchin spoke with Protocol about "buy now, pay later."

Photo: John Lamparski/Getty Images

Shortly after Affirm went public last year, CEO Max Levchin told Protocol that he saw “an ocean of opportunities” for the “buy now, pay later” pioneer. Wall Street agreed.

Affirm’s stock soared in its trading debut as the company blazed a trail for a fast-growing alternative to the credit cards that Levchin says consumers are increasingly rejecting.

Keep Reading Show less
Benjamin Pimentel

Benjamin Pimentel ( @benpimentel) covers crypto and fintech from San Francisco. He has reported on many of the biggest tech stories over the past 20 years for the San Francisco Chronicle, Dow Jones MarketWatch and Business Insider, from the dot-com crash, the rise of cloud computing, social networking and AI to the impact of the Great Recession and the COVID crisis on Silicon Valley and beyond. He can be reached at bpimentel@protocol.com or via Google Voice at (925) 307-9342.

Businesses are evolving, with current events and competition serving as the catalysts for technology adoption. Events from the pandemic to the ongoing war in Ukraine have exposed the fragility of global supply chains. The topic of sustainability is now on every board room agenda. Industries from manufacturing to retail and everything in between are exploring the latest innovations like process automation, machine learning and AI to identify potential safeguards against future disruption. But according to a recent survey from Boston Consulting Group, while 80% of companies are adopting digital solutions to navigate existing business challenges or opportunities like the ones mentioned, only about 30% successfully digitally transform their business.

For the last 50 years, SAP has worked closely with our customers to solve some of the world’s most intricate problems. We have also seen, and have been a part of, rapid accelerations in technology in response. Across industries, certain paths have emerged to help businesses manage the unexpected challenges over the last few years.

Keep Reading Show less
DJ Paoni

DJ Paoni is the President of SAP North America and is responsible for the strategy, day-to-day operations, and overall customer success in the United States and Canada. Dedicated to helping customers become best-run businesses, DJ has established himself as a trusted advisor who places a high priority on their success. He works with many of SAP North America's 155,000 customers and helps them adopt business and technology best practices across 25 different industries.

Workplace

The post-layoff playbook: How to avoid 'survivor's guilt'

Taking care of your laid-off employees is important. But how can you restore trust with the employees who make it through?

Employees who survive layoffs are charged with holding the company together. Whether or not managers listen to their concerns can make or break a company’s culture.

Photo: Justin Pumfrey/The Image Bank/Getty Images

Jennifer Burke was on her way to Hawaii for her daughter’s wedding when Zillow followed through on its long-anticipated layoff. She asked her manager to break the news to her by message in the car. You’re one of the safe ones, her manager responded.

“I felt relieved, of course,” Burke said. “I felt apprehensive. I felt sympathy for my co-workers that I knew were going to be laid off.”

Keep Reading Show less
Lizzy Lawrence

Lizzy Lawrence ( @LizzyLaw_) is a reporter at Protocol, covering tools and productivity in the workplace. She's a recent graduate of the University of Michigan, where she studied sociology and international studies. She served as editor in chief of The Michigan Daily, her school's independent newspaper. She's based in D.C., and can be reached at llawrence@protocol.com.

Enterprise

Why chip companies need the college students dazzled by software jobs

New chip fabricating plants will need tens of thousands of skilled workers who don’t currently exist. Training them means persuading students to look away from jobs at big tech companies.

Intel employees in clean room "bunny suits" work at Intel's D1X factory in Hillsboro, Oregon.

Photo: Intel Corporation

Every morning, Isaiah Morris drives his white Nissan Altima eight miles down Arizona state Route 101 to a sprawling, low-level office park in South Tempe. Inside one of the unassuming buildings adjacent to GoDaddy’s headquarters and a couple of Amazon offices, the Arizona State University student dons a lab coat, safety shoes and prescription goggles as he helps engineer chemicals for a chip manufacturing process called planarization.

Morris is an unusual 21-year-old. When they graduate college, many of his tech-minded peers will opt to work for the likes of Apple, Google and other household names that have enjoyed meteoric growth over the last decade. Jobs at those tech companies symbolize prestige for graduates and their parents in a way that careers with chipmakers like Intel do not.

Keep Reading Show less
Anna Kramer

Anna Kramer is a reporter at Protocol (Twitter: @ anna_c_kramer, email: akramer@protocol.com), where she writes about labor and workplace issues. Prior to joining the team, she covered tech and small business for the San Francisco Chronicle and privacy for Bloomberg Law. She is a recent graduate of Brown University, where she studied International Relations and Arabic and wrote her senior thesis about surveillance tools and technological development in the Middle East.

Policy

A new UK visa could steal your top tech talent

Without meaningful immigration reform, U.S.-trained foreign graduates could head across the pond.

The U.S. immigration system turns away hundreds of thousands of highly skilled tech workers every year.

Photo: Ben Fathers/AFP via Getty Images

Almost as soon as he took office, President Biden began the work of undoing a lot of the damage the Trump administration did to the U.S. H-1B visa program. He allowed a Trump-era ban on entry by H-1B holders to expire and withdrew a Trump proposal to prohibit H-1B visa holders’ spouses from working in the U.S. More recently, his administration has expanded the number of degrees considered eligible for special STEM OPT visas.

But the U.S. immigration system still turns away hundreds of thousands of highly skilled — and in many cases U.S.-educated — tech workers every year. Now the U.K. is trying to capitalize on the United States’ failure to reform its policy regarding high-skilled immigrants with a new visa that could poach American-trained tech talent across the pond. And there’s good reason to believe it could work.

Keep Reading Show less
Kwasi Gyamfi Asiedu

Kwasi (kway-see) is a fellow at Protocol with an interest in tech policy and climate. Previously, he covered global religion news at the Associated Press in New York. Before that, he was a freelance journalist based out of Accra, Ghana, covering social justice, health, and environment stories. His reporting has been published in The New York Times, Quartz, CNN, The Guardian, and Public Radio International. He can be reached at kasiedu@protocol.com.

Latest Stories
Bulletins