Protocol | Policy

Here’s what happened when Facebook stopped protecting users — on purpose

Internal documents reveal the impact of withholding certain integrity protections from a subset of Facebook users.

Facebook Papers: an illustration of the Facebook logo cracking

Frances Haugen said Facebook withholds certain protections from a subset of users to see how they'll react.

Image: Protocol

In her testimony before Congress last month, whistleblower Frances Haugen told lawmakers Facebook has conducted experiments where it withholds certain protections from a subset of users to see how they'll react. Facebook refers to this experimental group internally, she said, as "integrity holdouts."

"These are people who don't get protections from integrity systems to see what happens to them," Haugen said. "And those people who deal with a more toxic, painful version of Facebook use Facebook less."

Internal documents reveal a more complex story. According to one internal report from April 2019, Facebook has studied the impact of removing some protections against problematic content like clickbait and untrustworthy news for some users, but the results, at least in that report, were decidedly mixed.

The report showed that during one March 2019 test, when the company rolled back some protections from millions of users' News Feeds, their exposure to some of the worst forms of harmful content, like graphic violence, barely changed. As the report's author wrote, referring to the company's news feed protections at the time, "We are likely having little (if any) impact on violence."

The report also suggested that far from using Facebook less, integrity holdouts actually commented more and had more sessions on the app. "Given that Integrity is changing a ranking algorithm that is optimized for engagement, it is not surprising that integrity has some negative engagement impact," the report read.

A spokesperson for Haugen said that other documents she collected showed that retention is stronger among regular users than holdouts, but those findings were not included in this report.

The report was included in disclosures made to the Securities and Exchange Commission and provided to Congress in redacted form by Frances Haugen's legal counsel. A consortium of news organizations, including Protocol, has reviewed the redacted versions received by Congress. They offer a glimpse at how Facebook has analyzed the efficacy of its user protections — and weighed them against their impact on other company priorities like growth.

"Testing product and safety features is an important part of improving our platform and something that's standard in tech and many other industries," Facebook spokesperson Drew Pusateri told Protocol, noting that the holdout affected about 2% of Facebook users. "It helps us build the tools to reduce the prevalence of hate speech and other types of problematic content on our platform."

'High-harm spaces'

Facebook conducts holdout experiments for a range of business goals, not just its integrity work. Holdouts are effectively control groups that Facebook can compare to its larger pool of users. As former Facebook data scientist Sophie Zhang recently told Protocol, Facebook has also studied the impact of withholding ads from users. "The company wants to know the very long-term impacts of advertising on retention and usage for Facebook," Zhang said. "The argument was usually that we need to know what the impact of this is. We need to know if people like it or not. But this is also motivated by wanting to know the impact for growth."

By early 2019, it appears, the company had begun applying this approach to integrity protections. The report, published in late April 2019, detailed the initial findings from an experiment that tinkered with certain integrity protections in News Feed. Some of the findings were encouraging: The report showed, for instance, that regular users got substantially less clickbait and ad-farm content than the holdouts did, something the author notes is "not surprising," given that the company was demoting clickbait and ad farms "quite a bit."

The report showed that regular users' exposure to what the company considers "low quality news" was down about 18% compared to the holdouts. The company also found it was boosting content from major publishers more when integrity protections were in place. Holdouts, by contrast, were more likely to click the dislike button and report the posts they saw, and they were also more likely to see content from public pages than regular users were.

But the main takeaway from the report, the author wrote, was that the News Feed protections that were in place at the time weren't having an equally significant effect on more severe types of harm, like graphic violence. "I believe strongly that this needs to change," the author wrote.

During the experiment, the company continued to demote hate speech and graphic violence in at-risk countries, the report said. But for holdouts who weren't in at-risk countries, those demotions didn't exist. And yet, the report found no impact on regular users' exposure to violence compared to the holdouts.

"11% of users see content that has been marked as disturbing every day; 16% of users see content that is likely to be bullying; 39% of users see hateful content (i.e. borderline hate); 32% of users see borderline 3+ nudity content," the author wrote. "These are significant proportions of [daily active users] and we have effectively no ranking interventions in place to mitigate this." The author added, however, that those particular numbers "should be taken with a grain of salt," as measuring bad experiences on the platform was still a work in progress.

The report also made no secret of the negative impact of News Feed integrity protections on engagement. "By definition, Integrity is going to cause some engagement regression," the author wrote, noting that there are "tradeoffs between Integrity and Engagement."

Integrity efforts, the report found, were a blow to the company's "meaningful social interactions" metric, which emphasizes interactions between friends over public-facing content. One reason for that, the author proposed, was that holdouts commented more than regular users did. While regular users were more likely to like posts on Facebook compared to holdouts, the author wrote, it was "not enough to make up for the decline in comments." The report also showed that content views and time spent on the app were down slightly among regular users compared to holdouts.

The report's limitations

It would be easy to construe the findings from this report as a total contradiction of Haugen's claims and a condemnation of integrity work's impact on the worst types of content in general. But that would be a misread, said Sahar Massachi, a former member of Facebook's integrity team and co-founder of the new Integrity Institute think tank. It's important to note, he said, that this document appears to be only looking at integrity protections that existed in News Feed rankings at the time, and doesn't take into account other integrity interventions that other teams at Facebook might have been working on.

It also only looks at the integrity interventions that the News Feed team had already deployed, not the full range of possible interventions that may have been proposed but were shot down. "Their view on what 'integrity' covers is likely scoped to whichever team they're on," Massachi said of the report's author. "I read this as: Integrity interventions that were allowed to ship — in the scoped set that this person considered — didn't affect views of that kind of content."

The report itself isn't clear on exactly what protections were being withheld from the holdouts, but a comment posted along with the document suggests that the experiment affected protections related to clickbait, ad farms, engagement bait and news trustworthiness, among other things. Given that fact, it shouldn't be all that surprising that exposure to graphic violence wasn't impacted by the experiment.

But what the report is calling attention to is the fact that, at the time at least, Facebook's integrity protections for News Feed weren't designed so they would capture more severe harms. The company had only begun demoting what it called "borderline" content that nearly violated its policies a few months before the report was published, and the rollout of those demotions was slow.

"This document says: We should expand the definitions more," said one current Facebook employee who has worked on News Feed ranking and reviewed the report. And according to that employee, the message stuck. "This person's argument was successful in that the program was expanded in various dimensions."

The employee said, however, that some of those expansions were rolled back before Facebook published a public list of content it demotes.

"The story of integrity is you try to do the good thing and you go to the execs, and they shoot you down, and you come back with something more conservative, and you realize you didn't do anything, so you try again," the employee said. "What you're seeing [in this document] is that middle part."

Facebook wouldn't comment on whether the company changed its demotions before publishing its list, but Pusateri said the demotions included on that list are still in place today.

Both Zhang and Massachi — as well as Facebook's own public relations team — cautioned Protocol not to cast Facebook's decision to withhold these protections at all as a scandal in and of itself. Measuring the effectiveness of these interventions, they said, is critical to strengthening them. As Massachi put it: "In the vaccine trials, some people have to get the placebo."

[Editor's note: Below, OCQ stands for "objective content quality," which refers to clickbait and ad-farm content. A high OCQ score means likely clickbait.]


A MESSAGE FROM QUALCOMM

www.protocol.com

Just as the power of the PC fueled the early leaps of the tech revolution and the accessibility of the web built on that, the smartphone and 5G networking technology will reshape our world with blazingly fast connected devices. Leading that charge is 5G, the high-speed next generation of mobile wireless connectivity that will connect virtually everyone and everything, including machines, objects and devices.

LEARN MORE

Protocol | Workplace

Google contractor says she was fired for "ungoogley" behavior

According to a charge filed with the National Labor Relations Board, "ungoogley" is Google's term for having a bad attitude.

A contractor at Google staffing firm Modis claims she was fired from her job after asking about pay.

Photo: Future Publishing/Getty Images

A contractor at Google staffing firm Modis claims she was fired from her job for "ungoogley" behavior after asking about holiday pay at a meeting with management, according to a charge filed with the National Labor Relations Board by a lawyer for the Alphabet Workers Union.

Tuesday Carne said in an interview with Protocol that she was fired after just nine days of working in the data contracting facility in South Carolina. Carne's termination letter (which Protocol reviewed) called her behavior at the meeting "unacceptable and 'ungoogley'" and claimed that her behavior was the reason for her firing.

Keep Reading Show less
Anna Kramer

Anna Kramer is a reporter at Protocol (Twitter: @ anna_c_kramer, email: akramer@protocol.com), where she writes about labor and workplace issues. Prior to joining the team, she covered tech and small business for the San Francisco Chronicle and privacy for Bloomberg Law. She is a recent graduate of Brown University, where she studied International Relations and Arabic and wrote her senior thesis about surveillance tools and technological development in the Middle East.

The fintech developers who made mobile banking as routine as texting or online shopping aren't done. The next frontier for innovation is open banking – fintech builders are enabling consumers to be at the center of where and how their data is used to provide the services they want and need.

Most people don't even realize they're using open banking services today. If they connected their investment and banking accounts in a personal financial management solution or app, they're using open banking. Perhaps they've seen ads about how they can improve their credit score by uploading pay stubs or utility records to that same app – this is also powered by open banking.

Keep Reading Show less
Bob Schukai
Bob Schukai is Executive Vice President of Technology Development, New Digital Infrastructure & Fintech at Mastercard, where he leads the technical design, execution and support of innovative open banking and fintech solutions, as well as next generation technologies to support global payment and data capabilities. Prior to Mastercard, Schukai’s work focused on cognitive computing, financial technology, blockchain, user experience and digital identity. He is also a member of the Institute for Electrical and Electronics Engineers.
Protocol | Policy

Biden FCC nominee Sohn is walking a tightrope with Republicans

Gigi Sohn faces plenty of GOP opposition, but the longtime net-neutrality advocate is hoping to pick up a little Republican support as she deals with Democrats’ narrow margins.

Gigi Sohn’s work for net neutrality has become an issue in her confirmation hearings for the FCC.

Photo: Alex Wong/Getty Images

Gigi Sohn wouldn’t mind getting support from a Republican or two, and it’d certainly make her path back to the Federal Communications Commission easier.

During her Senate Commerce Committee confirmation on Wednesday, Sohn, a progressive favorite and longtime net-neutrality advocate, touted her commitment to ensuring a diversity of voices on the airwaves, her past fights for small conservative networks she personally disagrees with and her habit of socializing with those she battles on policy.

Keep Reading Show less
Ben Brody

Ben Brody (@ BenBrodyDC) is a senior reporter at Protocol focusing on how Congress, courts and agencies affect the online world we live in. He formerly covered tech policy and lobbying (including antitrust, Section 230 and privacy) at Bloomberg News, where he previously reported on the influence industry, government ethics and the 2016 presidential election. Before that, Ben covered business news at CNNMoney and AdAge, and all manner of stories in and around New York. He still loves appearing on the New York news radio he grew up with.

Protocol | Workplace

Microsoft Teams is going after small businesses

Microsoft Teams Essentials offers longer, bigger meetings for a relatively small price tag.

Companies can now buy a standalone version of Teams.

Photo: Mika Baumeister/Unsplash

Microsoft announced Wednesday that companies can now buy a standalone version of Teams — one of its most important products and a major player in work messaging and video chat, alongside Slack and Zoom. The product, called Microsoft Teams Essentials, aims to give small or medium-sized businesses a communication hub that costs less than its competitors'.

Microsoft will charge small businesses $4 per user per month for Microsoft Teams Essentials, while Zoom’s cheapest paid plan is $14.99 per user per month and Slack’s is $6.67 per user each month, when billed annually. The free version of Microsoft Teams still exists, as do the various other Microsoft 365 plans that include Teams. Teams Essentials offers longer meeting times, larger group meetings and more cloud storage.

Keep Reading Show less
Lizzy Lawrence

Lizzy Lawrence ( @LizzyLaw_) is a reporter at Protocol, covering tools and productivity in the workplace. She's a recent graduate of the University of Michigan, where she studied sociology and international studies. She served as editor in chief of The Michigan Daily, her school's independent newspaper. She's based in D.C., and can be reached at llawrence@protocol.com.

Protocol | Policy

5 things to know about NTIA nominee Alan Davidson

If confirmed, the former Googler will play a key role in shaping the unprecedented expansion of broadband across the country.

Alan Davidson has been nominated to lead the NTIA.

Photo: Win McNamee/Getty Images

On Wednesday, the Senate Commerce Committee is holding a hearing to confirm President Joe Biden’s nominee to lead the National Telecommunications and Information Administration — a traditionally somewhat-sleepy role that is taking on new prominence in the wake of the passage of the bipartisan infrastructure bill.

That law gives the NTIA authority to write the rules and oversee the distribution of $42.5 billion in broadband infrastructure grants to states, a duty that will require it to massively scale its internal resources. To lead the charge, Biden has nominated Alan Davidson, a well-known figure in Washington who has spent his career cycling through government, industry and advocacy groups. If confirmed, Davidson would have perhaps the most important role in guiding an unprecedented expansion of internet access in America.

Keep Reading Show less
Issie Lapowsky

Issie Lapowsky ( @issielapowsky) is Protocol's chief correspondent, covering the intersection of technology, politics, and national affairs. She also oversees Protocol's fellowship program. Previously, she was a senior writer at Wired, where she covered the 2016 election and the Facebook beat in its aftermath. Prior to that, Issie worked as a staff writer for Inc. magazine, writing about small business and entrepreneurship. She has also worked as an on-air contributor for CBS News and taught a graduate-level course at New York University's Center for Publishing on how tech giants have affected publishing.

Latest Stories