Power

How COVID-19 helped — and hurt — Facebook’s fight against bad content

The amount of child sexual abuse material Instagram caught and removed fell dramatically, while hate speech removals on Facebook and Instagram grew.

Mark Zuckerberg looking sad

Facebook's data shows that the pandemic made content-moderation systems both better and worse.

Photo: Drew Angerer/Getty Images

When Facebook sent home its content moderators in March due to the COVID-19 pandemic, announcing it would rely on automation to at least temporarily do their job, the company predicted the decision would have a major impact on its ability to find and remove content that violates its policies. Now, according to newly released data from Facebook, we know just how big an impact it had.

During the second quarter of 2020 the company removed less than half of the child sexual abuse material from Instagram that it did the quarter before — not because there was less of it, but because the company was less equipped to catch it. And on both Facebook and Instagram, the amount of suicide and self-injury content it removed dropped precipitously too. On Instagram, it fell from 1.3 million pieces of suicide content removed last quarter to just 275,000 pieces this quarter.

But in other categories, like hate speech, Facebook's new reliance on automated systems actually led to a drastic increase in removals, from just 9.6 million pieces of hate speech removed from Facebook in the beginning of 2020 to 22.5 million pieces removed between April and June.

The drop in the removal of child sexual abuse material from Instagram wasn't due to a decrease in the amount of it on the platform. Neither was the decrease in takedowns of suicide related content. It was due to the limited number of human beings who were available to look at those posts, since, initially at least, they were all working from home. "The reason this content is challenging is because it's graphic content that, at home, is very hard for people to moderate," said Guy Rosen, Facebook's vice president of integrity. "We want to be very careful with the environment that people have in order to look at that content."

It's not that the human reviewers are required to spot all child sexual abuse material. Automated systems are already responsible for removing 97.5% of those types of posts that appear on Facebook. But according to Facebook spokesperson Emily Cain, human reviewers are critical when it comes to "banking" child sexual abuse material. That is, taking known images and logging them so that Facebook's AI systems can then go find and remove them.

"Without humans banking this content then our machines can't find it at scale," Cain said. "And this compounds after a while, so our content-actioned numbers decreased."

"Overall, this pandemic and this situation really reinforced to us that it is always people and technology working together," Rosen said on a call with reporters Tuesday. "We always need people who look and measure and help tune our automation to ensure that we're always up to speed and always up to date with how content is evolving."

The decrease in content removal is a blow to Facebook's ongoing efforts to fight the spread of child sexual abuse material on the platform at a time when the National Center for Missing and Exploited Children says that it's seeing an exponential increase in the number of reports about child exploitation. That said, the company did manage to remove more pieces of child sexual abuse material from the Facebook app than it did last quarter. And yet, overall, in 2020, removals in that category are down significantly from where they were at the end of last year.

During the COVID-19 crisis, Rosen said Facebook has developed a ranking system to prioritize the most critical content in these sensitive categories. That might include anything from a live video to a post in which someone indicates they plan to harm themselves imminently. This ranking system was already in the works before COVID-19, but Rosen said the company expedited its development in response to the crisis.

"This enables our teams to spend their time on the cases where we need their expertise the most, and it means there will be a shift towards more content being initially actioned by our automated systems," Rosen said.

As for the sharp increase in the amount of hate speech being removed from the platform, Rosen attributed that, too, to the ongoing development of Facebook's AI systems. Because hate speech is less graphic than, say, a video of child abuse, moderators are more able to handle that content remotely. As Facebook's chief technology officer Mike Schroepfer told Protocol in a tweet, "The more … sensitive and nuanced the content the more we need help from people."

Of course, the perennial question about hate speech, child sexual abuse material and other types of problematic content is not just how much Facebook is taking down and how fast, but how prevalent that content is to begin with. On the subject of hate speech, that's a question that Facebook hasn't been able to answer yet. Turns out, measuring prevalence requires a lot of human input, too.

Protocol | Workplace

Instacart workers are on strike. How far can it get them?

Instacart activists want a nationwide strike to start today, but many workers are too afraid of the company and feel they can't afford a day off of work.

Gig workers protest in front of an Amazon facility in 2020.

Photo: Michael Nagle/Bloomberg via Getty Images

Starting today, an Instacart organizing group is asking the app's gig workers to go on a nationwide strike to demand better payment structures, benefits and other changes to the way the company treats its workers — but if past strikes are any indication, most Instacart users probably won't even notice.

The majority of Instacart workers on forums like Reddit and Facebook appear either unaware of the planned strike or don't plan to participate because they are skeptical of its power, afraid of retaliation from the company or are too reliant on what they do make from the app to be able to afford to take even one day off of the platform. "Not unless someone is going to pay my bills," "It will never work, you will never be able to get every shopper to organize" and "Last time there was a 'strike' Instacart took away our quality bonus pay," are just a few of the comments Instacart shoppers have left in response to news of the strike.

Keep Reading Show less
Anna Kramer

Anna Kramer is a reporter at Protocol (Twitter: @ anna_c_kramer, email: akramer@protocol.com), where she writes about labor and workplace issues. Prior to joining the team, she covered tech and small business for the San Francisco Chronicle and privacy for Bloomberg Law. She is a recent graduate of Brown University, where she studied International Relations and Arabic and wrote her senior thesis about surveillance tools and technological development in the Middle East.

The way we work has fundamentally changed. COVID-19 upended business dealings and office work processes, putting into hyperdrive a move towards digital collaboration platforms that allow teams to streamline processes and communicate from anywhere. According to the International Data Corporation, the revenue for worldwide collaboration applications increased 32.9 percent from 2019 to 2020, reaching $22.6 billion; it's expected to become a $50.7 billion industry by 2025.

"While consumers and early adopter businesses had widely embraced collaborative applications prior to the pandemic, the market saw five years' worth of new users in the first six months of 2020," said Wayne Kurtzman, research director of social and collaboration at IDC. "This has cemented collaboration, at least to some extent, for every business, large and small."

Keep Reading Show less
Kate Silver

Kate Silver is an award-winning reporter and editor with 15-plus years of journalism experience. Based in Chicago, she specializes in feature and business reporting. Kate's reporting has appeared in the Washington Post, The Chicago Tribune, The Atlantic's CityLab, Atlas Obscura, The Telegraph and many other outlets.

Protocol | China

WeChat promises to stop accessing users’ photo albums amid public outcry

A tech blogger claimed that popular Chinese apps snoop around users' photo libraries, provoking heightened public concerns over privacy.

A survey launched by Sina Tech shows 94% of the some 30,000 responding users said they are not comfortable with apps reading their photo libraries just to allow them to share images faster in chats.

Photo: S3studio via Getty Images

A Chinese tech blogger dropped a bombshell last Friday, claiming on Chinese media that he found that several popular Chinese apps, including the Tencent-owned chat apps WeChat and QQ, as well as the Alibaba-owned ecommerce app Taobao, frequently access iPhone users' photo albums in the background even when those apps are not in use.

The original Weibo post from the tech blogger, using the handle of @Hackl0us, provoked intense debates about user privacy on the Chinese internet and consequently prompted WeChat to announce that it would stop fetching users' photo album data in the background.

Keep Reading Show less
Shen Lu

Shen Lu is a reporter with Protocol | China. Her writing has appeared in Foreign Policy, The New York Times and POLITICO, among other publications. She can be reached at shenlu@protocol.com.

Protocol | Enterprise

As businesses struggle with data, enterprise tech is cleaning up

Enterprise tech's vision of "big data" largely fell flat inside silos. But now, an army of providers think they've figured out the problems. And customers and investors are taking note.

Corporate data tends to settle in silos that makes it harder to understand the bigger picture. Enterprise tech vendors smell a lucrative opportunity.

Photo: Jim Witkowski/Unsplash

Data isn't the new oil; it's the new gold. And in any gold rush, the ones who make the most money in the long run are the tool makers and suppliers.

Enterprise tech vendors have long peddled a vision of corporate America centered around so-called "big data." But there was a big problem: Many of those projects failed to produce a return. An army of new providers think they've finally figured out the problem, and investors and customers are taking note.

Keep Reading Show less
Joe Williams

Joe Williams is a senior reporter at Protocol covering enterprise software, including industry giants like Salesforce, Microsoft, IBM and Oracle. He previously covered emerging technology for Business Insider. Joe can be reached at JWilliams@Protocol.com. To share information confidentially, he can also be contacted on a non-work device via Signal (+1-309-265-6120) or JPW53189@protonmail.com.

Protocol | Policy

What Frances Haugen’s SEC complaint means for the rest of tech

Haugen argues Facebook misled investors by failing to disclose its platforms' harms. If the SEC bites, the rest of tech could be next.

The question is whether the SEC will find the contents of Haugen's complaint relevant to investors' interests.

Photo: Matt McClain-Pool/Getty Images

Whistleblowers like former Facebook staffer Frances Haugen have pretty limited options when it comes to actually seeking redress for the harms they've observed and documented. There's no federal privacy law in the U.S. to speak of, Section 230 protects platforms for online speech and companies like Facebook are under no obligation to share any information with lawmakers, or anyone else, about what's happening on their sites.

But there is one agency that not only governs all publicly-traded companies, including in tech, but also offers whistleblowers like Haugen the opportunity for a payout: the Securities and Exchange Commission.

Keep Reading Show less
Issie Lapowsky

Issie Lapowsky ( @issielapowsky) is Protocol's chief correspondent, covering the intersection of technology, politics, and national affairs. She also oversees Protocol's fellowship program. Previously, she was a senior writer at Wired, where she covered the 2016 election and the Facebook beat in its aftermath. Prior to that, Issie worked as a staff writer for Inc. magazine, writing about small business and entrepreneurship. She has also worked as an on-air contributor for CBS News and taught a graduate-level course at New York University's Center for Publishing on how tech giants have affected publishing.

Latest Stories