When YouTube sent content moderators home in March due to the COVID-19 pandemic, it dramatically expanded its use of automated filters — and that led to twice as many videos being taken down in the second quarter of 2020 as the first. The spike stems from YouTube's decision to "cast a wider net" for potentially violative videos in the absence of human moderators, and highlights the imperfect science of automatically policing content.
"When reckoning with greatly reduced human review capacity due to COVID-19, we were forced to make a choice between potential under-enforcement or potential over-enforcement," the company wrote in a blog post accompanying its second quarter transparency report. "Because responsibility is our top priority, we chose the latter — using technology to help with some of the work normally done by reviewers."
YouTube removed more content last quarter in all but two categories: hateful videos and videos that encourage harmful or dangerous activities. But in the most sensitive content categories, including violent extremist content and content that could jeopardize child safety, YouTube saw a threefold increase in the number of videos it removed. YouTube explained that's because the company "accepted a lower level of accuracy to make sure that we were removing as many pieces of violative content as possible."
This means, of course, that YouTube removed plenty of videos that didn't actually violate its policies, leading to roughly double the number of appeals, from around 166,000 last quarter to 325,000 in the second quarter. The number of videos that were reinstated after appeal also nearly quadrupled, from around 41,000 in the first quarter to 161,000 last quarter.
YouTube's transparency report comes on the heels of a similar report from Facebook, which described markedly different results. Like YouTube, Facebook also opted to send its content moderators home in March. But unlike YouTube, which removed more content last quarter in almost every category, Facebook and Instagram saw steep declines — including in some of the most sensitive categories it polices.
On Instagram, for example, the company removed about half as much child sexual abuse material in the second quarter as it did in the first, while removals of suicide related content fell by a whopping 79%. That's not because there was less of it: According to Facebook, it's because moderators were unable to review this graphic imagery at home, and therefore couldn't log it in Facebook's automated systems, which is how the company is able to search the platform and remove other exact matches that pop up in the future. That means much of that content that would normally be removed was left online.
A YouTube spokesperson said the company ran into the same problem. But it compensated for that issue by removing far more content overall than it otherwise would have. Anticipating that would lead to a spike in appeals — and it did — YouTube maintained a skeleton crew to process appeals in a timely fashion, eventually reinstating about half of the videos that were removed.
Facebook, by contrast, scaled back appeals, suspending them entirely in some sensitive content categories, like violent and graphic content. That led to a massive drop in appeals and the amount of content that was restored after removal in almost every category. "We couldn't always offer [appeals]," Facebook's vice president of integrity Guy Rosen said on a call with reporters earlier this month. "We still gave people an option to tell us that they disagreed with our decision on a piece of content and our teams looked at these signals in aggregate to find potential issues and restore content where appropriate."
The comparison between Facebook and YouTube isn't exact. For one thing, YouTube doesn't report as much granular information as Facebook does: While Facebook shares stats on the amount of child nudity and sexual exploitation content it removes, for example, YouTube shares information more broadly on child safety, a category that also includes risky challenges and videos that could broadly "endanger minors." For another, Facebook did see a much bigger jump in the amount of hate speech it removed last quarter, compared to YouTube.
And yet, the two reports still illustrate an important point about how the COVID-19 era has affected what people see — and don't — online. Facebook and YouTube often get lumped together as two social networks filled with the same filth, both using a combination of AI and low-wage contractors to rid problematic posts from their platforms. But over the last six months, these two companies have taken two different approaches to the same problem, and they have yielded dramatically different outcomes.
Where YouTube has risked silencing users who have done nothing wrong, Facebook has risked not silencing them fast enough in the name of maintaining accuracy. Neither approach is perfect. Both show just how far automated systems still have to go.
