Protocol | Policy

Can Twitter warnings actually curb hate speech? A new study says yes.

Researchers found that warning Twitter users that someone they follow has been suspended — and they could be next — cuts down on hate speech.

A sign on a fence that reads: "WARNING: DO NOT FEED THE TROLLS"

A new study from NYU adds to the evidence that giving users warnings about hate speech can actually cut down their use of hate speech by 10-20%.

Image: Gwengoat via Getty Images

Twitter in May said it would begin prompting users who are about to tweet something nasty to either revise or delete the message before sending. The decision, the company said at the time, was based on a successful test of the messages in the run-up to the 2020 election.

Now, a new study — this one from researchers at New York University — adds to the evidence that giving users warnings about hate speech can actually cut down their use of hate speech by 10-20%. And those warnings can change users' behavior even when users aren't in the heat of the moment and about to tweet something regrettable.

The researchers at NYU's Center for Social Media and Politics developed their experiment last summer, in response to what was beginning to look like a mass migration of Twitter users to more extreme platforms like Parler. "We wanted to find a way that would basically prevent them from migrating to these platforms, but at the same time, that would result in the reduction of hate speech," said Mustafa Mikdat Yildirim, a PhD. candidate in NYU's department of politics and the lead researcher on the report.

So, last July, as racial justice protests were swelling, anti-Asian sentiment was filling social media and conservatives like Sen. Ted Cruz were threatening to abandon Twitter, the NYU researchers began monitoring a subset of 600,000 tweets and scanning for users who they thought might soon be suspended for hate speech. Eventually, the researchers whittled their list down to users who did get suspended and also met certain other criteria, including having more than 50 followers and having at least 7 followers who have also used hateful language in their tweets.

Then the researchers trained their attention on the people who followed those suspended accounts. They wanted to know whether warning these people that someone they followed had been suspended for hate speech — and that they could be next — would change the way those people behaved.

The researchers ended up with a list of 27 suspended users with 4,327 followers among them, and divided the followers up into six experimental groups and one control. The researchers then set up their own Twitter accounts with names like @hate_suspension and @expert_on_hate and began publicly tweeting directly at the users in all six groups with one of six different warning messages. They wanted to see which approach, if any, was most effective.

Two of the groups got messages designed to remind people of what they could lose if they used hate speech. Another two received tweets that emphasized "legitimacy," which more or less meant respectfulness. The last two groups got messages that framed the sender as an expert to lend credibility to the message. The messages came in two different flavors — high intensity and low intensity. The control group, meanwhile, received no warning at all.

A table showing a selection of sample tweets sent from the researchers' accounts to users selected for participation in the study, including language such as, "The user @account you follow was suspended, and I suspect this was because of hateful langauge. If you continue to use hate speech, you might get suspended." Image: NYU Center for Social Media and Politics

The researchers found that just one warning reduced the use of hateful language by 10% a week after the experiment. For the most effective message — which was also the most politely worded — the change was more like 15-20% a week later.

The fact that all of the messages had similar degrees of impact suggested to the researchers that simply receiving a warning may have had more of an impact than what the particular warning said. "Knowing that someone else sees their hate speech [...] may make people think once more about the language that they used," Yildirim said.

The NYU researchers' findings build on Twitter's own results from last year's experiment. The company found that when users were prompted to revise or delete a harmful tweet before sending it, a whopping 34% of them actually did. And in the future, Twitter said, those users sent 11% fewer offensive replies than they'd sent before.

"Our teams are reviewing the report and its findings," a Twitter spokesperson said of the NYU research. "Broadly, over the past year, we've taken an iterative approach to our work, from encouraging people to more thoughtfully consider sharing content to taking several measures to slow down the spread of misinformation. We'll continue that iterative approach and look forward to building on our efforts with a variety of third-party partners on this critical work."

The NYU report suggests that an even more proactive intervention — warning users even when they're not on the cusp of saying something rotten — could have a significant effect too. And yet, the researchers aren't urging Twitter to adopt their method wholesale.

The NYU researchers didn't see evidence to detect that their warnings, which were coming from accounts with less than 100 followers, might prompt people to send even more hateful tweets out of spite. But they acknowledge that it might be different if Twitter was sending the message itself. "We don't really know whether people would actually come back at Twitter with some type of backlash," Yildirim said.

It would also be tricky for Twitter to automatically send these warnings to users who follow accounts that have been suspended. Some suspensions, after all, happen by mistake and then get reversed.

Yildirim said it will be important for Twitter to test this type of system itself and be transparent about its findings. The fact is, no one is quite as equipped as Twitter to implement such a widespread intervention on the platform. Civil society groups could take up the task, but they don't have all of Twitter's data or its technical resources at their disposal. Yildirim has been encouraged by Twitter's transparency with its experiments in creating healthier conversations so far.

Twitter is not the only company experimenting with warnings and other kinds of "friction." Facebook has also been ramping up its use of information labels and interstitials, but Facebook, Yildirim said, is far harder to study.

All of these companies are leaning into this strategy as a way to avoid having to take more drastic action, like removing content or suspending entire accounts. The NYU approach offers yet another option, using those suspensions as a cautionary example for the users who are left.

Elizabeth Holmes says Sunny Balwani abused her

On the stand in her fraud trial, the former Theranos CEO alleged sexual and emotional abuse by her former professional and personal partner.

On the fourth day of her testimony, Elizabeth Holmes broke down for the first time as she discussed her relationship with Sunny Balwani.

Photo: Michael Short/Bloomberg via Getty Images

This story contains mention of sexual assault.

Elizabeth Holmes has blamed many people along the way for the swift downfall of her blood-testing startup, Theranos. But on the fourth day of her testimony in the fraud case against her, she pointed the finger at the man who was once her close professional and personal partner: Sunny Balwani.

Keep Reading Show less
Nat Rubio-Licht
Nat Rubio-Licht is a Los Angeles-based news writer at Protocol. They graduated from Syracuse University with a degree in newspaper and online journalism in May 2020. Prior to joining the team, they worked at the Los Angeles Business Journal as a technology and aerospace reporter.

The Bureau of Labor Statistics indicates that by 2026, the shortage of engineers in the U.S. will exceed 1.2 million, while 545,000 software developers will have left the market by that time. Meanwhile, business is becoming increasingly more digital-first, and teams need the tools in place to keep distributed teams aligned and able to respond quickly to changing business needs. That means businesses need to build powerful workplace applications without relying on developers.

In fact, according to Gartner, by 2025, 70% of new applications developed by enterprises will use low-code or no-code technologies and, by 2023, there will be at least four times as many active citizen developers as professional developers at large enterprises. We're on the cusp of a big shift in how businesses operate and how organization wide innovation happens.

Keep Reading Show less
Andrew Ofstad
As Airtable’s co-founder, Andrew spearheads Airtable’s long-term product bets and represents the voice of the customer in major product decisions. After co-founding the company, he helped scale Airtable’s original product and engineering teams. He previously led the redesign of Google's flagship Maps product, and before that was a product manager for Android.

Who is Parag Agrawal, Twitter’s new CEO?

The main thing you need to know: He’s an engineer’s engineer.

Twitter’s new CEO is its current chief technology officer, Parag Agrawal.

Photo: Twitter

When Parag Agrawal was at Stanford writing his computer science thesis, his adviser couldn’t imagine that any of her students would become the CEO of one of the world’s most powerful social media companies.

But much has changed since Agrawal graduated with his doctorate in 2012. On Monday morning, Twitter announced that Jack Dorsey had resigned and that Chief Technology Officer Agrawal had been promoted to CEO, effective immediately.

Keep Reading Show less
Anna Kramer

Anna Kramer is a reporter at Protocol (Twitter: @ anna_c_kramer, email: akramer@protocol.com), where she writes about labor and workplace issues. Prior to joining the team, she covered tech and small business for the San Francisco Chronicle and privacy for Bloomberg Law. She is a recent graduate of Brown University, where she studied International Relations and Arabic and wrote her senior thesis about surveillance tools and technological development in the Middle East.

Protocol | Policy

Jack Dorsey and breaking up the cult of the founder

Dorsey’s farewell note is a warning shot to all founder CEOs … especially you-know-who.

“There aren’t many companies that get to this level. And there aren’t many founders that choose their company over their own ego.”

Photo: Getty Images

In his note Monday announcing his departure from Twitter, Jack Dorsey delivered a warm welcome to the company’s new CEO, a fond farewell to the tweeps he’s leaving behind and a quick shout-out to his mom.

He also fired a warning shot at certain other founder-CEOs who shall remain nameless.

Keep Reading Show less
Issie Lapowsky

Issie Lapowsky ( @issielapowsky) is Protocol's chief correspondent, covering the intersection of technology, politics, and national affairs. She also oversees Protocol's fellowship program. Previously, she was a senior writer at Wired, where she covered the 2016 election and the Facebook beat in its aftermath. Prior to that, Issie worked as a staff writer for Inc. magazine, writing about small business and entrepreneurship. She has also worked as an on-air contributor for CBS News and taught a graduate-level course at New York University's Center for Publishing on how tech giants have affected publishing.

Protocol | Fintech

Twitter isn’t part of Jack Dorsey’s big bet on crypto

Bitcoin unleashed a huge wave, and Dorsey — no longer doing double duty at Twitter and Square — wants to ride it.

There’s still time for Square to expand its crypto footprint, though, which makes the timing of Dorsey’s move significant.

Photo: Joe Raedle/Getty Images

Jack Dorsey’s sudden exit from Twitter underlines the tech pioneer’s growing fixation with crypto — a passion that has forced a sudden resolution of the odd situation of a single individual leading two large tech companies.

It’s now clear that Square is Dorsey’s favorite child and needs all of his attention to advance the role it could play in popularizing bitcoin, the best-known cryptocurrency.

Keep Reading Show less
Benjamin Pimentel

Benjamin Pimentel ( @benpimentel) covers fintech from San Francisco. He has reported on many of the biggest tech stories over the past 20 years for the San Francisco Chronicle, Dow Jones MarketWatch and Business Insider, from the dot-com crash, the rise of cloud computing, social networking and AI to the impact of the Great Recession and the COVID crisis on Silicon Valley and beyond. He can be reached at bpimentel@protocol.com or via Signal at (510)731-8429.

Latest Stories