Policy

Why Twitch’s 'hate raid' lawsuit isn’t just about Twitch

When is it OK for tech companies to unmask their anonymous users? And when should a violation of terms of service get someone sued?

Phone with Twitch app

The case Twitch is bringing against two hate raiders is hardly black and white.

Photo: Caspar Camille Rubin/Unsplash

It isn't hard to figure out who the bad guys are in Twitch's latest lawsuit against two of its users. On one side are two anonymous "hate raiders" who have been allegedly bombarding the gaming platform with abhorrent attacks on Black and LGBTQ+ users, using armies of bots to do it. On the other side is Twitch, a company that, for all the lumps it's taken for ignoring harassment on its platform, is finally standing up to protect its users against persistent violators whom it's been unable to stop any other way.

But the case Twitch is bringing against these hate raiders is hardly black and white. For starters, the plaintiff here isn't an aggrieved user suing another user for defamation on the platform. The plaintiff is the platform itself. Complicating matters more is the fact that, according to a spokesperson, at least part of Twitch's goal in the case is to "shed light on the identity of the individuals behind these attacks," raising complicated questions about when tech companies should be able to use the courts to unmask their own anonymous users and, just as critically, when they should be able to actually sue them for violating their speech policies.

"Normally, what happens when someone violates your terms of service is you boot them off the service," said Aaron Mackey, senior staff attorney for the Electronic Frontier Foundation. "This lawsuit, while ostensibly good because it's aimed at bad actors who have done pretty awful things, is an escalation."

A Twitch spokesperson emphasized that the company "respect[s] anonymity" and that Twitch is using "all of our tools in the tool kit to stop the attack — including a legal claim."

"Our intent is not to reveal the actors' names but to identify these individuals for law enforcement or to compel them to cease attacking our community," the spokesperson said.

Tech companies sue their own users all the time for violating rules against spam, selling counterfeit goods and even data scraping. The Twitch suit has some of that, taking aim at the two anonymous users — known as CruzzControl and CreatineOverdose — for allegedly using bots to assault users with slurs and other offensive speech. But it's also coming after these users for the offensive speech itself, which Twitch argues violates its policies against "hateful conduct," an important but ever-evolving category of violation that companies themselves still struggle to define.

"It's not new for online service providers such as Twitch to sue their own users," said Riana Pfefferkorn, a research scholar at the Stanford Internet Observatory, who herself brought a case against spammers on behalf of Twitter while working in private practice. "What I find novel is that the purpose of the large-scale, bot-enabled violation of the platform's policies was spewing hate at other users, rather than spammy behavior or phishing."

That signals an important recognition by Twitch that hateful behavior on its platform can be as damaging to users' experience as spam or other bad behavior, if not more. And trying new ways to punish that behavior is laudable.

And yet, taking users to court over violations of terms of service has been particularly contentious in other contexts. Academics and civil rights groups, for instance, have fought back against tech platforms' crackdowns on scraping, arguing that data scraping, which lots of companies consider a privacy violation, is actually an important research method. The Supreme Court recently voted against the Department of Justice in a case where the DOJ argued that breaching terms of service constituted a violation of the Computer Fraud and Abuse Act.

Suing over violations of terms of service can seem virtuous when the defendants are so — well, not. But, Mackey said, "I think there could be larger implications to what's happening here."

Twitch's spokesperson noted that "anyone is subject to a lawsuit under a Terms of Service given that it is treated as a contract." And that's true. It's just that when it comes to hate speech issues, those lawsuits by companies are vanishingly rare.

Asking a court to intervene in identifying an anonymous person is also not a trivial thing. The defendants' actions in this case may be detestable, but there are any number of other good reasons for people's anonymity to be protected both online and off. That's why courts have long recognized that the First Amendment establishes a right to anonymity.

But while that protection is strong, it's "not absolute," said Jeff Kosseff, associate professor of cybersecurity law in the United States Naval Academy, who is writing a book on anonymity. "It comes down to looking at the strength of the case," Kosseff said.

Courts have developed a framework to determine when it is reasonable to issue subpoenas to unmask anonymous figures. That framework depends on things like whether the plaintiffs have a legitimate claim and whether they can show a necessity for the information. Other tests courts have used include assessing whether the information can be attained anywhere else or whether the harm of unmasking outweighs the need to unmask.

"The test is designed to not be absolute, but to be flexible, to allow for certain situations where perhaps someone doesn't have to be publicly identified in a court filing, but you could use their information," Mackey said.

Typically the plaintiffs in these cases are users seeking subpoenas that could compel online service providers to give up other users' identities. "The twist here is that the service provider is also the plaintiff," Pfefferkorn said.

Of course, there's also a case to be made that the speech these users have engaged in, which included "racial slurs and descriptions of violent acts against racial minorities and members of the LGBTQIA+ community," according to the complaint, is harassment, which is not protected by the First Amendment. "I think those are going to be tricky questions," Mackey said. "Do these folks have a First Amendment right to engage in their expression?"

Twitch argues they do not. "In this case, the actors are not using anonymity to conduct speech or expression activity or to participate in our community, but rather to deliberately attack our community, break our terms of service, and obfuscate their methods," the spokesperson said.

According to what information Twitch does have about the users, they appear to be outside of the United States. Foreign defendants are entitled to First Amendment protections in U.S. courts, but as Pfefferkorn put it, "the wheels of justice against overseas defendants move even more slowly than they do when everyone involved is located in the U.S."

The big question is what kind of precedent this case could set for other tech platforms that are also trying to figure out how to stop targeted harassment from their users. Hate raids may be a phenomenon associated with Twitch, but targeted hate speech is a problem all over the internet. Kosseff is dubious the case will open the floodgates to mass litigation, if it even goes forward. For one thing, it's expensive, and for another, it's not a great look for platforms to sue average users for slight offenses. The defendants in this case, Kosseff argues, happen to be particularly odious.

There's also no guarantee these defendants' identities can even be found, if they were really diligent about covering their tracks. "[Twitch] presumably has some sort of IP address, it could be a Tor exit node, in which case, it's useless," Kosseff said. He also pointed out Twitch filed another lawsuit against anonymous trolls in 2019, which it voluntarily dismissed a year later.

Even if the case doesn't go forward, though, the threat of this type of suit is, on its own, a signaling exercise. "Twitch is ready, willing and able to expend significant resources against hate-spewing abusers on its service," Pfefferkorn said. "If having an account terminated for terms of service violations doesn't scare the people who harass and victimize others on Twitch, perhaps the specter of being sued in federal court will."

Fintech

Judge Zia Faruqui is trying to teach you crypto, one ‘SNL’ reference at a time

His decisions on major cryptocurrency cases have quoted "The Big Lebowski," "SNL," and "Dr. Strangelove." That’s because he wants you — yes, you — to read them.

The ways Zia Faruqui (right) has weighed on cases that have come before him can give lawyers clues as to what legal frameworks will pass muster.

Photo: Carolyn Van Houten/The Washington Post via Getty Images

“Cryptocurrency and related software analytics tools are ‘The wave of the future, Dude. One hundred percent electronic.’”

That’s not a quote from "The Big Lebowski" — at least, not directly. It’s a quote from a Washington, D.C., district court memorandum opinion on the role cryptocurrency analytics tools can play in government investigations. The author is Magistrate Judge Zia Faruqui.

Keep ReadingShow less
Veronica Irwin

Veronica Irwin (@vronirwin) is a San Francisco-based reporter at Protocol covering fintech. Previously she was at the San Francisco Examiner, covering tech from a hyper-local angle. Before that, her byline was featured in SF Weekly, The Nation, Techworker, Ms. Magazine and The Frisc.

The financial technology transformation is driving competition, creating consumer choice, and shaping the future of finance. Hear from seven fintech leaders who are reshaping the future of finance, and join the inaugural Financial Technology Association Fintech Summit to learn more.

Keep ReadingShow less
FTA
The Financial Technology Association (FTA) represents industry leaders shaping the future of finance. We champion the power of technology-centered financial services and advocate for the modernization of financial regulation to support inclusion and responsible innovation.
Enterprise

AWS CEO: The cloud isn’t just about technology

As AWS preps for its annual re:Invent conference, Adam Selipsky talks product strategy, support for hybrid environments, and the value of the cloud in uncertain economic times.

Photo: Noah Berger/Getty Images for Amazon Web Services

AWS is gearing up for re:Invent, its annual cloud computing conference where announcements this year are expected to focus on its end-to-end data strategy and delivering new industry-specific services.

It will be the second re:Invent with CEO Adam Selipsky as leader of the industry’s largest cloud provider after his return last year to AWS from data visualization company Tableau Software.

Keep ReadingShow less
Donna Goodison

Donna Goodison (@dgoodison) is Protocol's senior reporter focusing on enterprise infrastructure technology, from the 'Big 3' cloud computing providers to data centers. She previously covered the public cloud at CRN after 15 years as a business reporter for the Boston Herald. Based in Massachusetts, she also has worked as a Boston Globe freelancer, business reporter at the Boston Business Journal and real estate reporter at Banker & Tradesman after toiling at weekly newspapers.

Image: Protocol

We launched Protocol in February 2020 to cover the evolving power center of tech. It is with deep sadness that just under three years later, we are winding down the publication.

As of today, we will not publish any more stories. All of our newsletters, apart from our flagship, Source Code, will no longer be sent. Source Code will be published and sent for the next few weeks, but it will also close down in December.

Keep ReadingShow less
Bennett Richardson

Bennett Richardson ( @bennettrich) is the president of Protocol. Prior to joining Protocol in 2019, Bennett was executive director of global strategic partnerships at POLITICO, where he led strategic growth efforts including POLITICO's European expansion in Brussels and POLITICO's creative agency POLITICO Focus during his six years with the company. Prior to POLITICO, Bennett was co-founder and CMO of Hinge, the mobile dating company recently acquired by Match Group. Bennett began his career in digital and social brand marketing working with major brands across tech, energy, and health care at leading marketing and communications agencies including Edelman and GMMB. Bennett is originally from Portland, Maine, and received his bachelor's degree from Colgate University.

Enterprise

Why large enterprises struggle to find suitable platforms for MLops

As companies expand their use of AI beyond running just a few machine learning models, and as larger enterprises go from deploying hundreds of models to thousands and even millions of models, ML practitioners say that they have yet to find what they need from prepackaged MLops systems.

As companies expand their use of AI beyond running just a few machine learning models, ML practitioners say that they have yet to find what they need from prepackaged MLops systems.

Photo: artpartner-images via Getty Images

On any given day, Lily AI runs hundreds of machine learning models using computer vision and natural language processing that are customized for its retail and ecommerce clients to make website product recommendations, forecast demand, and plan merchandising. But this spring when the company was in the market for a machine learning operations platform to manage its expanding model roster, it wasn’t easy to find a suitable off-the-shelf system that could handle such a large number of models in deployment while also meeting other criteria.

Some MLops platforms are not well-suited for maintaining even more than 10 machine learning models when it comes to keeping track of data, navigating their user interfaces, or reporting capabilities, Matthew Nokleby, machine learning manager for Lily AI’s product intelligence team, told Protocol earlier this year. “The duct tape starts to show,” he said.

Keep ReadingShow less
Kate Kaye

Kate Kaye is an award-winning multimedia reporter digging deep and telling print, digital and audio stories. She covers AI and data for Protocol. Her reporting on AI and tech ethics issues has been published in OneZero, Fast Company, MIT Technology Review, CityLab, Ad Age and Digiday and heard on NPR. Kate is the creator of RedTailMedia.org and is the author of "Campaign '08: A Turning Point for Digital Media," a book about how the 2008 presidential campaigns used digital media and data.

Latest Stories
Bulletins