To give you the best possible experience, this site uses cookies. If you continue browsing. you accept our use of cookies. You can review our privacy policy to find out more about the cookies we use.
chinaprotocol | chinaauthorShen LuNoneDavid Wertime and our data-obsessed China team analyze China tech for you. Every Wednesday, with alerts on key stories and research.9338dd5bb5
Subscribe to
Want to better understand the $150 billion gaming industry? Get our newsletter every Tuesday.
Are you keeping up with the latest cloud developments? Get Tom Krazit and Joe Williams' newsletter every Monday and Thursday.
David Wertime and our data-obsessed China team analyze China tech for you. Every Wednesday, with alerts on key stories and research.
Want your finger on the pulse of everything that's happening in tech? Sign up to get David Pierce's daily newsletter.
Do you know what's going on in the venture capital and startup world? Get the Pipeline newsletter every Saturday.
Do you know what's coming next up in the world of tech and entertainment? Get Janko Roettgers' newsletter every Thursday.
Hear from Protocol's experts on the biggest questions in tech. Get Braintrust in your inbox every Thursday.
Get access to the Protocol | Fintech newsletter, research, news alerts and events.
March 1, 2021
Ever since Elon Musk's appearance on Clubhouse juiced Chinese web users' enthusiasm for the app, chatter about how Chinese tech firms can create a home-grown version of Clubhouse has been incessant.
China's early Clubhouse users enjoyed a fleeting moment of free speech before Beijing blocked the app on Feb. 8. This served as an opportunity for China's tech firms to build their own versions of the popular drop-in audio tool. According to Chinese internet consultancy iiMedia Research, more than 100 teams in China are developing audio-based social media apps. China Venture, a platform focused on equity investments, reported that over 50 Chinese venture capital and private equity firms have made investments in audio-based communication products.
Over the past three weeks, several Chinese audio-based apps have emerged on Apple and Android stores, and some have subsequently disappeared. [Protocol | China will discuss this issue in depth with investor William Bao Bean the morning of March 2 during a Clubhouse event. -Ed.]
Here's a partial list of the contenders.
MiTalk (米聊): Smartphone maker Xiaomi's MiTalk is the latest audio drop-in app on the market. Previously an instant messaging tool that Xiaomi first launched in 2010, MiTalk ceased operations on Feb. 19. But within a week, MiTalk reappeared as "an audio-based communication tool for professionals" on Feb. 26. Like Clubhouse, MiTalk is now invite-only.
Capital Coffee: 36Kr, the parent company of its eponymous Chinese tech publication, launched Capital Coffee, a Clubhouse-like tool for investors, on Feb. 22. It took the team less than 10 days to develop and launch the tool, 36Kr, the publication, reported. Capital Coffee is still in beta, but 36Kr claims the tool has already attracted more than 10,000 users to participate in its internal test, including partners of prominent Chinese VC firms.
Duihuaba (对话吧): Hong Kong-listed streaming company Inke launched Duihuaba (对话吧) or "Let's Talk" on Apple and Android stores just two days after censors shut down Clubhouse in China. The app's project leader told 36Kr that he asked Inke's C-suite on Feb. 3 whether he should make a clone. By that evening, the COO gave him an answer: "Start now." By late February, Duihuaba had attracted 4,000 registered users and 1,000 active users within only a week since it was released on Feb. 7. On Feb. 22, users found Duihuaba have been removed from the app stores. Inke explained that the app's technology is being upgraded and improved.
Clubhorse: Yes, it sounds just like a counterfeit Clubhouse, though Clubhorse's founder told Chinese media the tool's name was inspired by the Netflix show BoJack Horseman. Its interface looks a lot like Clubhouse, too. Clubhorse launched on WeChat as a "mini program," i.e. an app within the WeChat app, on Feb. 9. But it didn't last long. WeChat quickly moved to ban the program, according to the Global Times, giving two reasons for the removal: Clubhorse was suspected of disrupting official service functions and it had been reported for "suspectedly infringing on the legal rights and interests of others."
The list of Chinese companies trying to create Clubhouse clones will likely grow. Alibaba is rumored to be developing a similar tool called MeetClub. It's too early to dismiss these products as Clubhouse copycats. Many successful Chinese apps started out as a clear imitation of foreign tech counterparts: Taobao v. eBay, Weibo v. Twitter, Alipay v. PayPal. But those Chinese companies companies later scaled and innovated. The question for now is whether these Chinese audio communication tools can match Clubhouse' soaring popularity in a country with pervasive censorship.
Correction: This article has been updated to reflect the year MiTalk was founded.
From Your Site Articles
Shen Lu
Shen Lu is a reporter with Protocol | China. She has spent six years covering China from inside and outside its borders. Previously, she was a fellow at Asia Society's ChinaFile and a Beijing-based producer for CNN. Her writing has appeared in Foreign Policy, The New York Times and POLITICO, among other publications. Shen Lu is a founding member of Chinese Storytellers, a community serving and elevating Chinese professionals in the global media industry.
VCG / Contributor/ Getty Images
April 4, 2021
David Pierce ( @pierce) is Protocol's editor at large. Prior to joining Protocol, he was a columnist at The Wall Street Journal, a senior writer with Wired, and deputy editor at The Verge. He owns all the phones.
April 4, 2021
On this episode of the Source Code podcast: First, we go over what's in the Biden administration's $2 trillion infrastructure plan, and what it might mean for tech. Then, Shen Lu joins the show to talk about the Chinese government's recent crackdowns on Alibaba, Baidu, Tencent and other local tech giants. Finally, Megan Rose Dickey discusses the Asian American experience in the tech industry right now, and why the "model minority" myth continues to be a problem.
For more on the topics in this episode:
<ul class="ee-ul"><li><a href="https://www.protocol.com/newsletters/sourcecode/biden-500-billion-infrastructure" target="_self">President Biden wants to spend $500 billion on tech</a></li><li><a href="https://twitter.com/shenlulushen" target="_blank">Shen Lu on Twitter</a></li><li><a href="https://www.protocol.com/china/chinese-big-techs-shadiest-practices" target="_self">Chinese Big Tech's shadiest practices</a></li><li><a href="https://twitter.com/meganrosedickey" rel="noopener noreferrer" target="_blank">Megan Rose Dickey on Twitter</a></li><li><a href="https://www.protocol.com/asian-americans-model-minority" target="_self">Asian Americans in tech say they face "a unique flavor of oppression"</a></li></ul><strong>Subscribe to the show: <a href="https://podcasts.apple.com/us/podcast/protocols-source-code/id1497440658" rel="noopener noreferrer" target="_blank">Apple Podcasts</a> | <a href="https://open.spotify.com/show/1mc6ODYq0nZ8hrVDuvQfeL?si=HDPznyjDSEqv5EvMZxhKxQ" rel="noopener noreferrer" target="_blank">Spotify</a> | <a href="https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkcy5zaW1wbGVjYXN0LmNvbS95OVNfTjJfaw==" rel="noopener noreferrer" target="_blank">Google Podcasts</a> | </strong><a href="https://feeds.simplecast.com/y9S_N2_k" rel="noopener noreferrer" target="_blank"><strong>RSS</strong></a>
Keep Reading
Show less
David Pierce ( @pierce) is Protocol's editor at large. Prior to joining Protocol, he was a columnist at The Wall Street Journal, a senior writer with Wired, and deputy editor at The Verge. He owns all the phones.
March 21, 2021
Kate Silver is an award-winning reporter and editor with 15-plus years of journalism experience. Based in Chicago, she specializes in feature and business reporting. Kate's reporting has appeared in the Washington Post, The Chicago Tribune, The Atlantic's CityLab, Atlas Obscura, The Telegraph and many other outlets.
March 19, 2021
While debate has raged in recent months about the pros and cons of raising the minimum wage, there's one particular set of people in agreement regarding the benefits: hourly wage workers, like Remington*.
In September 2020, Remington began working in an Amazon fulfillment center in DuPont, Washington, after his hours were cut at his restaurant job because of COVID-19. A few months into his new job, which has a starting wage of $15 an hour plus benefits, he said he feels more secure earning a steady paycheck at a rate higher than the federal minimum wage — especially with benefits. "The consistent pay is amazing," he said. "Instead of just wondering 'Am I going to make rent this month?' I'm like, 'Yeah, I got the bills this month. I can even go on a trip.'" Thanks to his job, he's finally been able to get new glasses and contacts and take his daughter skiing and fishing, all while saving for a house he's planning to buy with his siblings — two of whom work with him at Amazon.
<p><div class="ad-tag"><div class="ad-place-holder" data-pos="1">
</div>
</div></p><p>Earlier this month, the push for a federal minimum wage increase to $15 per hour failed to make it into the $1.9 trillion COVID-19 relief bill. But the battle isn't over. Multiple lawmakers have sponsored bills that seek to raise the federal minimum wage, including the<a href="https://edlabor.house.gov/imo/media/doc/2021-01-26%20Raise%20the%20Wage%20Act%20Fact%20Sheet.pdf" rel="noopener noreferrer" target="_blank"> Raise the Wage Act of 2021</a>, which aims for an incremental increase to $15 an hour over the next five years, and would give 32 million Americans an increase in pay. A growing body of research supports raising the minimum wage. </p><h4>The ripple effect of increased pay</h4><p>At $7.25, the federal minimum wage hasn't been raised in more than a decade. According to the<a href="https://www.epi.org/publication/raising-the-federal-minimum-wage-to-15-by-2025-would-lift-the-pay-of-32-million-workers/" rel="noopener noreferrer" target="_blank"> Economic Policy Institute</a>, a person who is working full-time today at the federal minimum wage earns 18% less than what they would have earned 10 years ago, if adjusted for the increased cost of living; in fact, they're also making less than their minimum-wage counterparts were making more than 50 years ago.</p><p>The EPI goes on to say that with an increase to $15 per hour, nearly one in three Black workers, one in four Hispanic workers and one in five white workers would see their wages rise. </p><p><cite class="pull-quote">All told, increasing the federal minimum wage to $15 by 2025 would result in $107 billion in higher wages for workers, which would trickle into economies across the country, fueling business growth and job growth.</cite> </p><p><div class="ad-tag"><div class="ad-place-holder" data-pos="2">
</div>
</div></p><p>According to the<a href="https://www.nelp.org/publication/u-s-needs-15-minimum-wage/" rel="noopener noreferrer" target="_blank"> National Employment Law Project</a>, the increase would be especially impactful to women, and it would boost the income of 59% of workers whose total family income is below the poverty line.</p><p>A<a href="https://www.ipsos.com/en-us/news-polls/minimum-wage-thought-leadership-2021" rel="noopener noreferrer" target="_blank"> new study</a> conducted by Amazon/Ipsos found that most Americans support raising the minimum wage. Results from the study, which involved interviews with more than 6,000 adults, revealed that two out of three support increasing the minimum wage to $15 an hour. Those polled believe that the higher pay would have a positive impact on workers, their community, the country and the economy. Further, those making more than $15 an hour were more likely to say they were earning more than they spend and they could come up with $500 in an emergency; while those making less than $15 an hour were more likely to say they can't afford basic household expenses, such as groceries, prescription drugs and doctor visits. Respondents largely agreed that they look to large businesses for leadership in pushing earnings higher, both when it comes to raising the federal minimum wage and raising the company's own minimum wage.</p><p>Across the country, legislators and business leaders have long learned to take matters into their own hands. To date, the minimum wage has increased in 29 states, plus Washington, D.C., and 45 localities have adopted minimum wages above the federal minimum wage. Businesses such as Amazon have also taken the lead on raising pay.</p><p>In 2018, Amazon was the first major retailer to raise its starting pay to $15 an hour. The impact of that raise has reverberated well beyond the walls of the online marketplace. Other major retailers, such as Best Buy, Walmart, Target and Costco followed suit in increasing employees' starting pay. A <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3793677" rel="noopener noreferrer" target="_blank">recent report</a> studying voluntary increases to wages by businesses found that when Amazon raised its pay to $15 in 2018, it led to a 4.7% increase in wages for employees at other companies in the same market. While critics frequently point to the risk of widespread job losses if the minimum wage increases, the researchers in this study found that that the probability of employment decreased just 0.8% following the boost.</p><p><div class="ad-tag"><div class="ad-place-holder" data-pos="3">
</div>
</div></p><p>Amazon's increase to $15 an hour has also proved an asset to the business itself, said Jay Carney, the senior vice president of Amazon Global Corporate Affairs. "Once we increased our starting wage to $15 an hour, the positive impact on employee morale and retention — and the surge in job applicants — was immediate. In fact, the month after we raised our starting wage, applications for hourly positions more than doubled. Employees who saw their paychecks increase told us that they had an easier time providing for their families and were able to spend on things like car repairs and home improvement projects. In short, the investments we made in our hourly employees were quickly transferred to local businesses and economies," he said. </p><h4>The faces of a living wage</h4><p>Luv Luv*, who works in an Amazon fulfillment center in Shakopee, Minnesota, said her job has helped her live the life she wants. A former hairstylist, she said she was drawn to the company for its generous health insurance policy, and the above-average wages, she added, are a bonus. "The pay at Amazon gave me the means to be able to afford the apartment I have. I get to live a comfortable life on my own terms. It's been a phenomenal journey for me," she said.</p><p><div class="ad-tag"><div class="ad-place-holder" data-pos="4">
</div>
</div></p><p>Chatonn*, who also lives in Shakopee, was drawn to Amazon for the income. She left her job in retail and hasn't looked back since she accepted a role at a fulfillment center. "The difference on my paycheck was huge. I'm earning $15 an hour instead of 10-something," she said. The increase in pay, she added, gives her more expendable income, so she can support her children's hobbies and endeavors. "It allows me to take better care of my family. My daughter is into art, I can pay for her art supplies. My son's in karate, I can afford karate now," she said. "Amazon allows me to be the parent that I want to be. That is everything to me."</p><p>So often, the argument around wages is based on facts and figures, and the facts and figures show that a $15 federal minimum wage would <a href="https://www.epi.org/publication/why-america-needs-a-15-minimum-wage/" rel="noopener noreferrer" target="_blank">generate more than $107 billion in higher wages and help reverse decades of pay inequality</a>. But those numbers don't have faces or stories attached to them. In the cases of Remington, Luv Luv and Chatonn, an increase in pay has made an immeasurable impact, and it's one that's already flowing down to the next generation. </p><p><div class="ad-tag"><div class="ad-place-holder" data-pos="5">
</div>
</div></p><p><em>*Last names were withheld for privacy purposes.</em></p>
Keep Reading
Show less
Kate Silver is an award-winning reporter and editor with 15-plus years of journalism experience. Based in Chicago, she specializes in feature and business reporting. Kate's reporting has appeared in the Washington Post, The Chicago Tribune, The Atlantic's CityLab, Atlas Obscura, The Telegraph and many other outlets.
Protocol | Policy
Tech spent years fighting foreign terrorists. Then came the Capitol riot.
"Nobody's going to have a hearing if a platform takes down 1,000 ISIS accounts. But they might have a hearing if you take down 1,000 QAnon accounts."
Photo: Roberto Schmidt/Getty Images
March 12, 2021
Issie Lapowsky (@issielapowsky) is a senior reporter at Protocol, covering the intersection of technology, politics, and national affairs. Previously, she was a senior writer at Wired, where she covered the 2016 election and the Facebook beat in its aftermath. Prior to that, Issie worked as a staff writer for Inc. magazine, writing about small business and entrepreneurship. She has also worked as an on-air contributor for CBS News and taught a graduate-level course at New York University’s Center for Publishing on how tech giants have affected publishing. Email Issie.
March 8, 2021
On a Friday in August 2017 — years before a mob of armed and very-online extremists took over the U.S. Capitol — a young Black woman who worked at Facebook walked up to the microphone to ask Mark Zuckerberg a question during a weekly companywide question-and-answer session.
Zuckerberg had just finished speaking to the staff about the white supremacist violence in Charlottesville, Virginia, the weekend before — and what a difficult week it had been for the world. He was answering questions on a range of topics, but the employee wanted to know: Why had he waited so long to say something?
<p>The so-called Unite the Right rally in Charlottesville had been planned in plain sight for the better part of a month on Facebook. Facebook took the event down only a <a href="https://www.businessinsider.com/facebook-removed-unite-the-right-charlottesville-rally-event-page-one-day-before-2017-8" target="_blank">day before</a> it began, citing its ties to hate groups and the threat of physical harm. That turned out to be more than a threat. The extremist violence in Charlottesville left three people dead and dozens more injured. Then-Attorney General Jeff Sessions later <a href="https://www.usnews.com/news/national-news/articles/2017-08-14/jeff-sessions-calls-charlottesville-attack-domestic-terrorism" target="_blank">called</a> it an act of "domestic terrorism." </p><p><div class="ad-tag"><div class="ad-place-holder" data-pos="1">
</div>
</div></p><p>Zuckerberg had already posted a contrite, cautious <a href="https://www.facebook.com/zuck/posts/10103969849282011?pnref=story" target="_blank">message</a> about the rally on Facebook earlier that week, saying the company would monitor for any further threats of violence. But his in-person response to the employee's question that day struck some on the staff as dismissive. "He said in front of the entire company, both in person and watching virtually, that things happen all over the world: Is he supposed to comment on everything?" one former employee recalled.</p><p>"It was something like: He can't be giving an opinion on everything that happens in the world every Friday," another former employee remembered.</p><p>Facebook's chief operating officer and resident tactician, Sheryl Sandberg, quickly swooped in, thanking the employee for her question and rerouting the conversation to talk about Facebook's charitable donations and how Sandberg herself thinks about what to comment on publicly. A Facebook spokesperson confirmed the details of this account, but said it lacked context, including that Zuckerberg did admit he should have said something sooner. </p><p>Still, to the people who spoke with Protocol, Zuckerberg's unscripted remarks that day underscored something some employees already feared: that the company had yet to take the threat posed by domestic extremists in the U.S. as seriously as it was taking the threat from foreign extremists linked to ISIS and al-Qaeda. "There wasn't a patent condemnation such that we would have expected had this been a foreign extremist group," a third former employee said.</p><p><div class="ad-tag"><div class="ad-place-holder" data-pos="2">
</div>
</div></p><p>At the time, tech giants were already hard at work figuring out how to crack down on the global terrorist networks that filled their sites with beheading videos and used social media to openly recruit new adherents. Just a few months before Charlottesville, Facebook, YouTube, Twitter and Microsoft had <a href="https://gifct.org/about/story/#june-26--2017---formation-of-gifct" target="_blank">announced</a> a novel plan to share intel on known terrorist content so they could automatically remove posts that had appeared elsewhere on the web. </p><p>Despite the heavy-handed approach to international jihadism, tech giants have applied a notably lighter touch to the same sort of xenophobic, racist, conspiratorial ideologies that are homegrown in the U.S. and held largely by white Westerners. Instead, they've drawn drifting lines in the sand, banning explicit calls for violence, but often waiting to address the deranged beliefs underlying that violence until something has gone terribly wrong. </p><p>But the Capitol riot on Jan. 6 and the spiraling conspiracies that led to it have forced a reckoning many years in the making on how both Big Tech and the U.S. government approach domestic extremists and their growing power. In the weeks since the riot, the Department of Homeland Security has issued a <a href="https://www.dhs.gov/ntas/advisory/national-terrorism-advisory-system-bulletin-january-27-2021" target="_blank">terrorism advisory</a> bulletin, warning of the increased threat of "domestic violent extremists" who "may be emboldened" by the riot. The head of the intelligence community has <a href="https://www.nytimes.com/2021/01/19/us/politics/avril-haines-domestic-terror-qanon.html" target="_blank">promised</a> to track domestic extremist groups like QAnon. And attorney general nominee Merrick Garland, who prosecuted the 1995 Oklahoma City bombing case, <a href="https://time.com/5941907/merrick-garland-domestic-terrorism/" target="_blank">said</a> during his recent confirmation hearing that investigating domestic terrorism will be his "first priority."</p><p><div class="ad-tag"><div class="ad-place-holder" data-pos="3">
</div>
</div></p><p>Tech companies have followed suit, cracking down in ways they never have before on the people and organizations that worked to motivate and glorify that extremist behavior, including former President Donald Trump himself. </p><p>But the question now is the same as it was when that employee confronted Zuckerberg three years ago: What took so long? </p><p>Interviews with more than a dozen people who have worked on these issues at Facebook, Twitter and Google or inside the government shed light on how tech giants' defenses against violent extremism have evolved over the last decade and why their work on domestic threats lagged behind their work on foreign ones. </p><p>Some of it has to do with the War on Terror sociopolitical dynamics that have prioritized violent Islamism above all else. </p><p>Some of it has to do with the technical advancements that have been made in just the last four years. </p><p>And yes, some of it has to do with Trump.</p><h2>The room full of lawyers
</h2><p>Nearly a decade before Q was a glimmer in some 4channer's eye, the tech industry was facing a different scourge – the proliferation of child sexual abuse material. In 2008, Microsoft called a Dartmouth computer science professor named Hany Farid to help the company figure out a way to do something about it. Farid, now a professor at University of California, Berkeley, traveled to Washington to meet with representatives from the tech industry to discuss a possible solution.</p><p><div class="ad-tag"><div class="ad-place-holder" data-pos="4">
</div>
</div></p><p>"I go down to D.C. to talk to them, and it's exactly what you think it is: a room full of lawyers, not engineers, from the tech industry, talking about how they can't solve the problem," Farid recalled. </p><p>To Farid, the fact that he was one of the only computer scientists at the meeting sent a message about how the industry thought about the problem, and still thinks about other content moderation problems — not as a technical challenge that rewarded speed and innovation, but as a legal liability that had to be handled cautiously.</p><p>At the time, tech companies were already attaching unique fingerprints to copyrighted material so they could remove anything that risked violating the Digital Millennium Copyright Act. Farid didn't see any reason why companies couldn't apply the same technology to automatically remove child abuse material that had been previously reported to the National Center for Missing & Exploited Children's tipline. That might not catch every piece of child abuse imagery on the internet, but it would make a dent. In partnership with Microsoft, he spent the next year developing a tool called PhotoDNA that Microsoft <a href="https://news.microsoft.com/2009/12/15/new-technology-fights-child-porn-by-tracking-its-photodna/" target="_blank">deployed</a> across its products in 2009.<br></p><p>But social networks dragged their feet. Facebook became the first company outside of Microsoft to announce it had adopted PhotoDNA in <a href="https://blogs.microsoft.com/on-the-issues/2011/05/19/facebook-to-use-microsofts-photodna-technology-to-combat-child-exploitation/" rel="noopener noreferrer" target="_blank">2011</a>. Twitter took it up in <a href="https://www.theguardian.com/technology/2013/jul/22/twitter-photodna-child-abuse" rel="noopener noreferrer" target="_blank">2013</a>. (Google had begun using its own <a href="https://www.blog.google/outreach-initiatives/google-org/our-continued-commitment-to-combating/" rel="noopener noreferrer" target="_blank">hashing system</a> in 2008.) "Everybody came to this reluctantly," Farid said. "They knew if you came for [child sexual abuse material], you're now going to come for the other stuff."</p><p><div class="ad-tag"><div class="ad-place-holder" data-pos="5">
</div>
</div></p><p>That turned out to be the right assumption. By 2014, that "other stuff" included a string of ghastly beheading videos, slickly filmed and distributed by ISIS's loud and proud propaganda arm. One of those videos in particular, which documented the beheading of journalist James Foley, horrified Americans as it filled their Twitter feeds and multiplied on YouTube. "It was the first instance of an execution video going really viral," said Nu Wexler, who was working on Twitter's policy communications team at the time. "It was one of the early turning points where the platforms realized they needed to work together."</p><p>As the Foley video made the rounds, Twitter and YouTube scrambled to form an informal alliance, where each platform swapped links on the videos it was finding and taking down. But at the time, that work was happening through user reports and manual searches. "A running theme for a number of services was that we had a very manual, very reactive response to the threat that ISIS posed to our services, coupled with the speed of ISIS' territorial organizational expansion and at the same time the response from industry and from government being relatively siloed," said Nick Pickles, Twitter's senior director of public policy strategy.</p><p>The Foley video and several other videos that appeared on the platform in quick succession only underscored how insufficient that approach was. "The way that content was distributed by ISIS online represented a manifestation of their online and physical threat in a way which led to a far more focused policy conversation and urgency to address their exploitation of digital services," Pickles said.</p><p><div class="ad-tag"><div class="ad-place-holder" data-pos="6">
</div>
</div></p><p>But there was also trepidation among some in the industry. "I heard a tech executive say to me, and I wanted to punch him in the face: 'One person's terrorist is another person's freedom fighter,'" Farid remembered. "I'm like, there's a fucking video of a guy getting his head cut off and then run over. Do you want to talk about that being a 'freedom fighter'?"</p><p>To Farid, the choice for tech companies was simple: automatically filter out the mass quantities of obviously abhorrent content using hashing technology like PhotoDNA and worry about the gray areas later.</p><p class="pull-quote">There's a fucking video of a guy getting his head cut off and then run over. Do you want to talk about that being a 'freedom fighter'?</p><p>Inside YouTube, one former employee who has worked on policy issues for a number of tech giants said people were beginning to discuss doing just that. But questions about the slippery slope slowed them down. "You start doing it for this, then everybody's going to ask you to do it for everything else. Where do you draw the line there? What is OK and what's not?" the former employee said, recalling those discussions. "A lot of these conversations were happening very robustly internally."</p><p>Those conversations were also happening with the U.S. government at a time when tech giants were very much trying to <a href="https://www.wired.com/2014/03/facebook-security/" rel="noopener noreferrer" target="_blank">distance</a> themselves from the feds in the wake of the Edward Snowden disclosures. "At first when we were meeting with social media companies to address the ISIS threat, there was some reluctance to feel like tech companies were part of the solution," said Ryan Greer, who worked on violent extremism issues in both the State Department and the Department of Homeland Security under President Obama. "It had to be a little bit shamed out of them."</p><p><div class="ad-tag"><div class="ad-place-holder" data-pos="7">
</div>
</div></p><h2>The Madison Valleywood Project
</h2><p>Then, in 2015, ISIS-inspired attackers shot up a theater in Paris and an office in San Bernardino. The next year, they carried out a series of bombings in Brussels and drove cargo trucks through crowds in Berlin and Nice. The rash of terrorist attacks in the U.S. and Europe changed the stakes for the tech industry. Suddenly, governments in those countries began forcefully pushing tech companies to prevent their platforms from becoming instruments of radicalization. </p><p>"You had multiple Western governments speaking with one voice and linking arms on this to put pressure to bear on these companies," said the former YouTube employee.</p><p>In the U.S., the Obama administration gave this pressure campaign a codename: <a href="https://www.nytimes.com/2016/02/25/technology/tech-and-media-firms-called-to-white-house-for-terrorism-meeting.html" rel="noopener noreferrer" target="_blank">The Madison Valleywood Project</a>, which was designed to get Madison Avenue advertisers, Silicon Valley technologists and Hollywood filmmakers to work with the government in the fight against ISIS. In February 2016, Obama invited representatives from all of those industries – Google, Facebook and Twitter among them – to a day-long summit at the White House that was laser-focused on ISIS. The day's opening speaker, former Assistant Attorney General John Carlin, applauded Facebook, Twitter and YouTube's nascent counterterrorism efforts but urged them to do more. "We anticipate — and indeed hope — that after today you will continue to meet without the government, to continue to develop on your own efforts, building on the connections you make today," Carlin said, according to a copy of the <a href="https://epic.org/foia/MadisonValleywood_2.pdf" rel="noopener noreferrer" target="_blank">speech</a> obtained by the Electronic Privacy Information Center.</p><p><div class="ad-tag"><div class="ad-place-holder" data-pos="8">
</div>
</div></p><p>"The ISIS threat really captivated both U.S. and international media at the time," said Greer, who now works as national security director for the Anti-Defamation League. "There was a constant drumbeat of questions: What are you doing about ISIS? What are you doing about ISIS?"</p><p>The mounting pressure seemed to have an impact. Just weeks before the White House summit, Twitter became the first tech company to <a href="https://blog.twitter.com/en_us/a/2016/combating-violent-extremism.html" rel="noopener noreferrer" target="_blank">publicly</a> enumerate the terrorist accounts it had removed in 2016. The communications team opted to publish the blog post laying out the stats without attaching the author's name, Wexler said, because they were fearful of directing more death threats to executives who were already being bombarded with them.</p><p>Over the course of the next year and a half, tech executives continued to hold meetings with the U.K.'s home secretary, the United Nations Counter-Terrorism Executive Directorate and the EU Internet Forum.</p><p><br></p><p class="shortcode-media shortcode-media-rebelmouse-image">
<a href="https://media.gettyimages.com/photos/from-left-to-right-uk-home-secretary-amber-rudd-head-of-public-policy-picture-id865330526?b=1&k=6&m=865330526&s=170x170&h=_H9lHNEtUrY1xP4Xr75Pj4F0BvKbUzR9M64Rp2dRaQo=" target="_blank"><img class="rm-shortcode" loading="lazy" src="https://www.protocol.com/media-library/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbWFnZSI6Imh0dHBzOi8vYXNzZXRzLnJibC5tcy8yNTc0NzQyNS9vcmlnaW4uanBnIiwiZXhwaXJlc19hdCI6MTY0NDQ5OTE1OH0.HluxS3UkzhEHVK7AjUGUmSMIjhcYdArHYvQbEdGG1GM/image.jpg?width=980" id="99b98" data-rm-shortcode-id="b78df8d5dcf6965b6363ea9429b987bd" data-rm-shortcode-name="rebelmouse-image"></a>
<small class="image-media media-caption" placeholder="Add Photo Caption...">Twitter's Nick Pickles (second from left) and Facebook's Brian Fishman (third from left) attend the G7 Interior Ministerial Meeting in Ischia, Italy in October of 2017. </small>
<small class="image-media media-photo-credit" placeholder="Add Photo Credit...">Photo: Vincenzo Rando/Getty Images</small>
</p><p><div class="ad-tag"><div class="ad-place-holder" data-pos="9">
</div>
</div></p><p><br></p><p>By June 2017, Microsoft, Facebook, Google and Twitter <a href="https://gifct.org/about/story/" rel="noopener noreferrer" target="_blank">emerged with</a> a plan to share hashed terrorist images and videos through a new group called the Global Internet Forum to Counter Terrorism, or GIFCT. That group has since grown to include smaller companies, including Snap, Pinterest, Mailchimp and Discord, and is led by Obama's former director of the National Counterterrorism Center, Nick Rasmussen.</p><p>Meanwhile, Google's internal idea lab, Jigsaw, which had been studying radicalization online for years, began running a novel <a href="https://www.wired.com/2016/09/googles-clever-plan-stop-aspiring-isis-recruits/" rel="noopener noreferrer" target="_blank">pilot</a> designed to stop people from getting pulled in by ISIS through search. Working with outside groups, Jigsaw began sponsoring Google search ads in 2016 that would run whenever users searched for terms that risked sending them down an ISIS rabbit hole. Those search ads, inspired by Jigsaw's interviews with actual ISIS defectors, would link to Arabic and English-language YouTube videos that aimed to counter ISIS propaganda. In 2017, even as Google and YouTube worked on ways to remove ISIS content algorithmically, YouTube <a href="https://blog.youtube/news-and-events/bringing-new-redirect-method-features" rel="noopener noreferrer" target="_blank">deployed</a> the Redirect Method to searches inside its own platform to help counter propaganda its automated filters had not yet found. </p><p>Facebook, meanwhile, hired an expert on jihadi terrorism, Brian Fishman, to head up Facebook's work on counterterrorism and dangerous organizations in April 2016. At the time, the list of dangerous organizations consisted mainly of foreign terrorist organizations, as well as well-known hate groups like the Ku Klux Klan and the neo-Nazi group Blood & Honour. These organizations were banned from the platform, as was any praise of those groups. But Fishman's hiring was a clear signal that cracking down on ISIS and al-Qaeda had become a priority for Facebook.</p><p><div class="ad-tag"><div class="ad-place-holder" data-pos="10">
</div>
</div></p><p class="pull-quote">There was a constant drumbeat of questions: What are you doing about ISIS? What are you doing about ISIS?</p><p>After Fishman began, Facebook started using an approach similar to what the intelligence community would use to go after ISIS, relying not just on user reports and automated takedowns of known terrorist content, but using artificial intelligence as well as off-platform information to chase down whole networks of accounts.</p><p>ISIS's overt and corporate branding gave tech platforms a clear focal point to start with. "Some groups like the ISISes of the world and the al-Qaedas of the world are very focused on protecting their brand," Fishman said. "They retain tight control over the release of information." ISIS had media outlets, television stations, slogans and soundtracks. That meant platforms could begin sniffing out accounts that used that common branding without having to look exclusively at the content itself. </p><p>"I look back on that threat, and I recognize now in hindsight there were attributes of it that made it easier to go after than the types of domestic terrorism and extremism we're grappling with today," said Yasmin Green, director of research and development at Google's Jigsaw. "There was one organization, basically, and you had to make public your allegiance to it. Obviously, all of those things made it possible for law enforcement and the platforms to model and pursue it."</p><p><div class="ad-tag"><div class="ad-place-holder" data-pos="11">
</div>
</div></p><p><br></p><p class="shortcode-media shortcode-media-rebelmouse-image">
<a href="https://media.gettyimages.com/photos/yasmin-green-director-of-research-and-development-jigsaw-speaks-the-picture-id1210644850?b=1&k=6&m=1210644850&s=170x170&h=5xo3EZLexgMsab0xvC7TRT-pyfxqqtWDSityLtoUvx8=" target="_blank"><img class="rm-shortcode" loading="lazy" src="https://www.protocol.com/media-library/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbWFnZSI6Imh0dHBzOi8vYXNzZXRzLnJibC5tcy8yNTc0ODU4Ny9vcmlnaW4uanBnIiwiZXhwaXJlc19hdCI6MTY2NjkxOTA0NH0.YNluKT1h_v-6kpS9FJexwgvFs1Q77ikBXzV4MndqhNo/image.jpg?width=980" id="99442" data-rm-shortcode-id="9448e621190bd34309d2730d21d23cdf" data-rm-shortcode-name="rebelmouse-image"></a>
<small class="image-media media-caption" placeholder="Add Photo Caption...">Jigsaw's Yasmin Green has recently focused her violent extremism research on white supremacists and conspiracy theorists.</small>
<small class="image-media media-photo-credit" placeholder="Add Photo Credit...">Photo: Craig Barritt/Getty Images</small>
</p><p><br></p><p>Around that time, Twitter also began investing in monitoring what Pickles calls "behavioral signals," not just tweets. "If you focus on behavioral signals, you can take action before they distribute content," Pickles said. "The switch to behavior meant that we could take action much faster at a much greater scale, rather than waiting."</p><p>The development of automated filters across the industry was almost stunningly successful. Within a year, Facebook, Twitter and YouTube went from manually removing foreign terrorist content that had been reported by users to automatically taking down the vast majority of foreign terrorists' posts before anyone flags them. Today, according to Facebook's transparency reports, 99.8% of terrorist content is <a href="https://transparency.facebook.com/community-standards-enforcement#dangerous-organizations" rel="noopener noreferrer" target="_blank">removed</a> before a single person has even reported it.</p><p>"If you actually look a bit farther back, you understand just how much has moved in this arena," Green said. "That always makes me feel a little bit optimistic." </p><h2>The domestic dilemma
</h2><p>It didn't hurt that both the United States and the United Nations keep lists of designated international terrorist organizations. To the "rooms full of lawyers" that help make these decisions, that kept things clean; use those lists as a guide and level a hammer on any organizations on them, and tech executives could be fairly confident they wouldn't face much second-guessing from the powers that be. "If a terrorist group is put on a watch list or terrorist list or viewed by the international community, by the UN, as a terrorist group, then that gives Facebook everything they need to have a very strong policy," said Yael Eisenstat, a former CIA officer who led election integrity efforts for Facebook's political ads in 2018. </p><p><div class="ad-tag"><div class="ad-place-holder" data-pos="11">
</div>
</div></p><p>The same can't be said for domestic extremists. In the United States, there's not an analogous list of domestic terrorist organizations for companies to work from. That doesn't mean acts of domestic terrorism go unpunished. It just means that people are prosecuted for the underlying crimes they commit, not for being part of a domestic terrorist organization. That also means that individual extremists who commit the crimes are the ones who face the punishment, not the groups they represent. "You have to have a violent crime committed in pursuit of an ideology," former FBI Acting Director Andrew McCabe said in a recent <a href="https://podcasts.apple.com/gb/podcast/dhs-warning-domestic-violent-extremists/id498897343?i=1000507279942" rel="noopener noreferrer" target="_blank">podcast</a>. "We hesitate to call domestic terrorists 'terrorists' until after something has happened." </p><p class="pull-quote">Nobody's going to have a hearing if a platform takes down 1,000 ISIS accounts. But they might have a hearing if you take down 1,000 QAnon accounts.</p><p>This gap in the legal system means tech companies write their own rules around what sorts of objectionable ideologies and groups ought to be forbidden on their platforms and often only take action once the risk of violence is imminent. "If something was illegal it was going to be handled. If something was not, then it became a political conversation," Eisenstat said of her time at Facebook.</p><p><div class="ad-tag"><div class="ad-place-holder" data-pos="11">
</div>
</div></p><p>Even in the best of times, it's an uncomfortable balancing act for companies that purport to prioritize free speech above all else. But it's particularly fraught when the person condoning or even espousing extremist views is the president of the United States. "Nobody's going to have a hearing if a platform takes down 1,000 ISIS accounts. But they might have a hearing if you take down 1,000 QAnon accounts," said Wexler, who worked in policy communications for Facebook, Google and Twitter during the Trump administration.</p><p>There never was a Madison Valleywood moment in the U.S. related to the <a href="https://www.newsweek.com/hate-crimes-under-trump-surged-nearly-20-percent-says-fbi-report-1547870" rel="noopener noreferrer" target="_blank">rising hate crimes</a> and domestic extremist events that marked the Trump presidency. Not after Charlottesville. Not after Pittsburgh. Not after El Paso. The former president <em>did</em> have what might be construed as the opposite of a Madison Valleywood moment when he held an event at the White House in 2019 where <a href="https://www.washingtonpost.com/technology/2019/07/11/who-was-who-trumps-social-media-summit/" rel="noopener noreferrer" target="_blank">far-right conspiracy theorists and provocateurs</a> discussed social media censorship. But this time, Facebook, Google and Twitter weren't invited.</p><p><br></p><p class="shortcode-media shortcode-media-rebelmouse-image">
<a href="https://media.gettyimages.com/photos/poster-with-a-tweet-is-seen-as-president-donald-j-trump-participates-picture-id1155382518?b=1&k=6&m=1155382518&s=170x170&h=rssm82D_89TYJJFr8l77T0pLrAOoN70cMvhW2p3Eqoo=" target="_blank"><img class="rm-shortcode" loading="lazy" src="https://www.protocol.com/media-library/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbWFnZSI6Imh0dHBzOi8vYXNzZXRzLnJibC5tcy8yNTc0NzQ0MC9vcmlnaW4uanBnIiwiZXhwaXJlc19hdCI6MTYzMzkyMzA5OX0.WyLHy4C-a0IV0S0HwMaQ75yknPytWxqbU1wG7ZR9Ygs/image.jpg?width=980" id="b9709" data-rm-shortcode-id="aea6abb03061310945c5554c829dada0" data-rm-shortcode-name="rebelmouse-image"></a>
<small class="image-media media-caption" placeholder="Add Photo Caption...">President Trump's Social Media Summit in 2019 focused on alleged social media censorship of conservatives. </small>
<small class="image-media media-photo-credit" placeholder="Add Photo Credit...">Photo: Jabin Botsford/Getty Images</small>
</p><p><br></p><p>"Platforms write their own rules, but governments signal which types of content they find objectionable, creating a permission structure for the companies to step up enforcement," Wexler said. "President Trump's comments after Charlottesville and his tacit support of the Proud Boys sent a deliberate message to tech companies: If you crack down on white nationalists' accounts, we'll accuse you of political bias and make your CEOs testify before Congress."</p><p><div class="ad-tag"><div class="ad-place-holder" data-pos="11">
</div>
</div></p><p>During the Trump years, tech companies repeatedly courted favor with the president and his party. In 2019, Google CEO Sundar Pichai met <a href="https://www.cnbc.com/2019/03/27/trump-met-with-google-ceo-sundar-pichai-on-political-fairness-china.html" rel="noopener noreferrer" target="_blank">directly</a> with Trump to discuss what Trump later characterized as "political fairness." Internally, Google employees <a href="https://www.nbcnews.com/news/us-news/current-ex-employees-allege-google-drastically-rolled-back-diversity-inclusion-n1206181" rel="noopener noreferrer" target="_blank">told</a> NBC News that the company had rolled back diversity training programs because the company "doesn't want to be seen as anti-conservative." (Google denied the accusation to NBC.) On Election Day in 2020, YouTube allowed the Trump campaign to book an ad on its <a href="https://www.businessinsider.com/trump-2020-campaign-spends-big-on-youtube-biden-facebook-2020-11?amp" rel="noopener noreferrer" target="_blank">homepage</a> for the entire day, and it was the only <a href="https://stacks.stanford.edu/file/druid:tr171zs0069/EIP-Final-Report-v2.pdf" rel="noopener noreferrer" target="_blank">one of the top three social platforms</a> that had no explicit policy regarding attempts to delegitimize the election results. After the election, <a href="https://www.theverge.com/2020/11/4/21550180/youtube-oann-video-election-trump-misinformation-voting-final-results-facebook-twitter" rel="noopener noreferrer" target="_blank">videos</a> claiming Trump won went viral, but YouTube only began <a href="https://blog.youtube/news-and-events/supporting-the-2020-us-election/" rel="noopener noreferrer" target="_blank">removing</a> widespread allegations of fraud and errors on Dec. 9.</p><p>Google and YouTube declined to make any executives available for comment. In a statement, YouTube spokesperson Farshad Shadloo said: "Our Community Guidelines prohibit hate speech, gratuitous violence, incitement to violence, and other forms of intimidation. Content that promotes terrorism or violent extremism does not have a home on YouTube." Shadloo said YouTube's policies focus on content violations and not speakers or groups, unless those speakers or groups are included on a government foreign terrorist organization list.</p><p>At times, tech giants bent their own policies or adopted entirely new ones to accommodate President Trump and his most conspiratorial supporters on the far right. In January 2018, Twitter, for one, published its "world leaders <a href="https://blog.twitter.com/en_us/topics/company/2018/world-leaders-and-twitter.html" rel="noopener noreferrer" target="_blank">policy</a>" for the first time, seemingly seeking to explain why President Trump wasn't punished for threatening violence when he tweeted that his nuclear button was "much bigger & more powerful" than Kim Jong Un's. Later that year, after Facebook, Apple and YouTube all <a href="https://www.vox.com/2018/8/6/17655658/alex-jones-facebook-youtube-conspiracy-theories" rel="noopener noreferrer" target="_blank">shut down</a> accounts and pages linked to Infowars' Alex Jones, Twitter CEO Jack Dorsey booked an interview with conservative kingmaker Sean Hannity, where he <a href="https://time.com/5361874/twitter-jack-dorsey-alex-jones-sean-hannity/" rel="noopener noreferrer" target="_blank">defended</a> Twitter's decision not to do the same. Just a few weeks later, the company would <a href="https://www.wired.com/story/twitter-bans-alex-jones-infowars/" rel="noopener noreferrer" target="_blank">reverse course</a> after Jones livestreamed a tirade against a CNN reporter on Twitter — while standing outside of Dorsey's own congressional hearing. </p><p><div class="ad-tag"><div class="ad-place-holder" data-pos="11">
</div>
</div></p><p><br></p><p class="shortcode-media shortcode-media-rebelmouse-image">
<a href="https://media.gettyimages.com/photos/alex-jones-of-infowars-background-attends-a-senate-intelligence-in-picture-id1027224386?b=1&k=6&m=1027224386&s=170x170&h=1IyYVN-U74jheta0t0kJbu0S6ik1HK0i2o6F2l_FBW8=" target="_blank"><img class="rm-shortcode" loading="lazy" src="https://www.protocol.com/media-library/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbWFnZSI6Imh0dHBzOi8vYXNzZXRzLnJibC5tcy8yNTc0NzQ2MS9vcmlnaW4uanBnIiwiZXhwaXJlc19hdCI6MTY1MDE2MzI2M30.Fn3bmuFCgNa8RHs9FbhpYRL79TFy0csfebnUMhCdv4E/image.jpg?width=980" id="03c68" data-rm-shortcode-id="2713b8f837f7dd9f37a2f330f1cb7184" data-rm-shortcode-name="rebelmouse-image"></a>
<small class="image-media media-caption" placeholder="Add Photo Caption...">Twitter defended its decision not to remove accounts tied to InfoWars Alex Jones. Weeks later, Twitter reversed that decision, following CEO Jack Dorsey's testimony before Congress in September of 2018.</small>
<small class="image-media media-photo-credit" placeholder="Add Photo Credit...">Photo: Tom Williams/CQ Roll Call</small>
</p><p><br></p><p>Jones was a lightning rod for Facebook, too. As BuzzFeed recently <a href="https://www.buzzfeednews.com/article/ryanmac/mark-zuckerberg-joel-kaplan-facebook-alex-jones?ref=bfnsplash&utm_term=4ldqpho" rel="noopener noreferrer" target="_blank">reported</a>, Facebook decided in 2019 to do more than just ban Jones' pages. The company wanted to designate him as a dangerous individual, a label that also ordinarily forbids other Facebook users from praising or expressing support for those individuals. But according to BuzzFeed, Facebook altered its own rules at Zuckerberg's behest, creating a third lane for Jones that would allow his supporters' accounts to go untouched. And when President Trump threatened to shoot looters in the aftermath of George Floyd's killing, Facebook staffers <a href="https://www.washingtonpost.com/technology/2020/06/28/facebook-zuckerberg-trump-hate/" rel="noopener noreferrer" target="_blank">reportedly</a> called the White House themselves, urging the president to delete or tweak his post. When he didn't, Zuckerberg <a href="https://www.vox.com/recode/2020/6/3/21279434/mark-zuckerberg-meeting-facebook-employees-transcript-trump-looting-shooting-post" rel="noopener noreferrer" target="_blank">told</a> his staff the post didn't violate Facebook's policies against incitement to violence anyway.</p><p><div class="ad-tag"><div class="ad-place-holder" data-pos="12">
</div>
</div></p><p>"I would argue the looter-shooter post was more violating than [the post] on Jan. 6," Eisenstat said, referring to the Facebook video that ended up getting the former president kicked off of Facebook indefinitely in his last weeks in office. In the video, Trump told the Capitol rioters he loved them and that they were very special, while repeating baseless claims of election fraud. </p><p>For Facebook at least, this instinct to accommodate the party in power wasn't unique to the U.S., said tech entrepreneur Shahed Amanullah, who worked with Facebook on a series of global hackathons through his company, Affinis Labs. The goal of the hackathons, Amanullah said, was to fight all forms of hate and extremism online, and the events had been successful in countries like Indonesia and the Philippines. But when he brought the program to India, Amanullah said he received pressure from Facebook India's policy team to focus the event specifically on terrorism coming out of the majority-Muslim region of Kashmir. </p><p>The woman leading Facebook India's policy team at the time, Ankhi Das, was a vocal supporter of Indian Prime Minister Narendra Modi, and, <a href="https://www.wsj.com/articles/facebook-hate-speech-india-politics-muslim-hindu-modi-zuckerberg-11597423346" rel="noopener noreferrer" target="_blank">according</a> to The Wall Street Journal, had a pattern of allowing anti-Muslim hate speech to go unchecked on the platform. "I said there's no way I'm ever going to accept a directive like that," Amanullah recalled. </p><p>Though he was supposed to run seven more hackathons in the country, Amanullah cut ties. "That was the last time we ever worked with Facebook," he said. </p><p>A Facebook spokesperson told Protocol, "We've found nothing to suggest this is true. We've looked into it on our end, spoken to people who were present at the hackathon and have no reason to believe that anyone was pressured to shift the focus of the hack." Das did not respond to Protocol's request for comment.</p><p>To Amanullah, the experience working with Facebook in India signaled that the company was giving in to the Indian government and giving Islamist extremism an inordinate amount of attention compared to other threats. "If you want to talk about hate," he said, "you have to talk about all kinds of hate."</p><h2>The reckoning
</h2><p>Looking back in the wake of the Capitol riot, it's easy to view Charlottesville as a warning shot that went unheard, or at least insufficiently answered, by tech giants. And in many ways it was. But inside, things were also changing, albeit far more slowly than almost anyone believes they should have.</p><p>For Fishman, who had focused almost entirely on jihadism on Facebook until that point, the Unite the Right rally was a turning point. "Charlottesville was a moment when extremist groups on the American far right clearly were trying to overcome the historical fractioning of that movement and express themselves in more powerful ways," he said. "It absolutely was something we tracked and realized we needed to invest more resources into."</p><p><br></p><p class="shortcode-media shortcode-media-rebelmouse-image">
<a href="https://media.gettyimages.com/photos/neo-nazis-altright-and-white-supremacists-take-part-a-the-night-the-picture-id830696202?b=1&k=6&m=830696202&s=170x170&h=kiBAUb9a6_lE_STxphpsjj0HB7qMFyB3Iay26r5ShpA=" target="_blank"><img class="rm-shortcode" loading="lazy" src="https://www.protocol.com/media-library/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbWFnZSI6Imh0dHBzOi8vYXNzZXRzLnJibC5tcy8yNTc0ODM3MC9vcmlnaW4uanBnIiwiZXhwaXJlc19hdCI6MTY3Mjg2OTAwMH0.uoccdzJpkOcTc1C4mEWPtfwkCRre2qhqmxF5q-2w01o/image.jpg?width=980" id="cc7be" data-rm-shortcode-id="b6f571c0d913a60ee8ccb11d9b0eb1f3" data-rm-shortcode-name="rebelmouse-image"></a>
<small class="image-media media-caption" placeholder="Add Photo Caption...">The Unite the Right rally in Charlottesville, which had been planned on Facebook, left three dead and dozens injured.</small>
<small class="image-media media-photo-credit" placeholder="Add Photo Credit...">Photo: Zach D Roberts/Getty Images</small>
</p><p><br></p><p>Immediately after the rally, Facebook <a href="https://twitter.com/Kantrowitz/status/897525520178397185?s=20" rel="noopener noreferrer" target="_blank">banned</a> eight far-right and white nationalist pages associated with it. That's in addition to hate groups like the National Socialist Movement, the KKK, The Daily Stormer and Identity Evropa, which were already designated as dangerous organizations and forbidden from the platform long before the rally. A few months later, in 2018, Fishman's team changed names, from the counterterrorism team to the dangerous organizations team, a signal that the company would double down on enforcement against hate groups, too. The team working primarily on violent extremism of all stripes at Facebook eventually grew to 350 people.</p><p>In late 2017, Twitter broadened its policy on violent extremism to prohibit all violent extremist groups, not just designated foreign terrorist organizations. And in 2019, YouTube <a href="https://blog.youtube/news-and-events/continuing-our-work-to-improve" rel="noopener noreferrer" target="_blank">promised</a> it would limit the spread of misinformation by keeping conspiracy theory videos out of recommendations, a promise it's had some <a href="https://www.nytimes.com/interactive/2020/03/02/technology/youtube-conspiracy-theory.html" rel="noopener noreferrer" target="_blank">success</a> in keeping.</p><p>Even so, these companies repeatedly bungled the definition of what constitutes hate and violent extremism. It was, after all, a year after Charlottesville when Zuckerberg boldly <a href="https://www.vox.com/2018/7/18/17575156/mark-zuckerberg-interview-facebook-recode-kara-swisher" rel="noopener noreferrer" target="_blank">defended</a> Facebook's policy of allowing Holocaust denial in an interview with Recode. It wasn't until 2020 that Zuckerberg changed his mind about that, <a href="https://www.facebook.com/zuck/posts/10112455086578451" rel="noopener noreferrer" target="_blank">citing</a> "data showing an increase in anti-Semitic violence." His post never fully acknowledged the role Facebook's earlier policies might have played in stoking that violence.</p><p>There were also legitimate reasons why identifying and removing domestic hate groups in the U.S. was harder than taking down networks of accounts tied to ISIS and al-Qaeda. For one thing, domestic groups and the people associated with them have traditionally been far more diffuse, making it tougher to draw neat lines around them or use clues in their branding to find and remove whole networks. "Groups that are less organized and less structured and don't put out official propaganda in the same sort of way, you have to use a different tool kit in order to get at those kinds of entities," Fishman said. "Chasing the networks becomes even more important than chasing known pieces of content that you can gather using vendors."</p><p>Then, there's the fact that tech companies defined hate in limited ways. Facebook, Twitter and YouTube have all introduced hate speech policies that generally prohibit direct attacks on the basis of specific categories like race or sexual orientation. But what to do with a new conspiracy theory like QAnon that hinges on some imagined belief in a cabal of Satan-worshipping Democratic pedophiles? Or a group of self-proclaimed "Western chauvinists" like the Proud Boys cloaking themselves in the illusion that white pride doesn't necessarily require racial animus? Or the #StoptheSteal groups, which were based on a lie, propagated by the former president of the United States, that the election had been stolen? These movements were shot through with hate and violence, but initially, they didn't fit neatly into any of the companies' definitions. And those companies, operating in a fraught political environment, were in turn slow to admit, at least publicly, that their definitions needed to change. </p><p>It's also true that it's philosophically trickier for American companies to ban people who are not faraway terrorists, but other Americans, whom the U.S. government couldn't punish for their beliefs even if it wanted to. For all of the bad-faith concerns about censorship, it's also possible to argue with a straight face that platforms shouldn't have even more power to police speech than Congress has. "I find that to be kind of troubling at a time when so many people are talking about tech platforms having too much power, we look to them to do what governments would be unable to do," said Matt Perault, director of Duke University's Center on Science & Technology Policy and Facebook's former director of public policy. </p><p>All of those considerations complicated tech companies' efforts to respond to domestic threats that could be harmful but haven't yet caused harm. "One of the biggest challenges around designing a policy and enforcement framework in these areas is when you have domestic [actors] who are part of this conversation, who are not promoting violence," Pickles said, after recently <a href="https://www.independent.co.uk/life-style/gadgets-and-tech/twitter-conspiracy-theory-donald-trump-b1790200.html" rel="noopener noreferrer" target="_blank">admitting</a> that Twitter was too slow to act on QAnon. It wasn't until after the Capitol attack that Twitter began <a href="https://blog.twitter.com/en_us/topics/company/2021/protecting--the-conversation-following-the-riots-in-washington--.html" rel="noopener noreferrer" target="_blank">permanently banning</a> accounts for primarily sharing QAnon content.</p><p>It's not always easy to predict which extreme beliefs will turn violent either, said Green, who has spent the last few years focusing on both violent <a href="https://jigsaw.google.com/the-current/white-supremacy/countermeasures/" rel="noopener noreferrer" target="_blank">white supremacists</a> and conspiracy theorists at Jigsaw. In late 2019, Green traveled to Tennessee and Alabama to meet with people who believe in a range of conspiracy theories, from flat earthers to Sandy Hook deniers. "I went into the field with a strong hypothesis that I know which conspiracy theories are violent and which aren't," Green said. But the research surprised her, as some conspiracy theorists she believed to be innocuous, like flat earthers, were far more militant followers than the ones she considered violent, like people who believed in white genocide. </p><p>"We spoke to flat earthers who could tell you which NASA scientists are propagating a world view and what they would do to them if they could," Green said.</p><p>Even more challenging: Of the 77 conspiracy theorists Green's team interviewed, there wasn't a single person who believed in only one conspiracy. That makes mapping out the scope of the threat much more complex than fixating on a single group.</p><p>The runup to the 2020 election did accelerate some of this work, in part because the real-world threat was accelerating, too. After extremists associated with the far right, anti-government boogaloo movement <a href="https://www.latimes.com/california/story/2020-06-17/far-right-boogaloo-boys-linked-to-killing-of-california-lawmen-other-violence" rel="noopener noreferrer" target="_blank">carried</a> out a series of killings in 2020, Facebook <a href="https://about.fb.com/news/2020/06/banning-a-violent-network-in-the-us/" rel="noopener noreferrer" target="_blank">designated</a> the boogaloos as a dangerous organization in June. "We saw something that we thought looked like a real network, a real entity, not just a bunch of guys using shared imagery in order to project frustration at the government," Fishman said.</p><p class="pull-quote">The reason we put policies [in place] like those we built over 2020 was because of concern about something like Jan. 6.</p><p>As the blurred lines between militia groups and conspiracy theorists became obvious throughout summer 2020, Facebook also <a href="https://about.fb.com/news/2020/08/addressing-movements-and-organizations-tied-to-violence/" rel="noopener noreferrer" target="_blank">barred</a> hundreds of militarized groups like the Oath Keepers. And, after initially attempting to merely limit the spread of QAnon, it wound up banning all accounts, pages and groups "representing QAnon" in October and launching its own version of the Redirect Method to point people searching QAnon-related terms toward more credible sources. Meanwhile, Facebook's automated filters against hate speech also made significant strides, with the company <a href="https://www.protocol.com/covid-facebook-content-moderation" target="_self">removing</a> more than double the amount of hate speech in the second quarter of 2020 as it did in the first.</p><p>YouTube <a href="https://www.washingtonpost.com/technology/2020/10/15/youtube-qanon-crackdown/" rel="noopener noreferrer" target="_blank">cracked down</a> on QAnon too, prohibiting "content that targets an individual or group with conspiracy theories that have been used to justify real-world violence." And it kicked out white supremacists like <a href="https://techcrunch.com/2020/06/29/youtube-ban-stefan-molyneux-david-duke-white-nationalism/" rel="noopener noreferrer" target="_blank">David Duke and Richard Spencer</a>, who had managed to evade the company's hate speech policies for years. Twitter, in addition to banning the <a href="https://www.thedailybeast.com/twitter-bans-far-right-militia-group-the-oath-keepers" rel="noopener noreferrer" target="_blank">Oath Keepers</a>, began aggressively labeling President Trump's tweets over the summer for glorifying violence and, after the election, for spreading misinformation about voter fraud. Twitter also launched a new policy prohibiting "<a href="https://help.twitter.com/en/rules-and-policies/coordinated-harmful-activity" rel="noopener noreferrer" target="_blank">coordinated harmful activity</a>," in June 2020, a policy that was meant to rope in dangerous mass movements and conspiracy theorists who weren't always explicitly violent. </p><p>Dorsey and Zuckerberg, at least, paid for these decisions in hours spent flagellating themselves before incensed Republicans in Congress. (YouTube CEO Susan Wojcicki has, for the most part, avoided their ire.) But to anyone who was watching closely, it was clear from the frequent announcements and policy changes coming from tech companies that these global behemoths were preparing for an election like they'd seen other countries — countries where there is no such thing as a peaceful transfer of power. "The reason we put policies [in place] like those we built over 2020 was because of concern about something like Jan. 6," Fishman said. </p><h2>What comes next?</h2><p>Then came Jan. 6, and none of those policies were enough. Like the string of ISIS attacks before it, the raid inside the U.S. Capitol showed just how permeable both the country's digital and physical defenses still were against the evolving threat of domestic extremists. "It's not just organizations," Fishman said. "It's not just structured ideologies, even. It includes folks that are haphazard adherents to various conspiracy theories. It extends from people engaged primarily in the political process to folks that are explicitly rejecting political resolution of disputes, and all of that was represented on the mall on Jan. 6."</p><p>Indeed, a recent George Washington University <a href="https://extremism.gwu.edu/sites/g/files/zaxdzs2191/f/This-Is-Our-House.pdf" rel="noopener noreferrer" target="_blank">study</a> found that the majority of rioters who have been charged so far for their involvement had no known ties to domestic violent extremist organizations. They were, instead, what the researchers call "inspired believers."</p><p>"The challenge is now to figure out what you do with all of those different pieces," Fishman said.</p><p class="media-headline"><br></p><p><br></p><p class="shortcode-media shortcode-media-rebelmouse-image">
<a href="https://media.gettyimages.com/photos/trump-supporters-clash-with-police-and-security-forces-as-they-push-picture-id1230455457?b=1&k=6&m=1230455457&s=170x170&h=aXZUKUHmyCSOiRdkxTX3TiTj6f1iS5rjCKoaEKYrVgU=" target="_blank"><img class="rm-shortcode" loading="lazy" src="https://www.protocol.com/media-library/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbWFnZSI6Imh0dHBzOi8vYXNzZXRzLnJibC5tcy8yNTc0ODkzOS9vcmlnaW4uanBnIiwiZXhwaXJlc19hdCI6MTYzOTgxMTA5Mn0.8mtmhIl18Qhp6Km4e7Rgx6H4hmSCymxFYW4Rb86Ytcw/image.jpg?width=980" id="94878" data-rm-shortcode-id="4ce12a6cc893405f9c93e51429408922" data-rm-shortcode-name="rebelmouse-image"></a>
<small class="image-media media-caption" placeholder="Add Photo Caption...">The Jan. 6 right inside the U.S. Capitol forced a reckoning on Big Tech's approach to domestic extremists.</small>
<small class="image-media media-photo-credit" placeholder="Add Photo Credit...">Photo: Roberto Schmidt/Getty Images</small>
</p><p><br></p><p>It's no coincidence that tech companies' increased willingness to crack down on far-right fringe movements comes as Democrats have taken <a href="https://www.protocol.com/facebook-ban-trump-democrats-power" target="_self">control</a> of both the White House and Congress. The same political pragmatism that stalled efforts to address domestic extremism during the Trump years will undoubtedly animate efforts to fight domestic extremism in the Biden years. But having government officials in place who actually recognize domestic extremism as a threat and prioritize it could help Big Tech's efforts, too. "Insight from government about what's happening off of Twitter is a really important part of making sure our policies keep pace with what's happening," Pickles said.<br></p><p>That doesn't mean tech companies will march in lockstep with the Biden administration when it comes to identifying domestic extremists — or that they should. After all, the United States has a long and checkered history of casting civil rights leaders as domestic extremists and using that to justify all manner of surveillance and violence. Just last year, President Trump traveled to Kenosha, Wisconsin, and <a href="https://apnews.com/article/4a58a15c9955bb6312c1fbe42215110d" target="_blank">accused</a> Black Lives Matter protesters of being domestic terrorists, too.</p><p>"I think the government has a really important role in helping educate civil society, including industry. And there are things the government can do and validate that we don't know," Fishman said. "But there are reasons Facebook designates organizations under our dangerous organizations policy ourselves."</p><p>As the Department of Homeland Security and other agencies issue warnings about domestic violent extremism, Fishman said it will be critical for those agencies to share more context about those threats so platforms can make informed decisions. "Show us the footnotes," he said. "This is going to be one of the things that I think the government needs to do more of."</p><p>Of course, there's a lot the industry could do more of too. For one thing, critics like Farid say it needs to acknowledge and take responsibility for the way its algorithms often amplify violent extremism through recommendations. "Why are they so good at targeting you with content that's consistent with your prior engagement, but somehow when it comes to harm, they become bumbling idiots?" asked Farid, who remains dubious of Big Tech's efforts to control violent extremists. "You can't have it both ways."</p><p>Facebook, for one, recently <a href="https://www.wsj.com/articles/facebook-knew-calls-for-violence-plagued-groups-now-plans-overhaul-11612131374" rel="noopener noreferrer" target="_blank">said</a> it would stop recommending political and civic groups to its users, after reportedly <a href="https://www.wsj.com/articles/facebook-knew-calls-for-violence-plagued-groups-now-plans-overhaul-11612131374" rel="noopener noreferrer" target="_blank">finding</a> that the vast majority of them included hate, misinformation or calls to violence leading up to the 2020 election. Facebook is also now experimenting with <a href="https://about.fb.com/news/2021/02/reducing-political-content-in-news-feed/" rel="noopener noreferrer" target="_blank">limiting</a> the amount of political content in some users' news feeds.</p><p>"I do think things are trending up," said Greer, whose work at the Anti-Defamation League involves pushing companies on issues of violent extremism. "There are a lot of people working to solve these challenges. We are making some headway."</p><p>But more cross-industry collaboration on this front is crucial, particularly as violent extremists move from the big three platforms to smaller sites with less formidable defenses. Today, the GIFCT database includes roughly 300,000 unique hashed images and videos that all of its members can tap into. But the majority of those images and videos are tied to organizations on the United Nations' designated entities list, which includes almost entirely Islamist extremists. </p><p>Last month, the organization's executive director, Nick Rasmussen, acknowledged this shortcoming in a letter to the GIFCT community, asking for concrete proposals for expanding the hash database. "While GIFCT member companies are able to tag or 'hash' content associated with U.N.-designated terrorist groups like ISIS and Al-Qaeda, GIFCT does not currently have a framework in place for responding to terrorist and violent extremist content from all parts of the ideological spectrum, representing a critical gap in our collective efforts to prevent terrorists and violent extremists from exploiting digital platforms," Rasmussen wrote. The organization recently brought on Erin Saltman, a leading expert in far-right extremism and Facebook's former head of the dangerous organizations team in Europe, Africa and the Middle East, to be its director of programming.</p><p>Tackling this problem will require tech companies to take a broader view of what constitutes harm, looking not just at violent threats, but other common language that links violent movements. In her research on conspiracy theorists, Green said one commonality she's found that separates passive adherents to violent ones is their sense of urgency. She cited the Pittsburgh synagogue shooter's final post, which <a href="https://abcnews.go.com/US/pittsburgh-synagogue-alleged-mass-shooter-told-swat-officers/story?id=58803485" rel="noopener noreferrer" target="_blank">read</a>, "Screw the optics. I'm going in," as evidence. "He felt that it was so urgent," Green said. "Same with Pizzagate. Same with QAnon. Same with Stop the Steal. That's why that conspiracy theory turned violent. It wasn't just the very large scale threat of an election being stolen, but also that you have to go now and take back your rights."</p><p>On March 25, Zuckerberg, Dorsey and Pichai will testify before Congress for the first time since the Capitol attack. The three men will undoubtedly come armed with talking points about how their policies have evolved and how they're committed to doing better. But that conversation will likely only be the first of many as the threat from violent extremists here in the U.S. continues to grow.</p><p>"This isn't over," Fishman said of the aftermath of Jan. 6. "This is going to be with us for a long time."</p>
Keep Reading
Show less
Issie Lapowsky (@issielapowsky) is a senior reporter at Protocol, covering the intersection of technology, politics, and national affairs. Previously, she was a senior writer at Wired, where she covered the 2016 election and the Facebook beat in its aftermath. Prior to that, Issie worked as a staff writer for Inc. magazine, writing about small business and entrepreneurship. She has also worked as an on-air contributor for CBS News and taught a graduate-level course at New York University’s Center for Publishing on how tech giants have affected publishing. Email Issie.
Latest Stories
See more
Most Popular
Bulletins
Get Source Code in your inbox
David Pierce's daily analysis of the tech news that matters.
