Power

DARPA is pouring millions into a new AI defense program. Here are the companies leading the charge

Intel and Georgia Tech are spearheading Pentagon-backed efforts to defend against attacks that could trick tomorrow's self-driving cars, facial recognition and weapons detection software.

A machine learning algorithm being spoofed by an adversarial attack.

Is that a person or a stop sign? Intel and Georgia Tech are spearheading efforts against adversarial attacks that fool machine learning systems into making such mistakes.

Photo: Georgia Institute of Technology

The Pentagon is teaming up with some of the biggest names in tech to combat hacks designed to mess with the automated systems we'll rely on in the near future.

In February, DARPA issued a call for proposals for a new program. Like most DARPA projects, it had a fantastic acronym: Guaranteeing Artificial Intelligence (AI) Robustness against Deception (GARD). It's a multimillion-dollar, four-year initiative that's aiming to create defenses for sensor-based artificial intelligence — think facial recognition programs, voice recognition tools, self-driving cars, weapon-detection software and more.

Today, Protocol can report that DARPA has selected 17 organizations to work on the GARD project, including Johns Hopkins University, Intel, Georgia Tech, MIT, Carnegie Mellon University, SRI International and IBM's Almaden Research Center. Intel will be leading one part of the project with Georgia Tech, focusing on defending against physical adversarial attacks.

Sensors that use AI computer vision algorithms can be fooled by what researchers refer to as adversarial attacks. These are basically any hack to the physical world that tricks a system into seeing something other than what's there.

In the nascent self-driving car industry, there's already concern about how these attacks could manifest. There could be stickers, effectively invisible to the human eye, that render a stop sign unrecognizable. The stickers could trick other cars into clearing the road for your own vehicle, changing a roadside sign to make a vehicle turn into a certain business's parking lot, or, in the future, even more nefarious acts.

"We certainly know from the current world that there are people who want to create mischief, and there are one-off bad actors," Bruce Draper, GARD program manager at DARPA, told Protocol. "We also know that there are larger threats out there, and if you imagine a city that in the future has many, many self-driving cars in it, then if you suddenly cause all those self-driving cars to misbehave, you could really tie up a city. You could have a major impact. An even more frightening scenario would be if the first responder vehicles were self-driving and you could disable the first responders."

The project is split among three groups. One set of organizations will be looking at the theoretical basis for adversarial attacks on AI, why they happen and how a system can be vulnerable. Another group will be building the defenses against these attacks, and the last set of teams will serve as evaluators. Every six months, they'll test the defenses others built by throwing a new attack scenario their way and looking at criteria like effectiveness and practicality.

Over the next four years, DARPA will check in with each organization to evaluate their progress. But this is a research project — at least for now, there's no hard list of deliverables those involved need to provide to the agency.



Protocol Cloud, your weekly guide to the future of enterprise computing. Sign up now.



Intel was chosen to lead the physical adversarial attacks aspect of the project, as DARPA saw promise in the company's experience in simulating external environments for self-driving cars. Intel acquired Mobileye, a vehicle computer-vision sensor company, for $15 billion in 2017. Some consider Intel to be a dark horse in the race to build autonomous vehicles. It's made other bets in AI recently, and it also shed its smartphone modem business — which was playing catch-up to others like Qualcomm as the company missed the world's computing needs shift to mobile. Whether it has enough to win future battles in AI remains to be seen, but it's sitting on a strong base now. The company beat expectations in its last earnings report, posting over $20 billion in revenue for the quarter — nearly $1 billion more than expected. (That was, of course, before the world tumbled into a pandemic.)

The compounding computing problem

Arun Chandrasekaran, vice president and analyst at Gartner, said his research firm has seen an uptick in generative adversarial network (or GAN)-generated malicious attacks. Although the sophistication of both attack and defense technology varies greatly, he said, overall, "the detection and response to it… is not as advanced as the attacks themselves."

We're still "right on the cusp" of this problem, Draper said. The use of algorithms in military and civilian settings has skyrocketed in recent years, from providing movie recommendations to managing the stock market to making decisions on the battlefield. "What you see now is that dam breaking," Draper said. "I have a hard time projecting forward five or 10 years and thinking of any major aspect of either military or civilian life that doesn't have AI embedded into it."

And the problem is compounding: As the machine-learning framework and training data available to vendors become more widely available to hackers, Chandrasekaran said he believes malicious adversarial attacks will increase. And as the number of attacks increase, so too will their effectiveness.

"Fundamentally, this is really about trying to evade the existing detection systems," Chandrasekaran said. "You have existing systems that rely on a combination of images — in some cases, this could be voice and voice recognition… The fact that you could recreate something that's eerily similar to your voice or your image means that in many cases, you can essentially bypass the accuracy of your existing prevention system." These existing mechanisms must "significantly evolve," he added, to ensure confidence in their security.

Intel's currently focusing on the future — plugging in vulnerability holes and getting ahead of the threats downstream. "An important thing to know about this particular topic is this isn't a today threat," Jason Martin, a senior staff research scientist at Intel Labs, said. But it's a rarity in research to be able to spend time worrying about tomorrow's problems. "It's a nice place to be; it's not a 'panic now' sort of scenario," he said. "It's a 'calmly do the research and come up with the mitigations.'"

Intel and Georgia Tech have partnered on adversarial attack research for years. One of their focuses has been the ease with which bad actors can trick an algorithm into thinking a bird is a bicycle, for example, or mislabeling a stop sign — just by changing a few pixels.

The research so far, led by Duen Horng "Polo" Chau, associate professor of computing at Georgia Tech, has landed on an especially relevant takeaway: If you can't make something invulnerable from an attack, then make it computationally infeasible. For example, in some cryptography systems, there's some probability of an attacker figuring out the code key by using up considerable computing resources, but it's so improbable that it approaches impossible. Martin wants to approach the defense of physical adversarial attacks in a similar way: "The hope is that the combination of techniques in the defensive realm will make the cost of constructing an adversarial example too expensive," he said.

Intel and Georgia Tech's plan of attack

Intel and Georgia Tech plan to use some of the largest open-source image datasets — ImageNet and Microsoft's Common Objects in Context, for example. Open-source data is a DARPA requirement, and they're prioritizing research reproducibility. But those datasets are public and widely used. It begs the question: How does this effort differ from past ones?

Chau argues their innovation lies in how they plan to use the data. The plan boils down to teaching AI "coherence" — helping it see things more holistically, or, in a way, exercise common sense. It's something humans typically are born with or learn early on — and it's extremely difficult, if near impossible, to recreate with technology. Intel and Georgia Tech aim to provide three concrete solutions for a nebulous issue: temporal, semantic and spatial coherence.

Temporal coherence here relates to understanding of physics — things don't typically suddenly appear or disappear out of nowhere. For example, if a self-driving car registers a human, a stop sign or another object flickering into its view and then vanishing, then a hacker could be tampering with its system.

Semantic coherence relates to meaning. Humans identify things as a sum of their parts — a bird comprises eyes, wings and a beak, for example. The research team's plan is to incorporate a second line of defense into a sensing system — if it registers a bicycle, then it should next check for the wheel, handlebar and pedals. If it doesn't find those components, then something is likely wrong.



Get in touch with us: Share information securely with Protocol via encrypted Signal or WhatsApp message, at 415-214-4715 or through our anonymous SecureDrop.



Then there's spatial coherence, or knowledge of the relative positioning of things. If an object detector senses people floating in midair, for example, then that should be a red flag. And for all three of these strategies, the team hopes to not only teach object detectors to flag an attack but also correct it.

"It's definitely a good place to start," Chandrasekaran said of Intel and Georgia Tech's plan. He compared it to Cornell University research that used a deep forgery discriminator (a system that discriminates between real and fake items) to use human-like judgment to gauge whether or not an image of a face was fake.

Implications for the future

Chau's long-term priority is scalability, he said — making the team's research and techniques transparent by delivering open-source software. He acknowledged the risk involved in publicizing information that bad actors could use to their advantage, but argued it's even more important for people in charge of the technology to be informed. And AI systems are often a black box anyway.

"The reality is that sometimes these AI and machine learning techniques do not work — and more dangerously, sometimes it works this second, and the next second a completely different [outcome is] produced," Chau said. "It's important to know, when it works well, why does it work so well and, more importantly, when someone is really deliberately attacking it, intentionally, what is it really exploiting?"

In order for the team to counter threats, it's vital for them to proactively discover vulnerabilities that bad actors aren't yet aware of. If they don't, bad actors could end up with the tools to disassemble any new techniques they use.

"Because we're not convinced that we'll necessarily find the perfect defense, we're trying to advance the theory [and] figure out, 'What are the limits?'" Draper said. "We're going to try to defend them as best we can, make them as invulnerable as possible, but we also want to have enough of a theoretical background to develop the theory in such a way that we can tell people, when they're deploying an AI system, the extent to which it may be vulnerable or not."

And for the critical systems of tomorrow, that could make all the difference. "If you're doing a system whose job it is to recommend movies, maybe you're not that terrified of an attack — how much damage could an adversary do?" Draper said. "But if you're doing a self-driving car, then someone's life is on the line."

Fintech

Judge Zia Faruqui is trying to teach you crypto, one ‘SNL’ reference at a time

His decisions on major cryptocurrency cases have quoted "The Big Lebowski," "SNL," and "Dr. Strangelove." That’s because he wants you — yes, you — to read them.

The ways Zia Faruqui (right) has weighed on cases that have come before him can give lawyers clues as to what legal frameworks will pass muster.

Photo: Carolyn Van Houten/The Washington Post via Getty Images

“Cryptocurrency and related software analytics tools are ‘The wave of the future, Dude. One hundred percent electronic.’”

That’s not a quote from "The Big Lebowski" — at least, not directly. It’s a quote from a Washington, D.C., district court memorandum opinion on the role cryptocurrency analytics tools can play in government investigations. The author is Magistrate Judge Zia Faruqui.

Keep ReadingShow less
Veronica Irwin

Veronica Irwin (@vronirwin) is a San Francisco-based reporter at Protocol covering fintech. Previously she was at the San Francisco Examiner, covering tech from a hyper-local angle. Before that, her byline was featured in SF Weekly, The Nation, Techworker, Ms. Magazine and The Frisc.

The financial technology transformation is driving competition, creating consumer choice, and shaping the future of finance. Hear from seven fintech leaders who are reshaping the future of finance, and join the inaugural Financial Technology Association Fintech Summit to learn more.

Keep ReadingShow less
FTA
The Financial Technology Association (FTA) represents industry leaders shaping the future of finance. We champion the power of technology-centered financial services and advocate for the modernization of financial regulation to support inclusion and responsible innovation.
Enterprise

AWS CEO: The cloud isn’t just about technology

As AWS preps for its annual re:Invent conference, Adam Selipsky talks product strategy, support for hybrid environments, and the value of the cloud in uncertain economic times.

Photo: Noah Berger/Getty Images for Amazon Web Services

AWS is gearing up for re:Invent, its annual cloud computing conference where announcements this year are expected to focus on its end-to-end data strategy and delivering new industry-specific services.

It will be the second re:Invent with CEO Adam Selipsky as leader of the industry’s largest cloud provider after his return last year to AWS from data visualization company Tableau Software.

Keep ReadingShow less
Donna Goodison

Donna Goodison (@dgoodison) is Protocol's senior reporter focusing on enterprise infrastructure technology, from the 'Big 3' cloud computing providers to data centers. She previously covered the public cloud at CRN after 15 years as a business reporter for the Boston Herald. Based in Massachusetts, she also has worked as a Boston Globe freelancer, business reporter at the Boston Business Journal and real estate reporter at Banker & Tradesman after toiling at weekly newspapers.

Image: Protocol

We launched Protocol in February 2020 to cover the evolving power center of tech. It is with deep sadness that just under three years later, we are winding down the publication.

As of today, we will not publish any more stories. All of our newsletters, apart from our flagship, Source Code, will no longer be sent. Source Code will be published and sent for the next few weeks, but it will also close down in December.

Keep ReadingShow less
Bennett Richardson

Bennett Richardson ( @bennettrich) is the president of Protocol. Prior to joining Protocol in 2019, Bennett was executive director of global strategic partnerships at POLITICO, where he led strategic growth efforts including POLITICO's European expansion in Brussels and POLITICO's creative agency POLITICO Focus during his six years with the company. Prior to POLITICO, Bennett was co-founder and CMO of Hinge, the mobile dating company recently acquired by Match Group. Bennett began his career in digital and social brand marketing working with major brands across tech, energy, and health care at leading marketing and communications agencies including Edelman and GMMB. Bennett is originally from Portland, Maine, and received his bachelor's degree from Colgate University.

Enterprise

Why large enterprises struggle to find suitable platforms for MLops

As companies expand their use of AI beyond running just a few machine learning models, and as larger enterprises go from deploying hundreds of models to thousands and even millions of models, ML practitioners say that they have yet to find what they need from prepackaged MLops systems.

As companies expand their use of AI beyond running just a few machine learning models, ML practitioners say that they have yet to find what they need from prepackaged MLops systems.

Photo: artpartner-images via Getty Images

On any given day, Lily AI runs hundreds of machine learning models using computer vision and natural language processing that are customized for its retail and ecommerce clients to make website product recommendations, forecast demand, and plan merchandising. But this spring when the company was in the market for a machine learning operations platform to manage its expanding model roster, it wasn’t easy to find a suitable off-the-shelf system that could handle such a large number of models in deployment while also meeting other criteria.

Some MLops platforms are not well-suited for maintaining even more than 10 machine learning models when it comes to keeping track of data, navigating their user interfaces, or reporting capabilities, Matthew Nokleby, machine learning manager for Lily AI’s product intelligence team, told Protocol earlier this year. “The duct tape starts to show,” he said.

Keep ReadingShow less
Kate Kaye

Kate Kaye is an award-winning multimedia reporter digging deep and telling print, digital and audio stories. She covers AI and data for Protocol. Her reporting on AI and tech ethics issues has been published in OneZero, Fast Company, MIT Technology Review, CityLab, Ad Age and Digiday and heard on NPR. Kate is the creator of RedTailMedia.org and is the author of "Campaign '08: A Turning Point for Digital Media," a book about how the 2008 presidential campaigns used digital media and data.

Latest Stories
Bulletins