Enterprise

Okta CEO looks to rebuild trust after Lapsus$ breach: 'We should have done a better job'

In an interview with Protocol, Okta CEO Todd McKinnon said the cybersecurity firm could’ve done a lot of things better after the Lapsus$ breach of a third-party support provider earlier this year.

Okta CEO Todd McKinnon

From talking to hundreds of customers, “I've had a good sense of the sentiment and the frustrations,” McKinnon said.

Photo: David Paul Morris via Getty Images

Okta co-founder and CEO Todd McKinnon agrees with you: Disclosing a breach that impacts customer data should not take months.

“If that happens in January, customers can't be finding out about it in March,” McKinnon said in an interview with Protocol.

In January, the hacker group Lapsus$ found its way on to the laptop of an engineer at a third-party Okta support provider — initially thought to have given the group access to potentially hundreds of Okta customers. A later investigation that incorporated additional information found that just two customers were impacted, according to Okta.

But the breach itself was never the main concern anyway. Many honed in on the fact that it was Lapsus$, not Okta, that told the world about the incident, posting screenshots as evidence on Telegram in March. This raised more than a few questions about Okta's handling of the months-old breach.

The irony is that as a prominent identity and access management vendor, Okta is in the business of stopping the type of attack that struck its now-former support provider, Sitel. The firm was not using the Okta product or multifactor authentication on the compromised engineer’s VPN and Office 365 accounts, McKinnon said. (Sitel declined to comment on which authentication product it was using, and said in a statement that “multifactor authentication tools were and are used throughout Sitel Group's environment.” The company declined to specify if all of the compromised engineer’s accounts were secured with MFA.)

Protocol spoke to McKinnon about the customer concerns, how the breach's impact turned out to be a lot lower than previously feared and the security changes Okta has made in response.

This interview has been edited and condensed for clarity.

Looking back on Okta’s handling of the incident, what could you have done better?

The top thing in my mind is making all of the environments completely secure for the support people who are accessing Okta. We've spent so much effort making sure the Okta product and platform are secure, and then making sure the employees of Okta operate in secure environments. The third-party support organization was in another ring outside of that. So we need to make sure that's secure as well. That was really something that we could have done better. That wasn't as secure as it should have been. And so we're doing a lot of things to make sure that these technical support tools aren't used in insecure environments — making sure those environments themselves are authenticated via Okta.

This was a failure of technical enforcement.

And for the application that those support engineers use, we’re making sure that the Okta product enforces that the endpoint has a secure posture, and has the right management tools and the right malware detection and so forth — before we let anyone log in there. This was a failure of technical enforcement. So it's all about making sure we extend that security focus out one more ring, to these third parties, because that's what impacted us here.

Sitel didn’t use Okta, and that was part of the problem?

Exactly. We know this after the fact, because they brought in a forensic firm to do a full breach assessment. What we learned from that was, the way the attacker got in originally was through a VPN gateway, which didn't have multifactor authentication on it. So the most basic thing you do when you implement Okta is you make sure that all of your systems, whether it's email, or VPNs, or any of your SaaS apps, or your cloud infrastructure — all use an authentication policy that's strong. And MFA is the basic [policy] there. And then once [Lapsus$] got in, they were able to use a bunch of Windows vulnerabilities to move around and escalate privileges. They were also able to get into Office 365 — because again, it didn't have multifactor authentication on it. One of the basic things Okta does is it puts multifactor authentication on Office 365. So it's very ironic.

Why did you end your relationship with Sitel?

After the incident happened, there were too many questions about the extent of what had happened, so we decided to stop working with them. After that, I will say they were better, in terms of helping us understand the extent of it. In the beginning, they were a little slow to share all of the information. But in the end, they helped us.

This went from a five-day incident potentially affecting 366 customers to a 25-minute incident affecting two customers. Could you explain the discrepancy there, why it’s such a big change?

It’s [due to having] more information. There were two [investigation] reports. There was the original report that was done for Sitel. And then there was the report that we've sent to our customers, that was done by a different firm, that had access to all of the forensics information across both services — Okta and everything inside of Sitel. So there was more information. The original report, that established the five days and the 366 customers — the forensics firm didn't understand the potential impact to Okta. So they didn't drill into all those detailed forensics that could narrow down this impact. They just saw which machines were compromised and which Office 365 accounts were compromised. So we're actually thankful to Sitel for collaborating and giving that information to this third party, to further narrow down the impact.

One of the things I'm proud of is that, I think throughout this, we did make decisions that were helpful for customers. It would have been really easy, after the first 24 hours, to just take a super optimistic and narrow view of what the potential impact could be — given that we didn't have all the information that came out over time. We could have easily said, "It's probably just a few customers" — hoping that would be true. But we said, “The full, maximum potential impact is these 366 customers,” knowing that there was probably a 99% chance that it was going to be less than that. And then we went to all those customers, and told them that, and gave them this detailed log file analysis. Because the hardest thing was just trying to help customers through the unknown.

On the topic of your customers, there was at least one instance of a customer — the CEO of Tenable — posting a critical response to the incident.

Yes, Amit [Yoran]. I called him up. I’d never met him before, but I called him up after he posted that.

Right — and he’d felt like you had “brushed off” the incident, and you didn’t provide “actionable information” to customers. In retrospect, do you feel like that was unfair, or do you think you could’ve actually done a better job there?

I think there's many things we could’ve done better. Absolutely. There were tons of learnings. And with what he wrote, there were other, similar types of comments. I've probably talked to close to 400 CIOs and CISOs in small group meetings [about the incident]. I've had a good sense of the sentiment and the frustrations. So it's not just what he wrote, but I think it's pretty accurately representative of the frustrations of many people.

The challenge is that people looked at the timeline — January incident, hacker puts it on Telegram in March — and they made a bunch of assumptions. And if you make the assumption that Okta knew about it the whole time, and that we had it completely understood and diagnosed, and we were just not saying anything until the hacker disclosed it — then if that was true, then the way we behaved, his complaint would have been a valid complaint. But that's not true.

We didn't know about the extent of this in January. It's not an excuse. We should have done a better job to prevent it from happening and at doing all the things we could have done to potentially know more. But the fact is, we didn't know. And when the screenshots were released, it was, for all intents and purposes, when the incident started at Okta.

People may have thought, "You should’ve been telling us exactly what could have happened and what the impacts were and what access they had — because you've known about this since January." Well, we didn't know about it since January. So we had to figure it out.

But as you said, this wasn’t an uncommon reaction by customers to feel concerned about Okta’s handling of the disclosure.

I understand why they thought that. Because whether it's Okta, or a partner, or a third party — when there's a compromise like this where an attacker can see any kind of Okta support information, or customer user IDs, or email addresses — if that happens in January, customers can't be finding out about it in March. It almost doesn't matter why. We have to make sure that that does not happen. And that's what we're focused on doing.

So the first step, as I mentioned, is: We can't have the support application used in insecure environments. We’ve got to make sure we're not in that situation. And the second step is: If there are any issues, we have to make sure that we follow up on it. [In the January incident] our security operations center detected a failed account takeover attempt, and we notified the third party that there was something going on. Now these failed account takeover attempts happen pretty frequently. But if we detect one of these in our own environment, we [need to] make sure we run it down and make sure that there's nothing going on there.

The most important thing of all of this is that customers understand how seriously we're [taking this], and making sure this doesn't happen again.

Did you lose any customers over this?

Nothing significant. And part of that is because there are not a lot of good alternatives out there. What we do is pretty unique, [with] the integrations we have to different technology, and the value we can provide. And I think as long as we can explain what happened, and explain what we're doing, and make sure we build that trust back, we're going to be fine. And we’ll eventually be even stronger from this.

Fintech

Judge Zia Faruqui is trying to teach you crypto, one ‘SNL’ reference at a time

His decisions on major cryptocurrency cases have quoted "The Big Lebowski," "SNL," and "Dr. Strangelove." That’s because he wants you — yes, you — to read them.

The ways Zia Faruqui (right) has weighed on cases that have come before him can give lawyers clues as to what legal frameworks will pass muster.

Photo: Carolyn Van Houten/The Washington Post via Getty Images

“Cryptocurrency and related software analytics tools are ‘The wave of the future, Dude. One hundred percent electronic.’”

That’s not a quote from "The Big Lebowski" — at least, not directly. It’s a quote from a Washington, D.C., district court memorandum opinion on the role cryptocurrency analytics tools can play in government investigations. The author is Magistrate Judge Zia Faruqui.

Keep ReadingShow less
Veronica Irwin

Veronica Irwin (@vronirwin) is a San Francisco-based reporter at Protocol covering fintech. Previously she was at the San Francisco Examiner, covering tech from a hyper-local angle. Before that, her byline was featured in SF Weekly, The Nation, Techworker, Ms. Magazine and The Frisc.

The financial technology transformation is driving competition, creating consumer choice, and shaping the future of finance. Hear from seven fintech leaders who are reshaping the future of finance, and join the inaugural Financial Technology Association Fintech Summit to learn more.

Keep ReadingShow less
FTA
The Financial Technology Association (FTA) represents industry leaders shaping the future of finance. We champion the power of technology-centered financial services and advocate for the modernization of financial regulation to support inclusion and responsible innovation.
Enterprise

AWS CEO: The cloud isn’t just about technology

As AWS preps for its annual re:Invent conference, Adam Selipsky talks product strategy, support for hybrid environments, and the value of the cloud in uncertain economic times.

Photo: Noah Berger/Getty Images for Amazon Web Services

AWS is gearing up for re:Invent, its annual cloud computing conference where announcements this year are expected to focus on its end-to-end data strategy and delivering new industry-specific services.

It will be the second re:Invent with CEO Adam Selipsky as leader of the industry’s largest cloud provider after his return last year to AWS from data visualization company Tableau Software.

Keep ReadingShow less
Donna Goodison

Donna Goodison (@dgoodison) is Protocol's senior reporter focusing on enterprise infrastructure technology, from the 'Big 3' cloud computing providers to data centers. She previously covered the public cloud at CRN after 15 years as a business reporter for the Boston Herald. Based in Massachusetts, she also has worked as a Boston Globe freelancer, business reporter at the Boston Business Journal and real estate reporter at Banker & Tradesman after toiling at weekly newspapers.

Image: Protocol

We launched Protocol in February 2020 to cover the evolving power center of tech. It is with deep sadness that just under three years later, we are winding down the publication.

As of today, we will not publish any more stories. All of our newsletters, apart from our flagship, Source Code, will no longer be sent. Source Code will be published and sent for the next few weeks, but it will also close down in December.

Keep ReadingShow less
Bennett Richardson

Bennett Richardson ( @bennettrich) is the president of Protocol. Prior to joining Protocol in 2019, Bennett was executive director of global strategic partnerships at POLITICO, where he led strategic growth efforts including POLITICO's European expansion in Brussels and POLITICO's creative agency POLITICO Focus during his six years with the company. Prior to POLITICO, Bennett was co-founder and CMO of Hinge, the mobile dating company recently acquired by Match Group. Bennett began his career in digital and social brand marketing working with major brands across tech, energy, and health care at leading marketing and communications agencies including Edelman and GMMB. Bennett is originally from Portland, Maine, and received his bachelor's degree from Colgate University.

Enterprise

Why large enterprises struggle to find suitable platforms for MLops

As companies expand their use of AI beyond running just a few machine learning models, and as larger enterprises go from deploying hundreds of models to thousands and even millions of models, ML practitioners say that they have yet to find what they need from prepackaged MLops systems.

As companies expand their use of AI beyond running just a few machine learning models, ML practitioners say that they have yet to find what they need from prepackaged MLops systems.

Photo: artpartner-images via Getty Images

On any given day, Lily AI runs hundreds of machine learning models using computer vision and natural language processing that are customized for its retail and ecommerce clients to make website product recommendations, forecast demand, and plan merchandising. But this spring when the company was in the market for a machine learning operations platform to manage its expanding model roster, it wasn’t easy to find a suitable off-the-shelf system that could handle such a large number of models in deployment while also meeting other criteria.

Some MLops platforms are not well-suited for maintaining even more than 10 machine learning models when it comes to keeping track of data, navigating their user interfaces, or reporting capabilities, Matthew Nokleby, machine learning manager for Lily AI’s product intelligence team, told Protocol earlier this year. “The duct tape starts to show,” he said.

Keep ReadingShow less
Kate Kaye

Kate Kaye is an award-winning multimedia reporter digging deep and telling print, digital and audio stories. She covers AI and data for Protocol. Her reporting on AI and tech ethics issues has been published in OneZero, Fast Company, MIT Technology Review, CityLab, Ad Age and Digiday and heard on NPR. Kate is the creator of RedTailMedia.org and is the author of "Campaign '08: A Turning Point for Digital Media," a book about how the 2008 presidential campaigns used digital media and data.

Latest Stories
Bulletins