How I decided on the right AI auditor for my hiring tech company

Like other recruitment tech companies that use AI-based systems, Hired was in the market earlier this year for an algorithmic bias audit. But its chief technology officer, Dave Walters, didn’t want to merely check the AI fairness compliance box. Here’s how his team chose an auditing partner for the long haul.

Dave Walters headshot

"I would definitely suggest that companies start this process early — now," says Hired chief technology officer Dave Walters.

Photo: Hired

Click banner image for more How I decided series

Automated and algorithmic systems used to decide whether people get loans or jobs have attracted heightened scrutiny, compelling legislators to propose laws requiring audits of those systems. In New York City, there’s already a law requiring companies that provide automated employment decision tools to get their tech inspected, and they have just a few months left to comply before it goes into effect in January 2023.

Hired is one of those companies. Hired’s chief technology officer, Dave Walters, spent the early months of this year evaluating a growing pool of auditing services in the hopes of finding one that would not only help Hired check the AI fairness compliance box, but also offer deeply technical services for a long-term partnership. In the end, Hired chose London-based Holistic AI, which currently is evaluating the machine-learning models Hired uses for matching job seekers with roles that suit their skill sets.

Walters’ story, as told to Protocol, has been edited for clarity and brevity.

We cast a pretty wide net earlier this year, speaking to a number of different companies that were getting into the space. Of course, one of the things we found is generally everybody was getting into the space. This is a new demand, a new area that companies are spinning up to meet. Many of those companies [have come] from academia; they've been in that research area and in that mindset of doing this type of work, but just not necessarily monetizing in this way.

Because of how new it was, there was no single standard approach that every auditor was taking. That is the same with any new paradigm in any industry; there's going to be a certain amount of differential between approaches until some of those approaches get vetted out or rise to the top as the preferred approach.

We spoke with each of them. We started to narrow down that pool. Some of the key metrics we looked at:

  • How new was the company: Did they just spin up this year? Or maybe they've been around for a year or two at least — anticipating some of the early adopters in the industry?
  • What kind of funding did they have … We wanted to make sure that whoever we partnered with was going to have that level of stability.
  • We looked at the auditing, not as a point-in-time audit, [where] we just check this box, go through an audit, then we move on and forget about it. But for us, this is a long-term initiative and partnership.

And as we narrowed down the field to the last couple of potential partners, we also then asked those candidates to provide us with sample reports from audits that they've done previously. We were able to see those public-facing, unredacted reports and see some of the more internal, [redacted] reports to have a better feel for what our end result would be.

There definitely was one that we saw that was not as comprehensive as we would have liked. We had some anticipation that might be the case based on some conversations, but then that helped solidify and crystallize that for us.

Seeing that report and understanding how that's going to translate into your business, and how that's going to benefit you, I think is critical.

Some we did not feel in their proposed approach were going to be digging as deep into the models. We felt very strongly that whoever we partnered with, it was going to be important that they have access to our models and access to our data without creating any security risks, or [personally identifiable information] risks. And so, they are able to fully evaluate the model and the results of the model. We worked with our internal IT group and security teams as well their team. I’ve leveraged some third-party tools to ensure that our working with them is done securely.

The easier path would have been to say, “You know what — we're going to avoid any kind of data and expect that you try to audit us based purely on the models themselves.” But we ultimately felt that was not going to give us the best result.

I would definitely suggest that companies start this process early — now. We’re already halfway through the year, so that is a little bit difficult for companies at this point that haven't started. But maybe for future years, because this will be an annual commitment for New York state to stay current, it’s going to take time to do that research on the possible partners to work with and to vet them out, and to kick off the actual audit and work through that audit.


Inside Amazon’s free video strategy

Amazon has been doubling down on original content for Freevee, its ad-supported video service, which has seen a lot of growth thanks to a deep integration with other Amazon properties.

Freevee’s investment into original programming like 'Bosch: Legacy' has increased by 70%.

Photo: Tyler Golden/Amazon Freevee

Amazon’s streaming efforts have long been all about Prime Video. So the company caught pundits by surprise when, in early 2019, it launched a stand-alone ad-supported streaming service called IMDb Freedive, with Techcrunch calling the move “a bit odd.”

Nearly four years and two rebrandings later, Amazon’s ad-supported video efforts appear to be flourishing. Viewership of the service grew by 138% from 2020 to 2021, according to Amazon. The company declined to share any updated performance data on the service, which is now called Freevee, but a spokesperson told Protocol the performance of originals in particular “exceeded expectations,” leading Amazon to increase investments into original content by 70% year-over-year.

Keep Reading Show less
Janko Roettgers

Janko Roettgers (@jank0) is a senior reporter at Protocol, reporting on the shifting power dynamics between tech, media, and entertainment, including the impact of new technologies. Previously, Janko was Variety's first-ever technology writer in San Francisco, where he covered big tech and emerging technologies. He has reported for Gigaom, Frankfurter Rundschau, Berliner Zeitung, and ORF, among others. He has written three books on consumer cord-cutting and online music and co-edited an anthology on internet subcultures. He lives with his family in Oakland.

Sponsored Content

Great products are built on strong patents

Experts say robust intellectual property protection is essential to ensure the long-term R&D required to innovate and maintain America's technology leadership.

Every great tech product that you rely on each day, from the smartphone in your pocket to your music streaming service and navigational system in the car, shares one important thing: part of its innovative design is protected by intellectual property (IP) laws.

From 5G to artificial intelligence, IP protection offers a powerful incentive for researchers to create ground-breaking products, and governmental leaders say its protection is an essential part of maintaining US technology leadership. To quote Secretary of Commerce Gina Raimondo: "intellectual property protection is vital for American innovation and entrepreneurship.”

Keep Reading Show less
James Daly
James Daly has a deep knowledge of creating brand voice identity, including understanding various audiences and targeting messaging accordingly. He enjoys commissioning, editing, writing, and business development, particularly in launching new ventures and building passionate audiences. Daly has led teams large and small to multiple awards and quantifiable success through a strategy built on teamwork, passion, fact-checking, intelligence, analytics, and audience growth while meeting budget goals and production deadlines in fast-paced environments. Daly is the Editorial Director of 2030 Media and a contributor at Wired.

Wall Street is warming up to crypto

Secure, well-regulated technology infrastructure could draw more large banks to crypto.

Technology infrastructure for crypto has begun to mature.

Illustration: Christopher T. Fong/Protocol

Despite a downturn in crypto markets, more large institutional investors are seeking to invest in crypto.

One factor holding them back is a lack of infrastructure for large institutions compared to what exists in the traditional, regulated capital markets.

Keep Reading Show less
Tomio Geron

Tomio Geron ( @tomiogeron) is a San Francisco-based reporter covering fintech. He was previously a reporter and editor at The Wall Street Journal, covering venture capital and startups. Before that, he worked as a staff writer at Forbes, covering social media and venture capital, and also edited the Midas List of top tech investors. He has also worked at newspapers covering crime, courts, health and other topics. He can be reached at tgeron@protocol.com or tgeron@protonmail.com.


How I decided to go all-in on a federal contract — before assignment

Amanda Renteria knew Code for America could help facilitate access to expanded child tax credits. She also knew there was no guarantee her proof of concept would convince others — but tried anyway.

Code for America CEO Amanda Renteria explained how it's helped people claim the Child Tax Credit.

Photo: Code for America

Click banner image for more How I decided series

After the American Rescue Plan Act passed in March 2021, the U.S. government expanded child tax credits to provide relief for American families during the pandemic. The legislation allowed some families to nearly double their tax benefits per child, which was especially critical for low-income families, who disproportionately bore the financial brunt of the pandemic.

Keep Reading Show less
Hirsh Chitkara

Hirsh Chitkara ( @HirshChitkara) is a reporter at Protocol focused on the intersection of politics, technology and society. Before joining Protocol, he helped write a daily newsletter at Insider that covered all things Big Tech. He's based in New York and can be reached at hchitkara@protocol.com.


This carbon capture startup wants to clean up the worst polluters

The founder and CEO of point-source carbon capture company Carbon Clean discusses what the startup has learned, the future of carbon capture technology, as well as the role of companies like his in battling the climate crisis.

Carbon Clean CEO Aniruddha Sharma told Protocol that fossil fuels are necessary, at least in the near term, to lift the living standards of those who don’t have access to cars and electricity.

Photo: Carbon Clean

Carbon capture and storage has taken on increasing importance as companies with stubborn emissions look for new ways to meet their net zero goals. For hard-to-abate industries like cement and steel production, it’s one of the few options that exist to help them get there.

Yet it’s proven incredibly challenging to scale the technology, which captures carbon pollution at the source. U.K.-based company Carbon Clean is leading the charge to bring down costs. This year, it raised a $150 million series C round, which the startup said is the largest-ever funding round for a point-source carbon capture company.

Keep Reading Show less
Michelle Ma

Michelle Ma (@himichellema) is a reporter at Protocol covering climate. Previously, she was a news editor of live journalism and special coverage for The Wall Street Journal. Prior to that, she worked as a staff writer at Wirecutter. She can be reached at mma@protocol.com.

Latest Stories