Andrew Schroeder has been trying for a while now to figure out which of California's community health centers are equipped to handle and distribute the COVID-19 vaccine. These are the places that serve California's poorest people — more than 7 million of them, to be exact — and, as vice president of research and analysis for the humanitarian aid organization Direct Relief, Schroeder wants to make sure they're ready for the vaccine as more supply becomes available.
So, working with the California Primary Care Association, he recently sent a survey around to the organizations that manage some 1,370 federally qualified healthcare centers in the state, asking questions about their power sources, their refrigeration capabilities and their staffing. In the end, the survey yielded data on just 106 sites — less than 8% of the total.
"We respond to so many disasters across California, and we just run into this repeated frustration at not having enough detailed information at the site level," he said, noting that surveys are notoriously unhelpful in disaster response scenarios.
But sending vaccines to facilities that can't store them isn't an option when supply is so limited and the public health need is so great. So this time, Schroeder is taking a new approach. Last week, Direct Relief and the California Primary Care Association launched a project with a Washington-based firm called Macro-eyes that's promising to use machine learning to predict the vaccine readiness of every community health center in the state.
Founded in 2013 by an MIT professor and former hedge fund trader, Macro-eyes uses a combination of publicly available data, government-provided data and satellite imagery to help healthcare officials "predict the present," as CEO Benjamin Fels put it.
If a government wants to determine whether a given healthcare facility has backup power, for instance, Macro-eyes might start by, say, reviewing satellite imagery to see if there's a solar panel on the roof or using a web scraper to mine stories about recent investments in that facility. The company uses every scrap of information it finds as an input in its model, so it can then predict the characteristics of other, similarly situated facilities where data is more scarce.
"There's a portion of what we're doing where we are aggregating data, and there's a portion of what we're doing where we're using that data we aggregated to predict what's missing," Fels said.
Macro-eyes has already begun this work in parts of Africa where it's predicting healthcare infrastructure, population characteristics and healthcare utilization in countries like Sierra Leone and Tanzania, backed by more than $3 million in funding from the Bill & Melinda Gates Foundation. But this is the first deployment of this technology in the United States.
"The more data we have, the more we can share with the state and the county, and the better it is for our community health centers, which we know are the key players in reaching diverse populations and the really hard-to-reach populations," said Robert Beaudry, executive vice president of the California Primary Care Association.
Using machine learning in any healthcare setting has traditionally been a fraught endeavor. While these tools hold the promise of improved diagnostics and resource allocation, stories abound of algorithms inadvertently entrenching racial biases in healthcare data and prioritizing care for white patients over Black patients with similar chronic illnesses.
When it comes to vaccine distribution, the stakes couldn't be any higher. A miscalculation about a given healthcare facility's readiness could result in well-prepared facilities and their patients being passed over for life-saving vaccines or, worse yet, vaccines going to waste in facilities that can't properly store them.
The potential downside isn't lost on Schroeder or Fels, so they're starting small. In the initial phase of the project, their goal is to predict three things about each facility: how much refrigeration it has, whether it has a backup power source and whether it serves a population that is at high risk of complications with COVID-19. Then, they plan to go back to a subset of those facilities to validate their predictions. That, they say, will be easier than constantly calling or surveying thousands of facilities at the height of a health crisis. It will at least create a pool of ground truth data and help them understand how accurate their predictions really were in the first place. Those answers will be fed back into the model to improve the rest of the predictions.
This, Fels said, is the difference between a static algorithm and a true machine-learning model.
"You're baking in that engagement with being proven wrong all the time, and that's a good thing," he said.
Besides, Schroeder argues, it's better than the alternative, which is working with next to no information at all. That has left community health centers sidelined when it comes to vaccine distribution. Until now, the state and local health agencies have mostly sent vaccines to large healthcare systems that they know are equipped to handle vaccinations. It also happens to be where the densest population of doctors and nurses, who were first in line for vaccines, happen to work. But as California's distribution plans expand, community health centers will need to play an increasingly important role, Schroeder argued, because they already have relationships with the very people who need those vaccines most.
"If we were leading with things like building community trust, the health centers would be at the forefront of how we think about the vaccine," Schroeder said, "but they're really in the back seat."
