Artificial intelligence tools have the potential to become as essential to medical research and patient care as centrifuges and x-ray machines. Advances in high-accuracy predictive modeling can enable providers to analyze a range of patient risk factors to facilitate better health care outcomes — from preventing the onset of complications during surgery, to assessing the risk of developing various diseases.
When it comes to emergency services or critical care settings, however, the potential benefits of AI in the treatment room are often outweighed by the costs. And in this case, the talk about cost in health care isn’t just about money.
“With sufficient data and the right parameters, AI models perform quite well when asked to predict clinical outcomes. But in this case, ‘sufficient data’ often translates as an impractical number of patient features to collect in many care settings,” noted Gabriel Erion (Ph.D., ‘21), who is combining an M.D. with a Ph.D. in computer science as part of the University of Washington’s Medical Scientist Training Program. “The cost, in terms of the time and effort required to collect that volume and variety of data, would be much too high in an ambulance or intensive care unit, for example, where every second counts and responders need to prioritize patient care.”
But thanks to Erion and collaborators at the UW’s Paul G. Allen School and the UW School of Medicine, providers needn’t make a choice between caring directly for patients and leveraging advances in AI to identify the interventions with the highest likelihood of success. In a paper published in Nature Biomedical Engineering, the team presents CoAI, short for Cost-Aware Artificial Intelligence, a new framework for dramatically reducing the time, effort and resources required to predict patient outcomes and inform treatment decisions without sacrificing the accuracy of more cost-intensive tools.
To reduce the number of clinical risk factors required to be collected in real time, the researchers trained CoAI on a massive dataset combining patient features, prediction labels, expert annotations of feature cost, and a budget representing total acceptable cost. They applied Shapley values to calculate a quantitative measure of the predictive power of every single feature in the dataset; since Shapley values are additive, this approach enables CoAI to calculate the importance of a group of features relative to their cost. CoAI then recommends which subset of features would enable the most accurate prediction of patient risk within a specified budget. And some of those budgets are very tight, indeed.
“Fifty seconds. That’s how long first responders told us they can spare to score patient risk factors when they are in the midst of performing a life-saving intervention,” said co-senior author and professor Su-In Lee, who leads the Allen School’s AIMS Lab focused on integrating AI and the biomedical sciences. “CoAI deals with this constraint by prioritizing a subset of features to gather while achieving the same or better accuracy in its predictions as other, less cost-aware models. And it is generalizable to a variety of care settings, such as cancer screening, where different feature costs come into play — including financial considerations.”
As co-author Joseph Janizek (Ph.D., ‘22) explained, CoAI has a significant advantage over even other cost-sensitive methods owing to its efficiency and flexibility.
“A notable difference between CoAI and other approaches is its robustness to ‘cost shift,’ wherein features become more or less expensive after the model has been trained. Since our framework decouples feature selection from training, CoAI continues to perform well even when this shift occurs,” noted Janizek, who is also pursuing his M.D. in combination with a Ph.D. from the Allen School via the MSTP. “And because it’s model-agnostic, CoAI can be used to adapt any predictive AI system to be cost-aware, enabling accurate predictions at lower cost within a wide variety of settings.”
Janizek and his AIMS Lab colleagues teamed up with clinicians at the UW School of Medicine and first responders with Airlift Northwest, American Medical Response and the Seattle Fire Department to validate the CoAI approach. In a series of experiments, the researchers evaluated CoAI’s performance compared to typical AI models in predicting the increased bleeding risk of trauma patients en route to the hospital and the in-hospital mortality risk of critical care patients in the ICU. They also surveyed first responders and nurses to understand how patient risk scoring works in practice — hence the aforementioned 50-second rule. In the case of trauma response, their experiments showed that CoAI dramatically reduces the cost of data acquisition — by around 90% — while still achieving levels of accuracy comparable to other, more cost-intensive approaches. They achieved similar results for the inpatient critical care setting.
According to co-senior author Dr. Nathan White, associate professor of Emergency Medicine at the UW School of Medicine, these results speak to what is possible when researchers break down barriers between disciplines and prioritize how new technologies will be put to real-world use.
“A key contributor to the success of this project included the great synergy afforded by working across traditional silos of medicine and engineering,” said White. “AI is an important component of healthcare today, but we must always be aware of the clinical situations where AI is being used and seek out input from frontline health care workers involved directly in patient care. This will ensure that AI is always working optimally for the patients it intends to benefit.”
Lee agreed, noting that the UW’s MSTP serves to enhance this synergy with each new student who enters the program.
“Gabe and Joe were the first UW MSTP students to earn their Ph.D. in the Allen School. They exemplify the best of both worlds, combining rigorous computer science knowledge with hands-on clinical expertise,” Lee said. “This nexus of knowledge, spanning two traditionally disparate disciplines, will be essential to our future progress in developing AI as an effective and efficient tool used in biomedical research and treatment decisions.”
Dr. White’s colleagues in the Department of Emergency Medicine, Drs. Richard Utarnachitt, Andrew McCoy and Michal Sayre, along with Dr. Carly Hudelson of the Division of General Internal Medicine, are co-authors of the paper. An early preview of the project earned the Madrona Prize sponsored by Madrona Venture Group at the Allen School’s annual research day in 2019. The research was funded by the National Science Foundation, American Cancer Society, and National Institutes of Health.