The Rise of Medical AI: Are We Testing Them Right?

Artificial intelligence is making its way into almost every corner of healthcare. From breast cancer screenings to virtual nurses and even transcribing conversations between doctors and patients, AI tools are becoming more prevalent. The promise is that these technologies will make healthcare more efficient and lighten the load on medical professionals. However, there's a growing concern about whether these AI tools are as effective as advertised.

The Challenge of Testing AI in Healthcare

AI tools, particularly large language models (LLMs), rely heavily on the data they're trained on. Unfortunately, many of the assessments available today are based on medical student exams like the MCAT, which don't fully reflect real-world situations. A review found that only a tiny fraction of studies—about 5%—used actual patient data to evaluate these models. Most assessments focused on medical knowledge rather than practical tasks such as writing prescriptions or interacting with patients, which are crucial in real-world applications.

Why Current Benchmarks Fall Short

The benchmarks currently used to test these AI systems don't capture the complexities of real-life medical scenarios. Computer scientist Deborah Raji and her colleagues have pointed out that these tests are too rigid and don't adequately measure clinical abilities. They also tend to overlook the roles of nurses and other healthcare staff, focusing primarily on physicians' knowledge.

Raji notes that the optimism surrounding these benchmarks is leading to AI systems being deployed in real-world settings, despite their limitations. She advocates for developing evaluations that better reflect the diverse and complex tasks AI will face in healthcare environments.

Creating Better Evaluations

To improve how we evaluate healthcare AI, Raji suggests several strategies. One approach is to involve domain experts to understand practical workflows and gather naturalistic datasets from pilot interactions. This can help identify the range of queries people might use and the outputs generated by the AI. Another method is "red teaming," where a group actively challenges the model to see how it responds to different prompts.

Additionally, gathering data from hospitals about how these AI tools are actually used could inform future benchmarking practices. Drawing on methods from other fields, like psychology, could also help ground evaluations in real-world observations.

The Need for Specialized Testing

Raji emphasizes the importance of tailoring benchmarks to specific tasks. For instance, testing a model's ability to summarize doctors' notes is quite different from assessing its knowledge recall. While every task doesn't need a unique benchmark, the tests should be more relevant than simple multiple-choice questions, which don't truly reflect a doctor's performance.

Moving Toward Realistic Evaluations

Raji calls for researchers to invest more in developing evaluations that align with real-world expectations for AI systems. She also suggests that hospitals could be more transparent about the AI tools they use, sharing information about how they're integrated into clinical practices. This transparency could lead to better evaluations and help bridge the gap between current practices and more realistic assessments.

Advice for AI Practitioners

For those working with AI models, Raji advises being more thoughtful about the evaluations used. While medical exams are readily available, they don't represent the full scope of what these models should achieve in practice. She challenges the field to construct evaluations that truly reflect the intended use and expectations of these AI systems once they're deployed.