The benchmark passed. The deployment didn't

Healthcare systems and banks are selecting AI on evaluation scores designed for a different problem than the one they're actually trying to solve. The consequences are showing up in audits, not in dashboards.

In 2013, MD Anderson Cancer Center began one of the most ambitious clinical AI projects in healthcare history. The partnership with IBM was designed to bring Watson for Oncology into cancer care, a system that would synthesize medical literature, interpret clinical records, and recommend treatment protocols for leukemia and other cancers. On the curated benchmark cases used for evaluation, Watson performed impressively. By 2017, MD Anderson had spent approximately $62 million and canceled the project. Watson's recommendations had conflicted with oncologist judgment in too many real cases to be trusted in clinical care. The system was never deployed with patients. The benchmark had been built on textbook cases. Real patients are not textbook cases.

A decade later, healthcare systems and banks are making AI deployment decisions, at significantly larger scale and faster speed, with the same fundamental mismatch between how models are evaluated and the conditions they actually operate in. And the gap is widening, not closing, because the pace of AI development has outstripped the institutions we've built to measure it.

When the same model performs differently in a different hospital

The IBM Watson story is sometimes treated as an outlier, a cautionary tale about one overreaching project. The Epic Sepsis Model is harder to dismiss. Epic's deterioration index, a sepsis prediction tool embedded in its EHR and deployed across hundreds of U.S. health systems, showed strong performance in its developer's internal validation. When researchers at the University of Michigan externally validated the same model against 38,455 real hospitalizations, they published the results in JAMA Internal Medicine in 2021. The model's sensitivity was 33%. It missed two of every three sepsis cases. The area under the curve, the standard performance measure, was 0.63, substantially worse than the developer's original reporting. It was the same model. It was a different patient population. The benchmark did not transfer.

For a CIO or CMO deploying that model across a health system, the practical implication is direct. A sepsis alert system with 33% sensitivity is generating a significant number of missed cases that a clinical team believes the system is covering. The model is running. The dashboards show no red flags. The performance gap is invisible until a patient outcome or a quality review surfaces it.

The same mechanism plays out differently in financial services

In banking, the distribution shift problem takes a different form but produces the same structural risk. A bank evaluating an AML transaction monitoring system or a credit underwriting model is comparing vendor benchmark scores generated on curated financial datasets: standard transaction structures, US federal regulatory frameworks, loan types common enough to appear in academic training corpora. A bank with significant cross-border volume or exposure to multiple regulatory regimes will encounter a production distribution that diverges from that benchmark in ways that only surface in deployment. By then, the contract is signed and the compliance team is asking questions.

A 2025 GAO report examining AI use across US financial regulators confirms that the OCC, CFPB, and FCA are all actively applying existing governance frameworks to AI systems in financial services. The expectation, increasingly, is that organizations can demonstrate that their AI systems perform as intended across the specific populations and contexts they're deployed in, not just against a general benchmark. A leaderboard score is not a compliance posture. For organizations where that bridge between benchmark and production reality doesn't yet exist, the gap is a regulatory finding waiting to happen.

The benchmark half-life problem

The MD Anderson and Epic cases both involve a version of this problem that predates the current era of generative AI. The version in 2026 is faster and harder to track. Humanity's Last Exam, a benchmark specifically designed to be difficult for years, saw its top model scores rise from roughly 8% at launch in early 2025 to best models clearing 50% by April 2026, according to the Stanford AI Index 2026. Benchmarks that represent genuine cutting-edge capability today are being saturated within 12 to 18 months. Enterprise procurement cycles move more slowly. Organizations that bought on last quarter's benchmark may be running a model that has already been surpassed on the metric they selected it for, while the next generation of the benchmark has already moved on.

The Stanford AI Index 2026 notes explicitly that evaluation frameworks are being outpaced by the progress they were built to measure. In healthcare and financial services, where model behavior carries direct consequences for patient outcomes and regulatory compliance, that outpacing is not an abstract research concern. It is a governance gap.

What evaluation designed for your context actually requires

The organizations navigating this successfully are supplementing vendor benchmarks with evaluation designed around their actual operating context, tested on their patient population or transaction distribution, not a vendor's test set, and reviewed by clinical or domain practitioners who understand where failure is costly. That kind of evaluation is not something most AI vendors include in their standard offer, and it's not something most internal procurement teams are resourced to design themselves.

This work requires domain expertise that sits at the intersection of regulated industry knowledge and AI evaluation methodology, the ability to construct a test that reflects what actually happens when a model is wrong in a clinical setting, or wrong in a way a compliance team hasn't anticipated yet. That expertise is rarely inside the same organization buying the model.

The MD Anderson project is nearly a decade old. The lesson it offers, that the benchmark used to select a system should represent the problem the system will actually face, has not yet become standard practice in healthcare or banking AI procurement. In 2026, with the pace of model development and the scale of deployment both accelerating, the cost of that gap is higher than it has ever been. The organizations closing it are the ones working with partners who can evaluate what the benchmark never tested, before a deployment is the thing that does the testing.