A single factuality score is a single point of failure

Hallucination rates vary by an order of magnitude across domains. The model card number that gave your procurement team confidence was never tested against the questions your clinicians, compliance analysts, or loan officers actually ask.

Here is the number that matters more than any model card: hallucination rates vary by an order of magnitude depending on the domain you test. A model that fabricates information 1.5% of the time when summarizing primary-care consultations can fabricate it 17–33% of the time when answering legal research queries. That spread is not a rounding error. It is the distance between a system that works and one that generates regulatory exposure every time it runs.

Yet most enterprise AI procurement still relies on a single factuality benchmark, a headline number from a model card or a vendor-run leaderboard, to decide whether a model is safe to deploy. That number was almost certainly not generated against the questions that matter in your vertical. It is a vanity metric dressed up as due diligence.

The order-of-magnitude spread is not a bug. It is the default.

Stanford researchers published the first preregistered empirical evaluation of AI-driven legal research tools in 2024, testing Lexis+ AI, Westlaw AI-Assisted Research, and GPT-4 against verifiable legal queries. The results: AI research tools made by LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI) each hallucinate between 17% and 33% of the time. These are tools marketed with language like "hallucination-free" ,  and the spread between the best and worst performer was more than 2.5× within the same domain.

Now compare that to healthcare. A 2025 study in npj Digital Medicine evaluated LLMs on medical text summarization across 450 primary-care consultation transcripts: 49,590 transcript sentences manually labeled by clinicians. Hallucination rate: 1.47%. The omission rate: 3.45%. That is an order of magnitude lower than the legal domain, on a task that is arguably less complex than statutory reasoning but carries its own patient-safety weight.

Code generation shows the pattern from yet another angle. A USENIX Security 2025 study analyzed 576,000 code samples across 16 LLMs and found that 19.7% of recommended software packages were hallucinations, libraries that do not exist. Commercial models hallucinated at 5.2%; open source at 21.7%.

These are often the same underlying architectures producing wildly different error rates depending on what you ask them about. A single factuality benchmark cannot capture this.

Why one eval suite cannot defend a frontier model

The reason is structural. General-purpose factuality benchmarks test a model's ability to be consistent with a source document or to recall widely known facts. They do not test whether it can correctly distinguish between two nearly identical ICD-10 codes in a clinical note, or recognize that a regulatory citation sounds plausible but does not exist, a failure mode the Stanford legal study found accounted for a significant share of hallucinations.

The MedHallu benchmark, published in 2025, illustrates this precisely. Researchers built 10,000 medical question-answer pairs to test hallucination detection in clinical contexts. The best model achieved an F1 score of just 0.625 on the hardest hallucinations, the ones semantically close to the correct answer. These matter most in practice, because they are the ones clinicians are least likely to catch.

In banking, the problem takes a different shape. A 2025 MIT thesis published found that even when AI models are trained for banking tasks, they can still generate inaccurate or unsupported outputs. In a customer churn prediction test, the study showed that adding banking-specific reasoning and hallucination checks reduced hallucinations and improved accuracy, highlighting the need for domain-specific controls rather than relying on generic AI safeguards alone

This is the mechanism that procurement teams miss. A model that scores 97% on a general factuality leaderboard can still hallucinate at rates that would be disqualifying in your specific use case. The leaderboard did not test your use case. It tested something adjacent, and adjacency is not safety.

What a domain coverage matrix actually requires

The gap between a single factuality score and deployment readiness is a domain coverage matrix: expert-graded test suites per vertical, each built around the failure modes that matter in that vertical. For healthcare, that means clinician-validated test cases around diagnostic codes and medication interactions. For banking, compliance queries reviewed by regulatory specialists.

None of this is exotic. But it requires something most organizations buying AI models do not have internally: domain experts whose job is to evaluate model outputs against specific failure modes. The clinician who can spot a hallucinated ICD-10 code. The compliance analyst who recognizes a regulatory citation that does not exist.

That expertise is rarely inside the organization buying the model. It is certainly not inside the organization selling it. The model vendor optimizes for the benchmark. The enterprise buyer trusts the benchmark. And the domain-specific failure modes live in the space between the two, untested and unpriced until something goes wrong in production.

The Epic Sepsis Model is a useful reference point, not because it involved hallucination, but because it demonstrated the same structural gap. An external validation study tested the model across 38,455 hospitalizations and found a sensitivity of 33%. It had been deployed across hundreds of hospitals based on internal metrics that did not generalize. The distance between the vendor's evaluation and real-world performance is the same gap that exists today between general factuality benchmarks and domain-specific hallucination rates.

The uncomfortable arithmetic

A model that hallucinated at a flat rate across all domains would be a simpler problem. You could set a threshold and enforce it. But a model that hallucinated at 1.5% in one context, 19.7% in another, and 33% in a third demands something different: per-domain evaluation infrastructure built and maintained by people who understand what a wrong answer actually costs in each vertical.

That infrastructure does not come from the model provider. It does not come from a general-purpose eval framework. It comes from partners who can build expert-graded test suites calibrated to the failure modes of healthcare and banking, and who can update those suites as models, regulations, and clinical guidelines change.

A single factuality score gave the procurement team a number to put in the vendor assessment. It did not tell them whether the model would fabricate a regulatory citation, hallucinate a drug interaction, or invent a software library that becomes a supply-chain vulnerability. The score was never wrong. It was just never asked the right question.