To evaluate an LLM application for hallucinations, run it against real questions whose correct, supported answers reviewers have agreed in advance, score retrieval and answers separately, and break the results down by language, question type and user role. One average score can hide failure patterns that matter to particular users.
This article covers the method: what to measure, how to build and label a test set, which metrics to use and how to read them. Turning those results into release decisions, monitoring and rollback is the subject of our companion guide to release gates and incident response for AI systems.
"Hallucination" covers several different failures, and each needs its own label. NIST's Generative AI Profile discusses the broad risk under the name confabulation: false or erroneous content stated with confidence. For an application that answers from your documents, split it further:
- Unsupported claim. The answer states something its cited sources do not say, even if it happens to be true.
- Wrong answer. The answer contradicts the reference answer or the source.
- Misattributed citation. The claim may be right, but the citation points to a passage that does not support it.
- Answering instead of declining. No source the user may access covers the question, and the system answers anyway.
The difference between true and supported matters. A model can produce a correct fact from its training data that your approved documents do not contain. For a policy assistant that is still a failure, because nobody can trace the statement or update it when the policy changes. Write these definitions down before anyone labels a single answer.
The test set should resemble the questions people will actually ask, not the prompts the development team finds interesting. Good sources are search logs, support tickets, messages to the team that owns the documents and short interviews with future users. Remove personal data before any case enters the set.
Then add the cases that ordinary traffic underrepresents:
- questions no available document answers, where the right behavior is to decline;
- questions whose answer sits in a table, a scanned PDF or across two documents;
- the same question asked by users in different roles, so restricted content is tested;
- spelling variants, abbreviations and questions that mix Arabic and English.
Write questions natively in each language your users use instead of translating one set, because translated questions miss how people really phrase things. Our enterprise RAG guide adds a related case worth keeping: an Arabic question whose answer lives in an English document.
There is no universal right size. You need enough cases in each slice you care about to notice a difference that would change a decision. That argues for starting with a modest, carefully labelled set and growing it from every failure you find. Version the set, record which document versions each reference answer came from, and keep a held-out portion that nobody tunes prompts against.
Figure 1 — Use real questions. Include unanswerable cases.
Two careful reviewers can disagree about whether an answer is acceptable. A labeling guide turns those judgments into something repeatable. For each case, record the reference answer and acceptable variations, the passages a correct answer must rely on, whether the system should decline, and any wording that would be harmful even if accurate.
Define each label with an example. When does a partial answer count as correct? Does a missing citation fail an otherwise correct answer? What about an answer that is right but too long to use? Have two reviewers label the same sample independently and compare. Investigate disagreements: the guide, source material or required domain knowledge may be ambiguous. Resolve the case with a domain expert, clarify the rule and relabel where needed. Domain experts should own the labels for their domain: a finance reviewer for expense policy, a lawyer for contract clauses.
When a system retrieves documents, inspect both retrieval and answer generation; failures can arise at either stage or from the source material itself. Score each stage on its own:
- Retrieval. Did the passages the labels require appear in the results the model actually received, not merely somewhere in the index?
- Answer support. Split each answer into claims and check every claim against the passages it cites.
- Citation accuracy. Does each citation point to a passage that supports the sentence it is attached to?
- Correctness. Does the answer agree with the reference answer?
- Declining. Did the system decline when it should, and answer when it could?
If retrieval missed the passage, the generation prompt alone cannot supply the missing evidence. Review query construction, chunking, metadata, access filters and search settings. If retrieval found it and the answer still drifted, look at the instructions, the order of the context or the model. Keep simple automatic checks wherever they apply: cited passage IDs exist, output matches the expected format and quoted numbers and their units match the source.
For each metric, define its denominator. Retrieval coverage can be the required passages returned divided by all required passages for answerable cases. Supported-claim rate is the number of supported factual claims divided by all factual claims checked. Also report the share of whole answers with every factual claim supported: a strong claim-level rate can coexist with many partly unsupported answers. Handle unanswerable cases separately and document how you average across questions.
Figure 2 — Score each stage separately. Identify what needs fixing.
Grading thousands of answers by hand is slow, so many teams use a second model with a rubric as a grader. That can work, but the grader is another model whose errors you must measure. Run it first on cases humans have already labelled, compare its verdicts label by label and language by language, and read the cases where it disagrees. A grader that matches reviewers on English answers may not match them on Arabic ones.
Version the grader's prompt and model like any other component. When either changes, recheck it against the human-labelled sample before comparing new scores with old ones. Otherwise you cannot tell whether the system changed or the ruler did.
Report every metric per slice, with the number of cases beside each rate. Useful slices include language (Arabic, English and mixed), question type (lookup, comparison, multi-step and numeric), document format, user role and how recently the source changed. A system that looks acceptable overall can still fail on numeric questions in Arabic, and that may be the slice your finance team depends on.
Small slices are noisy, so treat a difference across a handful of cases as a reason to add cases, not as a conclusion. When comparing models, keep the test set and grader fixed and change one thing at a time. Our comparison of Arabic-capable model families applies the same discipline to model selection.
NIST describes its AI Risk Management Framework as voluntary guidance; it is not a certification or a law. It organizes risk work into four functions: Govern, Map, Measure and Manage. The method above is mostly Measure work, but it depends on Map, because you can only choose failure labels and slices once you understand the system's context, users and possible harms. NIST published the Generative AI Profile (NIST AI 600-1) on 26 July 2024 to extend the framework to generative AI. Both documents help structure the work. Neither guarantees that a system is safe or free of false content.
The recommendations in this article are TecnoStack's practical guidance, not NIST requirements. If you are planning an assistant, search tool or agent, our AI systems service can help you define the labels, metrics and slices before launch.
- National Institute of Standards and Technology. AI Risk Management Framework: voluntary guidance organized around the Govern, Map, Measure and Manage functions.
- National Institute of Standards and Technology. Generative Artificial Intelligence Profile (NIST AI 600-1), published 26 July 2024 as a companion to the AI RMF.
