Your document search returns the right policy page and passes a retrieval test. Yet the chatbot says files are kept for 90 days when the policy says 30. The link is real, but the claim is not supported by that link. Before putting the assistant in front of customers or staff, test the answer after retrieval. This is the next, distinct step after measuring retrieval Hit@3.

Separate retrieval from answer grounding

Hit@3 asks whether the correct document appeared in the first three retrieved results. A citation audit asks whether each specific claim in the answer is actually supported by the cited passage. The first test can pass while the second fails. Google Cloud describes grounding checks against reference texts and notes that a partly correct sentence can still be ungrounded. A small product can start with a manual case set and a deliberately narrow automated check.

Collect five anonymized real questions. Include one answered directly by a document, one with conflicting sources, one involving a date, one that has no answer, and one with a similar but wrong phrase. Preserve the retrieved passages, generated answer, citations, and prompt version. Rerun the same cases whenever the index, prompt, or model changes.

Demand evidence per claim and allow abstention

Use this instruction after retrieval. Adapt the output format to your app, but keep its contract:

Answer using only the retrieved passages below.
For every checkable claim, cite a source ID and a short verbatim quote
that supports it. Never cite an ID absent from the context.
If sources conflict, describe the conflict and their dates.
If support is insufficient, say "I cannot find the answer in these
sources" and identify what is missing. Do not invent policies, prices,
or deadlines. Return: answer, claims (text, source, quote), open issues.

A human reviewer scores five things separately: source ID exists, quoted text appears, the quote actually supports the claim, the source is current, and the assistant abstains appropriately. Exact text alone is not enough. “Up to 10 projects” does not prove “10 free projects,” for example. The short script below catches a missing or fabricated quote; it cannot decide whether a paraphrase is logically supported.

A small runnable filter

This is a wholly fictional example. Save it as citation_check_en.py and run it with standard Python. It verifies that each source ID exists and the quoted phrase occurs in that source. I ran it on the sample data:

sources = {
    "S1": "The Pro plan allows up to 10 active projects.",
    "S2": "Files are retained for 30 days.",
}
claims = [
    {"text": "Pro allows 10 projects.", "source": "S1", "quote": "up to 10 active projects"},
    {"text": "Files stay for 90 days.", "source": "S2", "quote": "90 days"},
]
for index, claim in enumerate(claims, 1):
    source = sources.get(claim["source"])
    print(index, "QUOTE_FOUND" if source and claim["quote"] in source else "REVIEW")

The result is 1 QUOTE_FOUND and 2 REVIEW. The second claim invents 90 days and its quoted phrase is absent. But QUOTE_FOUND does not mean an answer is true; it only means the literal text occurs in the cited source. Keep a human review sample and review every high-impact answer about price, policy, or sensitive data.

To avoid false passes, score each claim separately. A sentence with two numbers needs evidence for both. If documents have versions, retain the date with the source ID: a genuine quotation from an expired policy can still drive a wrong decision. In conflict cases, expect the assistant to present both sources rather than silently picking one. And when no source answers a question, a clear abstention is a successful result; do not mark it wrong just because it did not produce a confident answer. These distinctions make the evaluation useful for the product, not merely for formatting.

Set a launch gate

For an initial release, require zero nonexistent citations in the critical set, correct abstention when sources are silent, and human review of high-impact claims. If the bot fails, locate the layer: retrieval, stale document, prompt, or generation. Adding more links without checking what each sentence says does not fix unsupported answers. Save real failures as new cases and compare versions on the same distribution of questions.

Sources: Google Cloud's RAG grounding documentation and OpenAI's evaluation best practices. The policy and outputs are synthetic; this script calls no API and does not test semantic entailment.