Models 6 min readEvaluate an answer
Your AI found the right words. Why is the answer wrong?
An AI can quote a manual accurately and still apply the wrong rule. The missing detail may be a heading, a version number or an exception.
The short answer
A correct quote can still produce a wrong answer. The AI needs to know who or what the rule applies to—and whether an exception changes it. Check the passage it receives, not just the source link it displays.
On this page
The sentence is correct. The answer is not.
Imagine an AI assistant helping a developer use a software service. The developer asks, “How long can a request keep a user waiting?” The search step finds “The limit is 30 seconds.” The assistant repeats that number and links to the manual.
In our made-up Lyra service, the 30-second rule applies to exporting data in the background. Requests that a user waits for have a separate five-second limit. The assistant found a real sentence in our fictional manual, but used the rule for the wrong task.
This search-then-answer approach is called retrieval-augmented generation, or RAG. The system finds material in documents and passes it to the AI before it answers. But a link to the right manual is not enough: the passage must explain which task its rule applies to.
Conceptual example / A fictional manual
30 seconds is correct—for the wrong task.
How long may a request keep the user waiting?
- 01 · Missing heading
A rule without its task
The limit is 30 seconds.
We cannot tell which task this covers.
- 02 · Restore the heading
Background exports
30 seconds
Wrong task for this question.
- 03 · Retrieve the correct section
Requests a user waits for
5 seconds
This passage supports the answer.
The correct answer still needs a different section. If that section is missing, do not guess.
Inspect the evidence
- Document
- Lyra manual, version 2 (fictional)
- First section
- Background exports: “The limit is 30 seconds.”
- Applicable section
- Requests a user waits for: “The limit is 5 seconds.”
The words and limits are made up for this example; no retrieval system was tested.
About this visual
An original, fictional example. The 30-second limit applies to background exports; interactive requests have a separate five-second limit. Preserving scope helps distinguish a relevant passage from the wrong rule. No retrieval results are measured here.
The decision this changes
Before replacing the model behind a document assistant, check whether its search step strips away the headings and exceptions that make an answer valid. This is a proposed diagnostic, not a claim that repairing context always beats changing models.
Our interpretation
Connect the signals
We would test two things separately. Did search find the right passage? Did the AI receive enough of the surrounding text to use that passage correctly?
A search system may use surrounding text to choose a passage, yet send only the passage itself to the AI. The AI then lacks the details that made the passage understandable. The reverse can also happen: a complete, well-labeled passage is useless if search never finds it.
In the Lyra example, restoring the heading “Background exports” reveals that the 30-second rule answers a different question. The assistant still needs the section on requests that users wait for. If that section is unavailable, it should say that it cannot confirm the limit.
For builders Methods, implementation details and source evidence
How search can preserve context
Anthropic describes one way to preserve context: add a short note explaining where each passage belongs in its document. Include that note when preparing the passage for both meaning-based search and keyword search. The company found better retrieval in its tests. That does not tell us how well the method will work on your documents. [1]
The Late Chunking paper takes another route. An embedding model—a model that represents text as numbers for search—processes the longer text first. It then combines the numbers for each passage into a searchable representation. That representation can retain information from the surrounding text. The paper tests retrieval; it does not prove that an assistant will always answer correctly. [2]
What to send with a source passage
When passing a passage to the AI, we would include the document name and version, its section heading, the exact quoted text, and any nearby condition that changes the rule. Label AI-written explanatory notes separately from the original text.
Each detail answers a question. The heading tells us which task the rule covers. The version tells us whether the rule may be out of date. The exact quotation lets a reviewer check the wording. If a detail is missing, leave it unknown rather than ask the AI to guess.
This is a starting point, not a fixed checklist for every document. First inspect answers your system gets wrong. A missing heading may explain one error. Another may require keeping a table’s column headings or an exception printed farther away from the rule.
When the source changes, update its context
When a document changes, our proposed system would mark its saved passages and notes for review or rebuilding. Otherwise, new source text could be paired with an old explanation. Record which document version each passage and note came from.
An AI-written context note can also be wrong, so check it and label it clearly. Do not present it as a quotation. Sending the surrounding section may supply the missing rule, but it also gives the AI more text to sort through.
We would choose a fix after seeing which step failed. Longer passages, a different search model, or a second step that reorders search results are options to test. None is a guaranteed repair.
What we don’t know
We have not tested a working search system for this article. Lyra and its time limits are made up. The diagram explains one possible error; it does not show how often it happens or which search method works best.
Even a complete passage can come from a wrong or outdated document. Here we focus on whether the AI uses a passage correctly. We have not examined who may access the documents or how to verify that they are genuine.
Try this Open the full exercise and its limits
Try this
Write two short sections in a test manual: “Background exports: the limit is 30 seconds” and “User-waiting requests: the limit is 5 seconds.” Ask one question about each task. Save the expected answer and the supporting section beside each question.
Run the same questions twice. First give the AI passages without headings. Then include the verified headings and document name. Keep the documents and AI model unchanged. For each answer, record three checks separately: did search find the right section, did the AI receive its heading, and did it choose the right limit?
Next, remove the section containing the correct answer and run the question again. Does the assistant say the evidence is missing, or take a limit from the wrong task? Record the result. This small test can reveal a problem; it cannot show that the system is reliable in everyday use.
What to watch next
When checking an AI answer, inspect the exact passage the AI received. Did it include the heading or exception needed to apply the rule correctly?
Which missing heading or exception could change an answer your AI gives with confidence?
When comparing document assistants, look for tests that include missing headings, outdated versions and unanswered questions. Check whether the evaluation records both the passages retrieved and the text passed to the answering model.
One AI decision, made clearer.
Get a short email with one visual explanation and one practical check. Follow the full article when you want the evidence.
Send me the briefingOpens our email signup page. Confirm by email to receive the briefing. Free; unsubscribe anytime.
Preview a short email →Follow the evidence
Sources & further reading
- Anthropic: Introducing Contextual Retrieval ↗
Daniel Ford, September 19, 2024. Primary engineering report; checked September 25, 2026. Historical method, not a new release. Vendor evaluations were not reproduced by Curious Compute.
Back to cited passage ↑ - Günther et al.: Late Chunking ↗
arXiv:2409.04701, version 3, July 7, 2025. Primary research paper; checked September 25, 2026. Contextual chunk embeddings and their evaluation, not an end-to-end answer guarantee.
Back to cited passage ↑
A correction or a different perspective? Write to us.