Diagnose a RAG Retrieval Failure

Back to prompts
Diagnose a RAG Retrieval Failure
agentsadvancedClaudeGPT
rag/debugging/retrieval

Systematically figure out why a retrieval-augmented generation pipeline returned a wrong or irrelevant answer.

Prompt
A RAG pipeline returned a wrong or irrelevant answer to a user query. Help me diagnose why.

Query: {{query}}
Answer returned: {{answer}}
Chunks retrieved (in order): {{retrieved_chunks}}
Expected source of the correct answer (if known): {{expected_source}}

Walk through these failure modes in order and tell me which one(s) most likely occurred,
citing specific evidence from the inputs above:
1. Retrieval failure — the right chunk was never retrieved
2. Ranking failure — the right chunk was retrieved but ranked too low / crowded out
3. Chunking failure — the right information was split across chunks awkwardly
4. Generation failure — the right chunk was retrieved but the model ignored or misread it
5. Query mismatch — the query's phrasing doesn't semantically match how the source is written

For the most likely failure mode, give one concrete, testable fix (not a generic
"improve your embeddings" suggestion).

When to use this#

Use this when a RAG system gives a bad answer and you’re not sure if the bug is in retrieval, chunking, or generation. Feeding it the actual retrieved chunks is what makes the diagnosis useful instead of generic.

Tips#

  • If you don’t have {{expected_source}} yet, leave it blank — the model can still narrow down the failure mode from the retrieved chunks alone.
  • Once you get a diagnosis, ask “write the eval test case that would catch this regression.”
Profile Image of the Author
LockDraft
AI news, agents, automation & prompts.
Announcement
Welcome to LockDraft — practical AI news, agent patterns, automation workflows, and ready-to-use prompts.
Categories
Tags
Latest Moments
Site Statistics
Posts
6
Categories
4
Tags
20
Total Words
18,238
Running Days
0 days
Last Activity
0 days ago
Site Info
Build Platform
Cloudflare Pages
Blog Version
Firefly v6.16.8
License
CC BY-NC-SA 4.0
Table of Contents