Separate the RAG failure layers

When a user says a RAG answer is wrong, the visible response does not reveal the faulty layer. The product may not contain the needed knowledge. The retrieval query may be weak. Filters may exclude the correct source. Ranking may prefer an irrelevant chunk. The source may be stale or contradictory. The model may ignore good evidence or make an unsupported claim.

Knowledge coverageThe required information is missing, incomplete, inaccessible, or owned by another system.
Freshness and authorityThe index contains stale versions, duplicate policies, or sources with unclear precedence.
Query and retrievalThe search representation, filters, ranking, chunking, or top-k selection misses useful evidence.
GenerationThe model misreads retrieved context, ignores it, combines sources incorrectly, or invents an unsupported claim.
ExpectationThe user expects a capability, source, or permission scope the product does not provide.

Classifying the failure at the correct layer avoids two expensive responses: tuning prompts to compensate for missing content, or rewriting knowledge that retrieval never selects.

Ask users for the evidence they uniquely know

Users should not diagnose embeddings or chunking. Ask for the expected outcome and any source they trust:

  • What part of the answer is wrong or missing?
  • What should the answer say instead?
  • Is there a document, policy, page, or system that contains the correct information?
  • Did the answer use an outdated rule or ignore a supplied file?
  • What decision or workflow depends on getting this right?

If the user cannot name a source, the report is still useful. Their expected answer and business consequence help Product determine whether the missing knowledge is within scope and how urgently to investigate.

Feedback tells you which knowledge failure mattered to a real workflow. Retrieval telemetry tells you how the system reached it.

Attach retrieval context to the report

At report time, attach a trace ID and stable references to the retrieval run. Useful fields include query or query hash, collection and index version, filters, retrieved source IDs, chunk IDs, ranks, relevance scores, source timestamps, prompt version, model, answer citations, and latency.

Also capture product context: customer, plan, role, locale, permissions, selected workspace, uploaded files, and release. A report may appear to be a retrieval miss when the correct document was excluded by access rules or account configuration.

Minimize sensitive data. Store source identifiers and controlled links instead of copying private documents into feedback records. Permanent credentials should remain on the server, with short-lived browser tokens and origin validation.

Use a RAG feedback triage matrix

  1. Was the correct knowledge available? If not, assign a content coverage or ingestion owner.
  2. Was the source current and authoritative? If not, fix freshness, deduplication, or precedence.
  3. Was the correct source retrieved? If not, inspect query generation, filters, chunking, ranking, and index configuration.
  4. Did the answer use the retrieved evidence correctly? If not, inspect prompt, model behavior, context limits, and citation logic.
  5. Was the user expectation within scope? If not, clarify product boundaries or improve the experience around unavailable knowledge.

Every accepted case should have an owner and a verification method. Knowledge fixes need ingestion confirmation; retrieval fixes need recall or ranking tests; generation fixes need answer-level evaluations; expectation fixes need product communication or design.

Find knowledge gaps across feedback

Single reports identify examples. Clusters reveal operating priorities. Group RAG feedback by customer workflow, topic, source, source owner, collection, index version, model, prompt, locale, and time. Look for:

  • Repeated missing topics across high-value accounts.
  • Negative reports concentrated on one stale document or source owner.
  • Retrieval misses that begin after an index, embedding, or filter change.
  • Good retrieval but unsupported generation tied to a model or prompt version.
  • Permission-related gaps affecting a specific role or workspace.
  • Questions users repeatedly ask even though the product does not own the answer.

Prioritize with customer impact and evidence confidence, not raw count alone. A low-frequency policy error affecting a renewal-critical workflow may deserve faster action than a common low-stakes formatting complaint.

Measure the path from report to knowledge improvement

  • Percentage of RAG reports linked to retrieval and answer traces.
  • Accepted cases classified by knowledge, freshness, retrieval, generation, or expectation.
  • Time to assign the correct content or engineering owner.
  • Repeated feedback rate for the same source or topic after a fix.
  • Customer-reported usefulness for the affected workflow.
  • New evaluation cases created from real customer failures.

A strong RAG feedback process turns customer corrections into a living quality system. It improves not only the answer that failed, but also source ownership, retrieval evaluation, and the product's understanding of what customers expect it to know.

Connect RAG complaints to the evidence behind them

Use RedFeed to collect the expected outcome, attach retrieval and customer context, and route high-impact knowledge gaps during a free pilot.

Start free 14-day pilot →