Your Kedro pipelines are green, your RAG answers are wrong – here is a 16-problem map I use to debug them

[onestardao]

Hi everyone,

I ran into a pattern that I guess many Kedro users are seeing now:
the pipelines look perfect from Kedro’s point of view, but the RAG / LLM node at the end is still giving wrong or unstable answers.

To make this easier to debug, I wrote a long Medium article that treats this as a failure-diagnostics problem, not a “prompt tuning” problem:

👉 “Your Kedro pipelines are reproducible. Your RAG answers are wrong. Here is a 16-problem map to debug them.”
https://psbigbig.medium.com/your-kedro-pipelines-are-reproducible-ae42f775bfde

A quick summary of what is inside, from a Kedro user’s perspective:

1. The situation

• Kedro runs are green, Kedro-Viz looks clean, your Data Catalog is versioned and monitored.
• The only thing that is broken is the RAG / LLM behaviour: wrong time range, mixing customers, answering with the wrong data source, etc.
• It is hard to tell whether the root cause is retrieval, chunking, embeddings, prompt schema, or some infra / deployment issue around the LLM node.

2. A 16-problem failure map + global debug card

• The article introduces a 16-problem RAG failure map that I use when reviewing pipelines. Each problem has a number (No.1–No.16) and belongs to one of four “lanes”: input/retrieval, reasoning, state/memory, infra/deploy.
• There is a global debug card: a single image that encodes the objects, zones, and the full 16-problem table. You can upload this card + one failing run to any strong LLM and ask it to classify which problems are active and what structural fixes to try first.
• The same taxonomy has already been adapted (in different forms) into projects like RAGFlow, LlamaIndex, ToolUniverse (Harvard MIMS Lab) and a QCRI multimodal RAG survey, which gave me confidence that the map is general enough to be useful beyond one stack.

3. How it plugs into Kedro without changing your infra

The whole point is to keep Kedro as-is and add a semantic failure language on top. The article describes three levels:

1. Manual triage on a few pipelines

   • Pick a handful of recent runs where Kedro is happy but users are not.
   • For each run, collect: question, retrieval queries, retrieved chunks, prompt template, final answer, any evaluation signal.
   • Feed this bundle + the debug card to an LLM and ask it to tag problem numbers (No.1–No.16) and lanes (IN / RE / ST / OP).
   • Record those tags somewhere simple (issue tracker, CSV, metrics store) and look for clusters of failure types.

2. Structured diagnostics per node

   • Add a dataset like rag_failure_reports to your Data Catalog (JSON or Parquet).
   • For inspected nodes, save small documents that include pipeline name, node name, question, answer, wfgy_problem_no, wfgy_lane, and optionally a ΔS zone (semantic stress band).
   • Let the LLM “clinic” produce a short report per failing node and store it in that dataset so you can slice by pipeline, node, or failure type.

3. A Kedro hook that runs the clinic after LLM nodes

   • Once you trust the pattern, you can wire it into a after_node_run hook that only fires for nodes tagged llm_node.
   • The hook gathers question / retrieved chunks / answer, calls your internal “RAG failure clinic” client with the 16-problem map, and saves the diagnostic report into rag_failure_reports.
   • The rest of the Kedro project stays exactly the same. No new runner, no new orchestration layer.

The article includes a small sketch of such a hook and shows how to keep everything version-controlled inside your repo (for example in a docs/wfgy_rag_clinic/ folder with the debug card image + a system-prompt text file).

4. Instruments under the hood (optional, for people who like theory)

If you read further down, there is an explanation of how the map thinks about semantic stress ΔS, four zones of tension, and a few internal instruments (λ_observe, E_resonance and four repair operators) that give both humans and LLMs a consistent way to talk about “where tension accumulates” in the pipeline. You do not need to implement math to use them; the appendix system prompt lets an LLM approximate all of this from text.

5. Why I am sharing this here

I maintain an open-source project called WFGY that focuses on failure-first debugging for RAG / LLM systems. The 16-problem map started there, then got adapted into several other tools. This article is my attempt to write a Kedro-specific walkthrough, instead of a generic RAG rant.

I would really appreciate feedback from Kedro users:

• Does this match the kinds of failures you are seeing at the end of your pipelines?
• Would a small example repo with a Kedro project + this clinic wired in be useful, or is the article + debug card enough for now?
• If you have existing Kedro RAG projects and are willing to try the map on a few failing runs, I would love to hear which problem numbers show up most often.

Again, the full article with the image and the copy-pasteable system prompt is here:
https://psbigbig.medium.com/your-kedro-pipelines-are-reproducible-ae42f775bfde

Thanks for reading, and happy to iterate on this if the Kedro community finds it useful.

WFGY 3.0 · RAG 16 Problem Map — Global Debug Card