Knowledge Graph (KG) Creation and Applying HybridRAG on the KG for Simple Quantum Mechanics related FAQ
- The foundation of this RAG solution is a very small, curated Private Quantum Mechanics Document(s), only around the most fascinating topic of "Quantum Entanglement".
- This repository shows a semi-automatic way to create KG with open source technology, which is also usable to a Hybrid RAG that uses both strcutured and unstructured context.
- Keeping in mind the complexity of manual effort involved in semi-automatic Knowlegde Graph (KG) creation, the scope of the document is kept very small.
- To protect this source material, the corpus itself is not publicly shared in the repository.
- This highlights the importance of the method that can be applied to any domain or any data.
-
Notebook to run: "V3-Semiautomatic KG Ontology for QM FAQ.ipynb".
-
Main Goal
- This notebook semi-auttomatically creates the KG using a pre-built JSON file. Also the JSON can be iteratively improvised by visualizing the KG in Neo4j as explained below.
- The KG uses Neo4j Graph DB that contains both triplets (subject, relattion, object) and DocumentChunk nodes (raw chunks with chunk embeddings), with a MENTIONS relation connecting DocumentChunk nodes to the entity nodes mentioned in it.
- The KG also needs a vector index on the chunk embeddings for similarity search. For now, it is based on cosine similarity, but it can be improved by combining dense (e.g., cosine similarity) and sparse (e.g., BM25) similarity metrics by updating the CQL query.
- This HybridRAG pipeline formats the context of the prompt by finding the top-k chunks by Vector Search on Neo4j Chunk Embedding Index, then finds the triplets that has some node(s) mentioned in the Chunk and the 1-hop neighbours of these nodes.
-
Pre-processing steps to create JSON of triplets [Diagram Given below]:
- Summarize each section (1-2 paragraphs) using the Prompt-V3_1.
- Turned a 30-page PDF into 4-pages Summary PDF.
- Highlighted some of the selected sentences from the Summary PDF.
- Take each of the selected sentences from the Summary PDF and give the Prompt-V3_2.
- Analyse manually to be consistent with the 4-step verification.
-
JSON format:
- This is a sample of the JSON format used here for the triplets.
- {
"raw_sentence": "The time-dependent wave function,
$\Psi(x,t)$ , is determined by the Hamiltonian operator,$\hat{H}$ .", "suggested_triplets": [{ "subject_name": "time-dependent wave function", "subject_class": "Mathematical_Object", "relationship": "IS_DETERMINED_BY", "object_name": "Hamiltonian operator", "object_class": "Operator" }, }]
-
Important steps in this implementation for converting the JSON into Neo4j GraphDB:
- Neo4j Util Functions to get credentials and driver.
- Delete existing Graph and Indexes in Neo4j Graph DB.
- Add triplets from JSON into Neo4j Graph DB.
- Add DocumentChunk nodes with raw chunks in the "text" attribute and chunk embedding in the "embedding" attribute, and a MENTIONS relation connecting DocumentChunk node to all the nodes mentioned in it.
- Add a vector index on the "embedding" attribute of all the DocumentChunk nodes.
- After One round of this entire process some issues like disconnected islands were found and it was manually fixed in the JSON itself as mentioned in the later part of this notebook. After the JSON was updated, this notebook was run again. A few repetitions in this way gave the final KG.
-
KG Visualizations: 2 Important visualizations are given here:
- Final KG before adding the DocumentChunk nodes and MENTIONS relations. How the disconnected components relates to different sub-topics as shown here.
- Final KG after adding the DocumentChunk nodes and MENTIONS relations. How the entire KG became more connected in this way as shown here.
- Final KG before adding the DocumentChunk nodes and MENTIONS relations: How the disconnected components relates to different sub-topics as shown here.
- Final KG after adding the DocumentChunk nodes and MENTIONS relations: How the entire KG became more connected in this way as shown here.
- Notebook to run: "V4-HybridRAG for QM FAQ.ipynb".
- Prerequisite: Need to have the KG ready before running this notebook. The KG is created by running this notebook "V3-Semiautomatic KG Ontology for QM FAQ.ipynb" in this repository. This mentioned notebook semi-auttomatically creates the KG using a pre-built JSON file. Also the JSON can be iteratively improvised by visualizing the KG in Neo4j as explained in that notebook.
- The HybridRAG pipeline uses Neo4j Graph DB that contains both triplets (subject, relattion, object) and DocumentChunk nodes (raw chunks with chunk embeddings).
- This HybridRAG pipeline formats the context of the prompt by finding the top-k chunks by Vector Search on Neo4j Chunk Embedding Index, then finds the triplets that has some node(s) mentioned in the Chunk and the 1-hop neighbours of these nodes.
- So the context has 2 parts: Unstructured Raw chunks and Structured Triplets.
- The full prompt is formed by combining the User Query, Context, Instructions.
- The full prompt is fed into an LLM e.g., gpt-3.5-turbo to generate the RAG response.
- Concept:
- From Local to Global: A GraphRAG Approach to Query-Focused Summarization [https://arxiv.org/pdf/2404.16130]
- Implementation:
- Knowledge Graphs & LLMs: Real-Time Graph Analytics [https://neo4j.com/blog/developer/knowledge-graph-llm-analytics/]
- The GraphRAG Manifesto: Adding Knowledge to GenAI [https://neo4j.com/blog/genai/graphrag-manifesto/]
- Enhancing RAG-based application accuracy by constructing and leveraging knowledge graphs [https://blog.langchain.com/enhancing-rag-based-applications-accuracy-by-constructing-and-leveraging-knowledge-graphs/]
- Vectors and Graphs: Better Together [https://neo4j.com/blog/developer/vectors-graphs-better-together/]
- Tools:
- Python GraphRAG Package [https://neo4j.com/docs/neo4j-graphrag-python/current/]