GAIA Literature Knowledge Base

Graph RAG knowledge base for multi-disciplinary geoscience research spanning seismology, hydrology, geomorphology, atmospheric sciences, soil sciences, tectonics, geophysical sensing, and NLP for science.

Pipeline

DOI list ─┬─ Ingest (fetch HTML) ──► Parse (HTML/XML → JSON) ──► Chunk (section CSV)
          └─ Scrape (JS pages)  ─┘           │
                                             ▼
                            Extract (entities/relations) ──► Graph ──► RAG Query
                                             │
                                             ▼
                                    Eval (QA generation + prompt testing)

Quick start

pip install -e ".[dev,llm]"

# Seed the registry with DOIs
gaia-kb ingest seed -i dois.txt

# Fetch papers (HTTP) or scrape JS-rendered pages
gaia-kb ingest fetch
gaia-kb ingest scrape "https://seismica.library.mcgill.ca/article/view/195/287" --doi "10.26443/seismica.v1i1.195"

# Parse HTML or JATS XML to structured JSON
gaia-kb parse run
gaia-kb parse run --format xml

# Generate section-level chunks CSV
gaia-kb parse chunk

# Extract entities and relations
gaia-kb extract run

# Build knowledge graph
gaia-kb graph build

# Query
gaia-kb query "How does soil moisture affect seismic velocity?"

# Evaluate: generate QA pairs from chunks
gaia-kb eval generate-qa data/chunks.csv
gaia-kb eval test-prompts data/chunks.csv --chunks "0,1,2"

Project layout

configs/            Settings, domain taxonomy, logging
data/               (git-ignored) Raw HTML/XML, parsed JSON, entities, graph
  raw/              Papers organized by DOI (paper.html or paper.xml)
  parsed/           Structured JSON per paper
  entities/         Extracted entities and relations
  registry/         papers.jsonl — the paper manifest
  graph/            Graph snapshots and embeddings
  chunks.csv        Section-level chunks for embedding & QA generation
src/gaia_kb/        Python package
  ingest/           Paper acquisition, registry, Selenium scraper
  parse/            HTML parser, JATS XML parser, section chunker
  extract/          NLP entity and relation extraction
  graph/            Knowledge graph construction & storage
  rag/              Retrieval-augmented generation
  eval/             QA dataset generation & prompt testing
notebooks/          Jupyter notebooks for exploration
tests/              Test suite
scripts/            Operational scripts

Paper storage

What	Where	Format
Paper manifest	data/registry/papers.jsonl	JSONL, one record per paper (DOI + metadata + status)
Raw content	data/raw/{safe_doi}/paper.html	DOI with / → __
Parsed structure	data/parsed/{safe_doi}.json	Title, abstract, sections, references
Section chunks	data/chunks.csv	CSV: identifier, title, section_name, content
Entities	data/entities/{safe_doi}.json	Entities, relations, domain tags
Knowledge graph	data/graph/graph.graphml	NetworkX GraphML

Configuration

Edit configs/settings.yaml to set data paths, graph DB backend, and LLM provider. Edit configs/domains.yaml to extend the geoscience taxonomy.

Optional extras

pip install -e ".[llm]"        # Anthropic/OpenAI for LLM extraction & RAG
pip install -e ".[eval]"       # QA dataset generation (pandas + LLM)
pip install -e ".[scraper]"    # Selenium for JS-rendered pages
pip install -e ".[neo4j]"      # Neo4j graph backend
pip install -e ".[embeddings]" # sentence-transformers for vector search
pip install -e ".[notebooks]"  # Jupyter + matplotlib