Docling integration

[gabe-l-hart]

Description

This PR extends the local document parsing capabilities to allow the user to choose docling and optionally also use a VLM like smoldocling or granite3.2-vision for visual parsing. In addition to allowing users to choose docling for its output quality, this also allows users to input png and jpeg images as context.

Testing

I've tested this in conjunction with my #1278 branch for a fully local LLM / VLM researcher. My two methods of testing are via a custom script and via the nextjs CLI.

Custom Script

import asyncio
import json
import os
import sys
from gpt_researcher import GPTResearcher

os.environ["FAST_LLM"] = "ollama:granite3.2:2b"
os.environ["SMART_LLM"] = "ollama:granite3.2:8b"
os.environ["STRATEGIC_LLM"] = "ollama:granite3.2:8b"
os.environ["EMBEDDING"] = "ollama:granite-embedding:278m"
os.environ["OPENAI_API_KEY"] = "ollama"
os.environ["OLLAMA_BASE_URL"] = "http://localhost:11434"
os.environ["RETRIEVER"] = "duckduckgo"
os.environ["PROMPT_FAMILY"] = "granite"
os.environ["LLM_KWARGS"] = json.dumps({"num_ctx": 1024 * 128})

os.environ["REPORT_SOURCE"] = "hybrid"
os.environ["DOC_PATH"] = "./report_docs"
os.environ["CONVERT_WITH_DOCLING"]  = "True"
os.environ["DOCLING_VLM"] = "granite_vision_ollama"

if not (query := " ".join(sys.argv[1:])):
    query = input("?> ")

researcher = GPTResearcher(
    query=query,
    report_type="research_report",
    report_source="hybrid",
)

loop = asyncio.get_event_loop()
loop.run_until_complete(researcher.conduct_research())
loop.run_until_complete(researcher.write_report())

nextjs Frontend

I use the following env vars before running python main.py in the root of the repo

export FAST_LLM="ollama:granite3.2:2b"
export SMART_LLM="ollama:granite3.2:8b"
export STRATEGIC_LLM="ollama:granite3.2:8b"
export EMBEDDING="ollama:granite-embedding:278m"
export OPENAI_API_KEY="ollama"
export OLLAMA_BASE_URL="http://localhost:11434"
export RETRIEVER="duckduckgo"
export PROMPT_FAMILY="granite"
export LLM_KWARGS='{"num_ctx": 131072}'
export REPORT_SOURCE="hybrid"
export DOC_PATH="./report_docs"
export CONVERT_WITH_DOCLING="true"
export DOCLING_VLM="granite_vision_ollama"

From there, I've tested swapping to hybrid via the Preferences panel and then dropping in a .png screenshot to ensure it gets converted correctly.

Dependencies

This PR depends on getting the next release of docling out with ollama support for granite3.2-vision. I'm coordinating with @PeterStaar-IBM on that release, so it should go out soon!

Ollama support for Granite Vision 3.2 in Docling was added in 2.30.0

[assafelovic]

Hey @gabe-l-hart love your contributions! Same as before, can you please update related docs? Happy to merge afterward!

[gabe-l-hart]

@assafelovic Docs are updated. Thanks!