Spark History MCP Server & LangGraph AI Agent Integration

Repository Overview

This repository provides a complete infrastructure and an AI Agent for analyzing Apache Spark application logs using natural language.

It integrates:

1. Apache Spark History Server: For hosting and serving event logs.
2. Spark History MCP Server: A bridge exposing Spark metrics to AI models via the Model Context Protocol (MCP).
3. LangGraph Agent: A local AI assistant (powered by Ollama) that can query the MCP server to diagnose performance issues, identify bottlenecks, and summarize job statuses.
4. Spark Job Runner: A utility container to generate sample event logs from real graph algorithms (PageRank, Connected Components).

The entire stack runs in Docker, keeping your local environment clean, while the AI Agent runs in a local Python virtual environment for speed and flexibility.

---

Project Structure

Repo
├── docker-compose.yaml              # Infrastructure definition (Spark HS, MCP Server, Job Runner)
├── main.py                          # Main entry point for the AI Agent
├── requirements_agent.txt           # Python dependencies for the local AI Agent
├── requirements_spark.txt           # Python dependencies for the Dockerized Spark Job Runner
├── config/
│   ├── agent_config.yaml            # Agent settings (model selection, parameters, URLs)
│   ├── paths_config.yaml            # Data and log paths
│   └── prompts.yaml                 # System prompts and tool definitions for the LLM
├── data/
│   └── facebook_large/              # Input dataset (Download required)
│       ├── musae_facebook_edges.csv
│       └── musae_facebook_target.csv
├── scripts/
│   ├── close_infrastructure.sh      # Stops all Docker containers
│   ├── generate_data.sh             # Helper script to trigger data generation manually
│   ├── run_agent.sh                 # Sets up venv and runs the AI Agent
│   └── start_infrastructure.sh      # Starts Docker containers and data generation
├── spark-events/                    # Output folder for Spark Event Logs (mounted to Docker)
└── src/
    ├── agent/                       # Agent logic (LangGraph, Tools)
    │   ├── __init__.py
    │   ├── graph.py
    │   └── tools.py
    ├── spark_job/                   # PySpark script for generating sample logs
    │   ├── __init__.py
    │   └── generate_logs.py
    └── utils/                       # Helper functions (logging, config loading)
        ├── __init__.py
        ├── config_loader.py
        └── loggers.py

---

Installation & Setup

1. Prerequisites

• Docker & Docker Compose: Ensure Docker Desktop is running.
• Python 3.10+: Installed on your local machine.
• Ollama: Installed and running.

2. Prepare Data

Download the Facebook Large Page-Page Network dataset and place the CSV files in the data/facebook_large/ folder.

Download here: snap.stanford.edu/data/facebook-large-page-page-network.html

• Input: data/facebook_large/musae_facebook_edges.csv
• Output (Automatic): spark-events/ (created after running the infrastructure).

3. Prepare Local AI Model

Open a separate terminal and ensure Ollama is serving the model specified in config/agent_config.yaml (default is llama3.1:8b).

# 1. Start Ollama (if not running in background app)
ollama serve

# 2. Pull the model (in a new terminal tab)
ollama pull llama3.1:8b

4. Make Scripts Executable

Give execution permissions to the helper scripts:

chmod +x scripts/*.sh

---

Execution

Step 1: Start Infrastructure

This script initializes the Docker containers.

• If spark-events is empty, it runs the spark-job-runner container to generate data (this takes up to a couple of minutes).
• Then, it starts the spark-history-server and mcp-server.

./scripts/start_infrastructure.sh

*Wait until you see "Infrastructure ready!" and the URLs. *

Step 2: Run the AI Agent

This script handles the local Python environment.

• It checks for a .venv folder.
• If missing, it creates it and installs dependencies from requirements.txt.
• Then it launches the interactive Agent.

./scripts/run_agent.sh

Example Interaction:

User: "List all applications available in the history server"

Agent: (Returns list of apps with IDs)

User: "Analyze application local-17... Find the top 3 slowest stages."

Step 3: Stop Infrastructure

When finished, use this script to stop and remove the Docker containers.

./scripts/close_infrastructure.sh

Script Reference

• scripts/start_infrastructure.sh: Orchestrates the startup. Checks for existing data to avoid re-running heavy Spark jobs unnecessarily.
• scripts/run_agent.sh: Wrapper for main.py. Manages the virtual environment (.venv) automatically so you don't pollute your global Python.
• scripts/close_infrastructure.sh: A shortcut for docker-compose down to ensure a clean shutdown.

---

Configuration & Customization

Agent Configuration (config/agent_config.yaml)

This file controls the local AI model settings. To switch the LLM served by Ollama (e.g., to use qwen2.5:14b instead of llama3.1:8b), simply update the model_name parameter.

model_name: "llama3.1:8b"  # Change this to your pulled Ollama model
temperature: 0.0           # Keep low for deterministic tool usage
base_url: "http://localhost:11434"

System Prompts (config/prompts.yaml)

This file defines the System Prompt used by the Agent. It acts as a cheat sheet for the LLM, listing the available tools and their required JSON argument schemas.

If you need to verify the available tools or add new ones to the prompt instructions, you can reference the official source code definition here: github.com/kubeflow/mcp-apache-spark-history-server/blob/main/src/spark_history_mcp/tools/tools.py