Standardize local-dev example and fix library bugs

.kiro/steering/test-local-dev-notebooks.md

@@ -0,0 +1,48 @@
+---
+inclusion: manual
+---
+# Test Local-Dev Notebooks
+
+## When to Use
+When asked to test, validate, or run the lexical-graph-local-dev notebooks.
+
+## How to Run
+```bash
+cd examples/lexical-graph-local-dev
+bash scripts/test-local-dev-notebooks.sh
+```
+
+## Configuration
+Environment variables to customize behavior:
+- `SKIP_GITHUB=true|false` (default: true) — skip GitHub reader cells (no token available)
+- `SKIP_PPTX=true|false` (default: true) — skip PPTX reader cells (600s timeout)
+- `SKIP_LONG_RUNNING=true|false` (default: true) — skip JSON/Wikipedia reader cells (extract_and_build timeout)
+- `CLEANUP=true|false` (default: true) — cleanup all resources after test
+- `DOCKER_MODE=standard|dev` (default: standard)
+- `REPORT_DIR=path` (default: examples/lexical-graph-local-dev/test-reports/)
+
+## Prerequisites
+- AWS CLI configured with valid credentials
+- Docker running
+- Bedrock model access enabled (Claude Sonnet, Cohere Embed English v3)
+
+## What It Does
+1. Detects platform (ARM/x86)
+2. Creates .env from template with auto-detected AWS account/region
+3. Creates AWS resources (S3 bucket, Bedrock managed prompts)
+4. Starts Docker containers (Neo4j, pgvector, Jupyter)
+5. Executes all notebook cells (skipping GitHub and PPTX as configured)
+6. Generates per-cell execution report (JSON + markdown)
+7. Cleans up all resources (Docker, S3, Bedrock prompts, local .env)
+
+## Notebooks Tested
+- 00-Setup.ipynb — Environment setup, package installation, reader dependencies
+- 01-Combined-Extract-and-Build.ipynb — Reader providers (web, PDF, YouTube, docx, markdown, JSON, Wikipedia, CSV, directory)
+- 02-Querying.ipynb — TraversalBasedRetriever queries
+- 03-Querying-with-Prompting.ipynb — Custom prompts (file, S3, Bedrock managed)
+- 04-Advanced-Configuration-Examples.ipynb — Batch processing, custom metadata functions
+- 05-S3-Directory-Reader-Provider.ipynb — S3 directory reader with prefix filtering and metadata
+
+## Expected Results
+- 90+ cells SUCCESS, 8 SKIPPED (GitHub + PPTX + JSON + Wikipedia), 0 FAILED
+- Reports in test-reports/ directory

examples/lexical-graph-local-dev/README.md

@@ -11,13 +11,30 @@ This example provides a complete local development environment for the GraphRAG
 - [**00-Setup**](./notebooks/00-Setup.ipynb) – Environment setup, package installation, and development mode configuration
 - [**01-Combined-Extract-and-Build**](./notebooks/01-Combined-Extract-and-Build.ipynb) – Complete extraction and building pipeline using `LexicalGraphIndex.extract_and_build()`
 - [**02-Querying**](./notebooks/02-Querying.ipynb) – Graph querying examples using `LexicalGraphQueryEngine` with various retrievers
-- [**03-Querying with prompting**](./notebooks/03-Querying%20with%20prompting.ipynb) – Advanced querying with custom prompts and prompt providers
+- [**03-Querying-with-Prompting**](./notebooks/03-Querying-with-Prompting.ipynb) – Advanced querying with custom prompts and prompt providers
 - [**04-Advanced-Configuration-Examples**](./notebooks/04-Advanced-Configuration-Examples.ipynb) – Advanced reader configurations and metadata handling
 - [**05-S3-Directory-Reader-Provider**](./notebooks/05-S3-Directory-Reader-Provider.ipynb) – S3-based document reading and processing
 
 ## Quick Start
 
-### 1. Start the Environment
+### 1. AWS Prerequisites
+
+Before starting, ensure you have:
+- [AWS CLI configured with credentials](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html) — verify with `aws sts get-caller-identity`
+- Access to Amazon Bedrock models:
+  - `us.anthropic.claude-sonnet-4-6` (extraction, response, evaluation)
+  - `cohere.embed-english-v3` (embeddings)
+
+### 2. Configure Environment
+
+```bash
+cd notebooks
+cp .env.template .env
+```
+
+Review `.env` — defaults work for local Docker services. Set `S3_BUCKET_NAME` if using S3 features (notebooks 03, 05).
+
+### 3. Start the Environment
 
 **Standard (x86/Intel):**
 ```bash
@@ -37,15 +54,15 @@ cd docker
 ./start-containers.sh --dev --mac   # Enable live code editing
 ```
 
-### 2. Access Jupyter Lab
+### 4. Access Jupyter Lab
 
 Open your browser to: **http://localhost:8889**
 
 - No password required
 - Navigate to the `work` folder to find notebooks
 - All dependencies are pre-installed
 
-### 3. Run the Setup Notebook
+### 5. Run the Setup Notebook
 
 Start with `00-Setup.ipynb` to configure your environment and verify all services are working.
 
@@ -57,9 +74,6 @@ Start with `00-Setup.ipynb` to configure your environment and verify all service
 |--------|----------|-------------|
 | `start-containers.sh` | Unix/Linux/Mac | Main startup script with all options |
 | `start-containers.ps1` | Windows PowerShell | PowerShell version with same functionality |
-| `start-containers.bat` | Windows CMD | Command prompt version |
-| `dev-start.sh` | Unix/Linux/Mac | Development mode startup |
-| `dev-reset.sh` | Unix/Linux/Mac | Reset development environment |
 
 ### Script Options
 
@@ -86,9 +100,6 @@ Start with `00-Setup.ipynb` to configure your environment and verify all service
 
 # Windows PowerShell
 .\start-containers.ps1 -Mac -Dev
-
-# Windows Command Prompt
-start-containers.bat --mac --dev
 ```
 
 ## Services
@@ -99,7 +110,7 @@ After startup, the following services are available:
 |---------|-----|-------------|---------|
 | **Jupyter Lab** | http://localhost:8889 | None required | Interactive development |
 | **Neo4j Browser** | http://localhost:7476 | neo4j/password | Graph database management |
-| **PostgreSQL** | localhost:5432 | graphrag/graphragpass | Vector storage |
+| **PostgreSQL** | localhost:5432 | postgres/password | Vector storage |
 
 ## Development Mode
 
@@ -195,16 +206,16 @@ Key environment variables (configured in `docker/.env`):
 
 ```bash
 # Database connections (Docker internal names)
-VECTOR_STORE="postgresql://graphrag:graphragpass@postgres:5432/graphrag_db"
-GRAPH_STORE="bolt://neo4j:password@neo4j:7687"
+VECTOR_STORE="postgresql://postgres:password@pgvector-local:5432/graphrag"
+GRAPH_STORE="bolt://neo4j:password@neo4j-local:7687"
 
 # AWS Configuration (optional)
 AWS_REGION="us-east-1"
 AWS_PROFILE="your-profile"
 
 # Model Configuration
 EMBEDDINGS_MODEL="cohere.embed-english-v3"
-EXTRACTION_MODEL="us.anthropic.claude-3-7-sonnet-20250219-v1:0"
+EXTRACTION_MODEL="us.anthropic.claude-sonnet-4-6"
 ```
 
 ## Troubleshooting
@@ -218,12 +229,12 @@ EXTRACTION_MODEL="us.anthropic.claude-3-7-sonnet-20250219-v1:0"
 - PostgreSQL: 5432
 
 **Container networking:**
-- Use container names in connection strings (e.g., `neo4j:7687`, not `localhost:7687`)
+- Use container names in connection strings (e.g., `neo4j-local:7687`, not `localhost:7687`)
 - The `.env` file uses Docker internal networking
 
 **Development mode:**
 - Restart Jupyter kernel after enabling hot-reload
-- Check that lexical-graph source is mounted at `/home/jovyan/lexical-graph-src`
+- Check that lexical-graph source is mounted at `/home/jovyan/lexical-graph`
 
 ### Reset Environment
 
@@ -240,7 +251,7 @@ docker-compose down -v
 ## AWS Foundation Model Access (Optional)
 
 For AWS Bedrock integration, ensure your AWS account has access to:
-- `anthropic.claude-3-7-sonnet-20250219-v1:0`
+- `us.anthropic.claude-sonnet-4-6`
 - `cohere.embed-english-v3`
 
 Enable model access via the [Bedrock model access console](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html).
@@ -255,7 +266,7 @@ If you have existing FalkorDB configurations:
    GRAPH_STORE="falkordb://localhost:6379"
 
    # New Neo4j
-   GRAPH_STORE="bolt://neo4j:password@neo4j:7687"
+   GRAPH_STORE="bolt://neo4j:password@neo4j-local:7687"
    ```
 
 2. **Update imports** in your code:

examples/lexical-graph-local-dev/aws/bedrock-prompt-policy.json

@@ -0,0 +1,14 @@
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": [
+        "bedrock:CreatePrompt",
+        "bedrock:GetPrompt",
+        "bedrock:ListPrompts"
+      ],
+      "Resource": "*"
+    }
+  ]
+}

examples/lexical-graph-local-dev/aws/create_custom_prompt.ps1

@@ -0,0 +1,33 @@
+# Usage:
+# .\create_custom_prompt.ps1 <prompt_json_file> <region> [aws_profile]
+
+param(
+    [Parameter(Mandatory = $true)]
+    [string]$PromptJson,
+
+    [Parameter(Mandatory = $true)]
+    [string]$Region,
+
+    [string]$AwsProfile
+)
+
+if (-not (Test-Path $PromptJson)) {
+    Write-Host "Error: JSON file '$PromptJson' not found."
+    exit 1
+}
+
+Write-Host "Creating prompt from JSON file: $PromptJson"
+
+$cmd = @(
+    "aws", "bedrock-agent", "create-prompt",
+    "--region", $Region,
+    "--cli-input-json", "file://$PromptJson"
+)
+
+if ($AwsProfile) {
+    $cmd += @("--profile", $AwsProfile)
+}
+
+& $cmd
+
+Write-Host "Prompt created successfully."

examples/lexical-graph-local-dev/aws/create_custom_prompt.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+
+# Usage:
+# ./create_custom_prompt.sh <prompt_json_file> <region> [aws_profile]
+
+set -e
+
+PROMPT_JSON="$1"
+REGION="$2"
+AWS_PROFILE="$3"
+
+if [[ -z "$PROMPT_JSON" || -z "$REGION" ]]; then
+  echo "Usage: $0 <prompt_json_file> <region> [aws_profile]"
+  exit 1
+fi
+
+if [[ ! -f "$PROMPT_JSON" ]]; then
+  echo "Error: JSON file '$PROMPT_JSON' not found."
+  exit 1
+fi
+
+# Build AWS CLI command
+CMD=(aws bedrock-agent create-prompt --region "$REGION" --cli-input-json file://"$PROMPT_JSON")
+if [[ -n "$AWS_PROFILE" ]]; then
+  CMD+=(--profile "$AWS_PROFILE")
+fi
+
+echo "Creating prompt from JSON file: $PROMPT_JSON"
+"${CMD[@]}"
+echo "Prompt created successfully."

examples/lexical-graph-local-dev/aws/create_prompt_role.ps1

@@ -0,0 +1,67 @@
+# Usage:
+# .\create_prompt_role.ps1 -RoleName "my-bedrock-prompt-role" -Profile "my-aws-profile"
+
+param (
+    [Parameter(Mandatory = $true)]
+    [string]$RoleName,
+
+    [string]$Profile
+)
+
+if (-not $RoleName) {
+    Write-Host "Error: --role-name is required"
+    exit 1
+}
+
+$profileArgs = @()
+if ($Profile) {
+    $profileArgs = @("--profile", $Profile)
+}
+
+# Define the trust policy
+$trustPolicy = @"
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Principal": {
+        "Service": "bedrock.amazonaws.com"
+      },
+      "Action": "sts:AssumeRole"
+    }
+  ]
+}
+"@
+
+# Write to temporary trust policy file
+$tempTrustPolicyFile = "trust-policy-temp.json"
+$trustPolicy | Set-Content -Encoding UTF8 $tempTrustPolicyFile
+
+# Create the IAM role
+Write-Host "Creating IAM role '$RoleName' for Bedrock..."
+aws iam create-role `
+    --role-name $RoleName `
+    --assume-role-policy-document file://$tempTrustPolicyFile `
+    @profileArgs
+
+# Attach inline policy (assumes bedrock-prompt-policy.json is in same directory)
+Write-Host "Attaching inline policy (BedrockPromptMinimalPolicy)..."
+aws iam put-role-policy `
+    --role-name $RoleName `
+    --policy-name "BedrockPromptMinimalPolicy" `
+    --policy-document file://bedrock-prompt-policy.json `
+    @profileArgs
+
+# Get the role ARN
+$roleArn = aws iam get-role `
+    --role-name $RoleName `
+    --query "Role.Arn" `
+    --output text `
+    @profileArgs
+
+Write-Host "`nDone. Role ARN:"
+Write-Host $roleArn
+
+# Cleanup
+Remove-Item $tem

examples/lexical-graph-local-dev/aws/create_prompt_role.sh

@@ -0,0 +1,68 @@
+#!/bin/bash
+
+# Usage:
+# ./create_prompt_role.sh --role-name my-bedrock-prompt-role --profile my-aws-profile
+
+set -e
+
+# Default values
+ROLE_NAME=""
+PROFILE_OPTION=""
+
+# Parse arguments
+while [[ "$#" -gt 0 ]]; do
+    case $1 in
+        --role-name)
+            ROLE_NAME="$2"
+            shift
+            ;;
+        --profile)
+            PROFILE_OPTION="--profile $2"
+            shift
+            ;;
+        *)
+            echo "Unknown parameter passed: $1"
+            exit 1
+            ;;
+    esac
+    shift
+done
+
+if [[ -z "$ROLE_NAME" ]]; then
+    echo "Error: --role-name is required"
+    exit 1
+fi
+
+TRUST_POLICY=$(cat <<EOF
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Principal": {
+        "Service": "bedrock.amazonaws.com"
+      },
+      "Action": "sts:AssumeRole"
+    }
+  ]
+}
+EOF
+)
+
+# Create the role
+echo "Creating IAM role '$ROLE_NAME' for Bedrock..."
+aws iam create-role \
+    --role-name "$ROLE_NAME" \
+    --assume-role-policy-document "$TRUST_POLICY" \
+    "$PROFILE_OPTION"
+
+# Attach managed policy (adjust if using custom permissions)
+echo "Attaching managed policy (AmazonBedrockFullAccess)..."
+aws iam put-role-policy \
+  --role-name "$ROLE_NAME" \
+  --policy-name BedrockPromptMinimalPolicy \
+  --policy-document file://bedrock-prompt-policy.json \
+  "$PROFILE_OPTION"
+
+echo "Done. Role ARN:"
+aws iam get-role --role-name "$ROLE_NAME" --query "Role.Arn" --output text "$PROFILE_OPTION"