feat: add reset-repo ops scripts with dry-run and storage permission fixes

- Add ensure-storage.sh and reset-repo.sh for new/existing repos (prepare, inspect, dry-run)
- prepare updates existing indexes incrementally; soft/full-reset force rebuild
- Run ensure-storage on deploy and hetzner setup to keep /var/codewalk writable (uid 999)
- mkdir cloud index artifact dirs before Chroma/DuckDB open
- Document coverage matrix and workflows in SERVER_OPS.md

Author: aakash-anko

FULL_SETUP_GUIDE.md

@@ -574,11 +574,25 @@ curl -s -X POST "$API/admin/register" \
 
 ## 13. Delete / re-index
 
-### Index files only
+> **Preferred (server):** [deploy/reset-repo.sh](deploy/reset-repo.sh) — new + existing repos, `prepare`/`soft-reset`/`full-reset`, `--dry-run`, auto `chown 999:999`. See [SERVER_OPS.md §11](deploy/SERVER_OPS.md#11-re-index-reset--fix-stuck-indexing).
+
+```bash
+cd /opt/codewalk-src
+chmod +x deploy/reset-repo.sh deploy/ensure-storage.sh
+
+./deploy/reset-repo.sh inspect gupta29470/codewalk
+./deploy/reset-repo.sh prepare gupta29470/codewalk --index --dry-run   # preview
+./deploy/reset-repo.sh prepare gupta29470/codewalk --index             # update or create
+./deploy/reset-repo.sh soft-reset gupta29470/codewalk --index          # wipe index, rebuild
+./deploy/reset-repo.sh delete-repo gupta29470/codewalk
+```
+
+### Index files only (manual)
 
 ```bash
 REPO="gupta29470/codewalk"
 rm -rf /var/codewalk/indexes/${REPO}
+/opt/codewalk-src/deploy/ensure-storage.sh "${REPO}"   # required after rm as root
 
 docker compose exec postgres psql -U codewalk -d codewalk -c \
   "UPDATE repos SET last_indexed_sha=NULL, index_status='pending', index_version=0 WHERE full_name='${REPO}';"

README.md

@@ -1438,7 +1438,7 @@ Local MCP → GET /indexes/{owner}/{repo} → query locally
 6. Verify: `POST /admin/repos` with `X-Admin-Key` → `index_status: ready`
 7. Get `repo_token` from DB → set `CODEWALK_REPO_TOKEN` in MCP config
 
-**Day-to-day server ops:** [deploy/SERVER_OPS.md](deploy/SERVER_OPS.md) — health checks, indexing status, jobs/webhooks SQL, logs, reset/re-index.
+**Day-to-day server ops:** [deploy/SERVER_OPS.md](deploy/SERVER_OPS.md) — health, SQL, logs, [reset-repo.sh](deploy/reset-repo.sh) (prepare/reset/delete, `--dry-run`).
 
 ### Indexing a repo (checklist)
 
@@ -1475,7 +1475,7 @@ Cloud reads this on every index. Pushes to other branches are ignored. See [FULL
 |------|-----|
 | [FULL_SETUP_GUIDE.md](FULL_SETUP_GUIDE.md) | Complete A→Z setup |
 | [deploy/DEPLOY.md](deploy/DEPLOY.md) | Deployment guide |
-| [deploy/SERVER_OPS.md](deploy/SERVER_OPS.md) | Server ops cheat sheet — health, indexing, SQL, logs |
+| [deploy/SERVER_OPS.md](deploy/SERVER_OPS.md) | Server ops — health, indexing, SQL, [reset-repo.sh](deploy/reset-repo.sh) |
 | [env.server.example.txt](env.server.example.txt) | Hetzner `/opt/codewalk/.env` |
 | [env.local.example.txt](env.local.example.txt) | Local dev `.env` |
 | [mcp.json.example](mcp.json.example) | MCP config → `.vscode/mcp.json` |

deploy/DEPLOY.md

@@ -16,7 +16,7 @@
 6. [Future Deploys (CI/CD)](#future-deploys-cicd)
 7. [Rollback Procedure](#rollback-procedure)
 8. [Troubleshooting](#troubleshooting)
-9. [Server Ops Command Reference](./SERVER_OPS.md) — health, indexing, SQL, logs (cheat sheet)
+9. [Server Ops Command Reference](./SERVER_OPS.md) — health, indexing, SQL, `reset-repo.sh` / `ensure-storage.sh`
 
 ---
 

deploy/SERVER_OPS.md

@@ -396,12 +396,63 @@ cd /opt/codewalk && docker compose up -d --force-recreate codewalk-api
 
 ## 11. Re-index, reset & fix stuck indexing
 
+> **Preferred:** use `deploy/reset-repo.sh` — works for **new** and **existing** repos. Detects on-disk index: **update** (incremental) vs **create** (full embed). Every mode supports **`--dry-run`**.
+
+```bash
+cd /opt/codewalk-src
+chmod +x deploy/reset-repo.sh deploy/ensure-storage.sh
+
+# Inspect state + plans (no changes)
+./deploy/reset-repo.sh inspect gupta29470/codewalk
+
+# Dry-run any action first
+./deploy/reset-repo.sh prepare gupta29470/codewalk --index --dry-run
+./deploy/reset-repo.sh soft-reset gupta29470/codewalk --index --dry-run
+
+# Smart prepare: existing index → UPDATE (incremental); new → CREATE + register
+./deploy/reset-repo.sh prepare gupta29470/codewalk --index
+
+# Fix permissions — one repo or all
+./deploy/reset-repo.sh fix-perms
+./deploy/reset-repo.sh fix-perms gupta29470/codewalk --dry-run
+
+# Force wipe index (keep clone) + full re-create
+./deploy/reset-repo.sh soft-reset gupta29470/codewalk --index
+
+# Force wipe index + clone
+./deploy/reset-repo.sh full-reset gupta29470/codewalk --index
+
+# Delete from DB + disk
+./deploy/reset-repo.sh delete-repo gupta29470/codewalk --dry-run
+```
+
+**Coverage matrix** (see `inspect` for your repo):
+
+| Situation | Command |
+|-----------|---------|
+| New repo (no DB, no index) | `prepare owner/repo --index` → register + CREATE |
+| Existing index OK | `prepare owner/repo --index` → UPDATE (incremental) |
+| Permissions broken (code 14) | `fix-perms owner/repo` then `prepare --index` |
+| `index_status=failed` / stuck job | `prepare owner/repo --index` (unsticks + retries) |
+| Corrupt index (chroma, no manifest) | `soft-reset owner/repo --index` |
+| Force full rebuild | `soft-reset` (keep clone) or `full-reset` (wipe clone) |
+| Remove repo entirely | `delete-repo` → then `prepare --index` or git push |
+| All repos perms | `fix-perms` (no slug) |
+| Preview any of the above | add `--dry-run` or `inspect owner/repo` |
+
+**Not in script** (manual): private-repo clone auth (GitHub App PEM), DuckDB `.wal` lock delete, webhook branch allowlist (`codewalk.yaml`), laptop `rm -rf .codewalk` + MCP pull.
+
+`deploy-server.sh` and `hetzner-setup.sh` call `ensure-storage.sh` so deploys keep `/var/codewalk` writable.
+
+**Manual equivalents** (if script unavailable):
+
 ```bash
 cd /opt/codewalk
 export REPO="gupta29470/codewalk"
 
 # ── Soft re-index (keep clone, wipe index artifacts) ─────────────────
 rm -rf /var/codewalk/indexes/${REPO}
+/opt/codewalk-src/deploy/ensure-storage.sh "${REPO}"
 
 docker compose exec postgres psql -U codewalk -d codewalk -c \
   "UPDATE repos SET last_indexed_sha=NULL, index_status='pending', index_version=0
@@ -414,6 +465,7 @@ curl -s -X POST https://api.codewalk.xyz/admin/index \
 
 # ── Full reset (index + clone) ─────────────────────────────────────
 rm -rf /var/codewalk/indexes/${REPO} /var/codewalk/repos/${REPO}
+/opt/codewalk-src/deploy/ensure-storage.sh "${REPO}"
 # then UPDATE repos + admin/index as above
 
 # ── Unstick zombie row (indexing for hours, job still queued) ───────
@@ -440,7 +492,12 @@ docker compose exec postgres psql -U codewalk -d codewalk -c \
 ## 12. Permissions & common fixes
 
 ```bash
-# Permission denied on /var/codewalk/repos
+# Permission denied on /var/codewalk/repos (Chroma code 14: unable to open database file)
+/opt/codewalk-src/deploy/ensure-storage.sh
+# or per-repo:
+/opt/codewalk-src/deploy/ensure-storage.sh gupta29470/codewalk
+
+# Manual equivalent:
 chown -R 999:999 /var/codewalk
 chmod 600 /var/codewalk/secrets/*.pem
 

deploy/deploy-server.sh

@@ -114,6 +114,11 @@ echo "Restarting containers..."
 cd "$COMPOSE_DIR"
 docker compose up -d --force-recreate --remove-orphans
 
+# ── Index storage permissions (root rm/chown breaks Chroma writes) ─
+if [ -f "$SRC_DIR/deploy/ensure-storage.sh" ]; then
+    bash "$SRC_DIR/deploy/ensure-storage.sh" || echo "⚠️  ensure-storage.sh failed (non-fatal)"
+fi
+
 # ── Health check ────────────────────────────────────────────────────
 echo ""
 echo "Running health check..."

deploy/ensure-storage.sh

@@ -0,0 +1,115 @@
+#!/bin/bash
+# Ensure /var/codewalk layout exists and is writable by the API container (uid 999).
+# Safe to run anytime — idempotent. Works for one repo, all known repos, or base only.
+#
+# Usage:
+#   ./ensure-storage.sh [--dry-run]                  # base + all repos
+#   ./ensure-storage.sh [--dry-run] --all
+#   ./ensure-storage.sh [--dry-run] owner/repo
+set -euo pipefail
+
+STORAGE_ROOT="${INDEX_STORAGE_PATH:-/var/codewalk}"
+API_UID="${CODEWALK_API_UID:-999}"
+API_GID="${CODEWALK_API_GID:-999}"
+COMPOSE_DIR="${COMPOSE_DIR:-/opt/codewalk}"
+
+REPO_SLUG=""
+FIX_ALL=false
+DRY_RUN=false
+
+for arg in "$@"; do
+    case "${arg}" in
+        --all) FIX_ALL=true ;;
+        --dry-run) DRY_RUN=true ;;
+        */*) REPO_SLUG="${arg}" ;;
+    esac
+done
+
+if [ $# -eq 0 ] || { [ $# -eq 1 ] && [ "${1:-}" = "--dry-run" ]; }; then
+    FIX_ALL=true
+fi
+
+run_or_dry() {
+    local desc="$1"
+    shift
+    if ${DRY_RUN}; then
+        echo "  [dry-run] ${desc}"
+    else
+        "$@"
+    fi
+}
+
+ensure_repo_paths() {
+    local slug="$1"
+    [ -n "${slug}" ] || return 0
+    run_or_dry "mkdir -p ${STORAGE_ROOT}/indexes/${slug}/chroma" \
+        mkdir -p "${STORAGE_ROOT}/indexes/${slug}/chroma"
+    run_or_dry "mkdir -p ${STORAGE_ROOT}/repos/${slug}" \
+        mkdir -p "${STORAGE_ROOT}/repos/${slug}"
+    echo "    repo paths: indexes/${slug}, repos/${slug}"
+}
+
+discover_repos() {
+    local -a found=()
+    local slug=""
+
+    if [ -f "${COMPOSE_DIR}/docker-compose.yml" ]; then
+        while IFS= read -r slug; do
+            [ -n "${slug}" ] && found+=("${slug}")
+        done < <(docker compose -f "${COMPOSE_DIR}/docker-compose.yml" exec -T postgres \
+            psql -U codewalk -d codewalk -t -A -c "SELECT full_name FROM repos ORDER BY full_name;" \
+            2>/dev/null || true)
+    fi
+
+    if [ -d "${STORAGE_ROOT}/indexes" ]; then
+        while IFS= read -r slug; do
+            [ -n "${slug}" ] && found+=("${slug}")
+        done < <(find "${STORAGE_ROOT}/indexes" -mindepth 2 -maxdepth 2 -type d 2>/dev/null \
+            | sed "s|^${STORAGE_ROOT}/indexes/||" || true)
+    fi
+
+    if [ -d "${STORAGE_ROOT}/repos" ]; then
+        while IFS= read -r slug; do
+            [ -n "${slug}" ] && found+=("${slug}")
+        done < <(find "${STORAGE_ROOT}/repos" -mindepth 2 -maxdepth 2 -type d 2>/dev/null \
+            | sed "s|^${STORAGE_ROOT}/repos/||" || true)
+    fi
+
+    printf '%s\n' "${found[@]}" 2>/dev/null | sort -u
+}
+
+prefix="==>"
+${DRY_RUN} && prefix="[dry-run] ==>"
+
+echo "${prefix} Ensuring storage at ${STORAGE_ROOT} (owner ${API_UID}:${API_GID})"
+
+run_or_dry "mkdir base dirs under ${STORAGE_ROOT}" \
+    mkdir -p "${STORAGE_ROOT}/repos" "${STORAGE_ROOT}/indexes" "${STORAGE_ROOT}/secrets"
+run_or_dry "mkdir ${STORAGE_ROOT}/hf-cache" \
+    mkdir -p "${STORAGE_ROOT}/hf-cache" 2>/dev/null || true
+
+if [ -n "${REPO_SLUG}" ]; then
+    ensure_repo_paths "${REPO_SLUG}"
+elif ${FIX_ALL}; then
+    echo "${prefix} Ensuring paths for all registered / on-disk repos"
+    while IFS= read -r slug; do
+        [ -n "${slug}" ] && ensure_repo_paths "${slug}"
+    done < <(discover_repos)
+fi
+
+run_or_dry "chown -R ${API_UID}:${API_GID} ${STORAGE_ROOT}" \
+    chown -R "${API_UID}:${API_GID}" "${STORAGE_ROOT}"
+run_or_dry "chmod 755 ${STORAGE_ROOT}" chmod 755 "${STORAGE_ROOT}"
+run_or_dry "chmod 755 repos/ indexes/" \
+    chmod 755 "${STORAGE_ROOT}/repos" "${STORAGE_ROOT}/indexes" 2>/dev/null || true
+
+if [ -d "${STORAGE_ROOT}/secrets" ]; then
+    run_or_dry "chmod 700 secrets/" chmod 700 "${STORAGE_ROOT}/secrets"
+    if ! ${DRY_RUN}; then
+        find "${STORAGE_ROOT}/secrets" -type f -name '*.pem' -exec chmod 600 {} + 2>/dev/null || true
+    else
+        echo "  [dry-run] chmod 600 ${STORAGE_ROOT}/secrets/*.pem"
+    fi
+fi
+
+echo "${prefix} Storage OK"

deploy/hetzner-setup.sh

@@ -143,9 +143,13 @@ ${DOMAIN} {
 EOF
 
 # ── 7. Index storage (API container runs as uid 999) ─────────────────
-mkdir -p /var/codewalk/repos /var/codewalk/indexes /var/codewalk/secrets
-chown -R 999:999 /var/codewalk
-chmod 755 /var/codewalk
+if [ -f "$(dirname "$0")/ensure-storage.sh" ]; then
+    bash "$(dirname "$0")/ensure-storage.sh"
+else
+    mkdir -p /var/codewalk/repos /var/codewalk/indexes /var/codewalk/secrets
+    chown -R 999:999 /var/codewalk
+    chmod 755 /var/codewalk
+fi
 
 echo "=== Setup complete ==="
 echo ""