Add Ruff linting for the Python scripts

The _scripts/ helpers had no linter, and the pre-commit hook's Ruff stage
called a bare `ruff` — which is not on PATH here, since nothing Python is
installed globally. That stage has been exiting 127 and blocking every
commit that touches a .py file. It now goes through uv.

Markdown and files/ are excluded: Ruff 0.16 formats Python code fences
inside .md, and post code blocks are published prose set deliberately at
two-space indent. The notebooks under files/ are frozen artifacts of old
posts and produce hundreds of findings nobody will act on.

The B023 fix in publish.py is a real bug: the listRecords retry lambda
captured the loop variable by reference, so a retry on any page but the
last would have re-requested with the final page's cursor.

Changes:
- Add Ruff to the _scripts dev group with the shared rule set.
- Add make lint-scripts and format-scripts; call the former from CI.
- Point the CI test step at make test-scripts rather than restating it.
- Fix the hook to run Ruff via uv run --project _scripts.
- Convert html_diff.py to pathlib throughout; verified end to end against
  synthetic trees.
- Bind the listRecords params per iteration.
- Drop ten unused pytest.raises bindings and assorted dead locals.
- Bump setup-uv to v8.

Author: agude

.github/workflows/jekyll.yml

@@ -83,11 +83,13 @@ jobs:
           python-version: '3.12'
 
       - name: Install uv
-        uses: astral-sh/setup-uv@v7
+        uses: astral-sh/setup-uv@v8
+
+      - name: Lint Python scripts
+        run: make lint-scripts
 
       - name: Run script tests
-        working-directory: _scripts
-        run: uv run pytest
+        run: make test-scripts
 
       # Fast, credential-free subset of the build job's --site-dir
       # cross-check (keep the two invocations' flags in sync).
@@ -123,7 +125,7 @@ jobs:
         uses: actions/configure-pages@v6
 
       - name: Install uv
-        uses: astral-sh/setup-uv@v7
+        uses: astral-sh/setup-uv@v8
 
       - name: Build with Jekyll
         run: bundle exec jekyll build --baseurl "${{ steps.pages.outputs.base_path }}"

AGENTS.md

@@ -15,6 +15,8 @@ Jekyll-based static site (alexgude.com) running in Docker.
 - **Build:** `make build` (Production build to `_site/`).
 - **Deps:** `make lock` (Update Gemfile.lock via Docker).
 - **Lint:** `make lint` / `make format-all`.
+- **Lint Python:** `make lint-scripts` (Ruff check + format check) /
+  `make format-scripts` (Ruff autofix + format).
 - **Format MD:** `make format-md` (Run Prettier on all Markdown files).
 - **Scripts:** `make scripts` (List available Python scripts with descriptions).
 - **Test Scripts:** `make test-scripts` (Run Python script tests via pytest).

Makefile

@@ -33,7 +33,7 @@ DOCKER_RUN := docker run --rm $(DOCKER_RUN_OPTS) -v $(PWD):$(MOUNT) -w $(MOUNT)
 TEST ?= $(shell find _tests -type f -name 'test_*.rb' -not -name 'test_helper.rb')
 
 # Tier 1: Daily drivers
-.PHONY: serve build test test-scripts lint clean debug scripts
+.PHONY: serve build test test-scripts lint lint-scripts format-scripts clean debug scripts
 
 # Tier 2: Command variants
 .PHONY: serve-drafts serve-profile test-cov test-summary lint-fix check-links check-liquid doc-index doc-show
@@ -209,6 +209,20 @@ lint: image-build
 	@echo "Running linter..."
 	@$(DOCKER_RUN) bundle exec rubocop
 
+# Run Ruff (lint + format check) on all Python scripts.
+# Runs from _scripts/ so its pyproject.toml supplies the config and the
+# relative excludes resolve against the repo root.
+lint-scripts:
+	@echo "Running Ruff on Python scripts..."
+	@cd _scripts && uv run ruff check ..
+	@cd _scripts && uv run ruff format --check ..
+
+# Apply Ruff formatting and safe fixes to all Python scripts.
+format-scripts:
+	@echo "Formatting Python scripts with Ruff..."
+	@cd _scripts && uv run ruff format ..
+	@cd _scripts && uv run ruff check --fix ..
+
 # Build the site and check for broken links/HTML issues.
 check-links: build
 	@echo "Checking generated site for broken links and HTML issues..."
@@ -244,7 +258,8 @@ hooks-install: image-build prettier-image-build _bin/pre-commit.sh
 	@cp _bin/pre-commit.sh .git/hooks/pre-commit
 	@chmod +x .git/hooks/pre-commit
 	@echo "Pre-commit hook installed at .git/hooks/pre-commit."
-	@echo "It will run RuboCop on staged .rb files and Prettier on staged .md files."
+	@echo "It will run RuboCop on staged .rb files, Prettier on staged .md files,"
+	@echo "and Ruff on staged .py files."
 
 # Run RuboCop --autocorrect on all Ruby files to establish a clean formatting baseline.
 # This target modifies files on the host via the volume mount.

_bin/pre-commit.sh

@@ -116,10 +116,12 @@ if [ ${#STAGED_PY_FILES[@]} -gt 0 ]; then
   printf '%s\n' "${STAGED_PY_FILES[@]}"
   echo "---"
 
-  ruff check --fix "${STAGED_PY_FILES[@]}"
+  # Ruff lives in the _scripts uv project; --project makes it resolvable from
+  # the repo root, and its pyproject.toml supplies the config for every path.
+  uv run --project _scripts ruff check --fix "${STAGED_PY_FILES[@]}"
   RUFF_CHECK_EXIT=$?
 
-  ruff format "${STAGED_PY_FILES[@]}"
+  uv run --project _scripts ruff format "${STAGED_PY_FILES[@]}"
   RUFF_FORMAT_EXIT=$?
 
   git add -- "${STAGED_PY_FILES[@]}"

_scripts/atproto/publish.py

@@ -29,10 +29,11 @@
 import re
 import sys
 import time
-from datetime import datetime, timezone
+from collections.abc import Callable
+from datetime import UTC, datetime
 from pathlib import Path
+from typing import Any
 from zoneinfo import ZoneInfo
-from typing import Any, Callable
 
 import requests
 import yaml
@@ -53,13 +54,13 @@
 
 class PublishError(Exception):
     """Fatal pipeline error; main() converts it to a message and exit 1."""
+
+
 _CONFIG_FILE = Path(__file__).parent.parent.parent / "_config.yml"
 
 POST_FILENAME_RE = re.compile(r"^(\d{4}-\d{2}-\d{2})-(.+)\.md$")
 
-MANAGED_FIELDS = frozenset(
-    {"$type", "site", "path", "title", "description", "tags", "publishedAt"}
-)
+MANAGED_FIELDS = frozenset({"$type", "site", "path", "title", "description", "tags", "publishedAt"})
 
 # Publication fields the pipeline manages; values come from _config.yml so
 # site metadata has a single home (title → name, description, url).
@@ -78,13 +79,14 @@ def desired_publication_record(config: dict) -> dict[str, Any]:
         "preferences": {"showInDiscover": True},
     }
 
+
 # ---------------------------------------------------------------------------
 # Config helpers
 # ---------------------------------------------------------------------------
 
 
 def load_config(config_path: Path = _CONFIG_FILE) -> dict:
-    with open(config_path, encoding="utf-8") as fh:
+    with config_path.open(encoding="utf-8") as fh:
         return yaml.safe_load(fh) or {}
 
 
@@ -199,9 +201,7 @@ def _extract_frontmatter_strict(text: str) -> dict:
         return {}
     fm = yaml.safe_load(m.group(1)) or {}
     if not isinstance(fm, dict):
-        raise ValueError(
-            f"front matter is not a YAML mapping (got {type(fm).__name__})"
-        )
+        raise ValueError(f"front matter is not a YAML mapping (got {type(fm).__name__})")
     return fm
 
 
@@ -336,7 +336,8 @@ def list_records(self, collection: str) -> list[dict]:
             if cursor:
                 params["cursor"] = cursor
             resp = self._retry_request(
-                lambda: self._http.get(
+                # Bind params per iteration; the loop rebinds it each page.
+                lambda params=params: self._http.get(
                     f"{self._pds}/xrpc/com.atproto.repo.listRecords",
                     params=params,
                     headers=self._auth(),
@@ -438,14 +439,11 @@ def _verify_publication(client: AtprotoClient, publication_uri: str) -> dict:
     uri_did, rkey = m.group(1), m.group(2)
     if uri_did != client.did:
         raise PublishError(
-            f"publication_uri belongs to {uri_did} but the session is "
-            f"authenticated as {client.did}"
+            f"publication_uri belongs to {uri_did} but the session is authenticated as {client.did}"
         )
     remote = client.get_record(client.did, "site.standard.publication", rkey)
     if remote is None:
-        raise PublishError(
-            f"publication record does not exist on the PDS: {publication_uri}"
-        )
+        raise PublishError(f"publication record does not exist on the PDS: {publication_uri}")
     return remote
 
 
@@ -460,9 +458,7 @@ def _records_differ(local: dict, remote: dict) -> bool:
 def _document_sources(
     posts_dir: Path, books_dir: Path | None
 ) -> list[tuple[Path, Callable[[Path], dict | None]]]:
-    sources: list[tuple[Path, Callable[[Path], dict | None]]] = [
-        (posts_dir, parse_post)
-    ]
+    sources: list[tuple[Path, Callable[[Path], dict | None]]] = [(posts_dir, parse_post)]
     if books_dir is not None:
         bd = books_dir
         sources.append((bd, lambda f: parse_book(f, bd)))
@@ -474,6 +470,7 @@ def _source_files(src_dir: Path) -> list[Path]:
     All candidate markdown files under src_dir, recursively, mirroring
     Jekyll: directories starting with '_' (templates) are not read.
     """
+
     def included(f: Path) -> bool:
         rel = f.relative_to(src_dir)
         if any(part.startswith("_") for part in rel.parts[:-1]):
@@ -519,10 +516,7 @@ def _collect_documents(posts_dir: Path, books_dir: Path | None):
             if not rec["title"].strip():
                 errors.append("missing or empty 'title'")
             if not rec.get("publishedAt"):
-                errors.append(
-                    "cannot derive publishedAt from the 'date' front matter "
-                    "or filename"
-                )
+                errors.append("cannot derive publishedAt from the 'date' front matter or filename")
             if rec["path"] in seen_paths:
                 errors.append(
                     f"duplicate path {rec['path']!r} "
@@ -557,9 +551,7 @@ def sync_documents(
     # --- Keep the publication record itself in sync with _config.yml ---
     desired_pub = desired_publication_record(config)
     remote_pub_value = remote_publication["value"]
-    pub_subset = {
-        k: v for k, v in remote_pub_value.items() if k in PUBLICATION_MANAGED_FIELDS
-    }
+    pub_subset = {k: v for k, v in remote_pub_value.items() if k in PUBLICATION_MANAGED_FIELDS}
     if pub_subset != desired_pub:
         merged_pub = dict(remote_pub_value)
         merged_pub.update(desired_pub)
@@ -580,9 +572,7 @@ def sync_documents(
     local: dict[str, tuple[Path, dict]] = {}
     for doc_file, rec, errors in _collect_documents(posts_dir, books_dir):
         if errors:
-            raise PublishError(
-                "; ".join(f"{doc_file.name}: {msg}" for msg in errors)
-            )
+            raise PublishError("; ".join(f"{doc_file.name}: {msg}" for msg in errors))
         if rec is None:
             raise AssertionError("collector yielded no record and no errors")
         rec["site"] = publication_uri
@@ -610,9 +600,7 @@ def sync_documents(
     data_map: dict[str, str] = {}
     for path_key, (rkey, _, _) in remote.items():
         if path_key in local:
-            data_map[path_key] = (
-                f"at://{client.did}/site.standard.document/{rkey}"
-            )
+            data_map[path_key] = f"at://{client.did}/site.standard.document/{rkey}"
 
     # --- Diff and sync ---
     created = updated = unchanged = orphaned = 0
@@ -640,15 +628,11 @@ def sync_documents(
                         merged[key] = local_rec[key]
                     elif key in merged:
                         del merged[key]
-                merged["updatedAt"] = datetime.now(timezone.utc).strftime(
-                    "%Y-%m-%dT%H:%M:%SZ"
-                )
+                merged["updatedAt"] = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ")
                 if dry_run:
                     print(f"[dry-run] Would update: {path_key}")
                 else:
-                    client.put_record(
-                        "site.standard.document", rkey, merged, swap_cid=cid
-                    )
+                    client.put_record("site.standard.document", rkey, merged, swap_cid=cid)
                     print(f"Updated: {path_key}")
                 updated += 1
             else:
@@ -670,13 +654,13 @@ def sync_documents(
     if dry_run:
         print(f"[dry-run] Would write {len(data_map)} entries to {data_out}")
     else:
-        output = {path_key: data_map[path_key] for path_key in sorted(local) if path_key in data_map}
+        output = {
+            path_key: data_map[path_key] for path_key in sorted(local) if path_key in data_map
+        }
         data_out.parent.mkdir(parents=True, exist_ok=True)
         data_out.write_text(json.dumps(output, indent=2) + "\n", encoding="utf-8")
 
-    print(
-        f"created={created} updated={updated} unchanged={unchanged} orphaned={orphaned}"
-    )
+    print(f"created={created} updated={updated} unchanged={unchanged} orphaned={orphaned}")
 
 
 # ---------------------------------------------------------------------------
@@ -790,8 +774,7 @@ def validate_documents(
                 error_count += 1
             elif wk.read_text(encoding="utf-8") != expected_publication_uri:
                 print(
-                    f"ERROR: {wk} content does not match "
-                    "standard_site.publication_uri",
+                    f"ERROR: {wk} content does not match standard_site.publication_uri",
                     file=sys.stderr,
                 )
                 error_count += 1
@@ -821,9 +804,7 @@ def delete_orphans(
     local_paths: set[str] = set()
     for doc_file, rec, errors in _collect_documents(posts_dir, books_dir):
         if errors:
-            raise PublishError(
-                "; ".join(f"{doc_file.name}: {msg}" for msg in errors)
-            )
+            raise PublishError("; ".join(f"{doc_file.name}: {msg}" for msg in errors))
         if rec is None:
             raise AssertionError("collector yielded no record and no errors")
         local_paths.add(rec["path"])
@@ -868,9 +849,7 @@ def delete_orphans(
 def init_publication(client: AtprotoClient, config: dict) -> None:
     existing = get_publication_uri(config)
     if existing:
-        raise PublishError(
-            f"publication_uri is already set in _config.yml: {existing!r}"
-        )
+        raise PublishError(f"publication_uri is already set in _config.yml: {existing!r}")
     # The config guard is local-only; also check the PDS so running twice
     # before committing cannot create two publication records.
     remote_pubs = client.list_records("site.standard.publication")
@@ -880,9 +859,7 @@ def init_publication(client: AtprotoClient, config: dict) -> None:
             f"publication record(s) already exist on the PDS: {uris} — "
             "put the URI in _config.yml instead of creating another"
         )
-    at_uri = client.create_record(
-        "site.standard.publication", desired_publication_record(config)
-    )
+    at_uri = client.create_record("site.standard.publication", desired_publication_record(config))
     print(f"Publication record created: {at_uri}")
     print("Add this URI to _config.yml → standard_site.publication_uri and commit.")
 
@@ -911,9 +888,7 @@ def _dispatch(argv: list[str] | None = None) -> None:
     parser = argparse.ArgumentParser(description=__doc__)
     sub = parser.add_subparsers(dest="cmd", required=True)
 
-    pub_p = sub.add_parser(
-        "publish", help="Sync posts and books to PDS document records"
-    )
+    pub_p = sub.add_parser("publish", help="Sync posts and books to PDS document records")
     pub_p.add_argument("--posts-dir", required=True, type=Path)
     pub_p.add_argument("--books-dir", required=True, type=Path)
     pub_p.add_argument("--data-out", required=True, type=Path)
@@ -923,8 +898,7 @@ def _dispatch(argv: list[str] | None = None) -> None:
 
     val_p = sub.add_parser(
         "validate",
-        help="Validate posts and books for AT Protocol compatibility "
-        "(no network, no credentials)",
+        help="Validate posts and books for AT Protocol compatibility (no network, no credentials)",
     )
     val_p.add_argument("--posts-dir", required=True, type=Path)
     val_p.add_argument("--books-dir", required=True, type=Path)

_scripts/content/backdate_rating.py

@@ -24,7 +24,7 @@
 import subprocess
 import sys
 from datetime import datetime
-
+from pathlib import Path
 
 RATING_RE = re.compile(r"^rating:\s*(.+)$", re.MULTILINE)
 DATE_RE = re.compile(r"^date:\s*(.+)$", re.MULTILINE)
@@ -49,7 +49,7 @@ def get_repo_root() -> str:
 
 def repo_relative_path(filepath: str) -> str:
     """Convert any path to a repo-relative path for consistent git operations."""
-    abs_path = os.path.abspath(filepath)
+    abs_path = Path(filepath).resolve()
     return os.path.relpath(abs_path, REPO_ROOT)
 
 
@@ -196,7 +196,7 @@ def format_datetime(iso_str: str) -> str:
 
 def update_date(filepath: str, new_date: str, dry_run: bool) -> bool:
     """Update the date field in the file's front matter. Returns True if changed."""
-    with open(filepath, "r") as f:
+    with Path(filepath).open() as f:
         content = f.read()
 
     date_match = DATE_RE.search(content)
@@ -209,15 +209,13 @@ def update_date(filepath: str, new_date: str, dry_run: bool) -> bool:
         print(f"  SKIP (already correct): {filepath}")
         return False
 
-    new_content = (
-        content[: date_match.start(1)] + new_date + content[date_match.end(1) :]
-    )
+    new_content = content[: date_match.start(1)] + new_date + content[date_match.end(1) :]
 
     if dry_run:
         print(f"  DRY RUN: {filepath}")
         print(f"    {old_date} -> {new_date}")
     else:
-        with open(filepath, "w") as f:
+        with Path(filepath).open("w") as f:
             f.write(new_content)
         print(f"  UPDATED: {filepath}")
         print(f"    {old_date} -> {new_date}")
@@ -279,7 +277,7 @@ def main():
 
         # Skip files that already have a full timestamp (not just a bare date)
         if not args.force:
-            with open(filepath, "r") as f:
+            with Path(filepath).open() as f:
                 content = f.read()
             date_match = DATE_RE.search(content)
             if date_match and not BARE_DATE_RE.match(date_match.group(1).strip()):