GridWorks JournalKeeper

---

GridWorks JournalKeeper is responsible for long-term storage and management of GridWorks time-series and message data, beyond the initial persistence layer.

This repository is a work in progress. Current focus is on importing and managing the 2023–2024 Millinocket S3 data in a PostgreSQL database, with Alembic-managed schema migrations.

---

Development Quick Start

Prerequisites

• Python 3.12
• uv
• make
• Docker / Docker Compose
• PostgreSQL client (psql) recommended
• pre-commit (see below)

One-time tooling setup

This project uses pre-commit for local quality checks. The pre-commit runner must be installed once on your system (outside the project environment).

We recommend installing it with pipx:

# macOS / Linux
brew install pipx        # or see https://pipx.pypa.io/installation/
pipx ensurepath
pipx install pre-commit

Verify installation:

pre-commit --version

Note: pre-commit manages its own hook environments and does not rely on the project virtual environment. Once hooks are installed and cached, commits work offline.

---

Python environment (uv + Make)

This project uses uv for dependency management and a Makefile for workflow orchestration.

From the repository root:

make venv
make dev
pre-commit install
source .venv/bin/activate

After changing dependencies:

make lock
make dev

Common commands:

make lint
make test
make pre-commit

To run the test suite:

make test

Runtime Dependencies (Docker)

JournalKeeper expects:

• a running RabbitMQ broker
• a PostgreSQL database

We run both locally using Docker.

RabbitMQ

Follow the RabbitMQ setup instructions in gridworks-base

PostgreSQL (development database)

From the repository root:

docker compose up

You should see logs ending with:

database system is ready to accept connections

[Note: docker compose up -d will run in the background if you prefer that]

The dev database is exposed on:

• Host localhost
• Port 5433
• Database journaldb_dev
• User: journaldb
• Password: journaldb

Port 5433 is used to avoid conflicts with a local PostgreSQL instance on 5432.

Test connectivity:

psql -h localhost -p 5433 -U journaldb journaldb_dev

Database SCcema (Alembic)

Alembic is used to manage database schema migrations.

Verify Alembic connectivity

With your dev environment active:

alembic current

Expected output:

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
<revision_id>(head)

Apply migrations

alembic upgrade head

This will:

• 1. Connect to journaldb_dev
• 2. Create the alembic_version table if needed
• 3. Apply all migrations in alembic/versions
• 4. Mark the database as being at head

Publishing a Release

Publishing a Release

JournalKeeper uses tag-driven releases. The version in pyproject.toml is the single source of truth and must match the Git tag exactly.

Release checklist

1. Update the version

   Edit pyproject.toml:

   [project]
   version = "0.1.0"

2. Commit and merge to main

do this through standard branch -> dev -> main PRs

3. Add matching tag on main

git checkout main
git pull origin main

This ensures:

• tag points to the correct commit
• the release guard tag is on main will pass

4. ** Create and push the tag**

Now tag locally on main:

git tag v0.1.0
git push origin v0.1.0

This triggers the Release workflow.

Production Notes

JournalKeeper currently runs on an EC2-hosted PostgreSQL instance.

ssh ubuntu@journaldb.electricity.works
psql -U journaldb

Credentials are stored in 1Password

Remote access:

psql -h journaldb.electricity.works -U journaldb -d journaldb

Contributing

This repository uses:

• ruff for linting and formatting
• mypy for type checking
• pytest for tests
• pre-commit for enforcement

Before committing:

make pre-commit

License

MIT -- see LICENSE