Skip to content

feat(ledger): add v3 reconciler with Raft StatefulSet support #434

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
gfyrag wants to merge 9 commits into
base: main
Choose a base branch
from
Open

feat(ledger): add v3 reconciler with Raft StatefulSet support #434

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
9ca91be
feat(ledger): add v3 reconciler with Raft StatefulSet support
gfyrag Mar 25, 2026
dd00278
docs: add ledger v3 settings documentation
gfyrag Mar 25, 2026
dd2cca4
feat(ledger): add v3 integration tests and fix settings prefix
gfyrag Mar 25, 2026
24a53f1
feat(ledger): add preStop hook for proper Raft scale-down
gfyrag Mar 25, 2026
a9652ec
chore: regenerate helm RBAC template with StatefulSet permissions
gfyrag Mar 25, 2026
ab59f2b
feat(kubectl-stacks): add create command and install recipe
gfyrag Mar 31, 2026
8a63607
chore: Add setting
gfyrag Mar 31, 2026
ce73ffe
fix(ledger): align v3 tests with actual v3-mirror reconciliation path
gfyrag Apr 1, 2026
99ad9c2
chore: add settings doc
gfyrag Apr 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,3 +28,6 @@ Dockerfile.cross
charts

*.tgz
*.tar.gz

dist
3 changes: 3 additions & 0 deletions Justfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,5 +99,8 @@ generate-docs:
--templates-dir=./crd-doc-templates \
--config=./docs.config.yaml

install-kubectl-stacks:
cd tools/kubectl-stacks && go build -o {{env('GOPATH', `go env GOPATH`)}}/bin/kubectl-stacks .
Comment on lines +102 to +103
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Ensure ${GOPATH}/bin exists before writing the binary.

Line 103 can fail on clean environments where ${GOPATH}/bin is missing.

Proposed fix 
 install-kubectl-stacks:
-  cd tools/kubectl-stacks && go build -o {{env('GOPATH', `go env GOPATH`)}}/bin/kubectl-stacks .
+  mkdir -p {{env('GOPATH', `go env GOPATH`)}}/bin
+  cd tools/kubectl-stacks && go build -o {{env('GOPATH', `go env GOPATH`)}}/bin/kubectl-stacks .
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
install-kubectl-stacks:
cd tools/kubectl-stacks && go build -o {{env('GOPATH', `go env GOPATH`)}}/bin/kubectl-stacks .
install-kubectl-stacks:
mkdir -p {{env('GOPATH', `go env GOPATH`)}}/bin
cd tools/kubectl-stacks && go build -o {{env('GOPATH', `go env GOPATH`)}}/bin/kubectl-stacks .
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@Justfile` around lines 102 - 103, The install-kubectl-stacks target may fail
if ${GOPATH}/bin doesn't exist; update the install-kubectl-stacks recipe (the
block that does `cd tools/kubectl-stacks && go build -o {{env('GOPATH', `go env
GOPATH`)}}/bin/kubectl-stacks .`) to first create the bin directory (e.g., run
mkdir -p on the resolved {{env('GOPATH', `go env GOPATH`)}}/bin) before invoking
go build so the output path exists.


deploy: helm-update
earthly +deploy
12 changes: 12 additions & 0 deletions config/rbac/role.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,18 @@ rules:
- patch
- update
- watch
- apiGroups:
- apps
resources:
- statefulsets
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- batch
resources:
Expand Down
145 changes: 145 additions & 0 deletions docs/04-Modules/03-Ledger.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,3 +100,148 @@ Available fields:
- `push-retry-period`: Retry period for failed pushes
- `sync-period`: Synchronization period
- `logs-page-size`: Number of logs per page

## Ledger v3 Mirror

Ledger v3 can run alongside an existing v2 deployment as a **mirror**: it continuously replicates v2 ledger data into its own Raft-based storage. This allows gradual migration or read-offloading without disrupting v2.

When the `modules.ledger.v3-mirror` setting is present, the operator:
1. Deploys v2 normally (database, migrations, Deployment)
2. Deploys a v3 Raft StatefulSet in parallel
3. Runs a provisioning Job that creates mirror ledgers in v3, each sourcing data from the v2 PostgreSQL database

### Enabling v3 Mirror

Create a Settings resource with the key `modules.ledger.v3-mirror`. The value format is:

```
<v3-image-tag>:<ledger1>,<ledger2>,...
```
Comment on lines +117 to +119
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Add language specifier to fenced code block.

The fenced code block should have a language specified per markdownlint MD040. Use text or plaintext for format descriptions.

-```
+```text
 <v3-image-tag>:<ledger1>,<ledger2>,...
🧰 Tools
🪛 markdownlint-cli2 (0.22.0)

[warning] 117-117: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@docs/04-Modules/03-Ledger.md` around lines 117 - 119, The fenced code block
showing "<v3-image-tag>:<ledger1>,<ledger2>..." must include a language
specifier to satisfy markdownlint MD040; update the triple-backtick fence to
"```text" (or "```plaintext") so the block becomes ```text followed by
<v3-image-tag>:<ledger1>,<ledger2>... and the closing ``` to ensure the code
block is annotated.


- **v3-image-tag**: The container image tag of the ledger v3 binary (e.g. `v3.0.0-alpha.1`)
- **ledger names**: Comma-separated list of v2 ledger names to mirror

```yaml
apiVersion: formance.com/v1beta1
kind: Settings
metadata:
name: ledger-v3-mirror
spec:
stacks: ["my-stack"]
key: modules.ledger.v3-mirror
value: "v3.0.0-alpha.1:default,payments"
```

This example deploys a v3 cluster using image tag `v3.0.0-alpha.1` and creates two mirror ledgers (`default` and `payments`) that replicate from the v2 PostgreSQL database.

### How It Works

The provisioning Job connects to the v3 cluster's gRPC endpoint and calls `ledgerctl ledgers create` for each listed ledger with:
- `--mode mirror` — marks the ledger as a mirror (read-only, no direct writes)
- `--mirror-source-type postgres` — uses direct PostgreSQL access for replication
- `--mirror-dsn` — the PostgreSQL DSN of the v2 database (derived automatically from the Database resource)

The Job is idempotent: if a mirror ledger already exists, the error is ignored. It retries on failure (e.g. if the v3 cluster is not yet ready).

### Architecture

The operator creates the following resources for v3:

| Resource | Purpose |
|----------|---------|
| `StatefulSet/ledger` | Raft cluster nodes with `OrderedReady` pod management |
| `Service/ledger-raft` (headless) | DNS-based peer discovery for Raft consensus |
| `Job/v3-mirror-provision` | Creates mirror ledgers in the v3 cluster |
| 3 PVCs per pod | `wal`, `data`, `cold-cache` |

### Requirements

Ledger v3 does **not** require its own PostgreSQL or message broker. Storage is fully embedded (Pebble LSM). However, the v3 pods need network access to the v2 PostgreSQL database for mirror replication.

### Cluster Settings

```yaml
apiVersion: formance.com/v1beta1
kind: Settings
metadata:
name: ledger-v3-replicas
spec:
stacks: ["*"]
key: module.ledger.v3.replicas
value: "3"
```

- `module.ledger.v3.replicas`: Number of Raft nodes. **Must be odd** for quorum (default: 3).

The Raft cluster ID is automatically set to the stack name.

### Persistence Settings

Each pod gets three PVCs. Size and storage class are configurable:

```yaml
apiVersion: formance.com/v1beta1
kind: Settings
metadata:
name: ledger-v3-persistence
spec:
stacks: ["*"]
key: module.ledger.v3.persistence.wal.size
value: "5Gi"
---
apiVersion: formance.com/v1beta1
kind: Settings
metadata:
name: ledger-v3-data-size
spec:
stacks: ["*"]
key: module.ledger.v3.persistence.data.size
value: "10Gi"
---
apiVersion: formance.com/v1beta1
kind: Settings
metadata:
name: ledger-v3-cold-cache-size
spec:
stacks: ["*"]
key: module.ledger.v3.persistence.cold-cache.size
value: "10Gi"
```

| Key | Default | Description |
|-----|---------|-------------|
| `module.ledger.v3.persistence.wal.size` | 5Gi | WAL PVC size |
| `module.ledger.v3.persistence.wal.storage-class` | (cluster default) | WAL storage class |
| `module.ledger.v3.persistence.data.size` | 10Gi | Pebble data PVC size |
| `module.ledger.v3.persistence.data.storage-class` | (cluster default) | Data storage class |
| `module.ledger.v3.persistence.cold-cache.size` | 10Gi | Cold cache PVC size |
| `module.ledger.v3.persistence.cold-cache.storage-class` | (cluster default) | Cold cache storage class |

### Pebble Tunables

All Pebble settings are optional. When unset, the ledger binary defaults apply.

| Key | Example | Description |
|-----|---------|-------------|
| `module.ledger.v3.pebble.cache-size` | 1073741824 | Block cache size in bytes |
| `module.ledger.v3.pebble.memtable-size` | 268435456 | Memtable size in bytes |
| `module.ledger.v3.pebble.memtable-stop-writes-threshold` | 2 | Memtable count before stopping writes |
| `module.ledger.v3.pebble.l0-compaction-threshold` | 4 | L0 files to trigger compaction |
| `module.ledger.v3.pebble.l0-stop-writes-threshold` | 12 | L0 files before stopping writes |
| `module.ledger.v3.pebble.lbase-max-bytes` | 67108864 | L1 max size in bytes |
| `module.ledger.v3.pebble.target-file-size` | 67108864 | SST file target size |
| `module.ledger.v3.pebble.max-concurrent-compactions` | 2 | Compaction parallelism |

### Raft Tunables

All Raft settings are optional. When unset, the ledger binary defaults apply.

| Key | Example | Description |
|-----|---------|-------------|
| `module.ledger.v3.raft.snapshot-threshold` | 5000 | Log entries before snapshot |
| `module.ledger.v3.raft.election-tick` | 10 | Election timeout in ticks |
| `module.ledger.v3.raft.heartbeat-tick` | 1 | Heartbeat interval in ticks |
| `module.ledger.v3.raft.tick-interval` | 100ms | Duration of one tick |
| `module.ledger.v3.raft.max-size-per-msg` | 1048576 | Max message size in bytes |
| `module.ledger.v3.raft.max-inflight-msgs` | 256 | Max in-flight messages |
| `module.ledger.v3.raft.compaction-margin` | 1000 | Log retention after snapshot |
Loading