fix(server): separate migration ownership from startup

Author: leeroybrun

apps/docs/content/docs/self-hosting/docker.mdx

@@ -73,7 +73,7 @@ Notes:
 
 ### Prebuilt image: use Postgres (full flavor)
 
-The `relay-server` image is built from the same server code and can run full flavor too — you just override the defaults. Postgres is the external database supported by the prebuilt image. On startup, the image applies its bundled Postgres migrations before starting the server; set `RUN_MIGRATIONS=0` only when migrations are managed separately.
+The `relay-server` image is built from the same server code and can run full flavor too — you just override the defaults. Postgres is the external database supported by the prebuilt image. On startup, the image applies its bundled Postgres migrations before starting the server. Health-managed or multi-replica deployments should instead use the dedicated migration step described under [Migrations](#migrations).
 
 Example `docker-compose.yml` (Postgres + local files backend):
 
@@ -555,10 +555,23 @@ services:
 
 The source-built `server` target and prebuilt `relay-server` image run DB migrations by default. Migration selection follows `HAPPIER_DB_PROVIDER`, independently of the preset: Postgres, MySQL, and PGlite use their packaged/provider migration owner; SQLite uses the server's canonical in-process startup migration path.
 
+The default is convenient for a single container. For Postgres, MySQL, or PGlite deployments managed by an orchestrator, run migrations once before replacing application containers:
+
+```bash
+docker compose run --rm relay run-server --migrate-only
+docker compose up -d --force-recreate relay
+```
+
+Set `RUN_MIGRATIONS=0` on the long-running API and worker services. The explicit `--migrate-only` command still runs migrations when that environment value is present, exits only after migration completion, and never starts the server. Your deployment system must treat a non-zero exit as a failed deployment and leave the existing application containers running.
+
+If your platform cannot run and await a blocking pre-deploy command (for example, separate Dokploy Applications triggered by independent webhooks), designate exactly one API service as the migration owner instead: set `RUN_MIGRATIONS=1` only there, set `RUN_MIGRATIONS=0` on every worker and other replica, use a start-first rollout with automatic rollback, and give the migration-owning container enough health-check startup grace for the largest expected migration. This fallback relies on backward-compatible migrations so workers may continue serving while the API owner upgrades the schema.
+
 Notes:
 
 - Disable automatic migrations with `RUN_MIGRATIONS=0`. For SQLite, the entrypoint maps this to the server startup migration setting; it does not launch a second migration process.
-- In Postgres multi-replica setups, it’s OK if more than one replica tries to migrate at startup (the DB serializes via locks).
+- Keep SQLite on its normal single-process startup path; the packaged SQLite runtime does not support `--migrate-only`.
+- PostgreSQL advisory locks serialize competing migration attempts, but they do not protect a migration if an orchestrator terminates its container for failing startup health checks. Use one dedicated migration owner when migrations can outlive the platform's startup grace period.
+- If migration deployment is interrupted, inspect the database and migration ledger before recovery. Never mark an unknown failed migration as applied merely to make the server start.
 
 ## Reverse proxy checklist
 

apps/server/scripts/run-server.sh

@@ -1,9 +1,15 @@
 #!/bin/sh
 set -eu
 
+migrate_only=0
+if [ "${1:-}" = "--migrate-only" ]; then
+  migrate_only=1
+  shift
+fi
+
 server_binary="${1:-}"
 if [ "$#" -gt 1 ]; then
-  echo "[entrypoint] Usage: run-server.sh [packaged-server-binary]"
+  echo "[entrypoint] Usage: run-server.sh [--migrate-only] [packaged-server-binary]"
   exit 1
 fi
 if [ -n "$server_binary" ] && [ ! -x "$server_binary" ]; then
@@ -22,6 +28,9 @@ migrations_enabled=1
 if is_false "${RUN_MIGRATIONS:-1}" || is_false "${HAPPIER_STACK_PRISMA_MIGRATE:-1}"; then
   migrations_enabled=0
 fi
+if [ "$migrate_only" = "1" ]; then
+  migrations_enabled=1
+fi
 
 provider="$(printf "%s" "${HAPPIER_DB_PROVIDER:-${HAPPY_DB_PROVIDER:-postgres}}" | tr '[:upper:]' '[:lower:]' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')"
 flavor="$(printf "%s" "${HAPPIER_SERVER_FLAVOR:-${HAPPY_SERVER_FLAVOR:-full}}" | tr '[:upper:]' '[:lower:]' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')"
@@ -45,6 +54,11 @@ case "$provider" in
     ;;
 esac
 
+if [ "$migrate_only" = "1" ] && [ "$should_migrate" = "0" ]; then
+  echo "[entrypoint] --migrate-only is not supported by the packaged SQLite runtime."
+  exit 1
+fi
+
 export HAPPIER_DB_PROVIDER="$provider"
 export HAPPY_DB_PROVIDER="$provider"
 export HAPPIER_SERVER_FLAVOR="$flavor"
@@ -128,6 +142,11 @@ if [ "$should_migrate" = "1" ] && [ "$migrations_enabled" = "1" ]; then
   fi
 fi
 
+if [ "$migrate_only" = "1" ]; then
+  echo "[entrypoint] Migrations complete."
+  exit 0
+fi
+
 if [ -n "$server_binary" ]; then
   exec "$server_binary"
 fi

apps/server/scripts/run-server.sh.test.ts

@@ -185,6 +185,67 @@ describe('run-server.sh', () => {
     ]);
   });
 
+  it('runs only migrations when --migrate-only is requested', async () => {
+    const serverPath = await writeFakePackagedRuntime({ dir: tmpDir, logPath });
+    const res = spawnSync('sh', [getScriptPath(), '--migrate-only', serverPath], {
+      env: {
+        ...process.env,
+        HAPPIER_SERVER_FLAVOR: 'full',
+        HAPPIER_DB_PROVIDER: 'postgres',
+        DATABASE_URL: 'postgresql://postgres@db/happier',
+        RUN_MIGRATIONS: '0',
+        MIGRATIONS_MAX_ATTEMPTS: '1',
+        MIGRATIONS_RETRY_DELAY_SECONDS: '0',
+      },
+      stdio: 'pipe',
+      encoding: 'utf8',
+    });
+
+    expect(res.status).toBe(0);
+    expect(await readLogLines(logPath)).toEqual([
+      'MIGRATE provider=postgres url=postgresql://postgres@db/happier',
+    ]);
+  });
+
+  it('runs only the source-backed migration command when --migrate-only is requested', async () => {
+    const res = spawnSync('sh', [getScriptPath(), '--migrate-only'], {
+      env: {
+        ...process.env,
+        PATH: `${binDir}:${process.env.PATH ?? ''}`,
+        HAPPIER_SERVER_FLAVOR: 'full',
+        HAPPIER_DB_PROVIDER: 'postgres',
+        DATABASE_URL: 'postgresql://postgres@db/happier',
+        RUN_MIGRATIONS: '0',
+        MIGRATIONS_MAX_ATTEMPTS: '1',
+        MIGRATIONS_RETRY_DELAY_SECONDS: '0',
+      },
+      stdio: 'pipe',
+      encoding: 'utf8',
+    });
+
+    expect(res.status).toBe(0);
+    const yarnLines = (await readLogLines(logPath)).filter((line) => line.startsWith('YARN '));
+    expect(yarnLines).toEqual(['YARN --cwd apps/server migrate:deploy']);
+  });
+
+  it('fails closed when --migrate-only cannot own packaged SQLite migrations', async () => {
+    const serverPath = await writeFakePackagedRuntime({ dir: tmpDir, logPath });
+    const res = spawnSync('sh', [getScriptPath(), '--migrate-only', serverPath], {
+      env: {
+        ...process.env,
+        HAPPIER_SERVER_FLAVOR: 'light',
+        HAPPIER_DB_PROVIDER: 'sqlite',
+        RUN_MIGRATIONS: '0',
+      },
+      stdio: 'pipe',
+      encoding: 'utf8',
+    });
+
+    expect(res.status).toBe(1);
+    expect(res.stdout).toContain('--migrate-only is not supported by the packaged SQLite runtime');
+    expect(await readLogLines(logPath)).toEqual([]);
+  });
+
   it('delegates packaged SQLite migration to normal server startup', async () => {
     const serverPath = await writeFakePackagedRuntime({ dir: tmpDir, logPath });
     const res = spawnSync('sh', [getScriptPath(), serverPath], {

docs/deployment.md

@@ -90,6 +90,9 @@ Key notes:
 - The server defaults to port `3005` (set `PORT` explicitly in container environments).
 - The image includes FFmpeg and Python for media processing.
 - The server entrypoint (`apps/server/scripts/run-server.sh`) runs `prisma migrate deploy` on startup by default (set `RUN_MIGRATIONS=0` to disable). On Postgres, it retries on advisory-lock contention.
+- Health-managed and multi-replica deployments should run `run-server --migrate-only` as a single pre-deploy operation, then start API and worker replicas with `RUN_MIGRATIONS=0`. The migration command exits before server startup and explicitly overrides `RUN_MIGRATIONS=0`.
+- PostgreSQL advisory locks serialize concurrent migrators but cannot preserve a migration when the owning container is terminated. The pre-deploy operation's lifetime and timeout must be independent from application startup health checks.
+- If the platform cannot block application rollout on a pre-deploy operation, use exactly one API migration owner (`RUN_MIGRATIONS=1`), disable migrations on all workers and other replicas, and configure start-first rollout, rollback on failure, and a health startup grace long enough for the largest expected migration. Separate auto-deploy webhooks are not an ordering mechanism.
 
 ## Kubernetes manifests
 Example manifests live in `apps/server/deploy`:

docs/release-process.md

@@ -256,6 +256,8 @@ The reset option exists for rare cases where you intentionally want `target` to
 
 For the server, database migrations should be automated as part of the deployment runtime:
 
-- Run `prisma migrate deploy` at container startup (entrypoint) or via an explicit platform “pre-deploy” hook.
-- Running migrations from *both* API and worker is acceptable as long as you expect contention and handle it (Prisma uses a DB lock to serialize migrations; the non-holder should wait/retry).
+- For a single unmanaged container, the default entrypoint may run `prisma migrate deploy` before server startup.
+- For health-managed or multi-replica deployments, run `run-server --migrate-only` once in an explicit platform pre-deploy operation. Start API and worker replicas with `RUN_MIGRATIONS=0` only after that operation succeeds.
+- When an application platform cannot run and await a blocking pre-deploy operation, designate exactly one API service as the migration owner and set `RUN_MIGRATIONS=0` on workers and all other replicas. Protect that owner with start-first rollout, rollback on failure, and sufficient health-check startup grace; webhook acceptance alone does not prove migration or deployment completion.
+- Do not rely on API and worker startup races as migration ownership. Prisma's database lock serializes contenders, but it cannot preserve the winning migration when an orchestrator terminates that container for missing its startup-health window.
 - Avoid running migrations at image build-time (Dockerfile), since migrations require a live DB connection.