Phase 4: Security hardening + observability#105
Open
gonzafg2 wants to merge 8 commits intoZimengXiong:mainfrom
Open
Phase 4: Security hardening + observability#105gonzafg2 wants to merge 8 commits intoZimengXiong:mainfrom
gonzafg2 wants to merge 8 commits intoZimengXiong:mainfrom
Conversation
Replace SQLite with PostgreSQL 17 for production-grade persistence: - Change Prisma provider to postgresql, regenerate migrations - Rewrite docker-compose.yml with pg service, healthcheck, and pgdata volume - Simplify docker-entrypoint.sh: remove SQLite file logic, add PG wait loop - Update resolveDatabaseUrl() in config.ts and predev-migrate.cjs - Remove prisma_template copy from Dockerfile Add Cloudflare Tunnel integration for secure public access: - docker-compose.cloudflared.yml overlay (cloudflared + TRUST_PROXY) - GitHub Action workflow for tunnel/DNS/Zero Trust setup Update CI/CD to use PostgreSQL service containers in all test jobs.
- Backend healthcheck: send x-forwarded-proto header to bypass HTTPS redirect when TRUST_PROXY is enabled - Cloudflared: change depends_on from service_healthy to service_started to avoid startup deadlock with frontend healthcheck timing
- Add mem_limit and cpus to all services (db/backend: 512m/1cpu, frontend/cloudflared: 128m/0.5cpu) - Add connection_limit=10 to DATABASE_URL and disable UPDATE_CHECK_OUTBOUND - Fix frontend healthcheck: use 127.0.0.1 instead of localhost (alpine DNS) - Add daily PostgreSQL backup script with weekly snapshots and retention - Add backups/ to .gitignore - Add upstream sync documentation
- vitest.config: fileParallelism: false + singleFork: false so each test file runs in its own sequential fork (prevents beforeAll from one file truncating data set up by another file's beforeAll) - vitest.config: PostgreSQL fallback URL, ENABLE_AUDIT_LOGGING: true (config module caches env at load time, must be set before import) - testUtils: TRUNCATE + clearAuthEnabledCache instead of per-file SQLite databases (all files share one PostgreSQL database) - testUtils: Proxy around appPrisma prevents afterAll $disconnect from tearing down the shared PrismaClient - All test files: await setupTestDb() (now async)
PostgreSQL migration + Cloudflare Tunnel + hardening
POSTGRES_PASSWORD now requires .env (uses :? to fail explicitly). Fixes GitGuardian "Generic Password" alert on the DATABASE_URL connection string.
- verify-backup.sh: restores latest backup into ephemeral container, validates all 12 Prisma tables exist with coherent data - health-check.sh: checks public + local endpoints, alerts on state transitions via macOS notifications, no hardcoded secrets - Makefile: add db-verify-backup and health-check targets
…dation - Redact DATABASE_URL password in startup logs to prevent credential leaks - Enable audit logging by default (ENABLE_AUDIT_LOGGING=true) - Deep health check: /health now verifies DB connectivity (503 if down) - CI hardening: add npm audit, backend/frontend tsc --noEmit to lint job - Per-user cache invalidation: replace global drawingsCache.clear() with scoped invalidation so one user's edits don't flush everyone's cache - Add composite index on Drawing(userId, name) for search queries - Structured logging with pino: JSON in prod, pretty in dev, silent in tests; replaces console.* in startup, request middleware, error handler, CSRF, bootstrap
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| - | - | Generic Password | 8f7e355 | docker-compose.yml | View secret |
| - | - | Generic Password | ad12963 | docker-compose.yml | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secrets safely. Learn here the best practices.
- Revoke and rotate these secrets.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR focuses on “phase 4” security hardening and observability, including moving runtime/test infrastructure to PostgreSQL, enabling audit logging by default, adding deeper health checks, improving CI checks, and standardizing backend logging with Pino.
Changes:
- Introduces structured backend logging (Pino) with redaction, and updates multiple startup/runtime log sites.
- Switches local/dev/e2e/CI environments to PostgreSQL (compose updates, Prisma provider + new baseline migration, CI Postgres services).
- Adds ops scripts (backup, restore verification, health check) and tunes caching invalidation to be per-user.
Reviewed changes
Copilot reviewed 54 out of 57 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/verify-backup.sh | New script to restore latest compressed SQL backup into a temp Postgres container and validate expected tables/migrations. |
| scripts/health-check.sh | New one-shot monitor script for public/local endpoint reachability with transition-only notifications + log rotation. |
| scripts/backup-db.sh | New backup script using pg_dump + gzip with daily/weekly retention cleanup. |
| e2e/docker-compose.e2e.yml | Adds Postgres service and switches backend e2e DATABASE_URL to Postgres with health-based dependency. |
| docs/UPSTREAM-SYNC.md | Adds upstream sync notes and expected conflict areas for keeping fork aligned. |
| docker-compose.yml | Adds Postgres service, switches backend DATABASE_URL to Postgres, adds health-gated dependencies, resource limits, and upload volume. |
| docker-compose.cloudflared.yml | Adds Cloudflare Tunnel overlay compose for cloudflared + backend proxy-related env overrides. |
| backend/vitest.config.ts | Switches Vitest env DATABASE_URL default to Postgres and adjusts parallelism settings for shared DB usage. |
| backend/src/utils/tests/audit.test.ts | Updates test DB setup call to async/await. |
| backend/src/server/drawingsCache.ts | Adds per-user cache invalidation helper and exports it from the cache store. |
| backend/src/server/drawingsCache.test.ts | Adds unit test ensuring per-user cache invalidation doesn’t clear other users’ entries. |
| backend/src/server/csrf.ts | Switches CSRF debug logging from console to Pino and reduces logged candidate detail. |
| backend/src/routes/dashboard/types.ts | Extends dashboard deps to include invalidateDrawingsCacheForUser(userId). |
| backend/src/routes/dashboard/drawings.ts | Replaces global cache clear with per-user invalidation on drawing mutations; refactors response payload building. |
| backend/src/routes/dashboard/collections.ts | Uses per-user cache invalidation after collection deletion. |
| backend/src/middleware/errorHandler.ts | Switches error logging from console to Pino structured logs. |
| backend/src/logger.ts | New Pino logger configuration (prod JSON / dev pretty / test silent) with redaction paths. |
| backend/src/index.ts | Adds DATABASE_URL redaction in startup logs, converts several logs to Pino, adds deep /health DB check, wires per-user cache invalidation. |
| backend/src/config.ts | Defaults DB URL to Postgres, keeps legacy file: normalization for import, enables audit logging by default. |
| backend/src/auth/bootstrapSetupCode.ts | Switches bootstrap setup code emission log from console to Pino structured logging. |
| backend/src/tests/user-sandboxing.test.ts | Updates test DB setup call to async/await. |
| backend/src/tests/testUtils.ts | Reworks test DB utilities for Postgres: shared Prisma client proxy + TRUNCATE-based reset. |
| backend/src/tests/link-sharing-public.integration.ts | Updates test DB setup call to async/await. |
| backend/src/tests/imports-compat.integration.ts | Updates test DB setup call to async/await. |
| backend/src/tests/drawings.integration.ts | Updates test DB setup call to async/await. |
| backend/src/tests/drawings-shared.integration.ts | Updates test DB setup call to async/await. |
| backend/src/tests/auth-onboarding.integration.ts | Updates test DB setup call to async/await. |
| backend/src/tests/auth-enabled.integration.ts | Updates test DB setup call to async/await. |
| backend/scripts/predev-migrate.cjs | Updates dev migrate helper to default to Postgres URL and handles P3005 reset messaging for non-SQLite DBs. |
| backend/prisma/schema.prisma | Switches datasource provider to PostgreSQL and adds composite index on Drawing (userId, name). |
| backend/prisma/migrations/migration_lock.toml | Updates migration lock provider to PostgreSQL. |
| backend/prisma/migrations/20260318045122_init/migration.sql | Adds new Postgres baseline schema migration (tables, indexes, FKs). |
| backend/prisma/migrations/20260319000000_add_drawing_user_name_index/migration.sql | Adds composite index for drawing search (userId, name). |
| backend/prisma/migrations/20260217214759_drawing_sharing/migration.sql | Removes legacy SQLite migration. |
| backend/prisma/migrations/20260211232000_add_bootstrap_setup_code/migration.sql | Removes legacy SQLite migration. |
| backend/prisma/migrations/20260210190000_add_auth_identity/migration.sql | Removes legacy SQLite migration. |
| backend/prisma/migrations/20260210153000_add_auth_onboarding_completed/migration.sql | Removes legacy SQLite migration. |
| backend/prisma/migrations/20260207000000_add_query_indexes/migration.sql | Removes legacy SQLite migration. |
| backend/prisma/migrations/20260206195000_add_auth_login_rate_limit_config/migration.sql | Removes legacy SQLite migration. |
| backend/prisma/migrations/20260206173000_add_auth_enabled_system_config/migration.sql | Removes legacy SQLite migration. |
| backend/prisma/migrations/20260124152839_add_password_reset_audit_refresh_tokens/migration.sql | Removes legacy SQLite migration. |
| backend/prisma/migrations/20260124145151_add_user_auth/migration.sql | Removes legacy SQLite migration. |
| backend/prisma/migrations/20251124220546_add_library_model/migration.sql | Removes legacy SQLite migration. |
| backend/prisma/migrations/20251122065455_add_files_column/migration.sql | Removes legacy SQLite migration. |
| backend/prisma/migrations/20251122032308_add_preview/migration.sql | Removes legacy SQLite migration. |
| backend/prisma/migrations/20251122021659_init/migration.sql | Removes legacy SQLite migration. |
| backend/package.json | Adds Pino + ESLint-related deps and introduces backend lint/openapi scripts. |
| backend/package-lock.json | Updates lockfile for new deps and bumps backend package version. |
| backend/docker-entrypoint.sh | Updates container startup flow for Postgres readiness + migrations; changes secret generation behavior. |
| backend/Dockerfile | Removes prisma template copy and keeps only runtime prisma directory in image. |
| backend/.env.example | Updates example DATABASE_URL and comments for Postgres/Kubernetes usage. |
| Makefile | Adds db-verify-backup and health-check targets and updates help output grouping. |
| .gitignore | Ignores backups/ directory. |
| .github/workflows/test.yml | Adds lint job (eslint, audit, typecheck) and adds Postgres services + migrations to test jobs. |
| .github/workflows/cloudflare-tunnel-setup.yml | Adds workflow_dispatch automation for creating/configuring Cloudflare tunnels + DNS and optional Access policy. |
| .env.example | Adds root .env example for Postgres + app + optional cloudflared settings. |
Files not reviewed (1)
- backend/package-lock.json: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+40
to
+48
| const redactDatabaseUrl = (url?: string): string => { | ||
| if (!url) return "(not set)"; | ||
| try { | ||
| const parsed = new URL(url); | ||
| if (parsed.password) parsed.password = "***"; | ||
| return parsed.toString(); | ||
| } catch { | ||
| return url; | ||
| } |
Comment on lines
721
to
723
| } catch (error) { | ||
| console.error("Failed to issue bootstrap setup code:", error); | ||
| } |
Comment on lines
86
to
91
| const resolveDatabaseUrl = (rawUrl?: string) => { | ||
| const backendRoot = path.resolve(__dirname, "../"); | ||
| const defaultDbPath = path.resolve(backendRoot, "prisma/dev.db"); | ||
|
|
||
| if (!rawUrl || rawUrl.trim().length === 0) { | ||
| return `file:${defaultDbPath}`; | ||
| return "postgresql://excalidash:excalidash@localhost:5432/excalidash"; | ||
| } |
Comment on lines
+29
to
+30
| cd frontend && npm audit --audit-level=high --omit=dev || true | ||
| cd ../backend && npm audit --audit-level=high --omit=dev || true |
Comment on lines
+34
to
+41
| export const setupTestDb = async () => { | ||
| await appPrisma.$executeRawUnsafe(` | ||
| TRUNCATE TABLE | ||
| "AuditLog", "AuthIdentity", "RefreshToken", "PasswordResetToken", | ||
| "DrawingLinkShare", "DrawingPermission", "Drawing", | ||
| "Collection", "Library", "SystemConfig", "User" | ||
| CASCADE | ||
| `); |
Comment on lines
+6
to
18
| # --- Secret management --- | ||
| # Generate JWT_SECRET if not provided | ||
| if [ -z "${JWT_SECRET:-}" ]; then | ||
| echo "JWT_SECRET not provided, resolving persisted secret..." | ||
| if [ -f "${JWT_SECRET_FILE}" ]; then | ||
| JWT_SECRET="$(tr -d '\r\n' < "${JWT_SECRET_FILE}")" | ||
| fi | ||
|
|
||
| if [ -z "${JWT_SECRET}" ]; then | ||
| echo "No persisted JWT secret found. Generating a new secret..." | ||
| JWT_SECRET="$(openssl rand -hex 32)" | ||
| umask 077 | ||
| printf "%s" "${JWT_SECRET}" > "${JWT_SECRET_FILE}" | ||
| fi | ||
| else | ||
| # Persist explicitly provided secret to support future restarts without env injection. | ||
| umask 077 | ||
| printf "%s" "${JWT_SECRET}" > "${JWT_SECRET_FILE}" | ||
| echo "JWT_SECRET not provided. Generating an ephemeral secret..." | ||
| JWT_SECRET="$(openssl rand -hex 32)" | ||
| fi | ||
|
|
||
| export JWT_SECRET | ||
|
|
||
| # Ensure CSRF secret exists for stable token validation across restarts. | ||
| # (Still recommend setting explicitly for multi-instance deployments.) | ||
| # Generate CSRF_SECRET if not provided | ||
| if [ -z "${CSRF_SECRET:-}" ]; then | ||
| echo "CSRF_SECRET not provided, resolving persisted secret..." | ||
| if [ -f "${CSRF_SECRET_FILE}" ]; then | ||
| CSRF_SECRET="$(tr -d '\r\n' < "${CSRF_SECRET_FILE}")" | ||
| fi | ||
|
|
||
| if [ -z "${CSRF_SECRET}" ]; then | ||
| echo "No persisted CSRF secret found. Generating a new secret..." | ||
| CSRF_SECRET="$(openssl rand -base64 32)" | ||
| umask 077 | ||
| printf "%s" "${CSRF_SECRET}" > "${CSRF_SECRET_FILE}" | ||
| fi | ||
| else | ||
| umask 077 | ||
| printf "%s" "${CSRF_SECRET}" > "${CSRF_SECRET_FILE}" | ||
| echo "CSRF_SECRET not provided. Generating an ephemeral secret..." | ||
| CSRF_SECRET="$(openssl rand -base64 32)" | ||
| fi |
Comment on lines
+29
to
+33
| # Extract host:port from postgresql://user:pass@host:port/db | ||
| DB_HOST=$(echo "$DB_URL" | sed -n 's|.*@\([^:/]*\).*|\1|p') | ||
| DB_PORT=$(echo "$DB_URL" | sed -n 's|.*@[^:]*:\([0-9]*\).*|\1|p') | ||
| DB_PORT="${DB_PORT:-5432}" | ||
|
|
Comment on lines
+123
to
126
| logger.info( | ||
| { reason, code, expiresAt: expiresAt.toISOString() }, | ||
| "Bootstrap setup code issued" | ||
| ); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DATABASE_URLpassword masked as***in startup logsENABLE_AUDIT_LOGGING=true— security events now recorded out of the box/healthverifies DB connectivity withSELECT 1, returns 503 if Postgres is downnpm audit,tsc --noEmit(backend + frontend) added to lint jobinvalidateDrawingsCacheForUser(userId)replaces globaldrawingsCache.clear()— one user's edits no longer flush everyone's cache@@index([userId, name])on Drawing model + migrationconsole.*in startup, request middleware, error handler, CSRF, and bootstrapTest plan
eslint— 0 errors backend and frontendtsc --noEmit— compiles cleancurl localhost:8000/healthreturns{"status":"ok","db":"up"}/ 503 when DB down***instead of DB passwordnpx prisma migrate deployapplies new index migration