Migrate from sqlite3 to better-sqlite3

[sestinj]

Summary

• Replaces async sqlite3/sqlite packages with synchronous better-sqlite3 for improved performance and simpler API
• Updates all 11 source files that interact with SQLite databases
• Converts callback-based transaction patterns to db.transaction() API

Test plan

• Verify indexing works end-to-end with the new SQLite driver
• Run existing unit tests for affected modules
• Check that setup scripts still work (or flag if they need updates)

🤖 Generated with Claude Code

---

Continue Tasks: ▶️ 2 queued · ▶️ 35 not started · ✅ 6 no changes — View all

---

Summary by cubic

Migrate SQLite usage to better-sqlite3 for faster, simpler, fully synchronous DB access. Replaces async sqlite/sqlite3 across the codebase and standardizes queries and transactions.

• Refactors

  • Replaced open/sqlite3 with new Database() from better-sqlite3 and converted all calls to prepare().get/all/run, exec, and db.transaction().
  • Updated all indexing, docs, chunking, FTS, LanceDB cache, autocomplete, devdata, and test index modules to the sync API.
  • Kept PRAGMAs (busy_timeout, WAL) and adjusted migrations to the sync driver.

• Dependencies

  • Added better-sqlite3 and @types/better-sqlite3; removed sqlite and sqlite3.
  • After pulling, reinstall deps to build the native module (no data migration needed).

^{Written for commit 28b9cb1. Summary will update on new commits.}

[cubic-dev-ai]

3 issues found across 11 files

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="core/indexing/docs/migrations.ts">

<violation number="1" location="core/indexing/docs/migrations.ts:29">
P2: Fire-and-forget `void migrate(...)` discards the returned Promise, silently swallowing any errors that occur after the internal `await` in `migrate()`. The old code properly awaited completion of each migration before proceeding. Since `runSqliteMigrations` is no longer async, consider making these calls truly synchronous or keeping the function async and awaiting the `migrate` calls.</violation>
</file>

<file name="core/indexing/FullTextSearchCodebaseIndex.ts">

<violation number="1" location="core/indexing/FullTextSearchCodebaseIndex.ts:59">
P2: Guard against empty `results` before building the `IN (...)` query; `IN ()` is invalid SQL and will throw when no rows match the BM25 threshold.</violation>
</file>

<file name="core/indexing/LanceDbIndex.ts">

<violation number="1" location="core/indexing/LanceDbIndex.ts:477">
P2: Build the IN clause with bound parameters and handle the empty-results case to avoid SQL injection and invalid `IN ()` queries.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

P2: Fire-and-forget void migrate(...) discards the returned Promise, silently swallowing any errors that occur after the internal await in migrate(). The old code properly awaited completion of each migration before proceeding. Since runSqliteMigrations is no longer async, consider making these calls truly synchronous or keeping the function async and awaiting the migrate calls.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At core/indexing/docs/migrations.ts, line 29:

<comment>Fire-and-forget `void migrate(...)` discards the returned Promise, silently swallowing any errors that occur after the internal `await` in `migrate()`. The old code properly awaited completion of each migration before proceeding. Since `runSqliteMigrations` is no longer async, consider making these calls truly synchronous or keeping the function async and awaiting the `migrate` calls.</comment>

<file context>
@@ -25,84 +25,60 @@ export async function runLanceMigrations(table: Table) {
-            `PRAGMA table_info(${DocsService.sqlitebTableName});`,
-          );
+export function runSqliteMigrations(db: BetterSqlite3.Database) {
+  void migrate("sqlite_modify_docs_columns_and_copy_to_config", async () => {
+    const pragma = db
+      .prepare(`PRAGMA table_info(${DocsService.sqlitebTableName});`)
</file context>

[cubic-dev-ai]

P2: Guard against empty results before building the IN (...) query; IN () is invalid SQL and will throw when no rows match the BM25 threshold.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At core/indexing/FullTextSearchCodebaseIndex.ts, line 59:

<comment>Guard against empty `results` before building the `IN (...)` query; `IN ()` is invalid SQL and will throw when no rows match the BM25 threshold.</comment>

<file context>
@@ -56,25 +56,22 @@ export class FullTextSearchCodebaseIndex implements CodebaseIndex {
-        "SELECT * FROM chunks WHERE path = ? AND cacheKey = ?",
-        [item.path, item.cacheKey],
-      );
+      const chunks = db
+        .prepare("SELECT * FROM chunks WHERE path = ? AND cacheKey = ?")
+        .all(item.path, item.cacheKey) as any[];
</file context>

[cubic-dev-ai]

P2: Build the IN clause with bound parameters and handle the empty-results case to avoid SQL injection and invalid IN () queries.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At core/indexing/LanceDbIndex.ts, line 477:

<comment>Build the IN clause with bound parameters and handle the empty-results case to avoid SQL injection and invalid `IN ()` queries.</comment>

<file context>
@@ -475,11 +474,13 @@ export class LanceDbIndex implements CodebaseIndex {
-        .map((r) => `'${r.uuid}'`)
-        .join(",")})`,
-    );
+    const data = sqliteDb
+      .prepare(
+        `SELECT * FROM lance_db_cache WHERE uuid in (${allResults
</file context>

Suggested change

	const data = sqliteDb
	.prepare(
	`SELECT * FROM lance_db_cache WHERE uuid in (${allResults
	.map((r) => `'${r.uuid}'`)
	.join(",")})`,
	)
	.all() as any[];
	if (allResults.length === 0) {
	return [];
	}
	const uuids = allResults.map((r) => r.uuid);
	const placeholders = uuids.map(() => "?").join(",");
	const data = sqliteDb
	.prepare(
	`SELECT * FROM lance_db_cache WHERE uuid IN (${placeholders})`,
	)
	.all(...uuids) as any[];