feat: parquet loading, pgvector support, multi-table datasets, and cohere-10m vector dataset

[rebasedming]

Description

Adds everything needed to run vector benchmarks against the Cohere Wikipedia 10M dataset (the same corpus used by paradedb/paradedb CI), comparing the pg_search vector index against pgvector HNSW — plus multi-table dataset loading, groundwork for normalized-dataset benchmarks (e.g. StackOverflow joins).

Loader / driver changes

• Parquet input: loader load now accepts data.parquet, or a data/ directory of sharded parquet files (the S3 layout), alongside data.csv. Reading is streamed per row group; shards are chained with a column-consistency check.
• Multi-table datasets: schema.yaml accepts a tables list (one entry per table with its own columns) as an alternative to the single top-level table/columns form. Each table resolves its own data source under data/ with the same preference order (<table>.parquet, <table>.csv, <table>/ shard directory); tables load sequentially and drop removes all of them. Validation rejects mixed forms, nested entries, and duplicates, and names the failing table in per-backend column-type errors. Single-table datasets are unchanged.
• vector(n) schema type: CSV cells parse JSON arrays; parquet list<float> columns map directly to pgvector.Vector.
• smallint schema type: supported on both CSV and parquet paths with int16 bounds checks. Conversion happens in the shared RowSource layer, so every backend receives a Go int16 (pgx encodes int2 over binary COPY; ClickHouse DDL should use Int16).
• pgvector type registration: the shared postgres driver registers pgvector codecs in AfterConnect (best-effort), and a new TypeReloader hook resets the pool after pre scripts run — required because pre.sql does CREATE EXTENSION vector on connections that predate the extension.

New dataset: datasets/cohere-10m/

• Schema/table matching paradedb/paradedb benchmarks/datasets/cohere (cohere_wiki, emb vector(1024)).
• post.sql builds the real pg_search vector index (USING bm25 ... emb vector_cosine_ops) vs pgvector HNSW, with the recall knobs (paradedb.vector_cluster_max_probe, hnsw.ef_search) set via ALTER DATABASE and documented as the operating points to tune.
• Docker compose (paradedb v0.25.6-pg18, pgvector pg18), k6 KNN script with staggered phases, and a script converting the official cohere_queries.parquet into a k6 query pool.
• README covers pulling the sharded data from S3 and the recall-parity methodology.

Testing

• Unit tests: vector conversions, parquet round-trip asserting pgvector.Vector reaches the driver, sharded-directory loading, multi-table schema validation/expansion, per-table data resolution, and smallint conversion/overflow on both paths.
• End-to-end smoke test on 200k real rows (2 S3 shards): both backends loaded at 13–22k rows/sec, both indexes built (pg_search 49s, HNSW 99s), and a k6 run completed 108,999 queries with zero errors. Verified pgvector plans use the HNSW index and the pg_search KNN query returns correct rows.
• Multi-table smoke test against paradedb v0.25.0: a two-table dataset (one table from CSV, one from a directory of parquet shards) loaded in one run, join query returned correct counts, drop removed both tables. smallint verified end-to-end from both CSV and parquet into real int2 columns (including bounds value 32767).

🤖 Generated with Claude Code