feat(v2): records — schema-version-pinned rows, validated bulk-upsert, references view (Phase 2)

docs/superpowers/plans/2026-07-03-v2-records.md

@@ -0,0 +1,60 @@
+# v2 Records Implementation Plan (Phase 2 of 6)
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Executed 2026-07-03 on branch `feat/v2-records` (stacked on `feat/v2-schema-registry`, PR #219).
+
+**Goal:** Add the v2 `record` entity — one typed row pinned to a `schema_version`, carrying the cross-schema `reference` join key and Pandera-validated `fields` JSONB — with a validated bulk-upsert, paginated listing, bulk delete, and the `GET /references/{reference}` cross-schema document view (spec §5 record row, §6 write flow, §7 API surface).
+
+**Out of scope:** LanceDB search/sync (Phase 3), questions/suggestions/responses (Phase 4).
+
+## Architecture
+
+Extends the isolated v2 module set from Phase 1 (`models/v2/`, `contexts/v2/`, `api/v2/`, `api/schemas/v2/`). Postgres is the source of truth. On bulk-upsert, the context resolves each item's `schema_version_id` (default = `schema.current_version_id`), fetches each **distinct** version's Pandera body from the object store **once per request** (dict keyed by version id, via `contexts.files.get_object` pinned to the stored S3 `object_version_id`), validates every item via Phase 1's `validate_record_fields` *before any write* (all-or-nothing 422), then persists v1-style (fetch-existing-by-external_id → merge → flush → single commit).
+
+## Key decisions (as built)
+
+| Decision | Choice |
+|---|---|
+| Table name | `v2_records` — v1 owns `records`; renamed at Phase 6 retirement |
+| ORM class | `V2Record` (not `Record`): a second declarative class named `Record` breaks v1's string-based `relationship("Record")` lookups in the shared registry. `models/v2/__init__` also exports it as `Record` for v2-namespace ergonomics; `models/__init__` registers `V2Record` |
+| Status enum | New `V2RecordStatus` (`pending\|completed\|discarded`), PG type `v2_record_status_enum` — v1's `record_status_enum` untouched |
+| Bulk route | `POST /api/v2/schemas/{schema_id}/records:bulk-upsert` (spec §7, AIP-136 style; Starlette treats `:` as literal) → `{items, total}`, 200 |
+| List route | `GET /api/v2/schemas/{schema_id}/records?offset&limit&status&reference` → `{items, total}` (50 default / 1000 max, exact total) |
+| Delete route | `DELETE /api/v2/schemas/{schema_id}/records?ids=<csv>` → 204; empty param → specific 422; cap 100 |
+| References | `GET /api/v2/references/{reference:path}?workspace_id=<uuid>` — `:path` converter because DOIs contain slashes; unknown reference → 200 empty view (a reference is a join key, not an entity); required `workspace_id` mirrors `GET /schemas` scoping |
+| Upsert identity | `(schema_id, external_id)`, nullable `external_id` always inserts; duplicates within one payload → 422. Fetch-then-merge (not `upsert_many`): NULLs defeat ON CONFLICT, and validation must run before any write |
+| Update semantics | Patch-like for `metadata`/`status` (omitted preserves existing; documented on `RecordUpsert` and the context); `fields`/`reference`/version pin always overwritten |
+| Policies | `SchemaPolicy.{upsert_records,list_records,delete_records}` (writes: owner or admin member; reads: member); references view reuses `SchemaPolicy.list(workspace_id)`. `RecordPolicy` name belongs to v1 |
+| FKs | `schema_id` and `schema_version_id` both `ondelete=CASCADE` |
+| `metadata` column | Attr `metadata_` → column `"metadata"` (v1 pattern); `RecordRead` accepts either via `AliasChoices`, serializes as `metadata` |
+
+## Tasks (all completed, TDD: red → green → commit)
+
+1. **`V2RecordStatus` enum** — `tests/integration/test_enums_v2.py`; `enums.py`. `feat(v2): add V2RecordStatus enum`
+2. **`V2Record` ORM model** — `models/v2/records.py`, exports, aliased registration in `models/__init__.py`; `tests/integration/models/v2/test_record_models.py` (defaults, uniq constraint, NULL external_ids don't collide). `feat(v2): V2Record ORM model (v2_records) pinned to schema versions`
+3. **Alembic migration** — `8136bc88ee3a` (down_revision `9f3010c649c8`): table, enum, FKs, uniq, 4 indexes; upgrade→downgrade→upgrade round-trip verified. `feat(v2): alembic migration for v2_records table`
+4. **`V2RecordFactory`** — SubFactory(SchemaVersionFactory) with `_create` override awaiting the version to wire `schema_id`/`schema_version_id` (LazyAttribute sees a coroutine). `test(v2): V2RecordFactory`
+5. **Pydantic schemas** — `api/schemas/v2/records.py`: limits (500 bulk / 50-1000 list / 100 delete), `RecordUpsert`, `RecordsBulkUpsert`, `RecordRead`, `Records`, `ReferenceGroup`+`ReferenceView` (flattened `schema_id`/`schema_name` to avoid `BaseModel.schema` shadowing). `feat(v2): pydantic request/response schemas for v2 records and references`
+6. **Records context** — `contexts/v2/records.py`: `_fetch_body_json` (the test seam; monkeypatched in tests), `bulk_upsert_records`, `list_records`, `delete_records` (schema-scoped), `list_records_by_reference` (workspace-scoped join). 10 tests in `tests/integration/contexts/v2/test_records_context.py`. `feat(v2): records context with validated bulk-upsert, list, delete, reference query`
+7. **Policy actions** — `SchemaPolicy` additions. `feat(v2): record-scoped policy actions on SchemaPolicy`
+8. **Records router** — `api/v2/records.py` + mount in `create_api_v2()`; 9 API tests incl. member/non-member authz negatives and 422 validation detail. `feat(v2): /api/v2 records endpoints (bulk-upsert, list, delete) with authz`
+9. **References endpoint** — same router; 5 tests incl. DOI (slash) reference and workspace scoping. `feat(v2): GET /api/v2/references/{reference} cross-schema document view`
+10. **Gate** — 55 v2 tests green; `configure_mappers()` + app-import smoke (v1/v2 registry coexistence); ruff clean except pre-existing `helpers.py` ASYNC240 (untouched by this branch).
+
+## Review findings addressed (roborev jobs 63–71)
+
+- **HIGH missing migration (job 64)** — migration landed in the adjacent commit (same pattern as Phase 1); `__repr__` now says `V2Record(...)`.
+- **LOW upsert asymmetry (job 68)** — patch-like preserve-on-omit kept intentionally; contract documented on `RecordUpsert` and the context docstring.
+- **LOW dead empty-ids branch (job 70)** — empty `ids=` rejected before `parse_uuids`, so the specific "No record IDs provided" 422 surfaces; asserted in tests.
+- **MEDIUM slash references unreachable (job 71)** — `{reference:path}` converter + DOI test.
+
+## Accepted risks
+
+- Concurrent bulk-upserts racing on `(schema_id, external_id)` can raise IntegrityError (500) — same class as v1's behavior and Phase 1's accepted publish race; hardened when record writes move to the job queue.
+- `:` custom-verb route is Starlette-safe; if a proxy ever mangles it, fall back to `/records/bulk` (one-line change).
+
+## Downstream phases (unchanged from Phase 1 plan)
+
+3. **LanceDB index** — `index/` engine, inline sync on record write, `:search`/similarity; delete `search_engine/`.
+4. **Annotation v2** — `question` (column binding), `suggestion`, `response`.
+5. **Queue** — `queue`/`queue_item`/`queue_assignment`, `GET /queues/{id}/next`.
+6. **Migrator + retirement** — v1→v2 per-dataset migrator; delete v1 tables/handlers.