feat(v2): records — schema-version-pinned rows, validated bulk-upsert, references view (Phase 2)#226
Closed
JonnyTran wants to merge 14 commits into
Closed
feat(v2): records — schema-version-pinned rows, validated bulk-upsert, references view (Phase 2)#226JonnyTran wants to merge 14 commits into
JonnyTran wants to merge 14 commits into
Conversation
…mpy, PUT/defaults)
Pandera 0.32/pandas 3.0 drops Column.metadata through to_json, so the review widget is carried via a review_widgets side map (spec §13). validate_record_fields separates nulls so a None in a nullable numpy-int column no longer fails coercion.
Adds review_widgets JSONB overlay on SchemaVersion per spec §13 (Pandera to_json drops Column.metadata, so review widgets are carried out-of-band).
Includes review_widgets column; the circular schemas.current_version_id FK is added only on non-SQLite dialects (SQLite can't ALTER-add constraints; tests run on SQLite).
Adds a one-directional Schema.workspace relationship so the SubFactory pattern (and FK population) works as in DatasetFactory.
SchemaVersionCreate/Read carry review_widgets (spec §13).
publish_version flushes the new version before pointing schema.current_version_id at it (avoids the schemas<->schema_versions FK cycle and an unset version id). Threads review_widgets into the stored version and derived columns_cache (spec §13).
…r-workspace authz SchemaPolicy mirrors DatasetPolicy (owner always; admin+member for writes). Mounts a separate api_v2 sub-app at /api/v2. Adds an isolated tests/integration conftest with a v2-aware async_client (overrides get_async_db + get_s3_client on api_v2) and no OpenSearch fixture.
…VersionId) - validate_record_fields: convert numpy->native via .item()/ISO instead of lossy to_json (preserves high-precision floats + datetimes); reject omitted required cols. - put_object: propagate S3 VersionId so schema_versions.object_version_id is populated. - Add model default/uniqueness + float/datetime/missing-required tests. - Document accepted/deferred findings (publish race, 404-vs-403, commit granularity).
frame.iloc[0] collapses the validated row into a single Series whose dtype is the columns' common type; an int64 cell alongside a float64 column upcasts to float, silently degrading integer fidelity for record.fields. Extract each cell per-column (frame[col].iloc[0]) instead. Adds an Int+Float (no string column) regression test. Addresses roborev job 62.
…, references view (Phase 2) (#225) * feat(v2): add V2RecordStatus enum (pending/completed/discarded) * feat(v2): V2Record ORM model (v2_records) pinned to schema versions * feat(v2): alembic migration for v2_records table * test(v2): V2RecordFactory * feat(v2): pydantic request/response schemas for v2 records and references * feat(v2): records context with validated bulk-upsert, list, delete, reference query * feat(v2): record-scoped policy actions on SchemaPolicy * feat(v2): /api/v2 records endpoints (bulk-upsert, list, delete) with authz * feat(v2): GET /api/v2/references/{reference} cross-schema document view Also surfaces the specific 'No record IDs provided' 422 on empty delete ids (roborev job 70: the post-parse length check was unreachable). * fix(v2): match slash-containing references (DOIs) via :path converter; lint cleanups * docs: add Phase 2 (v2 records) implementation plan * docs: update schema-centric data model design with new v2 table definitions and record write process - Clarified table names and purposes for `schemas`, `schema_versions`, and `v2_records`. - Enhanced the description of the record write process, emphasizing validation and error handling. - Updated API endpoint descriptions for schemas, schema versions, and records to reflect recent changes.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #219 (Phase 1: schema registry). Implements Phase 2 of the schema-centric data model — spec §5 (record row), §6 (validated write flow), §7 (records/references endpoints):
docs/superpowers/specs/2026-06-27-schema-centric-data-model-design.md. Plan:docs/superpowers/plans/2026-07-03-v2-records.md.What's included
v2_recordstable +V2RecordORM model (class namedV2Recordbecause a secondRecordbreaks v1's string-based relationship lookups in the shared declarative registry) + Alembic migration8136bc88ee3a(upgrade/downgrade round-trip verified)V2RecordStatusenum (pending|completed|discarded) with its own PG type; v1 untouchedPOST /api/v2/schemas/{id}/records:bulk-upsert: resolves each item's pinned or current schema version, fetches each distinct version's Pandera body from the object store once per request, validates all items via Phase 1'svalidate_record_fieldsbefore any write (all-or-nothing 422 with per-item detail), then merges v1-style on(schema_id, external_id);metadata/statusare patch-like on update (documented)GET /schemas/{id}/records(offset/limit 50/1000,status/referencefilters, exact total) andDELETE /schemas/{id}/records?ids=(cap 100, 204)GET /api/v2/references/{reference:path}— cross-schema document view grouped per schema;:pathconverter so DOI references (containing slashes) resolve; workspace-scoped + authorized; unknown reference returns an empty 200 viewSchemaPolicy.{upsert_records,list_records,delete_records}: reads = workspace member, writes = owner/admin member; member/non-member negative testsOut of scope (later phases)
LanceDB search/sync (Phase 3); questions/suggestions/responses (Phase 4).
Accepted caveat
Concurrent bulk-upserts racing on
(schema_id, external_id)can surface an IntegrityError (500) — same class as v1's bulk behavior and Phase 1's accepted publish race; to be hardened when record writes move to the job queue.