Skip to content

fix(core,model-profiles): add missing ModelProfile fields, warn on schema drift#36129

Open
Mason Daugherty (mdrxy) wants to merge 5 commits intomasterfrom
mdrxy/model-profiles/schema
Open

fix(core,model-profiles): add missing ModelProfile fields, warn on schema drift#36129
Mason Daugherty (mdrxy) wants to merge 5 commits intomasterfrom
mdrxy/model-profiles/schema

Conversation

@mdrxy
Copy link
Member

PR #35788 added 7 new fields to the langchain-profiles CLI output (name, status, release_date, last_updated, open_weights, attachment, temperature) but didn't update ModelProfile in langchain-core. Partner packages like langchain-aws that set extra="forbid" on their Pydantic models hit extra_forbidden validation errors when Pydantic encountered undeclared TypedDict keys at construction time. This adds the missing fields, makes ModelProfile forward-compatible, provides a base-class hook so partners can stop duplicating model-profile validator boilerplate, and adds runtime + CI-time warnings for schema drift.

Changes

  • Add __pydantic_config__ = ConfigDict(extra="allow") to ModelProfile so unknown profile keys pass Pydantic validation even on models with extra="forbid" — forward-compatibility for when the CLI schema evolves ahead of core
  • Declare the 7 missing fields on ModelProfile: name, status, release_date, last_updated, open_weights (metadata) and attachment, temperature (capabilities)
  • Add _warn_unknown_profile_keys() in model_profile.py — emits a UserWarning when a profile dict contains keys not in ModelProfile, suggesting a core upgrade. Wrapped in a bare except so introspection failures never crash model construction
  • Add BaseChatModel._resolve_model_profile() hook that returns None by default. Partners can override this single method instead of redefining the full _set_model_profile validator — the base validator calls it automatically
  • Add BaseChatModel._check_profile_keys as a separate model_validator that calls _warn_unknown_profile_keys. Uses a distinct method name so partner overrides of _set_model_profile don't inadvertently suppress the check
  • Add _warn_undeclared_profile_keys() to the langchain-profiles CLI (cli.py), called after merging augmentations in refresh() — warns at profile-generation time (not just runtime) when emitted keys aren't declared in ModelProfile. Gracefully skips if langchain-core isn't installed
  • Add guard test test_model_data_to_profile_keys_subset_of_model_profile in model-profiles — feeds a fully-populated model dict to _model_data_to_profile() and asserts every emitted key exists in ModelProfile.__annotations__. CI fails before any release if someone adds a CLI field without updating the TypedDict

Testing

  • test_model_profile.py (new): covers extra="allow" behavior through Pydantic validation with extra="forbid" parent models, _warn_unknown_profile_keys warning/silence/crash-safety paths, and parametrized field existence checks for all 25 declared fields
  • test_base.py: tests _check_profile_keys fires even when a partner subclass overrides _set_model_profile, verifies unknown keys trigger warnings while valid keys stay silent
  • test_cli.py: guard test validates CLI output keys are a subset of ModelProfile fields; CLI-level _warn_undeclared_profile_keys covered transitively through existing refresh() integration tests

@github-actions github-actions bot added core `langchain-core` package issues & PRs fix For PRs that implement a fix internal model-profiles `langchain-model-profiles` package issues & PRs size: M 200-499 LOC labels Mar 20, 2026
@codspeed-hq
Copy link

codspeed-hq bot commented Mar 20, 2026

Merging this PR will degrade performance by 14.6%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

❌ 5 regressed benchmarks
✅ 8 untouched benchmarks
⏩ 27 skipped benchmarks1

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
WallTime test_import_time[ChatPromptTemplate] 523.7 ms 613.2 ms -14.6%
WallTime test_import_time[Runnable] 404.7 ms 459.9 ms -11.99%
WallTime test_import_time[BaseChatModel] 445.9 ms 502.4 ms -11.24%
WallTime test_import_time[RunnableLambda] 407.3 ms 467.8 ms -12.93%
WallTime test_import_time[InMemoryVectorStore] 508.7 ms 583.5 ms -12.83%

Comparing mdrxy/model-profiles/schema (a1be2d5) with master (7d05cfb)

Open in CodSpeed

Footnotes

  1. 27 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Wrap `self.profile` access in `_set_model_profile` and
`_check_profile_keys` with try/except AttributeError so subclasses
that override `__getattribute__` to raise on "profile" don't crash
during model construction.
@github-actions github-actions bot added langchain `langchain` package issues & PRs size: L 500-999 LOC and removed size: M 200-499 LOC labels Mar 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core `langchain-core` package issues & PRs fix For PRs that implement a fix internal langchain `langchain` package issues & PRs model-profiles `langchain-model-profiles` package issues & PRs size: L 500-999 LOC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant