fix(core,model-profiles): add missing ModelProfile fields, warn on schema drift#36129
Open
Mason Daugherty (mdrxy) wants to merge 5 commits intomasterfrom
Open
fix(core,model-profiles): add missing ModelProfile fields, warn on schema drift#36129Mason Daugherty (mdrxy) wants to merge 5 commits intomasterfrom
ModelProfile fields, warn on schema drift#36129Mason Daugherty (mdrxy) wants to merge 5 commits intomasterfrom
Conversation
Merging this PR will degrade performance by 14.6%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | test_import_time[ChatPromptTemplate] |
523.7 ms | 613.2 ms | -14.6% |
| ❌ | WallTime | test_import_time[Runnable] |
404.7 ms | 459.9 ms | -11.99% |
| ❌ | WallTime | test_import_time[BaseChatModel] |
445.9 ms | 502.4 ms | -11.24% |
| ❌ | WallTime | test_import_time[RunnableLambda] |
407.3 ms | 467.8 ms | -12.93% |
| ❌ | WallTime | test_import_time[InMemoryVectorStore] |
508.7 ms | 583.5 ms | -12.83% |
Comparing mdrxy/model-profiles/schema (a1be2d5) with master (7d05cfb)
Footnotes
-
27 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Wrap `self.profile` access in `_set_model_profile` and `_check_profile_keys` with try/except AttributeError so subclasses that override `__getattribute__` to raise on "profile" don't crash during model construction.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR #35788 added 7 new fields to the
langchain-profilesCLI output (name,status,release_date,last_updated,open_weights,attachment,temperature) but didn't updateModelProfileinlangchain-core. Partner packages likelangchain-awsthat setextra="forbid"on their Pydantic models hitextra_forbiddenvalidation errors when Pydantic encountered undeclared TypedDict keys at construction time. This adds the missing fields, makesModelProfileforward-compatible, provides a base-class hook so partners can stop duplicating model-profile validator boilerplate, and adds runtime + CI-time warnings for schema drift.Changes
__pydantic_config__ = ConfigDict(extra="allow")toModelProfileso unknown profile keys pass Pydantic validation even on models withextra="forbid"— forward-compatibility for when the CLI schema evolves ahead of coreModelProfile:name,status,release_date,last_updated,open_weights(metadata) andattachment,temperature(capabilities)_warn_unknown_profile_keys()inmodel_profile.py— emits aUserWarningwhen a profile dict contains keys not inModelProfile, suggesting a core upgrade. Wrapped in a bareexceptso introspection failures never crash model constructionBaseChatModel._resolve_model_profile()hook that returnsNoneby default. Partners can override this single method instead of redefining the full_set_model_profilevalidator — the base validator calls it automaticallyBaseChatModel._check_profile_keysas a separatemodel_validatorthat calls_warn_unknown_profile_keys. Uses a distinct method name so partner overrides of_set_model_profiledon't inadvertently suppress the check_warn_undeclared_profile_keys()to thelangchain-profilesCLI (cli.py), called after merging augmentations inrefresh()— warns at profile-generation time (not just runtime) when emitted keys aren't declared inModelProfile. Gracefully skips iflangchain-coreisn't installedtest_model_data_to_profile_keys_subset_of_model_profilein model-profiles — feeds a fully-populated model dict to_model_data_to_profile()and asserts every emitted key exists inModelProfile.__annotations__. CI fails before any release if someone adds a CLI field without updating the TypedDictTesting
test_model_profile.py(new): coversextra="allow"behavior through Pydantic validation withextra="forbid"parent models,_warn_unknown_profile_keyswarning/silence/crash-safety paths, and parametrized field existence checks for all 25 declared fieldstest_base.py: tests_check_profile_keysfires even when a partner subclass overrides_set_model_profile, verifies unknown keys trigger warnings while valid keys stay silenttest_cli.py: guard test validates CLI output keys are a subset ofModelProfilefields; CLI-level_warn_undeclared_profile_keyscovered transitively through existingrefresh()integration tests