chore: Fix `IntoSchema` typing, allow to pass `Sequence[tuple[str, IntoDType]] | SupportsKeysAndGetItem[str, IntoDType]` by FBruzzesi · Pull Request #3266 · narwhals-dev/narwhals

FBruzzesi · 2025-10-31T20:57:06Z

What type of PR is this? (check all applicable)

Related issues

Closes typing.IntoSchema too narrow #3257

Checklist

Code follows style guide (ruff)
Tests added
Documented the changes

If you have comments or can explain your changes, please do so below

FBruzzesi · 2025-10-31T20:58:31Z

narwhals/typing.py

-# TODO @dangotbanned: fix this?
-# Constructor allows tuples, but we don't support that *everywhere* yet
-IntoSchema: TypeAlias = "Mapping[str, dtypes.DType] | Schema"
+IntoSchema: TypeAlias = (
+    "Mapping[str, IntoDType] | Sequence[tuple[str, IntoDType]] | Schema"
+)


@dangotbanned I hope you are proud 😇

🥳

But now to ruin your day slightly ...

We got our typing for Schema.__init__ from polars, but even this is too strict.
Here's what typeshed has for dict, which we support (except **kwargs)

https://github.com/python/typeshed/blob/bf7214784877c52638844c065360d4814fae4c65/stdlib/builtins.pyi#L1158-L1187

Luckily we don't need to use any overloads, but we should have this in IntoSchema somewhere:

Iterable[tuple[str, IntoDType]] | SupportsKeysAndGetItem[str, IntoDType]

Interesting timing though, I just pushed a test where I passed a non-Mapping object to nw.Schema 😂

narwhals/tests/plan/schema_test.py

Lines 18 to 20 in 1579b32

# NOTE: Would type-check if `Schema.__init__` didn't make liskov unhappy

assert schema == nw.Schema(frozen_schema) # type: ignore[arg-type]

assert mapping == dict(frozen_schema)

Thanks @dangotbanned

Iterable[tuple[str, IntoDType]] | SupportsKeysAndGetItem[str, IntoDType]

SupportsKeysAndGetItem what the hell is this 😂 how's Mapping not enough anymore?

Regarding Iterable, I always have a hard time being comfortable with it, since it can be infinite (and lazy).

How does infinite converts to a dict?

Lazy makes it a bit more of a headache when we want to check it's content (e.g. first element is a string or a tuple), but that's alright

This is the reason why I went with Sequence, although polars Schema.__init__ signature is with the Iterable

We got our typing for Schema.init from polars, but even this is too strict.

can we resolve that later and just get the type[Dtype] / Dtype part sorted out in this one?

SupportsKeysAndGetItem what the hell is this 😂 how's Mapping not enough anymore?

Lol, it is an object that supports those methods ofc 😉

just import it in a TYPE_CHECKING block from here

https://github.com/python/typeshed/blob/be34e9201db75891a67d4d3ce5e5705ee6636f6f/stdlib/_typeshed/__init__.pyi#L161-L164

from _typeshed import SupportsKeysAndGetItem

Regarding Iterable, I always have a hard time being comfortable with it, since it can be infinite

If someone passes an infinite stream to an initializer - they need to learn the lesson that the call will never complete

(and lazy).

This is a desirable property for the caller and I think if we want to validate things, then that is our responsibility

Reducing scope

How about aligning just Schema.__init__ with dict.__init__?

We are only checking for None there, so the issues you have seem to not apply there

@dangotbanned attempt in 9ce61fe, which now fails for mypy when the dict has mix of initialized and un-initialized dtypes, which was the initial purpose of the PR 🤔

Edit: By changing Iterable into Sequence there are no issue (see 0e0d7a8). I am personally not interested in diving deeper than this.

Last to fix is the forward reference for tubular. Which I am unsure if it's our fault or them not importing something within a if TYPE_CHEKING: block

Last to fix is the forward reference for tubular. Which I am unsure if it's our fault or them not importing something within a if TYPE_CHEKING: block

tubular uses a runtime type checker (beartype).
I ran into a similar problem the last time I changed the definition of Schema:

uncap narwhals version azukds/tubular#455

chore: Remove 3.8 compat from Schema #2894

Regarding (https://github.com/narwhals-dev/narwhals/pull/3266/files#r2483418646), (https://github.com/narwhals-dev/narwhals/pull/3266/files#r2483400506)

This is my bad, but we should keep that part and my original TODO as a todo

narwhals/narwhals/typing.py

Lines 307 to 308 in d7bb050

# TODO @dangotbanned: fix this?

# Constructor allows tuples, but we don't support that *everywhere* yet

These things are related, but the timing of #3252 complicated both this PR and my TODO 😞

FBruzzesi · 2025-10-31T20:59:08Z

tests/frame/schema_test.py

Randomly removing some initialization to test the typechecking

FBruzzesi · 2025-10-31T20:59:24Z

narwhals/typing.py

+IntoNullableSchema: TypeAlias = (
+    "Mapping[str, IntoDType | None] | Sequence[tuple[str, IntoDType | None]]"
+)
+"""Schema specification with possible None values."""


TODO: Please let's come up with a better description

FBruzzesi · 2025-10-31T21:02:56Z

narwhals/_utils.py

    return (value,) * n_match if isinstance(value, bool) else tuple(value)
+
+
+class NullableSchema(OrderedDict[str, "IntoDType | None"]):


Reason for this class is mostly two folded:

Use as a utility to convert a Sequence[tuple[str, DType {|None}]] into a mapping. Hence make it easy to use the same API (key, dtype in obj.items()).

Easily set a flag to know if any value passed is None

The name Nullable* is making me think this is related to #3176 (comment) again 🫣

I'm not opposed to having classes to make our internal API cleaner btw 👍

The name Nullable* is making me think this is related to #3176 (comment) again 🫣

Not sure I see how it's related. Passing None seems more like to be a free card, "live and let live" kind of behavior.

I'm not opposed to having classes to make our internal API cleaner btw 👍

Any preference in having it prefixed by _?

Any preference in having it prefixed by _?

It is already in _utils, so no need for a prefix 🙂

The current _ names are an artifact from when we had everything in utils.py

Not sure I see how it's related. Passing None seems more like to be a free card, "live and let live" kind of behavior.

I'll try to give a more complete explanation later, but I was hinting at Nullable, Null, None being a bit overloaded.
Linking to (#3176 (comment)) was supposed to show that I made this mistake already 😂

Not sure I see how it's related. Passing None seems more like to be a free card, "live and let live" kind of behavior.

I'll try to give a more complete explanation later, but I was hinting at Nullable, Null, None being a bit overloaded.

I guess now still counts as later 😂

Somebody wrote docs about the term nullable, and this usage is not to do with that

https://fbruzzesi.github.io/anyschema/user-guide/metadata/#nullable

narwhals/dataframe.py

FBruzzesi · 2025-11-01T08:21:02Z

Regarding CI failures: the only one related to this PR seem to be tubular 👀

MarcoGorelli · 2025-11-02T11:43:03Z

narwhals/typing.py

    "IntoFrameT",
    "IntoLazyFrame",
    "IntoLazyFrameT",
+    "IntoNullableSchema",


Are we sure about making this a public type?

Is it possible to only focus on Dtype -> intodtype changes here without introducing new public types?

Ok I honestly think there might be something wrong in the PR to begin with. If I refocus on the basic changes to fix #3257, then passing a dictionary with mixed initialized and un-initialized dtypes does still complain (pyright is ok with it, mypy is not)

FBruzzesi · 2026-01-03T12:00:20Z

I will keep this PR open for a bit longer - if anyone is interested to bringing to the finish line, I am happy to hand it over 🙈

dangotbanned

Thanks for the PR (and reminding me about it 🙏) @FBruzzesi

I think there are parts of this that are ready to go!

But the combination of (#3262 and #3252) + my suggestion took the PR in a direction that you couldn't have anticipated when opening #3262

It seems like I made a suggestion on that PR that may have been lost (#3262 (comment)):

#3262 (comment)

@FBruzzesi I think if you were looking to clean things up following #3252, then #2486 may be what we're missing?

This issue is still open and we've both mentioned it recently, maybe there's something to that?

#2486

dangotbanned · 2026-01-07T11:43:30Z

narwhals/functions.py

+    return Version.MAIN.dataframe.from_dict(data, schema, backend=backend)



I like this change and think we can split those like it into a PR that makes a dent into #2786

dangotbanned · 2026-01-07T11:44:59Z

narwhals/functions.py

-        f"    nw.from_numpy(arr, backend='pyarrow').lazy('{implementation}')"
-    )
-    raise ValueError(msg)
+    return Version.MAIN.dataframe.from_numpy(data, schema, backend=backend)


(Same comment as from_dict)

dangotbanned · 2026-01-07T12:21:37Z

narwhals/typing.py

-# TODO @dangotbanned: fix this?
-# Constructor allows tuples, but we don't support that *everywhere* yet
-IntoSchema: TypeAlias = "Mapping[str, dtypes.DType] | Schema"
+IntoSchema: TypeAlias = (
+    "Mapping[str, IntoDType] | Sequence[tuple[str, IntoDType]] | Schema"
+)


Last to fix is the forward reference for tubular. Which I am unsure if it's our fault or them not importing something within a if TYPE_CHEKING: block

tubular uses a runtime type checker (beartype).
I ran into a similar problem the last time I changed the definition of Schema:

uncap narwhals version azukds/tubular#455

chore: Remove 3.8 compat from Schema #2894

dangotbanned · 2026-01-07T12:52:10Z

narwhals/schema.py

        return list(self.keys())

-    def dtypes(self) -> list[DType]:
+    def dtypes(self) -> list[IntoDType]:


dangotbanned · 2026-01-07T13:02:44Z

narwhals/typing.py

-# TODO @dangotbanned: fix this?
-# Constructor allows tuples, but we don't support that *everywhere* yet
-IntoSchema: TypeAlias = "Mapping[str, dtypes.DType] | Schema"
+IntoSchema: TypeAlias = (
+    "Mapping[str, IntoDType] | Sequence[tuple[str, IntoDType]] | Schema"
+)


Regarding (https://github.com/narwhals-dev/narwhals/pull/3266/files#r2483418646), (https://github.com/narwhals-dev/narwhals/pull/3266/files#r2483400506)

This is my bad, but we should keep that part and my original TODO as a todo

narwhals/narwhals/typing.py

Lines 307 to 308 in d7bb050

# TODO @dangotbanned: fix this?

# Constructor allows tuples, but we don't support that *everywhere* yet

These things are related, but the timing of #3252 complicated both this PR and my TODO 😞

dangotbanned · 2026-01-07T13:30:07Z

narwhals/_utils.py

    return (value,) * n_match if isinstance(value, bool) else tuple(value)
+
+
+class NullableSchema(OrderedDict[str, "IntoDType | None"]):


Not sure I see how it's related. Passing None seems more like to be a free card, "live and let live" kind of behavior.

I'll try to give a more complete explanation later, but I was hinting at Nullable, Null, None being a bit overloaded.

I guess now still counts as later 😂

Somebody wrote docs about the term nullable, and this usage is not to do with that

https://fbruzzesi.github.io/anyschema/user-guide/metadata/#nullable

chore: Fix IntoSchema typing, allow Sequence[tuple[str, IntoDType]

324a8dc

FBruzzesi added enhancement New feature or request typing labels Oct 31, 2025

FBruzzesi mentioned this pull request Oct 31, 2025

chore: IntoSchema overhaul #3262

Closed

10 tasks

Merge branch 'main' into chore/into-schema-overhaul-redo

983db99

FBruzzesi changed the title ~~chore: Fix IntoSchema typing, allow Sequence[tuple[str, IntoDType]~~ chore: Fix IntoSchema typing, allow to pass Sequence[tuple[str, IntoDType]] Oct 31, 2025

FBruzzesi commented Oct 31, 2025

View reviewed changes

missed one annotation

ab0bfab

FBruzzesi commented Oct 31, 2025

View reviewed changes

narwhals/dataframe.py Outdated Show resolved Hide resolved

FBruzzesi and others added 2 commits November 1, 2025 09:20

pragma no cover exception

88136c3

Merge branch 'main' into chore/into-schema-overhaul-redo

12bd5e9

FBruzzesi added 4 commits November 1, 2025 09:22

tricking ruff for better formatting

476dc69

is exporting enough for forward reference?

c90c110

delay NullableSchema import

38bac0e

Use SupportsKeysAndGetItem and Iterable

9ce61fe

FBruzzesi changed the title ~~chore: Fix IntoSchema typing, allow to pass Sequence[tuple[str, IntoDType]]~~ chore: Fix IntoSchema typing, allow to pass Iterable[tuple[str, IntoDType]] | SupportsKeysAndGetItem[str, IntoDType] Nov 1, 2025

one for two?

0e0d7a8

FBruzzesi changed the title ~~chore: Fix IntoSchema typing, allow to pass Iterable[tuple[str, IntoDType]] | SupportsKeysAndGetItem[str, IntoDType]~~ chore: Fix IntoSchema typing, allow to pass Sequence[tuple[str, IntoDType]] | SupportsKeysAndGetItem[str, IntoDType] Nov 1, 2025

merge main

15e08d7

MarcoGorelli reviewed Nov 2, 2025

View reviewed changes

lower diff, do not export SchemaDefinition

634bd33

FBruzzesi added the help wanted Extra attention is needed label Jan 7, 2026

dangotbanned reviewed Jan 7, 2026

View reviewed changes

	# NOTE: Would type-check if `Schema.__init__` didn't make liskov unhappy
	assert schema == nw.Schema(frozen_schema) # type: ignore[arg-type]
	assert mapping == dict(frozen_schema)

	# TODO @dangotbanned: fix this?
	# Constructor allows tuples, but we don't support that everywhere yet

		return (value,) * n_match if isinstance(value, bool) else tuple(value)


		class NullableSchema(OrderedDict[str, "IntoDType \| None"]):

		return Version.MAIN.dataframe.from_dict(data, schema, backend=backend)

Conversation

FBruzzesi commented Oct 31, 2025

What type of PR is this? (check all applicable)

Related issues

Checklist

If you have comments or can explain your changes, please do so below

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dangotbanned Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Reducing scope

Uh oh!

FBruzzesi Nov 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FBruzzesi Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FBruzzesi Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FBruzzesi Nov 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

FBruzzesi commented Nov 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FBruzzesi Nov 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FBruzzesi commented Jan 3, 2026

Uh oh!

dangotbanned left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

dangotbanned Oct 31, 2025 •

edited

Loading

FBruzzesi Nov 1, 2025 •

edited

Loading

FBruzzesi Oct 31, 2025 •

edited

Loading

FBruzzesi Oct 31, 2025 •

edited

Loading

FBruzzesi Nov 1, 2025 •

edited

Loading

FBruzzesi Nov 2, 2025 •

edited

Loading

dangotbanned left a comment •

edited

Loading