✨: add CanArrayX protocols #32

nstarman · 2025-06-22T19:01:00Z

No description provided.

nstarman · 2025-06-23T15:33:29Z

Ok This PR is doing too much. Let me pair it down to just a few Protocols and do the rest as a series of followups.

nstarman · 2025-06-23T22:00:38Z

Ping @NeilGirdhar, given related discussions.

src/array_api_typing/_array.py

nstarman · 2025-06-23T22:02:44Z

src/array_api_typing/_array.py

+        ...
+
+
+class CanArrayAdd(Protocol):


I was thinking about parametrizing by dtype. Self, other, output. Bit of a mess. Maybe tackle parametrizing as a followup?

nstarman · 2025-06-23T22:37:07Z

Should all the Protocols inherit from HasArrayNamespace?
Also should it be rename to CanArrayNamespace ?

NeilGirdhar · 2025-06-23T23:21:03Z

Should all the Protocols inherit from HasArrayNamespace?
Also should it be rename to CanArrayNamespace ?

I don't know what Joren will say, but I would guess no and no? (I think you got it right in this PR?)

Also, I'm guessing you're aware that int | float is float, and you're intentionally specifying both?

nstarman · 2025-06-24T00:06:11Z

don't know what Joren will say, but I would guess no

My thought was for building stuff like

class Positive(Protocol):
    def __call__(self, array: CanArrayPos, /) -> CanArrayPos: ...

is wrong.

It should be something like

class Positive(Protocol):
    def __call__(self, array: HasArrayNamespace, /) -> HasArrayNamespace: ...

But I think we want

class Positive(Protocol):
    def __call__(self, array: CanArrayPos, /) -> HasArrayNamespace: ...

Which I think works best if it's

class CanArrayPos(HasArrayNamespace, Protocol): ...

Also, I'm guessing you're aware that int | float is float, and you're intentionally specifying both?

Yes. :).

NeilGirdhar · 2025-06-24T01:56:05Z

I see, you're kind of using it as a poor man's intersection?

Also, I'm guessing you're aware that int | float is float, and you're intentionally specifying both?

Yes. :).

Okay, is that because you're going to generate some documentation from these annotations? Or you find it less confusing?

Also, are you going to add complex to the union?

nstarman · 2025-06-24T02:30:43Z

Okay, is that because you're going to generate some documentation from these annotations? Or you find it less confusing?

It's for 2 reasons: the array api does it in their docs and because I think the Python numerical tower is a mess and since ints and floats aren't subclasses of each other, it makes little sense for them to be interchangeable at the static type level. 😤😆

Also, are you going to add complex to the union?

Worth discussing. The array api does not.

NeilGirdhar · 2025-06-24T02:55:13Z

It's for 2 reasons: the array api does it in their docs

The docs are that way to help beginners who might be confused. (At least that was the argument that was presented.) But you aren't expecting beginners to read your code, are you?

And, you aren't using this repo to build docs?

The downside of populating the unions unnecessarily is overcomplicated type errors. So from a user standpoint, I think this is worse.

From a developer standpoint, it's a matter of taste. Personally, I think more succinct is easier to understand.

because I think the Python numerical tower is a mess and since ints and floats aren't subclasses of each other, it makes little sense for them to be interchangeable at the static type level.

As much as you might like to turn back time and change the typing decisions that were made, the fact is that the static type int is a subclass of float as far as type checkers are concerned, and that will not change for the foreseeable future.

I think I understand what you're doing and why. I spent years writing if x != 0 for a similar reason. But I think this is a fact that you just have to accept even if you dislike it.

Worth discussing. The array api does not.

Does it not?

array.__add__(other: int | float | complex | array, /) → array

Have I misunderstood the documentation?

nstarman · 2025-06-24T03:20:36Z

Ah. We're building towards v2021 first.
A release branch for every major version.
The versions have almost been entirely additive, so it's not too onerous.
This also makes backporting easier.

jorenham

It might be easier to use optype for this, as it already provides single-method generic protocols for each of the special dunders:

https://github.com/jorenham/optype/blob/master/optype/_core/_can.py

There's even documentation: https://github.com/jorenham/optype#binary-operations

And of course it's tested and thoroughly type-checked and stuff

nstarman · 2025-06-24T17:22:46Z

Sounds good to me...
It's good to have in-house expertise.

nstarman · 2025-07-01T20:51:13Z

@jorenham is this prep for using optype?

jorenham · 2025-07-01T23:38:19Z

@jorenham is this prep for using optype?

Yea, pretty much.

src/array_api_typing/_array.py

Signed-off-by: Nathaniel Starkman <[email protected]>

…istency

Support unary minus operator. Signed-off-by: Nathaniel Starkman <[email protected]>

jorenham · 2025-07-09T21:58:04Z

@jorenham do you want to switch some of these to be optype objects, or does the Self and docstring mean we should go ahead with rolling our own Protocols ?

I've thought about this, but I'm not sure what the best approach is. I considered four approaches:

Use optype but monkeypatch the __doc__ of the protocols. The downside is that we'd pollute these protocols, which might be annoying for users that use optype for other things as well.
Bundle optype as git submodule, so that we can monkeypatch __doc__ without polluting the "actual" optype protocols.
We write our own protocols (copy-pasting those of optype). This won't pollute optype, but we'd have to do quite a lot of work to write- test- and maintain them.
Use optype, but ignore the docstrings. If we later want docstrings after all, then we can revisit the 3 options above.

Now that I've written these down, I think I feel most for option 4. As far as I'm concerned, docstrings are a "should-have", not a "must-have" (MoSCow jargon). By postponing worrying about docstrings, we can focus on building the actual functionality first. This feels like the most agile approach to me.

Thoughts?

nstarman · 2025-07-10T01:13:46Z

For magic dunder methods I agree we can start with 4.

What about doing

@modify_docstring("", __float__="")
class CanFloat(opt.CanFloat): ...

@modify_docstring("", __int__="")
class CanInt(opt.CanInt[R]): ...

jorenham · 2025-07-10T13:00:17Z

For magic dunder methods I agree we can start with 4.

What about doing
@modify_docstring("", __float__="")
class CanFloat(opt.CanFloat): ...

@modify_docstring("", __int__="")
class CanInt(opt.CanInt[R]): ...

I like that!

nstarman · 2025-07-11T15:39:13Z

We still have the problem of Self in the type annotations. `

E.g.

class CanArrayAdd(Protocol):
    def __add__(self, other: Self | int | float, /) -> Self: ...

which isn't compatible with optype.CanAdd .

Edit: the closest I can get is

opt.CanAdd["HasArrayNamespace[NS_contra] | int | float", "Array[NS_contra]"],

Doing

opt.CanAdd["Array[NS_X] | int | float", "Array[NS_X]"], doesn't seem to work.

jorenham · 2025-07-11T15:45:30Z

We still have the problem of Self in the type annotations. `

E.g.
class CanArrayAdd(Protocol):
    def __add__(self, other: Self | int | float, /) -> Self: ...
which isn't compatible with optype.CanAdd .

I'll add them to optype then

update

https://github.com/jorenham/optype/releases/tag/v0.12.0

nstarman · 2025-07-11T15:49:55Z

I'll add them to optype then

Awesome, so then it'll be...

CanAddSelf[T, R=Self] = CanAdd[Self | T, Self | R]

so we can do CanAddSelf[int | float] ?

jorenham · 2025-07-11T15:50:07Z

Something like this, @nstarman?

class CanAddSelf(Protocol[_T_contra]):
    def __add__(self, rhs: Self | _T_contra, /) -> Self: ...

nstarman · 2025-07-11T15:51:03Z

Great! I guess the return type probably isn't necessary.

jorenham · 2025-07-11T15:52:26Z

Great! I guess the return type probably isn't necessary.

Yea indeed. And if anyone needs it after all, then we can always add it as optional type parameter later on.

jorenham · 2025-07-11T15:53:53Z

E.g.

class CanArrayAdd(Protocol):
    def __add__(self, other: Self | int | float, /) -> Self: ...

BTW, this wouldn't work in case of boolean arrays.

nstarman · 2025-07-11T15:57:01Z

E.g.

class CanArrayAdd(Protocol):
    def __add__(self, other: Self | int | float, /) -> Self: ...

BTW, this wouldn't work in case of boolean arrays.

Yeah. I noticed that. It's in the signature of the Array API, but without a way to detect boolean dtypes, how else do we write this statically?

Also we need CanRAddSelf, etc.

nstarman · 2025-07-11T16:01:55Z

I don't think we need to do single-method Protocols now that we're using optype

@docstring_setter(
    __pos__ = """...""",
    ...
)
class Array(
    HasArrayNamespace[NS_co],
    opt.CanPosSelf,
    opt.CanNegSelf,
    opt.CanAddSelf[int | float],
    opt.CanIAddSelf[int | float],
    opt.CanRAddSelf[int | float],
    opt.CanSubSelf[int | float],
    opt.CanISubSelf[int | float],
    opt.CanRSubSelf[int | float],
    opt.CanMulSelf[int | float],
    opt.CanIMulSelf[int | float],
    opt.CanRMulSelf[int | float],
    opt.CanTrueDivSelf[int | float],
    opt.CanRTrueDivSelf[int | float],
    opt.CanFloorDivSelf[int | float],
    opt.CanIFloorDivSelf[int | float],
    opt.CanRFloorDivSelf[int | float],
    opt.CanModSelf[int | float],
    opt.CanIModSelf[int | float],
    opt.CanRModSelf[int | float],
    opt.CanPowSelf[int | float],
    opt.CanIPowSelf[int | float],
    opt.CanRPowSelf[int | float],
    Protocol,
):

jorenham · 2025-07-11T16:02:58Z

It's in the signature of the Array API

Then that should be changed 🤷🏻‍♂️

how else do we write this statically?

I'd make it generic:

class CanAddSelf(Protocol[_T_contra]):
    def __add__(self, rhs: Self | _T_contra, /) -> Self: ...

😏

Also we need CanRAddSelf, etc.

Yea I'll add *Self variants or all binops 👌🏻.

But I'm thinking of leaving out the Self as input for the reflected ops, so it'll be

def __radd__(self, rhs: _T_contra, /) -> Self: ..

because it shouldn't be needed, ...right?

jorenham · 2025-07-11T16:05:12Z

I don't think we need to do single-method Protocols now that we're using optype

We'll still need some for the non-python dunders like __array_namespace_info__ and attributes like dtype

nstarman · 2025-07-11T16:06:26Z

We'll still need some for the non-python dunders like array_namespace_info and attributes like dtype

Yes, ones that don't have a natural fit in optype.

jorenham · 2025-07-11T16:07:41Z

We don't care about __divmod__, right?

nstarman · 2025-07-11T16:09:01Z

how else do we write this statically?
I'd make it generic:

That's a good idea. We can define a generic Array[InputT] and then also provide some common-sense defaults, like (names TBD)

Array[InputT]
NumericArray = Array[int | float]
BoolArray = Array[bool]

Signed-off-by: Nathaniel Starkman <[email protected]>

nstarman · 2025-07-11T16:26:11Z

Pushing a commit that won't work since it references non-existent optype classes, but does most of the things we'll need when those exist.

nstarman · 2025-07-14T16:49:25Z

src/array_api_typing/_array.py

+
+
+@docstring_setter(
+    __pos__="""Evaluates `+self_i` for each element of an array instance.


Should we push the docstrings to a JSON that gets read in? It would make this

@docstring_setter(**docstrings_json)

I opted for a toml file since it has nicely formatted multiline raw strings.

…strings from TOML file Signed-off-by: Nathaniel Starkman <[email protected]>

jorenham · 2025-07-16T16:49:42Z

I just released optype 0.12.0 :)

nstarman · 2025-07-16T18:05:17Z

@jorenham. It works!

… definitions

jorenham · 2025-07-16T19:26:26Z

src/array_api_typing/_array.py

+    op.CanAddSame[T_contra],
+    op.CanIAddSelf[T_contra],
+    op.CanRAddSelf[T_contra],
+    op.CanSubSame[T_contra],


This won't accept boolean numpy arrays:

>>> import numpy as np >>> np.array(True) - np.array(False) Traceback (most recent call last): File "<python-input-2>", line 1, in <module> np.array(True) - np.array(False) ~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~ TypeError: numpy boolean subtract, the `-` operator, is not supported, use the bitwise_xor, the `^` operator, or the logical_xor function instead.

Hm. Suggestions?

jorenham · 2025-07-16T19:33:24Z

src/array_api_typing/_array.py

+    op.CanPosSelf,
+    op.CanNegSelf,
+    op.CanAddSame[T_contra],
+    op.CanIAddSelf[T_contra],


+= also works if you just have an __add__ and no __iadd__:

>>> class Thingy: ... def __add__(self, rhs, /): ... return self if isinstance(rhs, Thingy) else NotImplemented ... >>> a = Thingy() >>> a + a <__main__.Thingy object at 0x7f9896498830> >>> a += a >>> a <__main__.Thingy object at 0x7f9896498830>

We already require Can{binop}Same, so can we remove CanI{binop}Self?

How do you read https://data-apis.org/array-api/2021.12/API_specification/array_object.html#in-place-operators.
In my reading I agree that __iadd__ isn't strictly necessary since x += 2 will fall back to x = x + 2, making a new object.
So yes?

The "May be implemented" makes it sound like it optional to me

But there's the mutability stuff as well; https://data-apis.org/array-api/2021.12/design_topics/copies_views_and_mutation.html#copyview-mutability

So I guess it depends on whether we want xpt.Array to be a flexible utility, or as a array api compliance check for static typing.

I'd opt for removing them. It's really only good for isinstance checks and not type-flow through a program.

jorenham · 2025-07-16T19:36:54Z

src/array_api_typing/_array.py

+    op.CanMulSame[T_contra],
+    op.CanIMulSelf[T_contra],
+    op.CanRMulSelf[T_contra],
+    op.CanTruedivSame[T_contra],


CanTruedivSame requires __truediv__: (Self, Self) -> Self. In NumPy, that only holds for np.inexact dtypes (floating and complex). So this would reject integer and boolean arrays:

>>> import numpy as np >>> np.array([1]) / np.array([1]) array([1.]) >>> np.array([True]) / np.array([True]) array([1.])

So we need to write a more flexible Protocol for Truediv?

It's just that we can't have it return Self, but something like xpt.Array would work I suppose. Something op.CanTruediv[int, xpt.CanArray] could work, but there's currently no optype protocol for __truediv__: (Self, Self) -> T.

If you think we'll need that, I wouldn't mind adding such protocols to optype. I'm not sure what to call them though 🤔

It's a real shame that Self and TypeVar don't play so nicely together.

Yea, and there's no need for that restriction either: https://discuss.python.org/t/self-as-typevar-default/909

jorenham · 2025-07-16T19:38:02Z

src/array_api_typing/_array.py

+    op.CanTruedivSame[T_contra],
+    op.CanITruedivSelf[T_contra],
+    op.CanRTruedivSelf[T_contra],
+    op.CanFloordivSame[T_contra],


This doesn't hold for boolean numpy arrays:

>>> import numpy as np >>> np.array([True]) // np.array([True]) array([1], dtype=int8)

jorenham · 2025-07-16T19:39:21Z

src/array_api_typing/_array.py

+    op.CanFloordivSame[T_contra],
+    op.CanIFloordivSelf[T_contra],
+    op.CanRFloordivSelf[T_contra],
+    op.CanModSame[T_contra],


mod and floordiv have identical signatures in numpy, so this won't work for boolean arrays:

>>> import numpy as np >>> np.array([True]) % np.array([True]) array([0], dtype=int8)

jorenham · 2025-07-16T19:40:32Z

src/array_api_typing/_array.py

+    op.CanModSame[T_contra],
+    op.CanIModSelf[T_contra],
+    op.CanRModSelf[T_contra],
+    op.CanPowSame[T_contra],


poor boolean arrays:

>>> np.array([True]) ** np.array([True]) array([1], dtype=int8)

jorenham · 2025-07-16T19:49:56Z

tests/integration/test_numpy1.pyi

+###
+# Ensure that `np.ndarray` instances are assignable to `xpt.Array`.
+
+arr_array: xpt.Array[Any, Any] = arr


What if you set the first typar to Never? Because that way, e.g. __add__ becomes (Self, Self | Never) -> Self which reduces to (Self, Self) -> Self.

In theory it shouldn't make a difference here. But I know that pyright has a bug where it (incorrectly) reduces Self | Any to Any in certain situations. So I wouldn't be surprised if mypy would also behave incorrectly in this case.

jorenham · 2025-07-16T19:51:45Z

tests/integration/test_numpy1.pyi

+# Ensure that `np.ndarray` instances are assignable to `xpt.Array`.
+
+arr_array: xpt.Array[Any, Any] = arr
+arr_floatarray: xpt.Array[float, Any] = arr


I'm also kinda curious if xpt.Array[float, Any] will reject boolean- and integer arrays.

jorenham · 2025-07-16T19:53:55Z

tests/integration/test_numpy2.pyi

+arr_array: xpt.Array[Any, Any] = arr
+arr_floatarray: xpt.Array[float, Any] = arr
+arr_boolarray: xpt.Array[bool, Any] = arr


these should probably stay in sync with the ones in test_numpy1.pyi

nstarman force-pushed the has_x branch 5 times, most recently from 96067a4 to a1be18e Compare June 23, 2025 19:19

nstarman marked this pull request as ready for review June 23, 2025 21:59

nstarman requested a review from jorenham June 23, 2025 21:59

nstarman changed the title ~~✨: add HasArrayX protocols~~ ✨: add CanArrayX protocols Jun 23, 2025

nstarman commented Jun 23, 2025

View reviewed changes

src/array_api_typing/_array.py Show resolved Hide resolved

nstarman commented Jun 23, 2025

View reviewed changes

jorenham reviewed Jun 24, 2025

View reviewed changes

nstarman mentioned this pull request Jul 1, 2025

feat: HasX attributes #34

Draft

jorenham reviewed Jul 1, 2025

View reviewed changes

src/array_api_typing/_array.py Outdated Show resolved Hide resolved

nstarman added 5 commits July 6, 2025 16:15

✨: move HasArrayNamespace to _array.py and update imports

3473691

✨: add Array class definition

5d56158

Signed-off-by: Nathaniel Starkman <[email protected]>

refactor: rename type variable in HasArrayNamespace protocol for cons…

c869543

…istency

✨: add CanArrayPos protocol

c1962cd

✨: add CanArrayNeg protocol

f21c055

Support unary minus operator. Signed-off-by: Nathaniel Starkman <[email protected]>

nstarman closed this Jul 11, 2025

nstarman reopened this Jul 11, 2025

🚧 transition to optype

ba8b4f5

Signed-off-by: Nathaniel Starkman <[email protected]>

nstarman commented Jul 14, 2025

View reviewed changes

This was referenced Jul 14, 2025

Can*Self binop protocols jorenham/optype#348

Merged

Can*Same binop protocols jorenham/optype#362

Merged

✨: add tomli dependency for Python version compatibility and load doc…

45cd14f

…strings from TOML file Signed-off-by: Nathaniel Starkman <[email protected]>

🔧 update optype version

855ddf2

✨: refactor test files to improve clarity and organization of NDArray…

56e4814

… definitions

jorenham reviewed Jul 16, 2025

View reviewed changes



		@docstring_setter(
		__pos__="""Evaluates `+self_i` for each element of an array instance.

✨: add CanArrayX protocols #32

Are you sure you want to change the base?

✨: add CanArrayX protocols #32

Conversation

nstarman commented Jun 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nstarman commented Jun 23, 2025

Uh oh!

nstarman commented Jun 23, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nstarman commented Jun 23, 2025

Uh oh!

NeilGirdhar commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nstarman commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NeilGirdhar commented Jun 24, 2025

Uh oh!

nstarman commented Jun 24, 2025

Uh oh!

NeilGirdhar commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nstarman commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jorenham left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nstarman commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nstarman commented Jul 1, 2025

Uh oh!

jorenham commented Jul 1, 2025

Uh oh!

Uh oh!

jorenham commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nstarman commented Jul 10, 2025

Uh oh!

jorenham commented Jul 10, 2025

Uh oh!

nstarman commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jorenham commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nstarman commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jorenham commented Jul 11, 2025

Uh oh!

nstarman commented Jul 11, 2025

Uh oh!

jorenham commented Jul 11, 2025

Uh oh!

jorenham commented Jul 11, 2025

Uh oh!

nstarman commented Jul 11, 2025

Uh oh!

nstarman commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jorenham commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jorenham commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nstarman commented Jul 11, 2025

nstarman commented Jun 22, 2025 •

edited

Loading

NeilGirdhar commented Jun 23, 2025 •

edited

Loading

nstarman commented Jun 24, 2025 •

edited

Loading

NeilGirdhar commented Jun 24, 2025 •

edited

Loading

nstarman commented Jun 24, 2025 •

edited

Loading

jorenham left a comment •

edited

Loading

nstarman commented Jun 24, 2025 •

edited

Loading

jorenham commented Jul 9, 2025 •

edited

Loading

nstarman commented Jul 11, 2025 •

edited

Loading

jorenham commented Jul 11, 2025 •

edited

Loading

nstarman commented Jul 11, 2025 •

edited

Loading

nstarman commented Jul 11, 2025 •

edited

Loading

jorenham commented Jul 11, 2025 •

edited

Loading

jorenham commented Jul 11, 2025 •

edited

Loading

nstarman commented Jul 11, 2025 •

edited

Loading

nstarman Jul 14, 2025 •

edited

Loading