Using Self with __init_subclass__ uses the base class as Self rather than the class being defined

[chrisbouchard]

Bug Report

I wanted to define two type hierarchies: model types, and handlers that are contravariant over models, where models can specify their handlers as class keyword arguments. I've included a simplified example in the To Reproduce section below.

The problem seems to be that, when __init_subclass__ runs for Foo, it's treating Self as if it were Model rather than Foo, which fails because T is contravariant and FooHandler is not a subtype of Handler[Model].

To Reproduce

Playground Link

from __future__ import annotations
from typing import Any, ClassVar, Self, TypeVar, Generic


T = TypeVar("T", bound="Model", contravariant=True)

class Handler(Generic[T]):
    pass

class Model:
    handler: ClassVar[type[Handler[Self]]]
    
    def __init_subclass__(cls, *, handler: type[Handler[Self]]) -> None:
        super().__init_subclass__()
        cls.handler = handler


class FooHandler(Handler[Foo]):
    pass

class Foo(Model, handler=FooHandler):
    pass

Expected Behavior

I expected mypy to treat Self as Foo when running Model.__init_subclass__(Foo, ...).

Actual Behavior

main.py:15: error: Incompatible types in assignment (expression has type "type[Handler[Self]]", variable has type "type[Handler[Model]]")  [assignment]
main.py:21: error: Argument "handler" to "__init_subclass__" of "Model" has incompatible type "type[FooHandler]"; expected "type[Handler[Model]]"  [arg-type]
Found 2 errors in 1 file (checked 1 source file)

Your Environment

• Mypy version used: 1.4.1 and master branch (both from playground)
• Python version used: 3.11

[SanjayManjunath]

Hey @chrisbouchard, is this what you're trying to do? https://mypy-play.net/?mypy=latest&python=3.11&gist=68363bd1f5ff97cb1d60d6499567f0f2

[chrisbouchard]

Hi @SanjayManjunath,

Thank you, but that's not quite the same thing I'm trying to do.

Contravariance

For one, it's important that T be contravariant, not covariant. I didn't include it since it's not relevant to the bug, but Handler[T] would actually have a method like

class Handler(Generic[T]):
    def handle_model(self, model: T) -> None:
        ...

I would like that FooHandler <: Handler[X] for any subtype X <: Foo, but not for any other type X <: Model. In other words, this example should (and does) pass type checking:

# Handler and Model as in the suggestion...

class FooHandler(Handler[Foo]):
    pass

# Same as above...
class Foo(Model, handler=FooHandler):
    pass

# This should also type-check because `SubFoo`s are still `Foo`s.
class SubFoo(Foo, handler=FooHandler):
    pass

but this example should fail:

Playground Link

# Handler and Model as in the suggestion...

class BarHandler(Handler[Bar]):
    pass

# An unrelated model type...
class Bar(Model, handler=BarHandler):
    pass

# This should fail because `BarHandler` can't handle `Foo`s -- but it passes.
class Foo(Model, handler=BarHandler):
    pass

This later case still passes type-checking with the suggested definitions of Model and Handler because the type of handler is not related to the Self type.

Self for Class Attribute

Also, in the suggestion you've replace the Self with Any in the type hint for Model.handler, but I don't think there's anything wrong with using Self there. For example, if we take out the __init_subclass__ and class keyword arguments for a moment, this version still using Self does pass type-checking.

Playground Link

from __future__ import annotations
from typing import ClassVar, Self, TypeVar, Generic


T = TypeVar("T", bound="Model", contravariant=True)

class Handler(Generic[T]):
    pass

class Model:
    handler: ClassVar[type[Handler[Self]]]


class FooHandler(Handler[Foo]):
    pass

class Foo(Model):
    pass

Foo.handler = FooHandler

(And this Playground fails as expected, just to show that it's not implicitly falling back to Self == Model or Self == Any or something.)

So the problem isn't with having Self in a ClassVar, but specifically with Self while type-checking the __init_subclass__ method.

[chrisbouchard]

I'll also mention, in case it's relevant: The other reason I would like Handler[T] to be contravariant is that in the actual implementation I would like the handler parameter to Model.__init_subclass__ to be optional, so that a class could omit it to inherit their parent type's handler. I left that out of the example because it was also not relevant to the bug.

class UnimplementedHandler(Handler[Model]):
    def handle_model(self, model: T) -> None:
        raise NotImplementedException

class Model:
    handler: ClassVar[type[Handler[Self]]] = UnimplementedHandler
    
    def __init_subclass__(cls, *, handler: Optional[type[Handler[Self]]] = None) -> None:
        super().__init_subclass__()
        if handler:
            cls.handler = handler

For that to work with Self, the handler attribute needs to be an instance of Handler[X] for all subtypes X of the type being defined—which is exactly contravariance.

[SanjayManjunath]

Thank you for the detailed description of the issue Chris. I was seeing this in a different way until now. Please let me know if you figure this out. I'm curious to see the fix.