Next steps for type introspection and stub generation

[Tpt]

This is an incomplete list of next steps to build proper stubs from PyO3 codebases:

• introspect all class methods (including adding a test for all magic methods)
• introspect classes associated constants (const)
• introspect modules associated constants (const)
• introspect simple enums built by #[pyclass]
• introspect complex enums built by #[pyclass]
• introspect class inheritance
• implement return type annotation
• fill PYTHON_TYPE constant on all implementations of the relevant traits (FromPyObject and IntoPyObject)
• figure out how to emit #[classattr] in the stubs
• add doc strings to the generated stubs
• proper test coverage
• proper formatting of stubs (not a blocker)
• integration into maturin
• proper type stubs for containers (list[T] instead of list). Blocked by generic_const_exprs being unstable
• allow to set custom type annotations (both inputs and output)
• allow to set custom stubs for eg. protocols
• add _typeshed.Incomplete to relevant places (modules with a #[pymodule_init] function...) (doc)
• choose between _typeshed.Incomplete and typing.Any in type annotations
• make sure to properly gate introspection element with cfg macros
• provide a way to set @overloads

Reference on stubs file

[davidhewitt]

For "custom type annotations", I think maybe adding it to #[pyo3(signature)] attribute is the way to go (xref #5138)

// `a` and return type set explicitly, b will be inferred
#[pyo3(pyfunction, signature = (a: int, b) -> int)]
fn foo(a: i32, b: i32) -> i32 {
    a + b
}

... I guess the alternative would be #[pyo3(type = "int")] annotations on each argument, but I think using the Python syntax is much better.

[yogevm15]

Hi,
I would like to help here, is there anything I should be aware of before I start working on it?

[Tpt]

@yogevm15 Thank you! It would be amazing!

I am currently working on function type annotations #5089

There are a bunch of topics that are quite independent from it like exposing const (without type annotation), supporting properly enums...

To help you deep dive in the code:

• Introspection data are emitted as const elements by the various PyO3 macros. Code for that is in pyo3-macros-backend/src/introspection.rs. Each element is a JSON string.
• The new pyo3-introspection parses the built dynamic libs objects, extracts the elements and builds the pyi stub files.
• Testing is done by building the pytests crate, run pyo3-introspection on it and check if the output is equal to the stub files in pytests/stubs. To run the test do nox -s test-introspection.

Note that introspection is only working with modules declared using inline Rust modules (mod XXX) and not with modules declared with functions (fn XXX)

[yogevm15]

Thanks,
I opened a PR for the modules associated consts #5150

I'm not sure about the classes associated consts. If I understood the code correctly, it exports them as class methods rather than actual class variables. So, how should we generate them in the stub?