python: Refine highlighting

[JaagupAverin]

WIP. Will test and add more.

import sys

# Issue #1: types in "binary unions" not highlighted:
x: int = 5
y: str = "5"
z: int | str = "5"

# Issue #2: Single-letter upper-case types confused for constants:
#   Given the current parser syntax, this issue cannot be fixed properly.
#   e.g:
#   Given a fn with prototype `def foo(x: type, y: int)`
#   And the call `foo(X, Y)`,
#   The parser cannot know if X and Y are constants or types.
E = 2.71828

type MyList[T] = T | list[T]

# Issue #3: except* not recognized as keyword:
try:
    pass
except* RuntimeError:
    sys.exit(1)


# Issue #4: Classes with underscore not always recognized as types:
class _PrivateClassA:
    def __call__(self) -> None:
        pass


isinstance(42, _PrivateClassA)

# Issue #5: Builtin types not recognized as types:
isinstance(42, list)
isinstance(42, float)

Release Notes:

• N/A or Added/Fixed/Improved ...

[JaagupAverin]

@osiewicz Currently this PR is still in draft, but already in advance I feel that this approach here is incredibly silly. I am not sure if it is possible to apply a rule for an arbitrarily nested case such as this?

type AnyTasks[T: ExecutableTask | Task] = T | _TaskList[T] | list[T | _TaskList[T]] | None

I feel like the problem boils down to not getting adequate output from the parser so we're having to do more complicated stuff in the highlighter.

[osiewicz]

Hmm. I'll revisit your PR tomorrow, but off the top of my head that's not going to be easy to pull off cleanly. Tree-sitter doesn't support matches of arbitrary depth.
Besides, I think we should fall back on semantic highlighting for cases like these. While in principle the only valid kind of identifier in that position is a type, users code might be invalid. Thus, if possible, I'd rather have us tackle semantic tokens instead of handling these cases at the tree-sitter level.

Don't get me wrong, I really appreciate your recent efforts on Python highlighting. It's just that this PR runs into a wall that should be brought down, not chipped away at :P

[osiewicz]

Hm, neither Pyright nor pylsp support semantic tokens at the moment.. That's a huge bummer.

[JaagupAverin]

Yeah I agree this is too big of a hack around the actual issue so I think I'll remove this part for now and maybe open an issue for it instead.

[JaagupAverin]

With the other adjustments my python project is 99.8% correctly highlighted though so its a good experience compared to a month ago already :)

[JaagupAverin]

Also regarding semantic tokens I believe basedpyright already has support for this https://docs.basedpyright.com/latest/benefits-over-pyright/pylance-features/.
This would make it another big feature in support of basedpyright at this stage.