Questions regarding multithreading from Go to Python.

[jshiwam]

Hi Everyone,

As we all know multithreading on multiple cores in python is not possible, since we have the GIL which makes sure that one python process will run at a single time on one CPU core.

From G0lang perspective, when we create multiple threads they can run on different cores, but since we are trying to execute python code from golang by making the use of this library, being that said I have the following questions:

1. where are we leveraging the go concurrency here?
2. Is anything running in parallel?
3. Isn't running multiple threads on a single core and context switching between them as good as running a sequential process given a scenario, where you only care about the end result?

[christian-korneck]

for Go specifically I tend to point to this blogpost and related conference talk, but you already know that one :-)
https://www.datadoghq.com/blog/engineering/cgo-and-python/

The blogpost is about locking the GIL so that only one Goroutine can use it at a time.

There are other ways (that I'm no expert of) to do things with the Python C-API. Like starting multiple GILs in separate threads.
I think a succesfull approach is to look at it from the C perspective (reading the Python C-API docs and articles about it, searching for OSS code on Github, sourcegraph that does what you want, asking in Python communities (Python IRC, lists, stackoverflow). And then figure out how to do things with the specifics/limitations of CGO. (I guess this is roughly how the author of the above blogpost figured out his variant).

For example, here's a (C related) write up on sub-interpreters:
https://stackoverflow.com/a/26570708

[jshiwam]

Thanks @christian-korneck , thanks for mentioning different approaches on how to figure out the issues mentioned in the discussion. Also the approach mentioned in this https://stackoverflow.com/a/26570708 post uses many sub-interpreters but eventually all of them will run on a single core because they share the same GIL. It seems like the concept of sub-interpreters is to create separate execution context with in the same interpreter, but Since the main interpreter will have the GIL, so whenever we lock/unlock it, all the other subinterpreters get locked/unlocked. And here we would never make use of multiple cores, and since we can't do that, we won't be leveraging the most powerful feature of golang.

I will update the discussion forum if I find out on how to make it work. But this seems to be a compatibility issue with python, as python was never designed to be used for multithreading, so I am afraid that it would work at all.

[christian-korneck]

@jshiwam this might be interesting for you (true multithreading subinterpreters are in Py 3.12 alpha): https://martinheinz.dev/blog/97