rayon::scope in an async context?

[bonsairobo]

I want to be able to concurrently execute CPU-heavy tasks (on a rayon pool) and wait for IO futures (on a blocking pool) that consume the results of the computational tasks. Because I'm in an async context, ideally I would be able to futures::try_join! those tasks. I've managed to wrap rayon::spawn in a future using async_channel, but this is painful because I can only pass 'static data into rayon::spawn. I'd rather use something like rayon::scope, but this would necessarily block my async executor thread.

I think I need some kind of async fn from rayon that would function like rayon::scope but instead of blocking the thread, it returns a future that completes when the scoped threads complete.

[cuviper]

It's pretty fundamental to the lifetime of scope that we do not return execution out of our control until all of the spawns are complete. I don't know how we could maintain safety if we were suspended with .await.

[bonsairobo]

@cuviper If we ignore the existing implementation of scope for the moment, do you think it's possible to await non-'static closures that run in a rayon pool? It seems possible to me, but maybe I am not seeing the full scope of the issue. I know that the blocking crate manages to do this to some extent (although it is designed specifically for blocking IO).

Maybe an example would help.

async {
    let mut captured_data = 0;
    // Hypothetical API
    let fut = rayon::scoped_future(|| { captured_data = 1; return "done"; });
    let message = fut.await;
}

I think we just require that fut carries the lifetime of the captured data. So long as the future does not escape this scope of captured_data, it should be safe.

[cuviper]

So long as the future does not escape this scope of captured_data, it should be safe.

But this can't be guaranteed -- safe code can completely drop or forget the future, and then the borrow is lost while the scope/spawn is still using it. This is the same problem that the original std::thread::scoped API had:
https://faultlore.com/blah/everyone-poops/

[bonsairobo]

Ah I see. That definitely throws a wrench into the mix. I'm not sure if what I'm asking for is possible.

I found this discussion helpful as well: https://stackoverflow.com/questions/65269738/spawn-non-static-future-with-tokio

[allsey87]

But this can't be guaranteed -- safe code can completely drop or forget the future, and then the borrow is lost while the scope/spawn is still using it.

This makes sense, but what is the idomatic way to bridge rayon with an async runtime with 'static only data? For example, if I am using wasm-bindgen-rayon and cannot block the main thread, how can I yield to the event loop while I am waiting for ParallelIterator::sum or rayon::spawn?

It seems there was a rayon-futures crate at one point that is now deprecated, but what was this replaced with?

[allsey87]

It seems there was a rayon-futures crate at one point that is now deprecated, but what was this replaced with?

Answering my own question after a bit more research. There seems to be an incomplete attempt at adding this functionality in PR #679. Apart from this, there is also tokio-rayon and async-rayon. These crates are mostly the same with latter depending on the futures crates rather than the tokio crate. It is not complete support for async interop, but does provide an spawn_async as an alternative to spawn that can be awaited.