Add full_blocking feature to thread pool

[nhukc]

This PR adds an option full_blocking to ThreadPoolBuilder that affects the behavior of nested thread pools.

When a job on a parent thread pool creates a job in a child thread pool and full_blocking is true, the parent job will block until the child job is completed. This is different from the default behavior where the parent thread is allowed to work on other jobs in the parent thread pool while the child job completes.

This behavior is useful for avoiding deadlocks caused by work-stealing and is helpful for instrumentation based profiling in multi-threaded settings.

See for more details:
#1174

[nhukc]

Feel free to push directly to this PR.

The documentation could be improved by matching the language to the rest of the rayon docs.

I'm not 100% confident in the implementation of in_worker_cross_blocking. It works for my test, but I could easily be missing something in the details.

[JoeyBF]

Thanks for the PR! One small suggestion would be to make full_blocking take a bool as an argument, so it can be toggled on or off. This seems to be more in line with the other builder methods.

I'll add that I tested my code using this branch and this does solve my own issue #957, which is essentially a variation of #592 🎉

[nhukc]

Thanks for taking a look at the PR!

The most similar function in the ThreadPoolBuilder is

pub fn use_current_thread(self) -> ThreadPoolBuilder<S>

Which sets a bool to true when called. I matched that function signature.

[JoeyBF]

That's fair. I would argue that use_current_thread would also need a bool, but as far as this PR is concerned it's probably better to just be consistent.

[nhukc]

@cuviper Are you interested in merging this? If not, can you close it?

[cuviper]

I need some time to think about the implications of this change, to make sure there aren't any big foot-guns here. But you're also helping solve a foot-gun, so that's going to be a balancing act.

I might also bikeshed the name, but that's not a big deal.