[FEA] look into best way to set off heap memory sections/limits

[revans2]

Is your feature request related to a problem? Please describe.
There is no good way for the rapids plugin to auto set things like the pinnedPool.size, spillStoreageSize, if off heap limits are enabled, and how much those limits should be.

We need some rules that are based on the node type/size to start out with. I know we have some for pinned memory, but not for the spill storage size. These would need to be sued when we are looking at a CPU history file. If we look at a GPU history file we should be extract the memory high water mark metrics for disk, host, and GPU memory to get an idea of the total pool size. Then we can decide how we want to set things like, increasing the host spill storage size so that we know everything can fit in there if it is close or increasing the pinned memory size if it too is close too. We might also be able to reduce some of this if we see that there is relatively little memory pressure.

This might also go into us deciding on what hardware we could recommend for a job, or how large of input partitions we are able to process at once.