settings recommendations?

[jayhohoho2019]

Hi, I'm wondering if there are any sizing guidelines with this plotter?

1. 1 single CPU, 10 Core/20 Threads, what is the recommended -r value? 128GB RAM with a 110GB ramdisk, what is the recommended bucket size (-u and -v)?
2. 2 CPUs, 28 Core each. what is the recommended -r? Is this plotter able to utilize both CPUs? Any caveats?
3. 2 CPUs, 28 Core each, 512GB RAM. In this a configuration, is it better to run one instance of the plotter maxing out -r, or multiple instances sharing CPU cores?

Actually what is -r? It's a multiplier of what?

Thank you.

[SebMoore]

Ok: first off, -r is meant to be the approx number of threads. Thing is, the plotter often doubles this number - so the advice is typically to set it to the number of physical cores your system has. I.e. on an 8C/16T system, set -r to 8.
Now for your questions:
1: -r should be set to 10. If using a 110G ramdisk, -u should be 512 to stop you from running out of RAM. I personally haven't noticed much (if any) difference fiddling with -v, so I usually just leave it.
2&3: (similar questions): there have been numerous people (myself included) who have issues getting the plotter to utilise both CPUs on dual-socket systems. Some have only seen the plotter use 1 cpu out of 2. I have seen the plotter use both CPUs, but only at 60%. So here's what I would do: set up a RAMDISK (for me, the smallest size you could use is 246G). Then, I would give it a shot, setting -r to 56 (both CPUs). If that results in 90-100% usage, great! If not, do the following:
Use the command numactl --hardware to obtain the list of threads running on each CPU. Then, use taskset -c [insert list of threads for one cpu here, separated by commas] ./chia-plotter -r 28 -u 512 etc. Then, repeat: set up a second ramdisk of the same size, and run the command again, but with the list of the threads of the second CPU. This will force two parallel instances of MadMax to run on the two separate CPUs. For me, this worked: I got up to 95% overall utilisation, and I went from 3 plots/hour to 5.
If you have 512G of ram, like in your question 3, you can probably run these two parallel plots in RAMDISKS - just be aware that you need 512 buckets, and your RAMDISK should be no bigger than 246G. You might still run out of memory though: I tested this on a dual 16C Xeon workstation, so you have more cores than me and so more ram usage. Best idea may be to stagger the starts of the plot: peak disk usage occurs at Phase 2 Table 7, so setting the second one to start at that point may be a good idea to prevent 100% memory usage.
Hope this helps!

[SebMoore]

I feel like numactl is much more efficient than taskset, and lets you bind memory from the node to the process as well. You can just run numactl --cpunodebind=0 --membind=0 -- ./build/chia_plot -r -u -t -p -f

Wow! I genuinely had no idea. Yes that looks way easier. Thanks!

[ghost]

I feel like numactl is much more efficient than taskset, and lets you bind memory from the node to the process as well. You can just run numactl --cpunodebind=0 --membind=0 -- ./build/chia_plot -r -u -t -p -f

@gryan315 know if a single 220gb tmpfs ramdisk mount works under numactl or is it still best to mount 2x 110GB tmpfs ramdisk, binding each to node with mpol?

[gryan315]

I'm honestly not sure if numactl will split the storage between the nodes in a single large tmpfs, it's worth testing (I don't have enough RAM to test that). Creating 2x tmpfs with mpol should work, but it's more difficult to bind, and easier to "prefer." mpol=local might be a way to get it to work as both a single large tmpfs or as 2x tmpfs.

[XxViper06xX]

Is this being run in Windows? I see no mention of OS in here and I have issues with Dual CPU in Windows.

[maxcom99]

So 2x 20core CPU with 512GB ram the final answer is? :)

[maxcom99]

If it may help someone in the future, here is where I ended up:

SYSTEM BASICS:

• OS: Ubuntu 21.04 (with GUI)
• 2x e52698 v4 - 20core / 40 thread each
• 512Gb RAM DDR4 2400 (16x 32GB)
• Raid 5 "chiaDone" SAS 12GB/sec ~2.5min copy time

RAM DISKS:
sudo mount -t tmpfs -o size=246G tmpfs /mnt/ram1/
sudo mount -t tmpfs -o size=246G tmpfs /mnt/ram2/

Findings:

• One large 492GB tmpfs running 2 instances manually offset at "[P2] Table 7 rewrite" finishing: Average time per instance ~29min
• Individual 246GB tmpfs, same offset: Average ~27min
• -t and -2 parameters - ~1min difference to not split these. Meaning -t /mnt/ram1 -2 /mnt/ram1 is about a minute faster overall than -t /mnt/ram1 -2 /mnt/ram2

THREADS:

• YMMV - please add your findings
• Single instance: Faster with -r 60-140 after 140 there was no difference except after -r 240, I saw slower times
• Single instance: Less than -r 60, noticeably slower
• Single instance: -r 60-80 the CPU load is 65-100% with most on the low side
• Two instances: Running both at -60 was a couple of minutes slower than both -r 80. CPU utilization 92%+

MISC NOTES:

• I had to enable the "-w true" flag because otherwise the swap would fill up and plot times went into ~32min
• Adding "numactl --cpunodebind=0 --membind=0 --" didn't have any effect on my plot speed, I have not tried "taskset" yet. Seems like I only have a single node (expected 2?).
• I "think" I can run -u 256 on 1 instance and -u 512 on the second - seems ok but needs more testing. Couple minute improvement overall but risk running out of RAM.
• $ numactl --hardware available: 1 nodes (0) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 ..

TASK PARAMETERS
./chia_plot -n -1 -w true -r 80 -u 512 -t /mnt/ram1/ -2 /mnt/ram1/ -d /mnt/chiaDone/ -c -f

Next Steps:

• I was not able to figure out "mpol=local" for mounting the tmpfs, makes sense this might be more efficient
• Need to try taskset and splitting the workload. My hypothesis is this would be slower because the OS appears to split and scale up the workload fairly well.
• Copying to a faster NVME drive and disable -w should get me a couple minutes per plot
• Super annoying that the GUI doesn't look for new plots - grr
• Push for 3 or 4 instances with 110GB RAM tmpfs and phase 3 and 4 on NVME?

[gryan315]

Should be able to get better split times by calling the mad max plotter with numactl, may not have any effect if it's still using ram from the other node in tmpfs. When mounting the tmpfs you can call it with mpol set to prefer node 0. Also, is your CPU scaling governor set to performance? You don't need to specify -2 if it's the same as -t.

[maxcom99]

Should be able to get better split times by calling the mad max plotter with numactl, may not have any effect if it's still using ram from the other node in tmpfs. When mounting the tmpfs you can call it with mpol set to prefer node 0. Also, is your CPU scaling governor set to performance? You don't need to specify -2 if it's the same as -t.

Thanks for the feedback!

It looks like I only have 1 node... what "ram from the other node"?

sudo mount -t tmpfs -o size=264G mpol=prefer:0 tmpfs /mnt/ram1/
mount: bad usage
Try 'mount --help' for more information.

Great idea on the CPU -

cat /proc/cpuinfo | grep MHz cpu MHz : 2700.000 cpu MHz : 2700.000 cpu MHz : 2699.924 cpu MHz : 83.338 cpu MHz : 2699.918 cpu MHz : 2699.913 cpu MHz : 2699.915 cpu MHz : 2699.917 cpu MHz : 2700.000 cpu MHz : 2618.235 cpu MHz : 2699.918 cpu MHz : 2400.000 cpu MHz : 2657.027 cpu MHz : 728.497 cpu MHz : 718.887 cpu MHz : 2418.187 cpu MHz : 698.608 cpu MHz : 701.178 cpu MHz : 2699.922 cpu MHz : 2699.925 cpu MHz : 2700.000 cpu MHz : 2700.000 cpu MHz : 2699.918 cpu MHz : 2700.000 cpu MHz : 2699.919 cpu MHz : 2700.000 cpu MHz : 701.617 cpu MHz : 2699.923 cpu MHz : 2577.595 cpu MHz : 2676.932 cpu MHz : 704.214 cpu MHz : 2289.839 cpu MHz : 2700.000 cpu MHz : 2100.000 cpu MHz : 2699.920 cpu MHz : 2700.000 cpu MHz : 2700.000 cpu MHz : 2700.000 cpu MHz : 2699.918 cpu MHz : 2700.000 cpu MHz : 2699.917 cpu MHz : 743.212 cpu MHz : 2700.000 cpu MHz : 2400.000 cpu MHz : 2699.918 cpu MHz : 693.618 cpu MHz : 696.303 cpu MHz : 2700.000 cpu MHz : 2699.917 cpu MHz : 2631.344 cpu MHz : 2700.000 cpu MHz : 2500.000 cpu MHz : 2664.034 cpu MHz : 2699.917 cpu MHz : 1600.000 cpu MHz : 700.645 cpu MHz : 693.833 cpu MHz : 2699.916 cpu MHz : 2699.917 cpu MHz : 2000.000 cpu MHz : 2699.921 cpu MHz : 695.945 cpu MHz : 2699.916 cpu MHz : 2699.917 cpu MHz : 2699.916 cpu MHz : 197.671 cpu MHz : 704.016 cpu MHz : 2699.920 cpu MHz : 2612.725 cpu MHz : 2676.252 cpu MHz : 2699.916 cpu MHz : 2212.128 cpu MHz : 2699.919 cpu MHz : 2700.000 cpu MHz : 2699.916 cpu MHz : 2699.920 cpu MHz : 2699.920 cpu MHz : 2699.918 cpu MHz : 699.864 cpu MHz : 2699.918

Not sure where I would lock this down other than disabling turbo. This link is less than helpful - https://www.dell.com/community/PowerEdge-Hardware-General/How-can-I-lock-my-Xeon-CPU-frequency-in-R730xd/td-p/6174957

I use both -t and -2 for ease of tinkering :)

[jayhohoho2019]

Thanks for sharing the info. Just to double check, your 2 cpus are 20c/40t but you set -r at 80 or more, on each instance of the plotter?

And in the 2 instance scenario, each instance is running with -r 80?

I have noticed in my case, if I set -r to be close to the total # of threads in both cpus, phase 3 gets much longer, even though phases 1,2 and 4 may get better. In particular write right entries takes much longer. Did you noticed anything similar?