256 buckets vs 128

[madMAx43v3r]

I've heard that 256 buckets is faster for some, let's figure out if we should make it the default.

[madMAx43v3r]

after feedback on discord and given that many ppl want to use ramdisk with 128 GiB RAM, I will increase default buckets to 256

[Hardcore-fs]

using a ramdisk is the WRONG way to go about it.....
since to use a "ram disk" ,you have to build an FS & put all transactions via the system file handling....
it also ties users to specific hardware builds.....

[SebMoore]

using a ramdisk is the WRONG way to go about it.....
since to use a "ram disk" ,you have to build an FS & put all transactions via the system file handling....
it also ties users to specific hardware builds.....

What do you suggest is the correct way to go about it then?

[maxpla3]

The thing with the madmax plotter is that its biggest advantage is complete or partial in-RAM plotting. So it makes sense to optimize the parameters for this scenario.

[homatthew22]

I've found 512 buckets to be the fastest setting on my core bottlenecked system (4c/4t, 16gb ram w/ NVMe drive). Total plot time ~6900s.

Likewise, I found 64 buckets to be the fastest on my I/O bottlenecked system (Macbook Pro 6c/12t, 16gb ram w/ NVMe over Thunderbolt3). Total plot time ~6600s.

[madMAx43v3r]

yes, more buckets are good for CPU, less buckets is good for SSD/HDD

[therealflinchy]

i assume higher buckets decreases system memory usage? just eyeballing my process now at least.

now not sure it's coincidence but looks like my phase 1 is waaay faster. about 100 seconds. see how it impacts overall plot time, promising so far hmmmm

buuut looks like i might lose every second saved in p1, by longer p2 hahaha

edit: oooh yucky, no, phase 3 very bad now hahah. P3-2 is still amazing, P3-1 tanks HARD

[madMAx43v3r]

ok now even better, latest phase3-stage1-tune

[therealflinchy]

Ahhh one of the few nights I've decided to actually go do real work so I can't be at home tinkering :(

[madMAx43v3r]

it's getting faster the longer you wait lol

[therealflinchy]

massive improvement, 256 buckets down from 2250 to 1818, 128 from 2050 best to 1930. including 215 second P4 times, may as well count 256 as a 1600 second plot, beats parallel easily for me now

[madMAx43v3r]

awesome

[fgc0109]

I have 384G RAM and set 300G ramdisk ,128 buckets
plot time is 2030s.
may be 16 buckets is fast for me ?

a .... it's not
i have 32 threads , 16 buckets only use 50 % cpu
[P1] Table 2 took 204.726 sec, found 4294936083 matches

32 buckets ,about 90 % cpu time
[P1] Table 2 took 162.617 sec, found 4294742033 matches
[P1] Table 3 took 200.101 sec, found 4294418187 matches

128 buckets
[P1] Table 2 took 131.608 sec, found 4295052059 matches
[P1] Table 3 took 174.346 sec, found 4295078653 matches

and 256 buckets
[P1] Table 2 took 122.176 sec, found 4294903340 matches
[P1] Table 3 took 148.528 sec, found 4294910119 matches

a , let's try 512 buckets
[P1] Table 2 took 126.45 sec, found 4294967042 matches
[P1] Table 3 took 143.494 sec, found 4295003588 matches

i didn't finish plot ,but i think the data is well enough makes sense
256 is better

[fer662]

times are not that consistent that i'd conclude 256 is better than 512 with just one try.

[therealflinchy]

I have 384G RAM and set 300G ramdisk ,128 buckets
plot time is 2030s.
may be 16 buckets is fast for me ?

a .... it's not
i have 32 threads , 16 buckets only use 50 % cpu
[P1] Table 2 took 204.726 sec, found 4294936083 matches

32 buckets ,about 90 % cpu time
[P1] Table 2 took 162.617 sec, found 4294742033 matches
[P1] Table 3 took 200.101 sec, found 4294418187 matches

128 buckets
[P1] Table 2 took 131.608 sec, found 4295052059 matches
[P1] Table 3 took 174.346 sec, found 4295078653 matches

and 256 buckets
[P1] Table 2 took 122.176 sec, found 4294903340 matches
[P1] Table 3 took 148.528 sec, found 4294910119 matches

a , let's try 512 buckets
[P1] Table 2 took 126.45 sec, found 4294967042 matches
[P1] Table 3 took 143.494 sec, found 4295003588 matches

i didn't finish plot ,but i think the data is well enough makes sense
256 is better

you need to try a whole plot. 256 was 100 seconds faster phase 1 for me, but 50 seconds slower phase 2, 200 seconds slower phase 3. specifically p3-1, p3-2 still very fast. as i was typing this he said he's working on p3-1, so maybe that'll impact large bucket sizes even more

[ESP4Ever]

Moreover, you shall create several plots in a row and get the average because the time is varying from one to another)
Anyway this SW is AWESOME! One plot in 30 minutes on an old IBM server (but with a lot of RAM and very fast SAS raid) is amazing!

[ADumaine]

It looks like using 256 buckets is about 15% faster for my setup using a ramdisk and the latest build (June 13).

Previously I was getting around a 1650s average. I am now getting 1408s avg. Without a ram disk I was getting about 2700s.

System is 5950x (x570 chipset) with 128G ram. tmp1 is a PNY 3040 2TB, tmp2 is 110G ram disk. Destination is external seagate usb 3.0.
Linux desktop with only chia GUI farming ( and firefox a the moment). 2GB swap

The first run is a little faster and I think it is because the subsequent plots are copying the file.

I have noticed that even when specifying 12 threads, all 32 cores are busy, but at only about 70-85%. I have not tried going over 12 as I wanted to ensure enough memory.

Multi-threaded pipelined Chia k32 plotter - 16f9793
Final Directory: /media/allan/Sea8-05/
Number of Plots: 28
Crafting plot 1 out of 28
Process ID: 7925
Number of Threads: 12
Number of Buckets: 2^8 (256)
Pool Public Key:   
Farmer Public Key:
Working Directory:   /media/allan/pny/
Working Directory 2: /mnt/ramdisk/
Plot Name: plot-k32-2021-06-13-10-39-da165bcd496cc9218ab2ce3c3039aaf03dfe3aec1f12ca99989c869b8e1af5a8
[P1] Table 1 took 8.70153 sec
[P1] Table 2 took 71.7063 sec, found 4294944111 matches
[P1] Table 3 took 105.447 sec, found 4294862653 matches
[P1] Table 4 took 132.529 sec, found 4294598903 matches
[P1] Table 5 took 128.129 sec, found 4294256254 matches
[P1] Table 6 took 120.51 sec, found 4293516247 matches
[P1] Table 7 took 81.4096 sec, found 4292123536 matches
Phase 1 took 648.438 sec
[P2] max_table_size = 4294967296
[P2] Table 7 scan took 5.70043 sec
[P2] Table 7 rewrite took 15.0367 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 23.1828 sec
[P2] Table 6 rewrite took 44.0538 sec, dropped 581428108 entries (13.542 %)
[P2] Table 5 scan took 29.4418 sec
[P2] Table 5 rewrite took 57.3699 sec, dropped 762153585 entries (17.7482 %)
[P2] Table 4 scan took 26.5094 sec
[P2] Table 4 rewrite took 43.3831 sec, dropped 829008531 entries (19.3035 %)
[P2] Table 3 scan took 31.6449 sec
[P2] Table 3 rewrite took 39.2567 sec, dropped 855204873 entries (19.9123 %)
[P2] Table 2 scan took 33.1506 sec
[P2] Table 2 rewrite took 40.582 sec, dropped 865593423 entries (20.1538 %)
Phase 2 took 397.884 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 25.7356 sec, wrote 3429350688 right entries
[P3-2] Table 2 took 18.2059 sec, wrote 3429350688 left entries, 3429350688 final
[P3-1] Table 3 took 28.6534 sec, wrote 3439657780 right entries
[P3-2] Table 3 took 20.2116 sec, wrote 3439657780 left entries, 3439657780 final
[P3-1] Table 4 took 31.485 sec, wrote 3465590372 right entries
[P3-2] Table 4 took 18.3484 sec, wrote 3465590372 left entries, 3465590372 final
[P3-1] Table 5 took 33.6928 sec, wrote 3532102669 right entries
[P3-2] Table 5 took 18.6981 sec, wrote 3532102669 left entries, 3532102669 final
[P3-1] Table 6 took 33.4997 sec, wrote 3712088139 right entries
[P3-2] Table 6 took 19.9081 sec, wrote 3712088139 left entries, 3712088139 final
[P3-1] Table 7 took 26.9946 sec, wrote 4292123536 right entries
[P3-2] Table 7 took 24.06 sec, wrote 4292123536 left entries, 4292123536 final
Phase 3 took 302.18 sec, wrote 21870913184 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 34.1797 sec, final plot size is 108797577628 bytes
Total plot creation time was 1382.73 sec
Started copy to /media/allan/Sea8-05/plot-k32-2021-06-13-10-39-da165bcd496cc9218ab2ce3c3039aaf03dfe3aec1f12ca99989c869b8e1af5a8.plot

Nice work on this plotter!

[fer662]

I wonder if thee ramdisk does anything other than saving the drives livespans. I'm getting similar or better times using 3 nvmes on the 5950x. I have 64 more gb of ram incoming so i'll be able to tell soon enough.

[fer662]

You should definitely try more cores... If anything, you'll swap a bit, but maybe your times will end up better anyway (make the swapfile be on the nvme drives)

[fer662]

Final Directory: /mnt/newplots/
Number of Plots: infinite
Process ID: 5282
Number of Threads: 22
Number of Buckets: 2^9 (512)
Pool Public Key:   a178d9aa3cfe584293daa4c5805c48f4c560a3e836edb93898039f2889643eed07c374809f419585ade0485ea280da92
Farmer Public Key: 864a01d5b1e0b0a9858562bbbb21f04b43cc30b74e6ddf4c3f27c1f9998bf6c7f0a01afc8fda243e9a066d947a5c7f5b
Working Directory:   /mnt/plotter1/
Working Directory 2: /mnt/plotter1/
Plot Name: plot-k32-2021-06-14-01-05-37fe651c77b281ae4d755d15415b05413661b72489fbdc5b250062ec31a1c1d7
[P1] Table 1 took 11.2762 sec
[P1] Table 2 took 84.1056 sec, found 4295023249 matches
[P1] Table 3 took 98.2388 sec, found 4294986960 matches
[P1] Table 4 took 115.099 sec, found 4295022749 matches
[P1] Table 5 took 112.633 sec, found 4295048559 matches
[P1] Table 6 took 102.459 sec, found 4294949582 matches
[P1] Table 7 took 85.1833 sec, found 4294865338 matches
Phase 1 took 609.14 sec
[P2] max_table_size = 4295048559
[P2] Table 7 scan took 6.57561 sec
[P2] Table 7 rewrite took 23.7997 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 24.2889 sec
[P2] Table 6 rewrite took 41.6966 sec, dropped 581259744 entries (13.5336 %)
[P2] Table 5 scan took 23.9027 sec
[P2] Table 5 rewrite took 39.8042 sec, dropped 762031625 entries (17.7421 %)
[P2] Table 4 scan took 22.8059 sec
[P2] Table 4 rewrite took 39.1667 sec, dropped 828870962 entries (19.2984 %)
[P2] Table 3 scan took 22.7342 sec
[P2] Table 3 rewrite took 38.7671 sec, dropped 855054518 entries (19.9082 %)
[P2] Table 2 scan took 22.8173 sec
[P2] Table 2 rewrite took 38.7769 sec, dropped 865653007 entries (20.1548 %)
Phase 2 took 351.651 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 30.5644 sec, wrote 3429370242 right entries
[P3-2] Table 2 took 20.9376 sec, wrote 3429370242 left entries, 3429370242 final
[P3-1] Table 3 took 31.9292 sec, wrote 3439932442 right entries
[P3-2] Table 3 took 20.8967 sec, wrote 3439932442 left entries, 3439932442 final
[P3-1] Table 4 took 33.4666 sec, wrote 3466151787 right entries
[P3-2] Table 4 took 20.8944 sec, wrote 3466151787 left entries, 3466151787 final
[P3-1] Table 5 took 34.2501 sec, wrote 3533016934 right entries
[P3-2] Table 5 took 21.7407 sec, wrote 3533016934 left entries, 3533016934 final
[P3-1] Table 6 took 36.3101 sec, wrote 3713689838 right entries
[P3-2] Table 6 took 22.1246 sec, wrote 3713689838 left entries, 3713689838 final
[P3-1] Table 7 took 34.5591 sec, wrote 4294865338 right entries
[P3-2] Table 7 took 26.683 sec, wrote 4294865338 left entries, 4294865338 final
Phase 3 took 336.368 sec, wrote 21877026581 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 33.8665 sec, final plot size is 108834568091 bytes
Total plot creation time was 1331.09 sec
Started copy to /mnt/newplots/plot-k32-2021-06-14-01-05-37fe651c77b281ae4d755d15415b05413661b72489fbdc5b250062ec31a1c1d7.plot

[ADumaine]

I did have 3 nvme in the system: 980 pro (gen 4 read 7k), pny 3040 (gen 4 5k read), and a xpg sx8200 pro (gen 3, 3.5k read).
The 980pro outperformed the other two with sustained writes (using latest firmware). It could do 7 to 8 in staged parallel. The other two would choke and could only do 3 to 4. I ended up using gui plotting for the 980pro and plotman for the other two. Even so the system would only do about 40/day. Other system specs in my last post for anyone interested.

Madmax makes it so much easier and faster. Now able to do about 21m plots (about 60/day).

I did a bit more testing of both threads and buckets. 512 helped. It brought plot time down to about 1300.

I tried more threads (up to 28) but that seemed to slow my system down. And the higher thread count used more memory. I bumped up the ram disk a little.

I also tried 1024 buckets, but that did not help much.

I removed the 2G swap from the ssd and created 6G swap on another nvme. I could not tell much difference since most of the runs stayed on the ramdisk.

The sweet spot seems to be 12 or 16 threads and 512 buckets for my system.

I ran a set of 24 plots with the above settings and copying the final file from the nvme to an external USB 3.0. Even with that slow copy the average or all 24 was 1279.12s. I could probably trim that down a little more if I put a 980pro back in and copied to another nvme, but my disks are full now.

Here are some of the test runs.

22 threads, 110 ramdisk
 
Multi-threaded pipelined Chia k32 plotter - 16f9793
Final Directory: /media/allan/Sea8-05/
Number of Plots: 1
Crafting plot 1 out of 1
Process ID: 56377
Number of Threads: 22
Number of Buckets: 2^8 (256)
Pool Public Key:   939f6f869b87542414b8d5f9ab94c157af6b4900e2d2906486b8e17cc9c9c4d5869fd4c167276e10f7281e167d7b2286
Farmer Public Key: 8b49a3190e88bb4a8d73bd08c5f9584bfdca0beb9b9784239ad2d3a6811085f7221cc85c6ab91e3166f8ef17ab610c0d
Working Directory:   /media/allan/pny/
Working Directory 2: /mnt/ramdisk/
Plot Name: plot-k32-2021-06-13-23-07-73522cfab41f724a66fdfa7173f06cb2d56654c7484099324cf4c78b81596a0e
[P1] Table 1 took 10.1012 sec
[P1] Table 2 took 72.9315 sec, found 4294837295 matches
[P1] Table 3 took 98.1112 sec, found 4294857735 matches
[P1] Table 4 took 121.075 sec, found 4294675215 matches
[P1] Table 5 took 117.853 sec, found 4294419994 matches
[P1] Table 6 took 110.614 sec, found 4293923410 matches
[P1] Table 7 took 77.5603 sec, found 4292891213 matches
Phase 1 took 608.257 sec
[P2] max_table_size = 4294967296
[P2] Table 7 scan took 4.66087 sec
[P2] Table 7 rewrite took 15.789 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 24.3493 sec
[P2] Table 6 rewrite took 44.2813 sec, dropped 581375737 entries (13.5395 %)
[P2] Table 5 scan took 29.1196 sec
[P2] Table 5 rewrite took 52.4877 sec, dropped 762088400 entries (17.746 %)
[P2] Table 4 scan took 26.0907 sec
[P2] Table 4 rewrite took 42.5708 sec, dropped 828952706 entries (19.3019 %)
[P2] Table 3 scan took 29.8159 sec
[P2] Table 3 rewrite took 39.5183 sec, dropped 855133406 entries (19.9106 %)
[P2] Table 2 scan took 31.2842 sec
[P2] Table 2 rewrite took 39.7383 sec, dropped 865549564 entries (20.1533 %)
Phase 2 took 388.438 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 24.4991 sec, wrote 3429287731 right entries
[P3-2] Table 2 took 18.8926 sec, wrote 3429287731 left entries, 3429287731 final
[P3-1] Table 3 took 29.694 sec, wrote 3439724329 right entries
[P3-2] Table 3 took 20.1135 sec, wrote 3439724329 left entries, 3439724329 final
[P3-1] Table 4 took 31.7472 sec, wrote 3465722509 right entries
[P3-2] Table 4 took 20.1697 sec, wrote 3465722509 left entries, 3465722509 final
[P3-1] Table 5 took 33.0858 sec, wrote 3532331594 right entries
[P3-2] Table 5 took 20.5967 sec, wrote 3532331594 left entries, 3532331594 final
[P3-1] Table 6 took 32.0921 sec, wrote 3712547673 right entries
[P3-2] Table 6 took 21.6556 sec, wrote 3712547673 left entries, 3712547673 final
[P3-1] Table 7 took 25.6705 sec, wrote 4292891213 right entries
[P3-2] Table 7 took 28.197 sec, wrote 4292891213 left entries, 4292891213 final
Phase 3 took 309.073 sec, wrote 21872505049 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 34.124 sec, final plot size is 108807428814 bytes
Total plot creation time was 1339.98 sec
Started copy to /media/allan/Sea8-05/plot-k32-2021-06-13-23-07-73522cfab41f724a66fdfa7173f06cb2d56654c7484099324cf4c78b81596a0e.plot
Copy to /media/allan/Sea8-05/plot-k32-2021-06-13-23-07-73522cfab41f724a66fdfa7173f06cb2d56654c7484099324cf4c78b81596a0e.plot finished, took 629.51 sec, 164.838 MB/s avg.


110G ramdisk, 28  hit 113G memory
 
allan@chia5950:~$ bash createplots.sh 
Multi-threaded pipelined Chia k32 plotter - 16f9793
Final Directory: /media/allan/Sea8-05/
Number of Plots: 1
Crafting plot 1 out of 1
Process ID: 59203
Number of Threads: 28
Number of Buckets: 2^8 (256)
Pool Public Key:   939f6f869b87542414b8d5f9ab94c157af6b4900e2d2906486b8e17cc9c9c4d5869fd4c167276e10f7281e167d7b2286
Farmer Public Key: 8b49a3190e88bb4a8d73bd08c5f9584bfdca0beb9b9784239ad2d3a6811085f7221cc85c6ab91e3166f8ef17ab610c0d
Working Directory:   /media/allan/pny/
Working Directory 2: /mnt/ramdisk/
Plot Name: plot-k32-2021-06-13-23-42-419c2e3bba366fb4fdacb65ea5f22e7477b17cd13531bcc9b6b12a8aae34cac9
[P1] Table 1 took 10.3359 sec
[P1] Table 2 took 72.8344 sec, found 4295056288 matches
[P1] Table 3 took 97.6076 sec, found 4295156127 matches
[P1] Table 4 took 120.57 sec, found 4295001932 matches
[P1] Table 5 took 117.644 sec, found 4295007947 matches
[P1] Table 6 took 111.011 sec, found 4295121993 matches
[P1] Table 7 took 78.2554 sec, found 4295116207 matches
Phase 1 took 608.312 sec
[P2] max_table_size = 4295156127
[P2] Table 7 scan took 4.21682 sec
[P2] Table 7 rewrite took 15.882 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 24.7697 sec
[P2] Table 6 rewrite took 43.9867 sec, dropped 581359842 entries (13.5354 %)
[P2] Table 5 scan took 28.268 sec
[P2] Table 5 rewrite took 49.3904 sec, dropped 761978215 entries (17.741 %)
[P2] Table 4 scan took 35.6911 sec
[P2] Table 4 rewrite took 40.2382 sec, dropped 828854202 entries (19.2981 %)
[P2] Table 3 scan took 28.6489 sec
[P2] Table 3 rewrite took 41.5919 sec, dropped 855230616 entries (19.9115 %)
[P2] Table 2 scan took 31.1618 sec
[P2] Table 2 rewrite took 39.3285 sec, dropped 865605240 entries (20.1535 %)
Phase 2 took 391.939 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 27.2428 sec, wrote 3429451048 right entries
[P3-2] Table 2 took 20.3238 sec, wrote 3429451048 left entries, 3429451048 final
[P3-1] Table 3 took 31.831 sec, wrote 3439925511 right entries
[P3-2] Table 3 took 20.5944 sec, wrote 3439925511 left entries, 3439925511 final
[P3-1] Table 4 took 32.1857 sec, wrote 3466147730 right entries
[P3-2] Table 4 took 20.6599 sec, wrote 3466147730 left entries, 3466147730 final
[P3-1] Table 5 took 38.1923 sec, wrote 3533029732 right entries
[P3-2] Table 5 took 21.3408 sec, wrote 3533029732 left entries, 3533029732 final
[P3-1] Table 6 took 34.9574 sec, wrote 3713762151 right entries
[P3-2] Table 6 took 22.4112 sec, wrote 3713762151 left entries, 3713762151 final
[P3-1] Table 7 took 28.1207 sec, wrote 4295116207 right entries
[P3-2] Table 7 took 27.6593 sec, wrote 4294967296 left entries, 4294967296 final
Phase 3 took 328.132 sec, wrote 21877283468 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 35.5259 sec, final plot size is 108836073651 bytes
Total plot creation time was 1363.98 sec
Started copy to /media/allan/Sea8-05/plot-k32-2021-06-13-23-42-419c2e3bba366fb4fdacb65ea5f22e7477b17cd13531bcc9b6b12a8aae34cac9.plot
Copy to /media/allan/Sea8-05/plot-k32-2021-06-13-23-42-419c2e3bba366fb4fdacb65ea5f22e7477b17cd13531bcc9b6b12a8aae34cac9.plot finished, took 628.722 sec, 165.088 MB/s avg.

------------------------------------------
ramdisk 115G,  6GB swap,  28 threads, 256 buckets

Multi-threaded pipelined Chia k32 plotter - 16f9793
Final Directory: /media/allan/Sea8-05/
Number of Plots: 1
Crafting plot 1 out of 1
Process ID: 85820
Number of Threads: 28
Number of Buckets: 2^8 (256)
Pool Public Key:   939f6f869b87542414b8d5f9ab94c157af6b4900e2d2906486b8e17cc9c9c4d5869fd4c167276e10f7281e167d7b2286
Farmer Public Key: 8b49a3190e88bb4a8d73bd08c5f9584bfdca0beb9b9784239ad2d3a6811085f7221cc85c6ab91e3166f8ef17ab610c0d
Working Directory:   /media/allan/pny/
Working Directory 2: /mnt/ramdisk/
Plot Name: plot-k32-2021-06-14-16-35-7da78527d52b9da493d4592fa4b014c82a92fbea3e896439e240ce6994e42053
[P1] Table 1 took 10.2757 sec
[P1] Table 2 took 72.1433 sec, found 4294826795 matches
[P1] Table 3 took 96.9594 sec, found 4294772783 matches
[P1] Table 4 took 119.6 sec, found 4294497549 matches
[P1] Table 5 took 116.507 sec, found 4294005459 matches
[P1] Table 6 took 109.505 sec, found 4293099586 matches
[P1] Table 7 took 76.5422 sec, found 4291219412 matches
Phase 1 took 601.539 sec
[P2] max_table_size = 4294967296
[P2] Table 7 scan took 3.88971 sec
[P2] Table 7 rewrite took 15.846 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 25.1002 sec
[P2] Table 6 rewrite took 43.5172 sec, dropped 581531399 entries (13.5457 %)
[P2] Table 5 scan took 28.5283 sec
[P2] Table 5 rewrite took 46.5049 sec, dropped 762235224 entries (17.7511 %)
[P2] Table 4 scan took 27.3255 sec
[P2] Table 4 rewrite took 40.6771 sec, dropped 829101031 entries (19.3061 %)
[P2] Table 3 scan took 32.778 sec
[P2] Table 3 rewrite took 39.3185 sec, dropped 855236487 entries (19.9134 %)
[P2] Table 2 scan took 29.6097 sec
[P2] Table 2 rewrite took 41.4029 sec, dropped 865662323 entries (20.1559 %)
Phase 2 took 384.268 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 29.9224 sec, wrote 3429164472 right entries
[P3-2] Table 2 took 22.1877 sec, wrote 3429164472 left entries, 3429164472 final
[P3-1] Table 3 took 30.3111 sec, wrote 3439536296 right entries
[P3-2] Table 3 took 20.7649 sec, wrote 3439536296 left entries, 3439536296 final
[P3-1] Table 4 took 35.1216 sec, wrote 3465396518 right entries
[P3-2] Table 4 took 20.2629 sec, wrote 3465396518 left entries, 3465396518 final
[P3-1] Table 5 took 32.9425 sec, wrote 3531770235 right entries
[P3-2] Table 5 took 21.0948 sec, wrote 3531770235 left entries, 3531770235 final
[P3-1] Table 6 took 34.9911 sec, wrote 3711568187 right entries
[P3-2] Table 6 took 22.5373 sec, wrote 3711568187 left entries, 3711568187 final
[P3-1] Table 7 took 26.2641 sec, wrote 4291219412 right entries
[P3-2] Table 7 took 25.8935 sec, wrote 4291219412 left entries, 4291219412 final
Phase 3 took 325.022 sec, wrote 21868655120 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 34.0743 sec, final plot size is 108784347231 bytes
Total plot creation time was 1344.96 sec
Started copy to /media/allan/Sea8-05/plot-k32-2021-06-14-16-35-7da78527d52b9da493d4592fa4b014c82a92fbea3e896439e240ce6994e42053.plot
Copy to /media/allan/Sea8-05/plot-k32-2021-06-14-16-35-7da78527d52b9da493d4592fa4b014c82a92fbea3e896439e240ce6994e42053.plot finished, took 641.982 sec, 161.601 MB/s avg.

--------------------------------------------------------  
ramdisk 115G, buckets 512, threads 12

Multi-threaded pipelined Chia k32 plotter - 16f9793
Final Directory: /media/allan/Sea8-05/
Number of Plots: 1
Crafting plot 1 out of 1
Process ID: 90491
Number of Threads: 12
Number of Buckets: 2^9 (512)
Pool Public Key:   939f6f869b87542414b8d5f9ab94c157af6b4900e2d2906486b8e17cc9c9c4d5869fd4c167276e10f7281e167d7b2286
Farmer Public Key: 8b49a3190e88bb4a8d73bd08c5f9584bfdca0beb9b9784239ad2d3a6811085f7221cc85c6ab91e3166f8ef17ab610c0d
Working Directory:   /media/allan/pny/
Working Directory 2: /mnt/ramdisk/
Plot Name: plot-k32-2021-06-14-17-36-12fda4958b88f96cd38e5f7a68a75e6c3eda087c6e4caa054495db234d9e0d80
[P1] Table 1 took 9.39853 sec
[P1] Table 2 took 73.9803 sec, found 4294912645 matches
[P1] Table 3 took 83.3576 sec, found 4294739935 matches
[P1] Table 4 took 101.558 sec, found 4294494170 matches
[P1] Table 5 took 99.6148 sec, found 4293974602 matches
[P1] Table 6 took 96.3077 sec, found 4293010795 matches
[P1] Table 7 took 74.5957 sec, found 4291203866 matches
Phase 1 took 538.823 sec
[P2] max_table_size = 4294967296
[P2] Table 7 scan took 5.83666 sec
[P2] Table 7 rewrite took 14.1423 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 22.9363 sec
[P2] Table 6 rewrite took 42.6694 sec, dropped 581474797 entries (13.5447 %)
[P2] Table 5 scan took 24.835 sec
[P2] Table 5 rewrite took 54.2208 sec, dropped 762211384 entries (17.7507 %)
[P2] Table 4 scan took 25.7664 sec
[P2] Table 4 rewrite took 41.1862 sec, dropped 829085035 entries (19.3058 %)
[P2] Table 3 scan took 30.338 sec
[P2] Table 3 rewrite took 38.6389 sec, dropped 855212077 entries (19.913 %)
[P2] Table 2 scan took 31.3962 sec
[P2] Table 2 rewrite took 39.2033 sec, dropped 865675500 entries (20.1558 %)
Phase 2 took 379.871 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 23.852 sec, wrote 3429237145 right entries
[P3-2] Table 2 took 17.9108 sec, wrote 3429237145 left entries, 3429237145 final
[P3-1] Table 3 took 28.1353 sec, wrote 3439527858 right entries
[P3-2] Table 3 took 17.4094 sec, wrote 3439527858 left entries, 3439527858 final
[P3-1] Table 4 took 31.2797 sec, wrote 3465409135 right entries
[P3-2] Table 4 took 17.0381 sec, wrote 3465409135 left entries, 3465409135 final
[P3-1] Table 5 took 29.6162 sec, wrote 3531763218 right entries
[P3-2] Table 5 took 17.2544 sec, wrote 3531763218 left entries, 3531763218 final
[P3-1] Table 6 took 32.5162 sec, wrote 3711535998 right entries
[P3-2] Table 6 took 18.5699 sec, wrote 3711535998 left entries, 3711535998 final
[P3-1] Table 7 took 22.8789 sec, wrote 4291203866 right entries
[P3-2] Table 7 took 23.7992 sec, wrote 4291203866 left entries, 4291203866 final
Phase 3 took 282.861 sec, wrote 21868677220 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 34.4635 sec, final plot size is 108784387673 bytes
Total plot creation time was 1236.06 sec
Started copy to /media/allan/Sea8-05/plot-k32-2021-06-14-17-36-12fda4958b88f96cd38e5f7a68a75e6c3eda087c6e4caa054495db234d9e0d80.plot
Copy to /media/allan/Sea8-05/plot-k32-2021-06-14-17-36-12fda4958b88f96cd38e5f7a68a75e6c3eda087c6e4caa054495db234d9e0d80.plot finished, took 641.693 sec, 161.674 MB/s avg.

---------------------------------------------
ramdisk 115G, buckets 512, threads 24

Multi-threaded pipelined Chia k32 plotter - 16f9793
Final Directory: /media/allan/Sea8-05/
Number of Plots: 1
Crafting plot 1 out of 1
Process ID: 92620
Number of Threads: 24
Number of Buckets: 2^9 (512)
Pool Public Key:   939f6f869b87542414b8d5f9ab94c157af6b4900e2d2906486b8e17cc9c9c4d5869fd4c167276e10f7281e167d7b2286
Farmer Public Key: 8b49a3190e88bb4a8d73bd08c5f9584bfdca0beb9b9784239ad2d3a6811085f7221cc85c6ab91e3166f8ef17ab610c0d
Working Directory:   /media/allan/pny/
Working Directory 2: /mnt/ramdisk/
Plot Name: plot-k32-2021-06-14-18-20-fd4cd14b2b7e18a552c18f88b254438a02876a18dac327b865bc3770d8361447
[P1] Table 1 took 10.1119 sec
[P1] Table 2 took 75.338 sec, found 4294946716 matches
[P1] Table 3 took 86.4338 sec, found 4294870257 matches
[P1] Table 4 took 104.757 sec, found 4294786018 matches
[P1] Table 5 took 103.134 sec, found 4294617648 matches
[P1] Table 6 took 98.2774 sec, found 4294146140 matches
[P1] Table 7 took 74.4565 sec, found 4293277187 matches
Phase 1 took 552.518 sec
[P2] max_table_size = 4294967296
[P2] Table 7 scan took 4.56867 sec
[P2] Table 7 rewrite took 15.6281 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 24.5403 sec
[P2] Table 6 rewrite took 42.8682 sec, dropped 581405767 entries (13.5395 %)
[P2] Table 5 scan took 25.1313 sec
[P2] Table 5 rewrite took 48.3613 sec, dropped 762118601 entries (17.7459 %)
[P2] Table 4 scan took 28.0562 sec
[P2] Table 4 rewrite took 39.5769 sec, dropped 828905301 entries (19.3003 %)
[P2] Table 3 scan took 30.3073 sec
[P2] Table 3 rewrite took 41.2362 sec, dropped 855077916 entries (19.9093 %)
[P2] Table 2 scan took 34.7005 sec
[P2] Table 2 rewrite took 39.7135 sec, dropped 865556570 entries (20.1529 %)
Phase 2 took 383.872 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 24.4811 sec, wrote 3429390146 right entries
[P3-2] Table 2 took 19.2036 sec, wrote 3429390146 left entries, 3429390146 final
[P3-1] Table 3 took 27.1584 sec, wrote 3439792341 right entries
[P3-2] Table 3 took 18.9964 sec, wrote 3439792341 left entries, 3439792341 final
[P3-1] Table 4 took 28.9598 sec, wrote 3465880717 right entries
[P3-2] Table 4 took 19.0173 sec, wrote 3465880717 left entries, 3465880717 final
[P3-1] Table 5 took 30.2678 sec, wrote 3532499047 right entries
[P3-2] Table 5 took 22.5402 sec, wrote 3532499047 left entries, 3532499047 final
[P3-1] Table 6 took 31.5078 sec, wrote 3712740373 right entries
[P3-2] Table 6 took 23.9701 sec, wrote 3712740373 left entries, 3712740373 final
[P3-1] Table 7 took 24.5273 sec, wrote 4293277187 right entries
[P3-2] Table 7 took 25.6171 sec, wrote 4293277187 left entries, 4293277187 final
Phase 3 took 298.902 sec, wrote 21873579811 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 34.6733 sec, final plot size is 108813528665 bytes
Total plot creation time was 1270.02 sec
Started copy to /media/allan/Sea8-05/plot-k32-2021-06-14-18-20-fd4cd14b2b7e18a552c18f88b254438a02876a18dac327b865bc3770d8361447.plot
Copy to /media/allan/Sea8-05/plot-k32-2021-06-14-18-20-fd4cd14b2b7e18a552c18f88b254438a02876a18dac327b865bc3770d8361447.plot finished, took 650.567 sec, 159.511 MB/s avg.

-----------------------------------------------
ramdisk 115G, buckets 512, threads 8

Multi-threaded pipelined Chia k32 plotter - 16f9793
Final Directory: /media/allan/Sea8-05/
Number of Plots: 1
Crafting plot 1 out of 1
Process ID: 95587
Number of Threads: 8
Number of Buckets: 2^9 (512)
Pool Public Key:   939f6f869b87542414b8d5f9ab94c157af6b4900e2d2906486b8e17cc9c9c4d5869fd4c167276e10f7281e167d7b2286
Farmer Public Key: 8b49a3190e88bb4a8d73bd08c5f9584bfdca0beb9b9784239ad2d3a6811085f7221cc85c6ab91e3166f8ef17ab610c0d
Working Directory:   /media/allan/pny/
Working Directory 2: /mnt/ramdisk/
Plot Name: plot-k32-2021-06-14-19-00-e4315772d31d2c3fe7bb387a571388102cf4b3e446a1e1b118f622822792c8a8
[P1] Table 1 took 8.96078 sec
[P1] Table 2 took 88.0818 sec, found 4295041277 matches
[P1] Table 3 took 94.2525 sec, found 4294985217 matches
[P1] Table 4 took 120.084 sec, found 4294808643 matches
[P1] Table 5 took 118.464 sec, found 4294572322 matches
[P1] Table 6 took 114.91 sec, found 4294136299 matches
[P1] Table 7 took 88.4795 sec, found 4293198343 matches
Phase 1 took 633.243 sec
[P2] max_table_size = 4295041277
[P2] Table 7 scan took 9.76585 sec
[P2] Table 7 rewrite took 14.5867 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 23.3676 sec
[P2] Table 6 rewrite took 46.8133 sec, dropped 581417980 entries (13.5398 %)
[P2] Table 5 scan took 23.6685 sec
[P2] Table 5 rewrite took 52.8596 sec, dropped 762115265 entries (17.746 %)
[P2] Table 4 scan took 31.8898 sec
[P2] Table 4 rewrite took 44.8005 sec, dropped 828990880 entries (19.3022 %)
[P2] Table 3 scan took 36.6456 sec
[P2] Table 3 rewrite took 46.598 sec, dropped 855164490 entries (19.9108 %)
[P2] Table 2 scan took 33.9088 sec
[P2] Table 2 rewrite took 45.8406 sec, dropped 865660819 entries (20.1549 %)
Phase 2 took 419.444 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 27.7309 sec, wrote 3429380458 right entries
[P3-2] Table 2 took 19.6853 sec, wrote 3429380458 left entries, 3429380458 final
[P3-1] Table 3 took 34.3956 sec, wrote 3439820727 right entries
[P3-2] Table 3 took 19.1378 sec, wrote 3439820727 left entries, 3439820727 final
[P3-1] Table 4 took 38.7842 sec, wrote 3465817763 right entries
[P3-2] Table 4 took 18.7153 sec, wrote 3465817763 left entries, 3465817763 final
[P3-1] Table 5 took 38.6031 sec, wrote 3532457057 right entries
[P3-2] Table 5 took 19.5043 sec, wrote 3532457057 left entries, 3532457057 final
[P3-1] Table 6 took 42.5553 sec, wrote 3712718319 right entries
[P3-2] Table 6 took 23.1816 sec, wrote 3712718319 left entries, 3712718319 final
[P3-1] Table 7 took 24.81 sec, wrote 4293198343 right entries
[P3-2] Table 7 took 24.034 sec, wrote 4293198343 left entries, 4293198343 final
Phase 3 took 333.869 sec, wrote 21873392667 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 36.7884 sec, final plot size is 108812415561 bytes
Total plot creation time was 1423.38 sec
Started copy to /media/allan/Sea8-05/plot-k32-2021-06-14-19-00-e4315772d31d2c3fe7bb387a571388102cf4b3e446a1e1b118f622822792c8a8.plot
Copy to /media/allan/Sea8-05/plot-k32-2021-06-14-19-00-e4315772d31d2c3fe7bb387a571388102cf4b3e446a1e1b118f622822792c8a8.plot finished, took 653.47 sec, 158.801 MB/s avg.

---------------------------------------------------------
ramdisk 115G, buckets 512, threads 16

Multi-threaded pipelined Chia k32 plotter - 16f9793
Final Directory: /media/allan/Sea8-05/
Number of Plots: 1
Crafting plot 1 out of 1
Process ID: 97113
Number of Threads: 16
Number of Buckets: 2^9 (512)
Pool Public Key:   939f6f869b87542414b8d5f9ab94c157af6b4900e2d2906486b8e17cc9c9c4d5869fd4c167276e10f7281e167d7b2286
Farmer Public Key: 8b49a3190e88bb4a8d73bd08c5f9584bfdca0beb9b9784239ad2d3a6811085f7221cc85c6ab91e3166f8ef17ab610c0d
Working Directory:   /media/allan/pny/
Working Directory 2: /mnt/ramdisk/
Plot Name: plot-k32-2021-06-14-19-35-124caa0b9f8a8878254c9114c530b069de66cb7d594475583b1de9c399426a3f
[P1] Table 1 took 9.42534 sec
[P1] Table 2 took 74.9341 sec, found 4295012385 matches
[P1] Table 3 took 88.7897 sec, found 4294885136 matches
[P1] Table 4 took 103.562 sec, found 4294786459 matches
[P1] Table 5 took 101.343 sec, found 4294643171 matches
[P1] Table 6 took 97.3351 sec, found 4294367692 matches
[P1] Table 7 took 74.6434 sec, found 4293688836 matches
Phase 1 took 550.043 sec
[P2] max_table_size = 4295012385
[P2] Table 7 scan took 6.36734 sec
[P2] Table 7 rewrite took 15.0704 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 24.7635 sec
[P2] Table 6 rewrite took 41.2483 sec, dropped 581339918 entries (13.5373 %)
[P2] Table 5 scan took 26.0399 sec
[P2] Table 5 rewrite took 48.3185 sec, dropped 762030559 entries (17.7437 %)
[P2] Table 4 scan took 29.775 sec
[P2] Table 4 rewrite took 39.7995 sec, dropped 828943436 entries (19.3012 %)
[P2] Table 3 scan took 35.7613 sec
[P2] Table 3 rewrite took 40.7876 sec, dropped 855117979 entries (19.9101 %)
[P2] Table 2 scan took 33.5238 sec
[P2] Table 2 rewrite took 39.1948 sec, dropped 865653853 entries (20.1549 %)
Phase 2 took 390.625 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 25.5753 sec, wrote 3429358532 right entries
[P3-2] Table 2 took 17.7288 sec, wrote 3429358532 left entries, 3429358532 final
[P3-1] Table 3 took 27.3679 sec, wrote 3439767157 right entries
[P3-2] Table 3 took 18.6405 sec, wrote 3439767157 left entries, 3439767157 final
[P3-1] Table 4 took 31.9836 sec, wrote 3465843023 right entries
[P3-2] Table 4 took 18.6415 sec, wrote 3465843023 left entries, 3465843023 final
[P3-1] Table 5 took 32.9369 sec, wrote 3532612612 right entries
[P3-2] Table 5 took 19.3803 sec, wrote 3532612612 left entries, 3532612612 final
[P3-1] Table 6 took 33.6758 sec, wrote 3713027774 right entries
[P3-2] Table 6 took 20.136 sec, wrote 3713027774 left entries, 3713027774 final
[P3-1] Table 7 took 21.962 sec, wrote 4293688836 right entries
[P3-2] Table 7 took 24.8877 sec, wrote 4293688836 left entries, 4293688836 final
Phase 3 took 295.539 sec, wrote 21874297934 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 31.6287 sec, final plot size is 108818255572 bytes
Total plot creation time was 1267.88 sec
Started copy to /media/allan/Sea8-05/plot-k32-2021-06-14-19-35-124caa0b9f8a8878254c9114c530b069de66cb7d594475583b1de9c399426a3f.plot
Copy to /media/allan/Sea8-05/plot-k32-2021-06-14-19-35-124caa0b9f8a8878254c9114c530b069de66cb7d594475583b1de9c399426a3f.plot finished, took 663.673 sec, 156.368 MB/s avg.

-----------------------------------------------
ramdisk 115G, buckets 1024, threads 12

Multi-threaded pipelined Chia k32 plotter - 16f9793
Final Directory: /media/allan/Sea8-05/
Number of Plots: 1
Crafting plot 1 out of 1
Process ID: 99673
Number of Threads: 12
Number of Buckets: 2^10 (1024)
Pool Public Key:   939f6f869b87542414b8d5f9ab94c157af6b4900e2d2906486b8e17cc9c9c4d5869fd4c167276e10f7281e167d7b2286
Farmer Public Key: 8b49a3190e88bb4a8d73bd08c5f9584bfdca0beb9b9784239ad2d3a6811085f7221cc85c6ab91e3166f8ef17ab610c0d
Working Directory:   /media/allan/pny/
Working Directory 2: /mnt/ramdisk/
Plot Name: plot-k32-2021-06-14-20-16-216ef43ec5269a5a93bcd383116b105cd4f48096d0c3db04565011938ba0cebc
[P1] Table 1 took 10.2866 sec
[P1] Table 2 took 85.1569 sec, found 4295018962 matches
[P1] Table 3 took 90.9024 sec, found 4294945316 matches
[P1] Table 4 took 107.218 sec, found 4294947981 matches
[P1] Table 5 took 105.429 sec, found 4294957217 matches
[P1] Table 6 took 102.171 sec, found 4294954843 matches
[P1] Table 7 took 82.5727 sec, found 4294950188 matches
Phase 1 took 583.756 sec
[P2] max_table_size = 4295018962
[P2] Table 7 scan took 6.05567 sec
[P2] Table 7 rewrite took 15.176 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 23.2756 sec
[P2] Table 6 rewrite took 43.5116 sec, dropped 581287910 entries (13.5342 %)
[P2] Table 5 scan took 26.8386 sec
[P2] Table 5 rewrite took 53.4729 sec, dropped 761963087 entries (17.7409 %)
[P2] Table 4 scan took 28.9243 sec
[P2] Table 4 rewrite took 46.6613 sec, dropped 828802672 entries (19.2972 %)
[P2] Table 3 scan took 25.8319 sec
[P2] Table 3 rewrite took 39.7773 sec, dropped 855072206 entries (19.9088 %)
[P2] Table 2 scan took 26.1032 sec
[P2] Table 2 rewrite took 41.3834 sec, dropped 865587842 entries (20.1533 %)
Phase 2 took 385.808 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 33.9327 sec, wrote 3429431120 right entries
[P3-2] Table 2 took 19.0141 sec, wrote 3429431120 left entries, 3429431120 final
[P3-1] Table 3 took 31.441 sec, wrote 3439873110 right entries
[P3-2] Table 3 took 18.7376 sec, wrote 3439873110 left entries, 3439873110 final
[P3-1] Table 4 took 29.3153 sec, wrote 3466145309 right entries
[P3-2] Table 4 took 18.3447 sec, wrote 3466145309 left entries, 3466145309 final
[P3-1] Table 5 took 31.4378 sec, wrote 3532994130 right entries
[P3-2] Table 5 took 18.6805 sec, wrote 3532994130 left entries, 3532994130 final
[P3-1] Table 6 took 35.2104 sec, wrote 3713666933 right entries
[P3-2] Table 6 took 19.4176 sec, wrote 3713666933 left entries, 3713666933 final
[P3-1] Table 7 took 22.6869 sec, wrote 4294950188 right entries
[P3-2] Table 7 took 25.5981 sec, wrote 4294950188 left entries, 4294950188 final
Phase 3 took 306.471 sec, wrote 21877060790 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 32.2404 sec, final plot size is 108835081943 bytes
Total plot creation time was 1308.32 sec
Started copy to /media/allan/Sea8-05/plot-k32-2021-06-14-20-16-216ef43ec5269a5a93bcd383116b105cd4f48096d0c3db04565011938ba0cebc.plot

-----------------------------------------------