Segfault in batch mode

[throwawaygithub1337]

I've been testing out batch mode extensively. Bitcoin-tool doesn't pipe out the result immediately which is probably the cause of the problem.

If you feed it a file that's ~20 mil keys, it will run for a while and segfault. Less than 15 mil keys is okay, though it will probably segfault on some system.

I've implemented some of your functions in my program, and it's working very well.
It doesn't segfault if you fprintf to stdout and free memory after every loop for each line.

You're right that generating the public key is the biggest bottleneck.
When generating compressed and uncompressed RIPEMD160, I'm getting ~1000 keys/sec.

This is with no IO bottleneck. The program is not piping to stdout.
I got this number by setting a loop counter and incrementing at the end of every loop.

Running multiple jobs with GNU Parallel helps, but is there no other way to speed up public key calculation?

Can we use some other secp256k1 library that is faster
(such as this one https://github.com/bitcoin/secp256k1)
or will the performance gain by negligible?

Thanks for all the work you've done. It really helped me.

[matja]

What is the full command line you're using and does it always segfault at the same place?
You're correct that the output is buffered, because libc stdio routines are being used for portability and performance, I'm not sure how this could introduce a segfault.

I've tried reproducing this with a file of 32M random private keys, generated with :
openssl rand $[32*1024*1024*32] | xxd -p -c 32 > keys.txt

And then convert with :
bitcoin-tool --batch --input-file keys.txt --input-type private-key --input-format hex --output-type address --output-format base58check --network bitcoin --public-key-compression compressed > output.txt

It completes with no errors on my x86-64 Linux machines.
I profiled the memory usage with valgrind and found it to be relatively static at about 128kB over the entire run, with no leaks.

If it is reproducible for you, could you possibly recompile with "-ggdb -O0", run in gdb and post the output of "bt full", showing which function it is segfaulting in? - Thanks.

[throwawaygithub1337]

Sorry, I should have made it clear I'm outputting RIPEMD160, not base58check. Also, it's uncompressed and compressed.

I don't understand how your memory usage stays so low when when you're not writing to disk. Where's all that data sitting at? At 32M keys, the output size is huge. I think if you try outputting ripemd160, it will segfault. I don't think I can recompile for debug since I don't know how to use Make.

BTW, I've also just tried Pieter Wuille's libsecp256k1. It's significantly faster (15x).
I've also found this library
https://github.com/hhanh00/secp256k1-cl

It seems interesting but there's no documentation.

[matja]

Using --output-type public-key-rmd involves less work than base58check. Redirecting stdout to a file and having it output to a terminal makes no difference in the memory usage on my machine. I'd be interested in what compiler you're using and how you're running it.

[throwawaygithub1337]

I used the Makefile that came with the code. It wasn't modified in any way. I assume it's compiled with GCC.
Anyways, I resorted to splitting up the SHA256 hash file in 100K lines chunks and using GNU parallel to run bitcoin-tool in the end.

I tested it at 17M and it didn't segfault. It segfault with 30M every single time. I tested it on 15 different hash files. It will run for several hours before segfault.

At first I thought it was because I was running several instances of it, but nope, it still segfault even with single instance.

I also thought it was because I set the result equal to a shell variable and echo "$variable" > file, but it still segfault even if you don't do that.

It doesn't segfault with 17M< and setting output to shell variable.