Observed intermittent "Serialization Error: too large read" errors when sending GRNs

[shuan9-polo]

Describe the bug
When sending GRINs to my address via Tor, the following errors were sometimes observed:

root@70ce6dbbdf8c:/data/wallet_data# grin-wallet --pass $(cat /data/.wallet_credentials.txt) --top_level_dir /data/ send -f -d "<address>" -s smallest 325700.65818005
20210530 03:32:14.922 WARN grin_wallet_api::owner - Attempting to send transaction via TOR
Wallet command failed: LibWallet Error: Wallet store error: Serialization Error: too large read

I have enough spendable GRINs according to the grin-wallet info output so it should not be caused by the balance.

I also tried with smaller amount 10000.65818005 for 10 times, but only the first attempt worked:

root@70ce6dbbdf8c:/data/wallet_data# for i in `seq 10`; do echo $i; grin-wallet --pass $(cat /data/.wallet_credentials.txt) --top_level_dir /data/ send -f -d "<address>" -s smallest 10000.65818005; sleep 10; done
1
20210530 03:15:48.196 WARN grin_wallet_api::owner - Attempting to send transaction via TOR
Tx sent successfully
Command 'send' completed successfully
2
20210530 03:17:19.380 WARN grin_wallet_api::owner - Attempting to send transaction via TOR
Wallet command failed: LibWallet Error: Wallet store error: Serialization Error: too large read
3
20210530 03:17:47.188 WARN grin_wallet_api::owner - Attempting to send transaction via TOR
Wallet command failed: LibWallet Error: Wallet store error: Serialization Error: too large read
4
20210530 03:18:15.071 WARN grin_wallet_api::owner - Attempting to send transaction via TOR
Wallet command failed: LibWallet Error: Wallet store error: Serialization Error: too large read
5
20210530 03:18:39.925 WARN grin_wallet_api::owner - Attempting to send transaction via TOR
Wallet command failed: LibWallet Error: Wallet store error: Serialization Error: too large read
6
20210530 03:19:04.079 WARN grin_wallet_api::owner - Attempting to send transaction via TOR
Wallet command failed: LibWallet Error: Wallet store error: Serialization Error: too large read
7
20210530 03:19:32.895 WARN grin_wallet_api::owner - Attempting to send transaction via TOR
Wallet command failed: LibWallet Error: Wallet store error: Serialization Error: too large read
8
20210530 03:19:58.124 WARN grin_wallet_api::owner - Attempting to send transaction via TOR
Wallet command failed: LibWallet Error: Wallet store error: Serialization Error: too large read
...

To Reproduce
Steps to reproduce the behavior:

1. call the grin-wallet cli to send GRINs to target address
2. see error "Wallet command failed: LibWallet Error: Wallet store error: Serialization Error: too large read"

Expected behavior
I expect the following logs printed after the send commands are executed:

Tx sent successfully
Command 'send' completed successfully

Screenshots
N/A

Desktop (please complete the following information):
(the grin-wallet and grin server are both hosted in containers)

• OS: Debian GNU/Linux
• Version 10
• grin-wallet version: 5.0.3
• grin server version: 5.0.4

Additional context
N/A

[phyro]

Thank you for opening an issue @shuan9-polo . It seems like this error appears when the transaction that is constructed is too big to fit in a block https://github.com/mimblewimble/grin/blob/f51b6e13761ac4c3c8e57904618ef431c14c6227/core/src/core/transaction.rs#L853-L855

The smallest strategy sorts the outputs by value increasingly and keeps taking them as inputs until input_sum >= amount so if you happen to have a lot of small outputs in your wallet, you could end up with a transaction that is too big to fit in a block. You could check this by outputting a slatepack without going through Tor and counting the number of inputs that were selected. The wallet should probably be aware of the block weight limit and check whether the transaction is within these limits. Perhaps consolidating the outputs (merging many of them to a single output) and making sure you don't have too many would be a good idea to also reduce your fee costs for future transactions.

[shuan9-polo]

Perhaps consolidating the outputs (merging many of them to a single output) and making sure you don't have too many would be a good idea to also reduce your fee costs for future transactions.

Would you please elaborate how I can do this? Is it possible to do this with the grin-wallet cli?

[phyro]

@shuan9-polo I'm not sure about the state of self-spend at the moment - it might not work over Tor Grinnode-live/2020-grin-bug-bash-challenge#59. Perhaps using manual send (not Tor), would work meaning you send to your own grin address, but with manual slatepacks. I'm not sure what the state of this is at the moment, so you should definitely confirm first that it works by trying smaller amounts.

[shuan9-polo]

Oh, I need to clarify that the address is not owned by me, instead, it's from an exchange. I mean the address associated with me in that exchange by "my address". Sorry for misleading you.

[phyro]

@shuan9-polo so the issue, as I understand it, is that one party in a transaction (probably the sender) is adding way too many inputs in the transaction so that the transaction becomes too big to include in a block. This can happen if you own a lot of inputs with very small values. One option to solve this would be to select the inputs manually but the wallet does not support this today. So another option is to make sure you don't have inputs with too low value so that when it builds a transaction, it's not using too many inputs. This can be achieved by sending to yourself multiple times by using "smallest" strategy. Each self-send would then take some small inputs and produce two outputs which would reduce the number of inputs with small amounts that you may have. Does this make sense? So you'd need to first consolidate the outputs you have and only then send such big transactions. Alternatively, you could try with even smaller amounts.

[shuan9-polo]

Thanks @phyro, as I understand, what you asked me to do is:

1. self-spend the spendable amount for multiple times to consolidate the output
2. once consolidation is done, continue to send the remaining amount to the address from the exchange

Hopefully, the send will complete successfully.

I have tried self-spend with Tor first but failed by the same errors observed in Grinnode-live/2020-grin-bug-bash-challenge#59. Then I tried it with manual steps by

1. use grin-wallet send -m -u file.tx -f -s smallest <amount> to send manually
2. use grin-wallet receive -i file.tx to receive from slatepack msgs
3. use grin-wallet finalize -i file.tx to finalize
4. use grin-wallet post -i file.tx to post

and it worked. After a while, I see the txs is confirmed and well received.

But I did notice that it still took some fee for self-spend. So my question is, do you have a chance to know which solution may cost less fee, the self-spend one or the below?
"send smaller amount of GRINs to the address owned by exchange for multiple times"

[phyro]

@shuan9-polo I'm glad you were able to consolidate them. Yes, self-spend is a transaction like any other, so it pays fees to miners just like a regular transaction. Unfortunately, I don't know how to solve this differently than what I have suggested above (or by having an option to manually select the outputs you want to use). The closest that comes to mind is to do smarter consolidation which means using as many inputs to create the 1 or 2 outputs that hold the sum of the input amounts.
Did the issue you were getting before go away and you were able to deposit/withdraw to the exchange?

[shuan9-polo]

Hi @phyro, I did not use the self-spend way to consolidate the outputs as it will cost a lot of fees. Reading through the following description of the all strategy:

using all consolidates all of your outputs into a single new output, thus reducing your wallet size

As I understand from it, calling send with all strategy will not get me into the same situation, where transaction is too big to fit in a block. Do you think if it's safe to use it to help the current problem, considering the downsides mentioned below?

The downside is that the entire contents of your wallet remain locked until the transaction is validated on-chain, and all outputs are linked to one another, a detriment to your privacy.

[phyro]

@shuan9-polo I'm not too familiar with this option so I can't really tell. I don't know what happens and what could go wrong in your scenario where we believe you have a lot of outputs, I'd assume that you'd again get to the same problem if you used all outputs as inputs and the issue is too many inputs? I would personally consolidate them in some iterations. If that's too costly, then I'm not sure what the answer to this is. If the transaction is too big, you have no choice but to split it in multiple transactions currently (afaik).

[cliik]

Should this ticket be closed? Reading through the dialog, it appears this is expected behavior. If a user builds a transaction that is too large, it makes sense to see a "too large" error.

[phyro]

Agree. If they have more questions they can open a new issue and link to this one