Test changing an endorsement policy

[davidkel]

Add Support to dynamically change the endorsement policy and test the following scenarios

• Move from a less restrictive to more restrictive endorsement policy
• Move from a more restrictive to a less restrictive endorsement policy

[davidkel]

From Saptha:
FINDINGS: Endorsement Policy Update: Majority to "AND('Org1MSP.member','Org2MSP.member','Org3MSP.member')"
Network : 3 Org, 2 peer each
-- During update

gateway throws error message:10 ABORTED: failed to assemble transaction: ProposalResponsePayloads do not match
No Events received as: status code 11—mvcc_read_conflict
Also randomly got endorsement policy failure ,status code 10.
-- After the policy is updated successfully,

client recovers well and transactions seems to work fine.

[davidkel]

Endorsement policy failures are expected when moving from a less restrictive to a more restrictive policy and the endorsements were collected before the policy change was completed.

Proposal Response Payload mismatches can also occur because proposal response payloads reference the endorsement policy eg a read which creates an entry in the read set and while that's different due to the commit to multiple peers not being atomic then it's entirely possible to get mismatched proposal response payloads.

For the same reason MVCC read conflicts can occur. This can happen when endorsements are done before the new definition is committed. When the peer comes to validate it the read set has one version of the lifecycle but now the version has changed so reading it again shows a difference resulting in the MVCC read conflict.

[davidkel]

From Saptha:
I retested the scenario with latest fabric-gateway version.

Findings:

Status code 10 - Observed for transactions happening during the update.
After the update all transactions are failing with either one of the following errors.

"4 DEADLINE_EXCEEDED: Deadline exceeded"
"14 UNAVAILABLE: Stream refused by server"
gatewayPeer.log
client.log
script.log

[davidkel]

@sapthasurendran This is a new issue and certainly unexpected, we need to try to recreate this and understand how a client permanently lost comms with the gateway server. Did a restart of the client address the issue ?

[davidkel]

Moving to a more restrictive policy I hit

{"component":"CLIENT","timestamp":"2021-11-25T17:00:04.066Z","txnId":"8d1920a9563418b467f831ca231040299a22d6c5710d39775acb9c5ef0deccda","stage":"Endorsing","message":"addUpdateAssets({\"arguments\":[\"1\",\"2000\"]})"}
{"component":"CLIENT","timestamp":"2021-11-25T17:00:09.001Z","txnId":"8d1920a9563418b467f831ca231040299a22d6c5710d39775acb9c5ef0deccda","stage":"Failed","message":"14 UNAVAILABLE: no combination of peers can be derived which satisfy the endorsement policy: required chaincodes are not installed on sufficient peers"}

Plus

{"component":"CLIENT","timestamp":"2021-11-25T17:00:04.112Z","txnId":"2ff7f00089bebd1e294444705bd76d233367df2451f07b4ab5fa6dc4af7739d0","stage":"Endorsing","message":"updateChaosAsset({\"arguments\":[\"cd1\",\"99\"]})"}
{"component":"CLIENT","timestamp":"2021-11-25T17:00:07.365Z","txnId":"2ff7f00089bebd1e294444705bd76d233367df2451f07b4ab5fa6dc4af7739d0","stage":"Submitting","message":""}
{"component":"CLIENT","timestamp":"2021-11-25T17:00:07.379Z","txnId":"2ff7f00089bebd1e294444705bd76d233367df2451f07b4ab5fa6dc4af7739d0","stage":"Submitted","message":""}
{"component":"CLIENT","timestamp":"2021-11-25T17:00:09.440Z","txnId":"2ff7f00089bebd1e294444705bd76d233367df2451f07b4ab5fa6dc4af7739d0","stage":"Failed","message":"unexpected validation code 10"}

[davidkel]

I can recreate the problem where the client loses connection to the gateway and the connection doesn't recover

{"component":"CLIENT","timestamp":"2021-11-25T17:05:58.440Z","txnId":"f4bcbe05d1c9257ebfbaa7492d3a853935ab69789c2d2feb661b413662d10a76","stage":"Endorsing","message":"addUpdateAssets({\"arguments\":[\"1\",\"2000\"]})"}
{"component":"CLIENT","timestamp":"2021-11-25T17:06:28.454Z","txnId":"f4bcbe05d1c9257ebfbaa7492d3a853935ab69789c2d2feb661b413662d10a76","stage":"Failed","message":"4 DEADLINE_EXCEEDED: Deadline exceeded"}
{"component":"CLIENT","timestamp":"2021-11-25T17:05:58.443Z","txnId":"bcd6cc53c262a76f23015a2003af3e72785079f44be01ba28840c6adaa31bc6a","stage":"Endorsing","message":"updateChaosAsset({\"arguments\":[\"cd1\",\"99\"]})"}
{"component":"CLIENT","timestamp":"2021-11-25T17:06:28.454Z","txnId":"bcd6cc53c262a76f23015a2003af3e72785079f44be01ba28840c6adaa31bc6a","stage":"Failed","message":"14 UNAVAILABLE: Stream refused by server"}

A restart of the client fixes the issue (but is most likely just down to the grpc connection being recreated).

going to try client and server side grpc debugging
export GRPC_TRACE=all
export GRPC_VERBOSITY=DEBUG

and peer grpc
FABRIC_LOGGING_SPEC=info:grpc=debug

[davidkel]

The issue where the client loses connection to the gateway hyperledger/fabric-gateway#332