Add support to send config transactions

[davidkel]

Apart from business transactions we should also be able to test the effect of config transactions happening at the same time to see the effect on the system

Unsure whether to make this part of the client or have a separately running app

Config transactions that would have an effect

• adding an org to a channel
• removing an org from a channel
• adding an orderer to a channel
• removing an orderer from a channel
• changing an orderer eg the address
• adding/changing/removing an anchor peer
• policy changes ?

Not specific to config transactions but might also want to consider

• joining a peer to the channel
• unjoining a peer to the channel
• deploying a chaincode update (maybe include adding/changing/removing SBEs)
• changing a chaincode endorsement policy
• changing a private collection definition
• adding chaincode to a peer without chaincode

[davidkel]

Discussion with Dave, the first 2 to try are

1. changing an orderer eg the address
2. changing a chaincode endorsement policy of a live channel (via lifecycle commands)

At a first pass need to just have scripts that can be run that will effect the change so automated but need to be driven manually

[sapthasurendran]

FINDINGS: Endorsement Policy Update: Majority to "AND('Org1MSP.member','Org2MSP.member','Org3MSP.member')"
Network : 3 Org, 2 peer each
-- During update

1. gateway throws error message:10 ABORTED: failed to assemble transaction: ProposalResponsePayloads do not match
2. No Events received as: status code 11—mvcc_read_conflict
3. Also randomly got endorsement policy failure ,status code 10.

-- After the policy is updated successfully,

1. client recovers well and transactions seems to work fine.

[denyeart]

@sapthasurendran the endorsement policy update test scenario worked fine for me, however I was not applying a steady transaction load, I just did one transaction before the update and one transaction after the update, both succeeded and got the correct number of endorsements based on the update.

So the issue may be with the test rather than a gateway issue, or it may be that some errors are always going to occur for a couple seconds while the endorsement policy change works its way though the network. Could you explain exactly what your test does? What does the chaincode do? Does it create or update a key? Does it read the same key? Do you have a steady transaction load? What rate of transactions? Do you have multiple transactions occurring concurrently? How many seconds does it take after updating the endorsement policy for transactions to start succeeding again?

[sapthasurendran]

@sapthasurendran the endorsement policy update test scenario worked fine for me, however I was not applying a steady transaction load, I just did one transaction before the update and one transaction after the update, both succeeded and got the correct number of endorsements based on the update.

So the issue may be with the test rather than a gateway issue, or it may be that some errors are always going to occur for a couple seconds while the endorsement policy change works its way though the network. Could you explain exactly what your test does? What does the chaincode do? Does it create or update a key? Does it read the same key? Do you have a steady transaction load? What rate of transactions? Do you have multiple transactions occurring concurrently? How many seconds does it take after updating the endorsement policy for transactions to start succeeding again?

Yes, the errors happened only for the transactions which was waiting for endorsement during the time of the policy update .
Basically my test ran a script which updated the endorsement policy from majority to policy of one peer from each org against a live client app .
The client app runs multiple transactions in parallel with a max transaction per sec count of 20.
Failures occurred for create asset and update asset transactions.
After 6 secs from the time chaincode definition was updated for new endorsement policy, transactions succeeded .

Also, few transactions were successful even during the update process.

[davidkel]

@sapthasurendran I would fully expect to see endorsement policy failures to occur when moving from a less restrictive endorsement policy to a more restrictive one. I would not expect to see those errors if going in reverse which would be an interesting test.

The other 2 errors of unmatched proposal responses and mvcc read conflicts are surprising to see and not something I would have expected to see if I am only changing the endorsement policy and no changes to chaincode were being made.

Can you post the log timelines from the client and the policy change script so that we could see at what stage of the policy change appears to drive these errors ?

[sapthasurendran]

@davidkel I retested the scenario with latest fabric-gateway version.

Findings:

• Status code 10 - Observed for transactions happening during the update.

After the update all transactions are failing with either one of the following errors.

• "4 DEADLINE_EXCEEDED: Deadline exceeded"
• "14 UNAVAILABLE: Stream refused by server"

client.log
gatewayPeer.log
script.log

[davidkel]

Clarification then, it would seem that the proposal response payloads may reference the endorsement policy eg a read which creates an entry in the read set and while that's different due to the commit to multiple peers not being atomic then it's entirely possible to get mismatched proposal response payloads.