Add off-chain-data go client application

[twoGiants]

Off Chain Data Go Application

This PR contains an implementation of the off-chain data store sample in go using the block event listening capability of the Fabric Gateway client API. The issue reference is here in the Fabric Gateway SDK repository.

It follows the same procedure and uses mostly the same naming as the off-chain data typescript sample here. There are some differences in the go package structure compared to the typescript modules and I added a couple of tests to the parser which helped me better understand the unpacking of the block data structure. The Go implementation is more similar to the Java version. More about that below in the Differences section.

@bestbeforetoday gave me this useful link where I found the iterator_test.go to be helpful and this link where the "Endorser Transaction" comments were helpful.

Summary

• created project directory, app.go, getAllAssets.go, transact.go and listen.go files
• reused and changed the logic for connect.go from the private data application
• implemented the asset transfer basic contract in the contract package
• implemented getAllAssets.go and transact.go
• implemented the block parser and all the supporting data structures
• implemented the block processor and all the supporting data structures
• implemented listen.go
• updated README.md with the command to run the go application
• updated script which runs the application in the Github Actions workflow
• manual tests

Differences

• I didn't use any interfaces. I removed them after the suggestion of @bestbeforetoday.
• I did not implement any methods which are not used anywhere, like toProto or getSignatureHeader.
• The block parser with all it's underlying structs (which are mostly private) is in the parser package.
• There are some simple tests for the cache utility and the block unwrapping.
• The block processor with it's underlying struct transaction (which is private) is in the processor package.
• The store is in the store package and because it resembles a bit a flat file db I called it flatFile.go
• The utilities are in the utils package.
• In all the additional packages I followed the encapsulation rule: if it's not referenced outside the package it's private.

Additional Information

I process the block events in a go routine and use context to stop the processing. Instead of panicking everywhere down the line where an error can occur I handle the errors by wrapping them, returning them, log them in the go routine, then return and shutdown gracefully. Panicking results in the context, gateway and checkpointer not getting closed.

Questions

1. Here is a TODO proposing to simplify the processor.transaction data structure.
2. And here is a TODO proposing to encapsulate the final unwrapping of the kvWrite in the parser instead of having it in transaction.writes.
3. I assume I should squash everything to one commit? It might be a big commit message.

[bestbeforetoday]

First thing, it runs great for me so this looks nicely done. Thank you! Sorry for large number of comments. This is just polishing the material to make it a really clean sample.

I would be careful to try and keep all the code that listens for blocks, navigates to their transactions and passes the read/write sets to some kind of store close together. Perhaps all in listen.go. This is the key learning point of the sample so it needs to be easy to access and navigate. The block parser should be separate (as you've done) so it is a reusable package on its own. The store implementation can also be a little saperate to avoid cluttering the key listener code. I might put it in a different file within the same package though, just as private (lowercase) content.

The Application section of off_chain_data/README.md should be updated to include references to key parts of the Go implementation, similar to the existing TypeScript and Java entries.

To catch some issues, it might be worth running this command in the application-go directory:

go run honnef.co/go/tools/cmd/staticcheck@latest -f stylish -checks all ./...

To avoid specific checks (like having package-level godoc), you can exclude specific rules. For example:

go run honnef.co/go/tools/cmd/staticcheck@latest -f stylish -checks 'all,-ST1000' ./...

[twoGiants]

The build failures look to be because of Docker Hub rate limiting. I have made a change that should avoid this in the future. If you rebase your change on the latest main branch content, hopefully we will see a clean build.

@bestbeforetoday one is still failing.

[bestbeforetoday]

This is looking really good now. Just a few very small niggles that would be nice to tidy up and I think it is ready to go.

[bestbeforetoday]

It is fine as-is. Just an option to have it a standalone function since it doesn't rely on any Block state. It is related to parsing of the block so I can see an argument for keeping it as a Block method to aid readability.

[bestbeforetoday]

An error is never returned so you don't need the error return value. The results are just passed to unmarshalProposalResponsePayloadsFrom so you might find it easier to include this logic in that method. chaincodeActionPayload.GetAction().GetProposalResponsePayload()

[twoGiants]

This is great, done.

[bestbeforetoday]

This is really a nitpick... I think the result slice you create here (or rather its underlying array) is going to get discarded as soon as you append to it in the loop since there isn't capacity to append more elements. You can save the work of creating storage that will be discarded by using a nil value, which still works as an empty slice:

var result []*peer.ChaincodeEndorsedAction

Or you can make a slice with capacity to hold len(chaincodeActionPayloads) elements up-front.

Same comment for the other instances of creating an empty (zero capacity) slice that you will then append to in this file.

Like I say, it's a really minor detail. Change it only if you want to.

[twoGiants]

I think the result slice you create here (or rather its underlying array) is going to get discarded as soon as you append to it in the loop since there isn't capacity to append more elements.

Had to reread a couple times but I think I understand now. Thanks for clarifying. I changed it.

[bestbeforetoday]

Using object state that you then need to manage instead of just passing arguments between private methods looks like a red flag to me. There is much more potential for things to go wrong since you are mutating more state in your object, and state is left lying around longer than you need it. For example, ocs.writes still contains data after you have finished persisting it and this method returns.

Also look out for error handling where you return only the error if it is non-nil; otherwise nil. Just return the error value directly. It is really easy to do this by accident.

Consider a simpler pattern, something like:

	writes, err := ocs.marshal(data.Writes)
	if err != nil {
		return err
	}

	return ocs.persist(writes)

[twoGiants]

For example, ocs.writes still contains data after you have finished persisting it and this method returns.

Actually, this was intended. But I was unsure about it. I thought it can make sense to keep the last write.

I believe I misunderstood this message then:

a store function (or store interface if you prefer to use a struct that maintains some state around the storing of writes, like the write and simulated failure count).

I removed it and used the simpler pattern. Thanks for pointing out! I am remembering more and more of the good practices thank to your reviews.

[bestbeforetoday]

I think I meant the write count (and simulated failure count) but explained it poorly! The simulatedFailureCount, transactionCount and path are state that makes sense to be there since they need to be maintained over over many calls to write().

[bestbeforetoday]

Suggested change

	if err := f.Close(); err != nil {
	return err
	}
	return nil
	return f.Close()

[twoGiants]

Done.

[bestbeforetoday]

Suggested change

	if err := app.run(); err != nil {
	return err
	}
	return nil
	return app.run()

[twoGiants]

Done.

[bestbeforetoday]

Suggested change

	if err := context.Cause(ctx); err != nil {
	return err
	}
	return nil
	return context.Cause(ctx)

[twoGiants]

Done.

[twoGiants]

Two things are left on my mind:

1. connect.go panics (like in the other samples) instead of returning errors like everywhere else in this sample. Keep or change?
2. block_test.go tests are rudimentary. They helped me building the parser but they are incomplete. Keep, remove or extend?

Once you approve I will squash everything to one commit.

[bestbeforetoday]

Excellent work. The code reads nicely to me and works perfectly.

The only things I noticed that should probably be added to make the block parser more reusable is:

1. ToProto() method on the Block, Transaction and NamespaceReadWriteSet. These should return a *common.Block, *common.Payload and *rwset.TxReadWriteSet respectively. They allow anyone reusing the parser to extract any additional information they might need from the underlying protobuf messages.
2. Transaction should have ValidationCode() and Creator() methods, since this information is commonly of interest and are either impossible or less easy to get from the associated *common.Payload protobuf message.

The sample works great as-is, and these are not good reasons to hold up the PR. They can be addressed in a future change.

Thank you for the contribution!